Java memory issues and HTTPUseNotesMemory

We’ve been suffering from various http memory related issues. We run on Solaris and Domino on Solaris is 32bit (and apparently staying this way). So despite having 64GB of memory in our servers, we’re constrained by the 4GB shared memory limit.

The issues seem to relate to Java. At various times agents stop working due to memory issues. Despite going through every Java agent with a fine tooth comb, the issues still re-occur. Increasing the Java pool size doesn’t help because this causes a different issue. Because we allow staff to upload files up to 45MB in size, we get file upload issues as the memory fragments and Domino cannot assign a contiguous memory block. (Setting our one 8.5 server to use the ‘buffer’ doesn’t appear to work either).

The problem has been with Lotus for some considerable time and we’ve been going though the process of adding more and more debug information. These settings are generating GBs of logs every week. So far Lotus Support have been unable to pinpoint which agent is causing the memory leak.

For the last two weeks we’ve been running with some new settings suggested by Lotus Support. The new settings corresponded to a peak period, end of Semester A exams. Despite the increased load, both servers have running without memory issues for the full week between backups. So I thought I’d share the settings:

JavaMaxHeapSize=268435456
iNotesUseNotesMemory=0
HTTPUseNotesMemory=0

The HTTP setting is the important one as we don’t use our Domino setup for email, and seems to move the http process out of the shared memory heap and therefore allows us to increase the Java heap without side effects. It would be nice to have more information as to what this setting means. For example, how much memory can the HTTP server consume? Is it the full 32 bit address space? Hopefully somebody at Lotusphere will know. But it does seem to be an incredibly useful setting that could be enabled by default on 32bit Domino servers with large amounts of memory.

8 Comments Add yours

  1. Here at the Belgian Federal Public Service of Finance we are running a 30.000 user Inotes site on Solaris and indeed you need to have some additional settings to keep everything happy.
    Feel free to drop me a e-mail

  2. Simon Boulton says:

    I don’t know whether this is exactly the same thing you are seeing, but since upgrading both our servers to 8.5.2 FP1 we have been finding that agents and other processes eventually start to fail because of lack of memory. Removing FP1 cures the problem.

  3. Paul says:

    @kurt de Feyter
    Thanks Kurt. I may do. The servers are staying up with the new settings. But it would be great to have a 64 bit version.

  4. Paul says:

    @Simon Boulton
    Our problems have been occurring with 7.x and 8.x servers. Our web site is heavily used handing about 1.5 millions requests per day. So small memory leaks can bring the whole thing down.

  5. Paul!

    Any news on this issue? We have alson seen some problems with memory and Java agents.

  6. Paul says:

    @Leif Lagebrand We’ve been running with these setting for a few months. The number of crashes has been significantly reduced. We also audited most of our java code and recycled everything we could. To be honest we won’t know how successful the changes are until we reach our peak load levels, just before exams in May.

  7. Eric Stewart says:

    Hi Paul, how did things go during the exam period? Are your instances of crashes still significantly reduced?

    We are experiencing a similar problem with Java agents giving JVM : Insufficient Memory issues on a regular basis. We are considering applying these ini settings to see if they have any positive impact, but would be interested to hear about your longer term experiences?

    Thanks.

  8. Paul says:

    @Eric Stewart We’re having significantly less problems using these settings. But we’re running low loads at the moment. The key time will be the beginning of October, when our students are back.

Leave a Reply