Last year I blogged about a permissions issue with Domino 7 running in a Solaris zone.  Domino’s own ID file’s permissions are reset so only root has access rights. This stops the server from executing it’s own code. The problem also effects other files modified by the HTTP task such as DTF files, cgi-bin, and even the web logs. The problem is still effecting us but we’ve discovered a way to alleviate our problems so our solution may help anyone suffering similar problems.

Background

Traditionally we’ve been very careful with our hardware updates, always making single changes at a time. However last year, strategic decisions to consolidated hardware and move towards greener server rooms meant that we had to move our Domino systems from dedicated servers with their own disk arrays to services hosted on Sun Enterprise servers along with moving our data on a SAN. Due to circumstances out of everyone’s control, the original plan to gradually move step by step to the new consolidated approach had to be moved to a very aggressive timetable (i.e. Everything done at once). At this time our servers were also moved from Solaris 9 to 10 and moved into Zones, as per our current Solaris virtualisation policy.

For me this was too many changes at once and even now I am still certain the change to using Zones is part of the underlying cause of our problems.

‘Identifying’ the problem

We reported the problem to Lotus last summer. Since then we’ve been back and forth between Lotus and Sun to identify where the issue lies, regularly updating our systems to record  more debugging information.

Since we’ve been unable to generate the problem on demand,  it’s often weeks before we could update Lotus with further data. But recently the issues increased to such an extent that the problem was occurring several times a day to both user-facing servers (where previously it was once a month to only one server). As the problems increased, we started to ‘poke and hope’ trying every idea possible while planning both moves to Domino 8.x and de-zoning one server.

Our thinking has always been that either the HTTP task is the cause of the problem or the Novell Identify Management Driver, that updates the address book. Initially the problem only affected the server IDM was connect to so this was our first culprit. When the problems started to occur on a server which did not have IDM, our thinking started to move toward the HTTP task, especially since our backup server does not have HTTP running and the problem has never occurred. Although our two live servers usually have 1000 concurrent users during office hours, the problem often occurred late evening when there’s less user activity. So whatever the cause of the problem was, it wasn’t easy to identify and we couldn’t trace the problem to any particular user activity.

During our ‘crisis’ week, we poked around the system typing to find anything that would alleviate our problems.

As we poked around the system and our own code, we realised a number of our agents have seasonal patterns that matched the increase in server issues. These heavily used agents, such as an agent that issues exam results, logged user’s actions and the data to text logs.

As part of our ‘poke and hope’ plan, we switched off all HTTP agent text file logging. Since then the problems have not reoccurred within our reboot periods (we reboot each server once a week). I have since intentionally brought a live server down using a text logging agent. However, after further testing against our development servers, the cause of the issue appears to be more complex than it first appears. Despite hammering our test servers using jmeter and the same agents, we rarely replicate the issue. So the actual issue is more complex than a simple bug in text logging code, there’s a more complicated interaction occuring. But switching off the code seem to reduce the likelihood of the ‘stars aligning’.

The text logging uses the standard method for opening files

fileNum% = FreeFile()
Open filename For Output As fileNum%
Print #fileNum%, strOutputString
Close #filenum%

So there’s nothing unusual about the code

Although the  problem is still sitting with Lotus, they’re still unable to identify what’s causing the problem. However the early indications are that the changes we’ve made have significantly reduced the number of occasions the permissions issue occurs.

We use the Domino redirect [url] to direct to users to different web pages.

One of our systems handles the Athens devolved authentication used many by UK Universities to provide access to journal subscriptions to students off-campus.

Athens basically works by a set of redirects and can be used by users either pre journal search or post. If they choose not to pre-authenticate. When a user accesses a journal they’re directed to Athen’s login page. They can then identify their host institution. At this point a series of redirects takes place.

Basically Athens sends a long URL to a Java agent on Domino. This url has a long MD5 session hash and information to pass back to Athens so that it can then direct the user to the correct page on publisher’s website.

We process the hash, validate it and then using a private key create a new hash that includes information about the user such as a unique id number and any permission sets that apply to them.

Today, it stopped working.

A publisher recently restructured their website and this created much longer url hashes. (2200 characters long)

Every time the agent was called from the Athens, Domino generated a http error – ‘Buffer Overflow’.

After some debugging. We identified that the error was generated by the in-built Domino redirect. It appears that there is a 2048 character limit on Java redirects.

The solution is simple to implement. You just need to return url string as part of the header via the http ‘Location’. But it’s odd that Domino will accept 4kb’s long URLs but less for redirects.

Technote: ‘Buffer Overflow Exception’ error using URL redirection with a string longer than 2048

We use a Google Search Appliance to index our Domino websites. We use Google’s OneBox module to execute FT searches to return results from a database we don’t want Google to index (basically 80,000 journal records we don’t want to be counted against our license total).

Yesterday we upgraded our Search Appliance to the latest version of Google’s software (6.2.0.G14). About an hour after switching it on, one of our Domino servers was hit by a sustained denial of service. The Agent called by the Google OneBox module had saturated all our HTTP threads. The Google box was making continuous requests to the search agent at a rate of nearly 200,000 per hour. The odd thing was, Google’s OneBox module passes across the end-user’s IP address as part of the search query and all requests were coming from 216.239.43.1 – a Google IP address.

It appears that there’s a known issue with the OneBox module that can cause this. (Bug report #2368523). Google immediately applied the patch, and after an hour the requests had stopped.

If you intend to upgrade to 6.2. I suggest you remove all OneBox modules from your front-ends before upgrading

So related to this problem, I find it easy to create a DOS on a Domino server. Calling any agent that takes a second to return results continuously creates a DOS (it might simply be a case of holding F5). Our server’s are set up as per Lotus’s recommendations. But does anyone have any tips for optimising the Domino http stack (Solaris) to avoid DOS?

Sorry, for posting a job advert. But it might be useful to any out of work Domino developers.

2 x Applications and Development Consultant (with a focus on Learning Technology)

We are looking for experienced web-based software development professionals with substantial programming experience (preferably with expertise in Lotus Domino, Lotuscript, Java but additional languages such as .NET would be of interest) to join our web and learning applications team; developing and supporting the University’s intranet and on-line learning and business support environments.

More details can be found on The University of Hertfordshire’s website

]]> http://blog.pahudson.net/archives/217/feed 0 http://blog.pahudson.net/archives/214 http://blog.pahudson.net/archives/214#comments Wed, 14 Oct 2009 12:01:41 +0000 Paul http://blog.pahudson.net/?p=214

With the release of 8.5.1 I thought I’d make the switch. So far I’ve been impressed. I’ve used 8.5 to play with XPages but I’ve not used it as my main development tool, so I’ve haven’t updated existing apps. Today I’ve been updating a 7 app and wanted to add some ‘computed text’ but couldn’t find it under the ‘create’ menu. After much hair pulling and unhelpful ‘help’ files telling me it’s still under ‘create’. I’ve (finally) found that the menus default to ‘simple’ menus and you have to switch on ‘advanced menus’. (‘View‘ -> ‘Advanced menus‘).

We’ve just generated our yearly overview stats for our Domino-based Learning Environment. Our Learning Environment is firmly embedded within the University’s learning and teaching so it’s heavily used. Some of the figures are:

Daily Student Logins: 13,000

Daily Staff Logins: 1300

Data Transferred: 14TB

Average Daily Transfer: 46GB

Page Requests: 300,000,000

2008/9 Module Websites: 4000 (over 20,000+ in total)

• Discussions Topics: 24,000
• Teaching Resources: 120,000
• Assignment submissions: 80,000
• Quizzes attempted: 33,000
• Wiki Pages: 4000
• Module-level managed groups: 3000+
  • Group Discussions: 12,000
  • Wiki pages: 4700
  • Shared Files: 13,000
  • Blog Entries: 5400

Student Created & Managed Groups: 1400

We started developing our in house learning environment in 2000. It’s completely web based and the only Notes clients are our development and admin clients. It’s linked into our student record system so modules are automatically created and student registered on them.

Even back in 2000, those ‘in the know’ (i.e. Computer Science lecturers) argued that the technology wouldn’t scale and would be abandoned by IBM. However, we found the technology a good fit for our team’s skill-base which was more ‘learning technologies’ based (i.e. Director, VB etc). I think after nearly 10 years, we can say Domino does scale!

Our Domino servers (7.03) run on Solaris and we’ve recently moved them onto new hardware. As part of the move, they have been moved in Solaris zones. Unfortunately since the changes we’ve experienced a serious issue that appears to be related to server permissions. Users are loosing the ability to upload files as it appears that the server is rewriting the permissions to the ID file so root access is required. Effectively the server is loosing access to it’s own ID file. So far we’ve been unable to identify why this happens.

We all experience issues with all software and Domino / Notes is not immune but it’s really infuriating when two issues strike at once.

We’ve been moving server’s around and my Design (8.02) client stopped replicating. Not in the normal way, an invalid connection document, but one where any replication event caused an immediate NSD. After much hair pulling – and at my age that’s something you want to avoid in case it doesn’t grow back – the only solution appears to be a complete reinstall.

The second more serious issue was that one of our servers lost permissions to it’s own ID file. Although HTTP continued to serve files anything requiring the ID file stopped functioning. A quick chmod solved the problem and all functionality returned. But this is the first live server we’ve run in a Solaris zone and the how zones function is beyond my basic unix skills. Hopefully this is a simple unix configuration issue and not some fundamental problem with domino on solaris (it wouldn’t be the first time we’ve run into solaris issues). One for our Domino admin to sort out tomorrow.

We have a facility for staff that allows them to download an Excel marking scheme, enter student marks and then to upload it as a tab delimited file (or xml file) back into the system. Last night a member of staff using the facility crashed both servers in our cluster.

On further inspection (and a lot of development server crashes!) I finally narrowed the problem down to a single line of code

Line Input #fileID, strLine

On further investigation it appears that the user exported their marking scheme from Excel using the ‘Unicode (.txt)’ rather than ‘ Text (.txt)’ and it’s the import of Unicode that was causing Domino to crash. My quick solution to stop the server crash is to use ‘Dos2Unix’ convert the file before importing. Although it seems to alter the format so that it no long imports correctly, it’s stopped the server from crashing.

A word of warning if you use a Google Search Appliance to index your Domino content. The latest version of the appliance software (5.2.0.G32) has a bug which means it is now case sensitive when checking for Domino rewrite and ignore rules. For example it will only exclude agents if they are correctly capitalised. i.e OpenAgent. If you have agents which might cause you some problems (say agents that delete content or send emails), make sure you add ignore statements to the exception lists before starting the index. Luckily our Google box runs with ‘student’ access so couldn’t do any damage!