Ramble On

Last year I blogged about a permissions issue with Domino 7 running in a Solaris zone.  Domino’s own ID file’s permissions are reset so only root has access rights. This stops the server from executing it’s own code. The problem also effects other files modified by the HTTP task such as DTF files, cgi-bin, and even the web logs. The problem is still effecting us but we’ve discovered a way to alleviate our problems so our solution may help anyone suffering similar problems.

Background

Traditionally we’ve been very careful with our hardware updates, always making single changes at a time. However last year, strategic decisions to consolidated hardware and move towards greener server rooms meant that we had to move our Domino systems from dedicated servers with their own disk arrays to services hosted on Sun Enterprise servers along with moving our data on a SAN. Due to circumstances out of everyone’s control, the original plan to gradually move step by step to the new consolidated approach had to be moved to a very aggressive timetable (i.e. Everything done at once). At this time our servers were also moved from Solaris 9 to 10 and moved into Zones, as per our current Solaris virtualisation policy.

For me this was too many changes at once and even now I am still certain the change to using Zones is part of the underlying cause of our problems.

‘Identifying’ the problem

We reported the problem to Lotus last summer. Since then we’ve been back and forth between Lotus and Sun to identify where the issue lies, regularly updating our systems to record  more debugging information.

Since we’ve been unable to generate the problem on demand,  it’s often weeks before we could update Lotus with further data. But recently the issues increased to such an extent that the problem was occurring several times a day to both user-facing servers (where previously it was once a month to only one server). As the problems increased, we started to ‘poke and hope’ trying every idea possible while planning both moves to Domino 8.x and de-zoning one server.

Our thinking has always been that either the HTTP task is the cause of the problem or the Novell Identify Management Driver, that updates the address book. Initially the problem only affected the server IDM was connect to so this was our first culprit. When the problems started to occur on a server which did not have IDM, our thinking started to move toward the HTTP task, especially since our backup server does not have HTTP running and the problem has never occurred. Although our two live servers usually have 1000 concurrent users during office hours, the problem often occurred late evening when there’s less user activity. So whatever the cause of the problem was, it wasn’t easy to identify and we couldn’t trace the problem to any particular user activity.

During our ‘crisis’ week, we poked around the system typing to find anything that would alleviate our problems.

As we poked around the system and our own code, we realised a number of our agents have seasonal patterns that matched the increase in server issues. These heavily used agents, such as an agent that issues exam results, logged user’s actions and the data to text logs.

As part of our ‘poke and hope’ plan, we switched off all HTTP agent text file logging. Since then the problems have not reoccurred within our reboot periods (we reboot each server once a week). I have since intentionally brought a live server down using a text logging agent. However, after further testing against our development servers, the cause of the issue appears to be more complex than it first appears. Despite hammering our test servers using jmeter and the same agents, we rarely replicate the issue. So the actual issue is more complex than a simple bug in text logging code, there’s a more complicated interaction occuring. But switching off the code seem to reduce the likelihood of the ‘stars aligning’.

The text logging uses the standard method for opening files

fileNum% = FreeFile()
Open filename For Output As fileNum%
Print #fileNum%, strOutputString
Close #filenum%

So there’s nothing unusual about the code

Although the  problem is still sitting with Lotus, they’re still unable to identify what’s causing the problem. However the early indications are that the changes we’ve made have significantly reduced the number of occasions the permissions issue occurs.

Domino

We use the Domino redirect [url] to direct to users to different web pages.

One of our systems handles the Athens devolved authentication used many by UK Universities to provide access to journal subscriptions to students off-campus.

Athens basically works by a set of redirects and can be used by users either pre journal search or post. If they choose not to pre-authenticate. When a user accesses a journal they’re directed to Athen’s login page. They can then identify their host institution. At this point a series of redirects takes place.

Basically Athens sends a long URL to a Java agent on Domino. This url has a long MD5 session hash and information to pass back to Athens so that it can then direct the user to the correct page on publisher’s website.

We process the hash, validate it and then using a private key create a new hash that includes information about the user such as a unique id number and any permission sets that apply to them.

Today, it stopped working.

A publisher recently restructured their website and this created much longer url hashes. (2200 characters long)

Every time the agent was called from the Athens, Domino generated a http error – ‘Buffer Overflow’.

After some debugging. We identified that the error was generated by the in-built Domino redirect. It appears that there is a 2048 character limit on Java redirects.

The solution is simple to implement. You just need to return url string as part of the header via the http ‘Location’. But it’s odd that Domino will accept 4kb’s long URLs but less for redirects.

Technote: ‘Buffer Overflow Exception’ error using URL redirection with a string longer than 2048

Domino Buffer Overflow, Domino, HTTP

We use a Google Search Appliance to index our Domino websites. We use Google’s OneBox module to execute FT searches to return results from a database we don’t want Google to index (basically 80,000 journal records we don’t want to be counted against our license total).

Yesterday we upgraded our Search Appliance to the latest version of Google’s software (6.2.0.G14). About an hour after switching it on, one of our Domino servers was hit by a sustained denial of service. The Agent called by the Google OneBox module had saturated all our HTTP threads. The Google box was making continuous requests to the search agent at a rate of nearly 200,000 per hour. The odd thing was, Google’s OneBox module passes across the end-user’s IP address as part of the search query and all requests were coming from 216.239.43.1 – a Google IP address.

It appears that there’s a known issue with the OneBox module that can cause this. (Bug report #2368523). Google immediately applied the patch, and after an hour the requests had stopped.

If you intend to upgrade to 6.2. I suggest you remove all OneBox modules from your front-ends before upgrading

So related to this problem, I find it easy to create a DOS on a Domino server. Calling any agent that takes a second to return results continuously creates a DOS (it might simply be a case of holding F5). Our server’s are set up as per Lotus’s recommendations. But does anyone have any tips for optimising the Domino http stack (Solaris) to avoid DOS?

Domino Google Search Appliance

Sorry, for posting a job advert. But it might be useful to any out of work Domino developers.

2 x Applications and Development Consultant (with a focus on Learning Technology)

We are looking for experienced web-based software development professionals with substantial programming experience (preferably with expertise in Lotus Domino, Lotuscript, Java but additional languages such as .NET would be of interest) to join our web and learning applications team; developing and supporting the University’s intranet and on-line learning and business support environments.

More details can be found on The University of Hertfordshire’s website

Domino Domino, Job

With the release of 8.5.1 I thought I’d make the switch. So far I’ve been impressed. I’ve used 8.5 to play with XPages but I’ve not used it as my main development tool, so I’ve haven’t updated existing apps. Today I’ve been updating a 7 app and wanted to add some ‘computed text’ but couldn’t find it under the ‘create’ menu. After much hair pulling and unhelpful ‘help’ files telling me it’s still under ‘create’. I’ve (finally) found that the menus default to ‘simple’ menus and you have to switch on ‘advanced menus’. (‘View‘ -> ‘Advanced menus‘).

Domino Configuration, Domino 8.5.1

We’ve just generated our yearly overview stats for our Domino-based Learning Environment. Our Learning Environment is firmly embedded within the University’s learning and teaching so it’s heavily used. Some of the figures are:

Daily Student Logins: 13,000

Daily Staff Logins: 1300

Data Transferred: 14TB

Average Daily Transfer: 46GB

Page Requests: 300,000,000

2008/9 Module Websites: 4000 (over 20,000+ in total)

• Discussions Topics: 24,000
• Teaching Resources: 120,000
• Assignment submissions: 80,000
• Quizzes attempted: 33,000
• Wiki Pages: 4000
• Module-level managed groups: 3000+
  • Group Discussions: 12,000
  • Wiki pages: 4700
  • Shared Files: 13,000
  • Blog Entries: 5400

Student Created & Managed Groups: 1400

We started developing our in house learning environment in 2000. It’s completely web based and the only Notes clients are our development and admin clients. It’s linked into our student record system so modules are automatically created and student registered on them.

Even back in 2000, those ‘in the know’ (i.e. Computer Science lecturers) argued that the technology wouldn’t scale and would be abandoned by IBM. However, we found the technology a good fit for our team’s skill-base which was more ‘learning technologies’ based (i.e. Director, VB etc). I think after nearly 10 years, we can say Domino does scale!

Domino

Our Domino servers (7.03) run on Solaris and we’ve recently moved them onto new hardware. As part of the move, they have been moved in Solaris zones. Unfortunately since the changes we’ve experienced a serious issue that appears to be related to server permissions. Users are loosing the ability to upload files as it appears that the server is rewriting the permissions to the ID file so root access is required. Effectively the server is loosing access to it’s own ID file. So far we’ve been unable to identify why this happens.

Domino Domino, Solaris

We all experience issues with all software and Domino / Notes is not immune but it’s really infuriating when two issues strike at once.

We’ve been moving server’s around and my Design (8.02) client stopped replicating. Not in the normal way, an invalid connection document, but one where any replication event caused an immediate NSD. After much hair pulling – and at my age that’s something you want to avoid in case it doesn’t grow back – the only solution appears to be a complete reinstall.

The second more serious issue was that one of our servers lost permissions to it’s own ID file. Although HTTP continued to serve files anything requiring the ID file stopped functioning. A quick chmod solved the problem and all functionality returned. But this is the first live server we’ve run in a Solaris zone and the how zones function is beyond my basic unix skills. Hopefully this is a simple unix configuration issue and not some fundamental problem with domino on solaris (it wouldn’t be the first time we’ve run into solaris issues). One for our Domino admin to sort out tomorrow.

Domino Domino, Solaris

We have a facility for staff that allows them to download an Excel marking scheme, enter student marks and then to upload it as a tab delimited file (or xml file) back into the system. Last night a member of staff using the facility crashed both servers in our cluster.

On further inspection (and a lot of development server crashes!) I finally narrowed the problem down to a single line of code

Line Input #fileID, strLine

On further investigation it appears that the user exported their marking scheme from Excel using the ‘Unicode (.txt)’ rather than ‘ Text (.txt)’ and it’s the import of Unicode that was causing Domino to crash. My quick solution to stop the server crash is to use ‘Dos2Unix’ convert the file before importing. Although it seems to alter the format so that it no long imports correctly, it’s stopped the server from crashing.

Domino Crash, Domino, Solaris, Unicode

A word of warning if you use a Google Search Appliance to index your Domino content. The latest version of the appliance software (5.2.0.G32) has a bug which means it is now case sensitive when checking for Domino rewrite and ignore rules. For example it will only exclude agents if they are correctly capitalised. i.e OpenAgent. If you have agents which might cause you some problems (say agents that delete content or send emails), make sure you add ignore statements to the exception lists before starting the index. Luckily our Google box runs with ‘student’ access so couldn’t do any damage!

Domino Domino, Google Search Appliance

I’ve still got document locking in test. New ‘features’ keep cropping up so I’ve held off running the design update!

The latest issue is that databases with locking enabled are generating email messages warning users when their messages have generated a conflict. So far I’ve been unable to stop these emails.

There are a few comments reporting the same issue on the support forums and there’s an IBM reply stating that it’s been recorded as a bug – but this feature has been around since V6 and it’s still in 7.03 and I’m not sure it is a bug, more a helpful feature without a facility to disable it.

I’d prefer to find a solution to this before I roll this out to our module websites – there are 4000 of them (per year) and they’re heavily used an save conflicts do happen. I suspect cryptic messages to our students about ‘Databases….’ might be confusing.

Domino

A survey by WebAIM on how users of screen readers interact with web pages is worth a read. It gives a small insight into how screen reader users navigate around pages and some of the problems they face. The survey failed to clarify the term Web 2.0 so the survey hasn’t helped to understand if AJAX based sites disenfranchise partially sighted users but it does confirm some of the advice accessibility experts make. It also reinforces some of the point I tried to make to the Domino developers at Lotusphere about where the new XPage technology fails to meet basic accessibility criteria, namely that screen reader users use the page’s semantics to navigate around the page. A good practice that XPage authoring using the visual interface fails to support.

When html was originally specified it was intended to be a docuument markup language and the tags selected define the page’s content – headings, paragraphs, lists, etc. As web developers we all know this, it’s really basic stuff.

Since headings are more important than paragraphs, a significant percentage of screen reader users use headings to quickly navigate around sections of the page and this confirms the advice accessibility experts have been giving for years. Define a readable document and then style it to look like an application. XPages gets this fundamentally wrong and provides no visual tools to generate standard page markup, a glaring omission that no other editor that I’m aware off fails to include.

All’s not lost. If you want to develop a semantic page it’s just a case of switching to code view and you can add normal markup and it wouldn’t take much for Lotus to add some simple page editing tools.

Hopefully surveys like this help developers see how some simple changes to their pages can make page browsing easier for a small forgotten number of web users

Accessibility, Domino, Web Design, XPages Accessibility, Domino, Web Design, XPages

Flickr is 5 years old today

I’m really glad Flickr has survived and grown from strength to strength.

For me, Flickr is the perfect ‘Web 2.0′ website. It doesn’t throw lots of rotating wait icons all over the place. It only uses dynamic features when they benefit the user and it’s inline editing is implemented perfectly.

Powerful but easy to use – I can’t give it a more glowing review than that.

Domino Flickr

Ok, I’ve been using Domino long enough to know that templates don’t do what you think they should do so I shouldn’t really be surprised.

I had a problem with document locking. Some documents couldn’t be edited because there were some temporary locks being set and my code didn’t unlock them. Okay, my code was at fault, I’d accidentally allowed temporary locks when the administration server was unavailable, but since our servers only get rebooted once a week about 2 in the morning and the documents in question were created during the day, I shouldn’t have been seeing temporary locks.

On further investigation it appears that reason temporary locks were being set was because there was no administration server set in the ACL. This was despite the fact I had set one in the template the database was created from.

I’ve never understood why some features are inherited at creation and others aren’t. If the tick option for document locking is inherited and designed to use the administration server to ensure locking in a cluster, is it wrong of me to think that this should also be inherited from the template?

Domino Domino

From my personal perspective the big feature in 8.5 is XPages, but 8.5 is not a one trick pony. There have been a number of other improvements, such as the ID Vault (something not applicable in our institution) and Domino Attachment and Object Service (DAOS).

DAOS

DAOS is described elsewhere. But basically, DAOS allows you to store the attachments in your Domino databases outside of the database, reducing the size of the nsf. 

Big figures are being thrown around; reducing storage size by 40-60%, removal of duplicate attachments and DAOS invisible to Domino apps

DAOS can be enabled per-database or across the entire server and it’s possible to undo the change at a later stage, which is excellent news. Also since it’s implemented at the API level, it’s invisible to the end user or developer.

Reflecting on our system, DAOS  offers:

• Quicker backups. DAOS are stored as files on the file system so can be incrementally backed up.
• NSF are smaller. Since these are loaded into the server’s cache for web delivery, this means more apps can stay resident in memory and so less disk IO.
• We create 4000 modules (databases) per year and carry across teaching resources from previous years. Often resources are uploaded as attachments, and don’t change year on year, so we should see another significant space saving

Other 8.5 features and benefits

• 8.5 contains 400 bug fixes to the nsf
• 50% reduction in CPU for transaction logging
• ID Vault (easier password change)
• Domino configuration tuner for identifying performance issues with your Domino setup. This will be updated as and when, rather than being tied to a Domino release, another positive step from Lotus. 

Developments post 8.5

There will finally be a new Eclipse based lotuscript editor release with 8.5.1

Further ahead the team are investigating how XPages can consume SOAP, REST, and XML (data sources). New controls for XPages such as menu bar, toolbar, outline and general performance improvements.

They are also working on directory independence such as using user info from ldap or active directory to improve single sign-on

Domino, XPages Domino, Domino 8.5, XPages

This is the biggie for us web developers. This is the first session on the technology at this year’s Lotusphere. It’s definitely popular, and hence very warm.

The session was ran by Maureen Layman and a newbie Maire Kehoe and demonstrtedb(live) many of the new XPage features in Designer 8.5. Maire was clearly nevous but did pretty well.

Most of the stuff was basic stuff. I’ve not looked at themes or localisation yet and I wasn’t aware of the server side inline errors (though configuring them didn’t seem to be obvious), so I got something out of the session.

Unfortunately, as expected, nothing was mentioned about accessibility.

Hopefully by the end of the week, i’ll see some good examples of how developers have used XPages.

Domino, Lotusphere, XPages

I thought I’d document over the next few months my baby steps in learning XPages. Although I won’t be using XPages for any development in the short to medium term, I’m going to start to investigate the accessibility implications of each option before developing anything in anger.

My first step – and it really is a first step almost embarrassing step, was to simply add html to my page so that I can develop pages with a semantic structure. Stop laughing at the back. We all have to start somewhere.

The Designer client offers some simple tools to style on-screen text

Don’t use them. The html this toolbar produces uses inline styles rather than standard html and explains why the discussion template’s html seems to be missing any form of structure.

Using the bold face option adds

<span style="font-weight:bold">Bold text</span>

The italic and underline also generate inline styles. So using these options could bloat your code considerably.

The increase and decrease font size option also generates in-line styles:

<span style="font-size:14pt">14 pt text<span>


As you can see, it also uses ‘points’ as the default unit rather than pixels or ems, a unit I’ve avoided because it renders differently across browsers. So another reason to avoid this edit bar.

As far as I can see, there doesn’t appear to be an option to set headings and paragraphs, or to insert rules and lists. So how do you add them?

One option appears to be to use the ‘computed field’ option and to set the ‘content type’ to html and for the ‘value’ to use the javascript return function to return html. For example

return ("<h4>This is the header</h4>")

This seems to be a too long winded for my liking so the other alternative seems to be to switch to ‘source’ view and to manually add the html directly in the source view. I have to admit, I’m surprised that there isn’t an WYSIWYG tools to help you produce a page with any form of semantic structure.

One word of warning. It appears 8.5 continues Domino’s habit of generating BRs for every carriage return.

Domino, XPages Domino 8.5, XPages

I wonder why it is that many believe Notes is dead?  Compare and contrast two recent announcements from IBM and Apple

Notes/Domino 8.5 release

(There is a little link in the news banner in the centre of the screen to a press release about Notes on the Mac)

iLife ’09

On Apple’s site, there are loads of videos and information telling us why iLife ’09 is worth upgrading to. On the IBM site we get

If I hadn’t been to Lotusphere 2008 and didn’t follow the Lotus related blogs, would this page really convince me that Notes 8.5 is a fantastic product?

Apple, Domino

…. confusing! I’m sure it will make sense once I start to use it. 

Had a brief look at the 8.5 discussion template and it appears all that all link properties are blank so no effort has been made to generate alternative views when Javascript is unavailable. Bad design. 

I’m not sure when I’ll be able to use XPages in anger. Our learning environment has 22,000 users with 14,000 individuals logging in daily. It’s completely web based – no Notes clients, and with the environment heavily used in teaching, we have to ensure the software is reliable before upgrading. We’ve experienced reliability issues with Domino on Solaris pre X.02/X.52 release so we tend to wait for these releases. We’re also are limited to making big system upgrades to the month of August. So it may even be 2010 before I see 8.5 live on our systems. Let’s hope 8.5 proves to be reliable and we can go sooner.

Domino Domino, Web Design

Ok, now this is beyond a joke. Sorry, but Domino 8.5 is going to be an accessibility nightmare. I know the discussion template is used many many companies and I’ve seen some University’s base their discussion facilities on this template (not us, we wrote our own), but the new template makes NO ATTEMPT to be accessible, both from point of view of handling javascript or by using semantic HTML.

Sean Cull has kindly left his 8.5 discussion template  open for us to play with.

If you’ve got Firefox with the web developer tool bar installed. Switch off css

Notice how the page just collapses into a mess. None of the generally accepted conventions for semantic pages have been obeyed (such as marking up related links as lists).

It’s even worse if you switch off javascript. Although many of the advanced screen readers such as JAWS, integrate with Internet Explorer and deal with certain types of Javascript events, it’s generally accepted in accessibility circles that wherever possible, web developers should attempt to make as much of the page work without javascript as possible. In fact nothing in the discussion template works without javascript. All the links are to ‘#’ with the clicks being handled through events. Why, when there’s a document stored in the database is it now possible to show the document when javascript is switched off.

If the template demonstrated significant usability improvements over the existing template it would almost be acceptable, but the template is very basic. Where’s the different view modes such as viewing discussions as linear threads or the ability to filter discussions to the current conversation (i.e. limiting the view to the direct reply hierarchy). Where’s the facility to view the message you’re writing your reply to? Where’s the lookup when tags are added?

A core template such as this, should be accessible. If I was evaluating this product for use within the University it would not make it past the tender. It would fail on every accessibility point.  Sorry, it’s simply that bad.

Domino, Web Design Accessiblity, Domino, Lotusphere 2008, Web Design