We’ve just upgraded a server to 8.5.3. We have an agent that runs as a ‘web user’ but has ‘full admin rights’. The agent basically reads a view with documents that have readers fields and assembles them for display to the user. The agent needs to grab info from a protected database so has ‘full administration rights’. Previously the documents were read with the ‘users’ access permissions. So readers fields were obeyed. Various versions of this code has run this way for 10 years. Under 8.5.3 this is not the case. The readers fields are being ignored. Has anyone else experienced this issue?

I’m posting this on behalf of my colleague David Harding (@dharding) who’s been investigating XPages Single Copy Design. I think it’s best I leave it to Dave to explain:

We are looking to roll out XPages Single Copy Design to several thousand databases.  So far, we have not been able to find a recommended practice for rolling out the XPages Single Copy Design flag and template path to all of these databases.

Documentation on the Lotus Notes and Domino Application Development wiki (http://www-10.lotus.com/ldd/ddwiki.nsf/dx/Single_Copy_XPage_Design#Does+employing+SCXD+typically+have+a+positive+or+negative+affect+on+speed+of+performance%3F) suggests that setting the options in the template should propagate to databases that inherit from that template.

Quote from page:

“Is it true that selecting or de-selecting SCXD in a template does not propagate to applications already created with that template?

This is not the case. If a template is set to use a SCXD database and an application is created from this template then the new application will have the XPages from the SCXD database. And changing the template to say another SCXD or no SCXD database set will reflect in the database that inherits its design from the template.”

In testing, we have not found this to be the case.  Testing adding in the Xpage template path and checking the flag in the template*, combined with all combinations of HTTP Restarts and Design updates do not seem to visibly update the database.properties file in the inheriting database to include the $XPageSharedDesign and $XpageSharedDesignTemplate items.

*When I say Template here, I am referring to the ntf file that the database inherits from, not the Xpages source database that will be the path in the $XpageSharedDesignTemplate item in the database.properties file.

In an attempt to find an alternative, reliable method of deploying this setting we have looked at using a NotesNoteCollection to access the database.properties file and adding in the values.  There doesn’t seem to be an obvious way to locate this file, though tests have indicated that it is the only one in the database with a $Daos item, so we have been able to get a hold of it by filtering for that in the NotesNoteCollection.

So the questions are, is there a ‘best practice’ for rolling out the Single Copy XPage flag? Is the NotesNoteCollection method a no-go area?  Why are our experiences with the template not in line with the claims in the aforementioned quote?

If you use Domino’s HTTP GZIP compression, you may have issues with your Google Search Appliance reporting:

Malformed HTTP header: empty content

It appears that the Google box will tell Domino that it accepts compressed content in the HTTP headers even though it is unable to process the content. Luckily I found an easy solution on the Google Search Appliance support forum.

The solution is to add headers to the Google Box’s http requests.

1. Open the GSA admin interface.
2. Select ‘Crawl and Index‘ -> ‘HTTP Headers‘
3. In the ‘Additional HTTP Headers for Crawler’ enter:

Accept-Encoding: *;q=0

If you are one of the few Solaris users and use the ‘Dir$’ function to read directories in your administrative routines to process databases in a directory. Be aware that it doesn’t work for databases over 2GBs in size. They’re omitted from list.

We PMRed it. But it’s already been closed. Apparently it’s a code limitation and won’t be getting fixed. I expect it will be all that 32-bit Solaris code we’re still having to use. Don’t you just love having your hardware tied down by software?

This blog post has been a long time in writing.

From about 1998, my team was responsible for supporting learning technology projects within the University. The range of projects was diverse from programmable computer chip simulations to supporting more traditional computer aided learning packages such as CD-ROMs or web-based ‘interactive’ learning objects. We also had a responsibility to identify technologies that could be implemented as University services.

A number of our early distance learning projects had requirements for on-line support. Our manager heard of a project running in Engineering that was using something called ‘Lotus Notes’ to support their distance learning course. Despite some misgivings on my part, Lotus Notes was adopted for our projects. I have to admit, my first impressions (and continued impression) is that the Notes client is awful. I’m probably the only Domino developer who’s never developed for the client. Luckily as a group, we weren’t interested in proliferating clients around the Uni and were more interested in the ability for making Domino database available to students through the web.

A support environment for MSc Computer Science built in Lotus Notes

As it turned out, I didn’t work on these early Notes projects. Watching colleagues develop the early projects, I really wasn’t impressed by development for the Notes client. It seemed to me that to make something work through the web and client was far too time consuming. It also seemed that the design of the web side was compromised by the client side (for example using framesets). The early projects also seemed to involve too much training and support, such as showing users how to use the Notes client and setting up users accounts. But, importantly, it also appeared the Notes offered a very easy to learn development platform where you could create complicated web interactions, much more easily than with Perl, PHP, or ASP. Something that was very useful for a team of Learning Technologist with a mixture of academic, technical and design backgrounds.

As we moved into 2000, we re-evaluated the work the team were doing. It was clear that the targeted ‘learning object’ developments were not working. We’d develop ‘learning objects’ with reasonably high production values and staff found it difficult to maintain the content. We even had a few projects cease after the academic we worked with left. Often interesting pedagogical solutions we would suggest were undermined by the need of staff to maintain them and to be perfectly honest, in 2000 few academics we worked with could think beyond a linear ‘book’ approach to teaching online.  However, the systems we’d built to support these projects had worked better than we expected. So as a group we took the decision to focus on developing a University-wide system that would support staff in their teaching.

At the time, most Universities were looking at ‘Learning Environments’. The two obvious choices were Blackboard and WebCT. We’d played with both, but they appeared to be focused on the academic ‘technology leaders’ and not the average academic, many of whom were still getting to grips with email (I know hard to believe). So as a team we resisted buying a solution and opted to develop our own with the idea that everything should be as simple as sending an email. To be honest, it was a selfish decision. I didn’t want my job to turn into an admin job, just setting up courses and adding students. Developing our own system would be more interesting and we could design the admin ‘out of the system’ by tying everything to our student system and academic structures.

It was at this point our group disintegrated. The manager left and wasn’t replaced. Our web manager left and although we’d recently employed the Notes developer from Engineering, they also found a new job. So as 2001 approached, we were left with two learning technologists and a graphic designer, a promise of more staff, and a commitment to the University to release a University-wide pilot in September 2001. It was important that the technology allowed us to work quickly.

Luckily, the previous summer, I had to develop my first ‘Notes’ project. Unlike colleagues previous Notes projects I took the decision to remove the Notes client from the equation and to design something that could form the basis of a future learning environment. So everything on the website had to be configurable by staff through a web interface. Everything from registering students, selecting functionality, to setting up groups had to be done through a web interface.

Biosciences Module Site

With this forming the basis of the main teaching areas and with the support of a senior manager who thankfully developed the basic integration with the data from our student record system, by May we had a portal structure that could be connected to a series of modules. i.e. the basics of a learning environment.

The Studynet Portal in 2001

Luckily at this point we employed a Notes Administrator who was able to point out some massive flaws in my design that would have crippled any large roll-out. I have to say, we hit the jackpot with our Notes Admin. We didn’t want an administrator to administer ‘Notes’. There are no Notes clients. We don’t use Domino for email. We wanted a administrator who would be an integral part of the development team who would provide the tools to integrate our system with data from other systems, something he was more than happy to do. With his advice, some minor restructuring, we managed to release a scalable prototype and my colleague was able to start the immense task of training hundreds of staff before the September roll-out.

The amazing thing with Domino is how easy it is to set up simple ‘form-capturing’ type websites and this meant that we could create a wide range of resources very quickly. So by the time of release we were able to provide a wide range of functionality from classifieds and discussion forums through to integrations with our library catalogue. So by (late) September we managed to achieve what few thought we would, we rolled-out this new environment to every student (20,000), every member of staff (2000+), and to every module (4000) in the University.

Although I was working 100+ hours every week for 9 months, I’m not sure such a small group of inexperienced ‘developers’ could have achieved such a tight timescale with any other technology available at the time. This to me has been the greatest thing about Domino. It’s amazing what you can achieve with just a little bit of knowledge. This project was testament to that.

It’s now ten years later and we’re still continuing to expand and develop our learning environment. The majority of it is still built in Domino. Our single Solaris server is now part of a cluster of three (two live, one backup). It’s extremely well used, with 13000 students and 1500 members of staff logging on daily, with an average of 1.4 million page requests per day. Our module website functionality has expanded with blogs, wikis, tests, better group work facilities, podcasts,etc and we now handle about 150,000 assignments submissions each academic year. We now have over 30,000 databases that provide websites for the various academic components (i.e. module websites, programme websites etc). There’s even a simple mobile interface. I could keep quoting numbers and features. But on reflection I’m amazed by what we’ve been able to achieve with Domino in the last decade. Especially as we were told by ‘experts’ across the University that Notes was ’ageing technology’ that would never scale and wouldn’t be around in 5 years time, never mind 10.

Studynet Portal (2010)

StudyNet Mobile Interface

So where does the project go from here?

Although I still think Domino can still form the basis of our environment, increasingly, we’re storing data in relational databases for more complex work and we’re building more facilities outside of Domino using ‘cheaper’ technology. 

We’re not yet using XPages. When we have hundreds of thousands lines of code, we’re unlikely to ‘convert’ to XPages, just create new additions as XPages. Plus we’ve learnt the hard way that Domino on Solaris is never reliable at a x.0 and there’s usually something that causes us issues at the x.1 release (usually virus checkers) so we’ve not yet moved all our servers to 8.5.2. Equally, the team’s composition has changed and staff are more development focused and rightly concerned about limiting themselves to a technology that doesn’t transfer elsewhere.

I’m also concerned by Lotus’ apparent decision to not develop a 64 bit version of Domino on Solaris. With us already struggling with memory limitations this could ultimately force us off the platform and if this happened we would have to move to a commercial or open source solution, away from the in-house development.

I also have to state that Lotus’ support for education compared to other vendors has been awful throughout the last decade and hopefully this will also improve. 

But personally, despite some misgivings when we first started to use the technology, I hope that Domino continues to provide us with the tools that we require. So that in 2016 I can write another update on how well it’s served us (I would say 2021, but I don’t think Uni’s will still need learning environments)

We use the Domino redirect [url] to direct to users to different web pages.

One of our systems handles the Athens devolved authentication used many by UK Universities to provide access to journal subscriptions to students off-campus.

Athens basically works by a set of redirects and can be used by users either pre journal search or post. If they choose not to pre-authenticate. When a user accesses a journal they’re directed to Athen’s login page. They can then identify their host institution. At this point a series of redirects takes place.

Basically Athens sends a long URL to a Java agent on Domino. This url has a long MD5 session hash and information to pass back to Athens so that it can then direct the user to the correct page on publisher’s website.

We process the hash, validate it and then using a private key create a new hash that includes information about the user such as a unique id number and any permission sets that apply to them.

Today, it stopped working.

A publisher recently restructured their website and this created much longer url hashes. (2200 characters long)

Every time the agent was called from the Athens, Domino generated a http error – ‘Buffer Overflow’.

After some debugging. We identified that the error was generated by the in-built Domino redirect. It appears that there is a 2048 character limit on Java redirects.

The solution is simple to implement. You just need to return url string as part of the header via the http ‘Location’. But it’s odd that Domino will accept 4kb’s long URLs but less for redirects.

Technote: ‘Buffer Overflow Exception’ error using URL redirection with a string longer than 2048

Sorry, for posting a job advert. But it might be useful to any out of work Domino developers.

2 x Applications and Development Consultant (with a focus on Learning Technology)

We are looking for experienced web-based software development professionals with substantial programming experience (preferably with expertise in Lotus Domino, Lotuscript, Java but additional languages such as .NET would be of interest) to join our web and learning applications team; developing and supporting the University’s intranet and on-line learning and business support environments.

More details can be found on The University of Hertfordshire’s website

]]> http://blog.pahudson.net/archives/217/feed 0 http://blog.pahudson.net/archives/200 http://blog.pahudson.net/archives/200#comments Tue, 16 Jun 2009 20:40:38 +0000 Paul http://blog.pahudson.net/?p=200

Our Domino servers (7.03) run on Solaris and we’ve recently moved them onto new hardware. As part of the move, they have been moved in Solaris zones. Unfortunately since the changes we’ve experienced a serious issue that appears to be related to server permissions. Users are loosing the ability to upload files as it appears that the server is rewriting the permissions to the ID file so root access is required. Effectively the server is loosing access to it’s own ID file. So far we’ve been unable to identify why this happens.

We all experience issues with all software and Domino / Notes is not immune but it’s really infuriating when two issues strike at once.

We’ve been moving server’s around and my Design (8.02) client stopped replicating. Not in the normal way, an invalid connection document, but one where any replication event caused an immediate NSD. After much hair pulling – and at my age that’s something you want to avoid in case it doesn’t grow back – the only solution appears to be a complete reinstall.

The second more serious issue was that one of our servers lost permissions to it’s own ID file. Although HTTP continued to serve files anything requiring the ID file stopped functioning. A quick chmod solved the problem and all functionality returned. But this is the first live server we’ve run in a Solaris zone and the how zones function is beyond my basic unix skills. Hopefully this is a simple unix configuration issue and not some fundamental problem with domino on solaris (it wouldn’t be the first time we’ve run into solaris issues). One for our Domino admin to sort out tomorrow.

We have a facility for staff that allows them to download an Excel marking scheme, enter student marks and then to upload it as a tab delimited file (or xml file) back into the system. Last night a member of staff using the facility crashed both servers in our cluster.

On further inspection (and a lot of development server crashes!) I finally narrowed the problem down to a single line of code

Line Input #fileID, strLine

On further investigation it appears that the user exported their marking scheme from Excel using the ‘Unicode (.txt)’ rather than ‘ Text (.txt)’ and it’s the import of Unicode that was causing Domino to crash. My quick solution to stop the server crash is to use ‘Dos2Unix’ convert the file before importing. Although it seems to alter the format so that it no long imports correctly, it’s stopped the server from crashing.