Domino and the future

I know what you’re thinking, not another blog article about Domino’s future. Well no. This is about Domino dealing with future dates and what happens if there’s a blip with your OS’s date

Friday, we had the unfortunate situation of one of our servers briefly believing it was operating in 2020. For the fraction of a second the server blipped, seven databases were accessed, including names.nsf. From this point forward the server believed these databases were operating in 2020. Although dates were correctly being set, all new and edited documents inherited the same last modified date of July 25th 2020 corrupting their replication history. Although the cluster replicator immediately pushed the changes across, it continued to report that it was unable to deal with dates so far in the future.

IBM helpfully has a tech note covering such an eventuality. Don’t bother searching for it because it rather helpfully tells you that your on your own and IBM has no idea what will happen to your system. I guess in someway that’s understandable, but not what you want to see when faced with rebuilding your servers from backups

This is how we (eventually) fixed the issues. It might not work for you and I honestly think we dodged a bullet, if the same happened during term time we would have been looking at lots of corrupted databases and incorrectly submitted assignments.

In our environment we do not have Notes clients, only web users. So to give us breathing space we shut down the effected server’s http task. By this point each database had numerous documents with incorrect dates and corrupted replication histories.

The only solution appears to be to delete the offending databases from the effected server and to replicate a clean version from another server or restore the database from a backup.

We elected for the former so that we didn’t lose changes made since the backup was captured. But this meant the corrupted data has to be cleaned. Luckily this was relatively simple. Simply a case of setting the $Revisions field for each database to 1 and re-save the documents in the database clearing the dates from the good copy.

The only complication was names.NSF which had to come from our backup server and copied while both servers were down

Luckily it was only seven databases and not the 40,000 databases on the server otherwise we’d have been faced with an extended downtime and probably visit to the VC’s office to explain why the University would be without it’s learning environment for a week (the maximum time we’ve been down is a day in the pilot year when we ran off a single server,13 years ago). If the same issue hit us during term time with 2000 concurrent users, I’d hate to think what would happen