It saddens me that the first news post I make of this academic year is about a data corruption incident.

On Friday the 10th of October some of you may have noticed that silver (our main server) went away for a bit, and so did your files. What happened was that sometime during Wednesday the 8th of October the part of the server where we keep your (and our) files got corrupted, we didn't notice. Maybe if we had noticed then this wouldn't have been so bad, maybe. However for the next 2 days or so, random files and data were being erased, overwrite or just not being written.

When we finally noticed on the morning of Friday the 10th we took the server offline, after examining the damage we decided the best course of action was to wipe everything and restore from a backup. Due to the fact that files were being randomly written to etc.. the last backup that we had and that we also trusted was taken on the morning of the 8th. That has been restored and you should all now have the vast majority of your files back.

The file restoration process is still going on and at a later date we shall make the backups from the 9th and the 10th available to you all, these are to be used at your own risk as we cannot guarantee that they are what you think they are. Instructions on how to access them will be posted later, watch this space.

We have taken steps to prevent data corruption potentially becoming so bad in the future, upon detecting an error, file systems will be remounted as read only and only after an admin has examined the situation will they become available for writing again.

In the occurrence of unexpected downtime places like facebook ( and twitter ( are where we will post updates.

I would also like to remind that you should have anything important backed up in multiple places.



Posted by imranh on Oct 11, 2014