Yesterday at 10.40am, Jamey, my fellow web developer, accidentally deleted out the entire database holding all our yearbooks online – photos, entries, pages, users… everything. Suddenly anyone trying to access the website would have seen a blank error page, not knowing the full extent of the problem. I found out about 20 seconds later when Jamey said “oh God… no!” or something to that effect, with his head buried in his hands.
Jamey’s unplugging of the office server so as to plug in a cheese toastie machine a week earlier became trivial in comparison.
Oh no… had we really just lost everything? If we had, the business would have pretty much been ruined. Our hundreds of schools, tens of thousands of users creating yearbooks with us this year would have to go back to square one in creating their books, and more likely they’d jump ship to our competitors, demanding deposits back.
Luckily – and diligently – we keep backups.
15 minutes later and the website was back online, seemingly as though nothing had ever happened. Regrettably, however, any and all data added and changes made since 2am that morning had irretrievably been lost. We really are very sorry about this, but we’re also very fortunate that this only affected approximately 30 users.
Anyone logging into the website in the few hours after the data lost saw a message explaining the problem and linking them to this blog. We communicated the problem honestly and transparently without hiding anything.
We’ve immediately implemented some new policies to ensure this never happens again, and also to ensure that if something similar does somehow happen, that the damage done is an absolute minimum.
So how safe is your data with us?
By safe here, I mean what’s the chance of something similar to the above happening with the end result being that your whole yearbook with us is deleted, all that work lost, and you have to start again from scratch?
As above, we keep backups. Every night, at around 2am when everyone’s fast asleep, our server backs up the database and all uploaded files and photos onto a separate physical disk on the server. So if the first disk blew up, there’d be a backup. Or if Jamey deleted the database again, there’s a backup.
But we go further. What if the whole server blew up, or the location the server’s in was burgled and the server was stolen? Right after backing everything up to the second disk, our office server connects to the main server and backs everything up to itself as well. So your data is now in three places: the live disk on the main server, a backup disk on the main server, and a disk on the office server.
But we go further. Say the office burns down the same night the server hosting facility burns down. Or say a virus makes its way onto the main server and also onto our office server. What then? Well, (almost) every day Nicola and David alternately carry portable USB hard drives with them to work. They plug them in, and everything gets backed up onto these drives. They then take them home with them at night.
Oh, and your data is also safe if you accidentally delete it with us. When you delete something like a photo or even your whole yearbook in our database, we don’t actually delete it straight away. We just mark it as deleted and then wait a while before properly deleting it. So if you change your mind in the meantime we can easily get it back 🙂
So worst case scenario, the office and main server blow up, Nic or David just bring their hard drive in and we get back up and running from there. At most we’d lose three days of data (after a weekend) and the site would be down for maybe half a day.
How safe are your files if making your own pages on your computer?
If you’re making the yearbook completely yourself offline, for example in Word or Publisher or using one of the software solutions our competitors sell, then how safe is your data? What’s the chance of your losing all that hard work?
Unfortunately, unless you’re extremely diligent with backups, your data is nowhere near as safe as it is with us. Your hard drive could (and they do, oh yes they do) stop working at any time. Your computer could be stolen. Your computer could get a virus. You could accidentally delete a file.
How safe is your data with our competitors?
Ask them. If you’re using a different yearbook company and they have an online system, ask them what their backup policy is. The last thing you want is to put so much hard work into your yearbook only to find it disappears just like that.
NOTE: I’ll do another blog post soon talking about how safe your data is with us in terms of privacy and data security. This post was just about the likelihood of losing all that data!