Unexpected backup failures
I've posted the first part of this before, so please forgive the repetition.
A number of years ago, I worked in IT support in London for a large company.
An important sales division based in Manchester that dealt with the quick-turnaround stuff - pouncing on new opportunities etc. (I don't know the details, but apparently their work was more time sensative than usual) was affected badly by the IRA bombing. Turns out, local ops had a well defined backup procedure, which unfortunately didn't include offsite backups, because they thought their 'bomb proof' room and firesafe were sufficient.
Replacement equipment arrived. The office was quickly assembled temporarily elsewhere, but all the data? processes, biddings, contracts etc.? No problem, we'll grab the backup tapes.
Unfortunately, the building was a crime scene or whatever, and basically no-one was allowed near the place for weeks.
Eventually, someone was allowed to retrieve the tapes. The backups were fine. The firesafe was fine. The bomb-proof ops-room was fine. But by the time anyone was actually allowed to retrieve them, the majority of the data was stale.
------
On a personal note, I've been running my own hosted servers for about 18 years - basically just play things, hosting websites etc., and also hosting all my work and email and everything else, so I can always access everything from everywhere, even a borrowed mobile phone or tablet etc... Think 'the cloud' long before marketing types invented it.
I did regular backups to 2 other servers, both in different countries, and with different providers.
However, I got all 3 servers around the same time, which meant payment renewal was around the same time. That was all done automatically to the credit card, so no problem....
However, unfortunately, about 3 years ago I had a very long hospital stay following illness. It was only 6 months later when I was getting back on my feet that I discovered that payment had been due, and had failed due to my credit card expiring/being renewed.
Basically, all 3 services were shutoff by the time I realised, and despite desperate phone calls, all machines had since been reprovisioned.
I lost everything from the previous five years.. emails, contracts, code, documentation, everything. The reason it wasn't longer was that I managed to recover a 5 year old full backup that was fully intact, off a machine at home that had died 2 years previously due to power supply failure. Ironically, I had originally been doing my backups to this home machine before I decided to switch to a more reliable backup solution...
Biting the hand that feeds IT
