The latest scary backup story comes from a firm called Danger that makes the Sidekick PDA/phone. The Sidekick stores the majority of its data in a central data center and the data is loaded each time to the phone is restarted. The idea is that the data center provides protection if you lose your phone. A good idea, right? Well yes, assuming that Danger adequately protects its customers’ data.
A number of outlets are reporting that Danger suffered a catastrophic data loss and all users’ data has been lost. I checked with a family friend who confirmed that her Sidekick was down for a week and is now finally working as a phone, but her data is inaccessible. This is unacceptable; Sidekick users paid a monthly fee for this service and Danger should have maintained reasonable precautions to protect their customers data. Clearly this is a bad situation for everyone, and lessons to be learned by all.
Here are some key takeaways from this event.
1.) Relying solely on local snapshots or even remote mirrors is not enough - If your snapshot lives on the same disk system that holds the primary data and you have a catastrophic data loss, the snapshots could be corrupted as well. Similarly if you are replicating data to a remote site, you also run the risk of replicating the corruption.
2.) Backup is a must – Had Danger had an effective backup strategy, the data would not have been lost. Sure they may have faced some downtime to recover the data, but that is much less painful than the complete loss of data and ensuing loss of business.
3.) Perform recovery tests – Even if you have a consistent backp strategy, it is vital to test recovery. A bad tape drive or bad media can prevent recovery and you want to know this before a disaster occurs! Recovery tests ensure that your backup and recovery processes are running consistently and reliably.
This event is likely to cost Danger millions in legal costs and brand damage. It is unlikely that anyone will want to buy a Sidekick any time soon and so it could even drive Danger (now part of Microsoft) out of business. It is amazing how something as simple as not backing up can have such extreme consequences.
The lesson is that data protection is vital. It can certainly be frustrating and annoying at times, but if you are thinking about taking shortcuts, remember the Sidekick. You do not want to be the person in charge of data protection for Danger or any other company that suffers this kind of loss.