No disaster recovery plan is going to cover every possibility—and some real data center disasters have been whoppers. Data center managers know to watch out for fires, burglaries and sometimes extreme natural disasters if you happened to build your data center in Tornado Alley (bad idea). However, the term “disaster” is largely subjective. It can be natural, manmade and it can come from a wide variety of sparks (both literally and figuratively). While a data center might plan for the possibility of an electrical fire, do you have a plan of action (POA) if an SUV barrels through your facility? Probably not, but it’s happened.
As data center managers hone their disaster recovery plan over the years, they need to make sure to build in some responses for less likely scenarios. Here are a few disasters that actually occurred in data centers, as well as tough lessons someone else has already learned:
- All’s Well That Ends “Weld”: You wouldn’t think a welder would be the source of a disaster, but it happened just this year at an Amazon data center site-to-be. One of the worker’s torches led to a three-alarm fire in Virginia, just days after the New Year. Black smoke covered the skies, and an Amazon spokesperson talked to a local ABC news channel, citing $100,000 in damages. However, Amazon reassured everyone that there was zero effect on operations because the data service was still being built. Well before a data center is built, there are plenty of opportunities for disaster.
- Talk About Solar Powered: Think you know all the natural disasters that can happen? Think again, because a solar storm wreaked havoc on a data center site. Solar storms are rare, but possible just about anywhere on earth. Solar flares can lead to solar storms, which is a coronal ejection stemming from the sun that comes to earth just like rays do. Brimming with charged particles, magnetic fields are created when these flares approach Earth, becoming essentially electricity-bearing cables. According to the Solar Storm Risk to the North American Electric Grid report from London, “One serious threat to the reliability of electric power is geomagnetic storms—severe disturbances caused by solar storms in the upper layers of our atmosphere. The currents they induce can overload the electric grid system to trigger voltage collapse, or worse, damage a significant number of expensive extra-high voltage transformers”
When a solar storm hits, transformers are instantly damaged. This happened in 1989 in Canada when the entire Hydro Quebec power grid was brought down. There’s no telling if and when a solar flare or storm can strike, which makes it impossible to be prepared.
- Pillaging and Plundering: Most people think burglars are spooky criminals that go bump in the night. However, at a data center located near King’s Cross Station in London, a team of burglars dressed up as policemen and convinced employees to let them stroll right in. Between three and five men were granted entry, saying that they received reports of people on the roof. Staff member were then tied up and expensive hardware was taken. The thieves were never caught.
While these disasters may seem outlandish, they all have one thing in common: nobody was seriously hurt and the data center recovered. It’s going to take more than a solar flare or a game of dress up to take down these resilient and impressive facilities. However, some of these disasters were actually preventable, particularly the burglary. Having best practices in place, a disaster plan with fast action and everyone trained in what to do during a crisis is key.
Data centers house incredibly costly equipment that 1) criminals know about and 2) is not immune to the wrath of Mother Nature. When creating a disaster plan, try to think of every possible scenario. This is also a reminder of how important location is when choosing a data center site. States like Oregon, Utah and Washington are relatively free of natural disasters and therefore are a good starting point. Aim for a temperate region with plenty of highly skilled technical professionals on hand. Being well-staffed, well-located and having a well-designed disaster recovery plan is key to running a successful data center in the long term.