It has been a great five years forMission Critical. Over those five years, we have focused on numerous topics essential to data centers and other critical operations; however, the devastation caused by Superstorm Sandy has clearly demonstrated that our mission is not complete. mc1112-cronin-fig1-615.jpg

What follows is a lesson in disaster management, communication, preparedness, and recovery.

The story is personal to my situation, but the issues raised apply to any disaster. This story is about a little community that did not take the direct brunt of the storm but was brushed by its effects. The community did not experience total loss of assets (or lives) as did other communities. In fact, its assets remained largely intact, but the support infrastructure failed. As a consequence, this community did not receive the support or attention it needed, as other areas in the state were so much worse off.

The failed infrastructure includes power, POTS (plain old telephone), and cable. Even worse, the community depends on pumps for its well water and oil and propane for heat. The electrical utility (a subsidiary of a large Midwest utility) is left nameless but not blameless; its response was pitiful at best. I’ve tried to note any positives associated with the performance of any utility provider. In addition, there is plenty to learn from this event that has little or nothing to do with the utilities.

Day 1

3:00 PM: Power fails in moderate wind conditions, and the outage is reported


8:00 PM: Midnight: High wind gusts (60+ mph) topple trees and destroy the power, cable, and POTS infrastructure.

Day 2

Morning: Survey damage and discover that everyone is stranded. The downed trees block all means of escape.

All day: Await repair crews. Neighbors with cellular access discover that the township’s website is down because someone failed to renew the IP address. Local government is incommunicado.

Day 3

No outside help in sight. The neighbors band together and clear the streets for emergency responders and food and fuel deliveries. We also assemble fuel cans and travel in search of generator fuel.

Day 4

Veolia Environmental Services, under contract with the electric utility, arrives and cleans up the oil from two crushed transformers (A great job, and they communicated with all the property owners).

The utility’s tree-trimming contractor drives through without addressing the remaining limbs on wires that were too high to reach by the neighbors.

Community cleanup and building repairs continue.

Day 5

Community cleanup and building repairs mostly done.

Twenty-minute lines for fuel.

Day 6

Comcast Cable is the first infrastructure utility to begin repairs. That’s Comcastic!

Century Link (phone lines) repairs main trunk cables on county road and installs new poles (super workers).

Days 7 and 8

Century link repairs POTS lines. Dial tone once again!

Lines for fuel disappear.

Day 9

A second storm brings heavy snow.

No further damage.

Day 10

Eighty percent of the wires remain down, three or more pole-mounted transformers remain unrepaired and destroyed, and more than a dozen poles still need replacement on just one two-mile stretch of road.

To date not a single electric utility vehicle has been spotted anywhere within town. There is at least a week’s worth of restoration work to be done once the electrical utility decides to show up.

Numerous radio broadcasts by the electric utility and government promise power by midnight on Day 13.

Day 11

After less than 12 hours from yesterday’s broadcasts,

the government and utility are backtracking on those

restoration promises, with nonspecific suggestions

that it may take several days longer.


Lee Electrical, utility restoration contractors from Aberdeen, NC, has two crews working in the next town (doing a great job over there).


A few township residents get power restored as the township committee holds hearings.


Township residents blast the committee for no communication and demand the electric utility be disenfranchised due to its poor performance (more to come).

LESSONS

So what are the mission critical lessons learned from this event?

POTS lines, while the most reliable of all our land-based phone systems, do fail, but a reasonable number of crews can make the repairs relatively quickly.

Weather experts have been warning for years about changing weather patterns and more severe storms in this area. Numerous disaster movies have demonstrated the effects of these changes with reasonable accuracy. Sandy was not a Category 1 hurricane when it came ashore in the northeast; however, because it tracked into an overbuilt area the damage and human suffering was severe. An unprepared populace (from individuals to major corporations to whole industries) exacerbated the situation.

Communication problems:. Local government officials did not communicate very well. Public websites were down as they were hosted in closets and not in the cloud. Cellular provided the only working communications on Days 2-7, but, due to the terrain, coverage was spotty and connections were slow.

People were trapped on their streets for days; those who were working in a mission-critical role had no way of getting to work. Those working at the time of the storm had no way to return home to take care of their families. If management were to make space available for displaced employees to shelter families, then the employees would be more inclined to remain at work with their families safe and sound nearby.

We all have expectations that the electric utility will always be there, and we really have not fully thought through the scenarios where it is not, when it will not be back by the time backup systems run out of fuel, or where the utility effectively abandons the area in order to address other areas. A few considerations are:

• Emergency generators could not get fuel due to blocked roads.

• On-site fuel storage at 24 to 72 hours is minimal. In this are, we have been running on backup for more than 240 hours, with local fuel supplies exceptionally low.

• Perhaps piped natural gas generation should be given greater consideration as a redundant fuel source (e.g., bi-fuel generators). 

• In addition to diesel, gasoline supplies must be part of disaster planning as some emergency response vehicles even ran dry.

• Wide areas of destruction require time to get repair teams and supplies from across the country. A more pro-active approach, even stocking up before a storm and staging tankers at your site would help, yet adequate response still takes time.

So as a mission-critical operator do you have a survival plan for your operation when the entire outside infrastructure that you depend on daily disappears overnight?

Do you have alternative means of communicating with your clients?

Do you have staff in-place that will take action?

Ultimately the question is not: How long can you operate off the grid? It is: are you prepared to operate indefinitely without people, fuel, and other supplies?

There is so much more to be said. But you can start by putting your organization through a stranded on an island drill before asking yourself: Am I really prepared for this reality?