Planning For Disaster
Can your electrical infrastructure weather a natural disas-ter?
While natural disasters cannot be avoided, their impact may be somewhat lessened if businesses are better prepared. Many companies may have disaster plans, but they fail to realize the underlying assumption is the electrical power and infrastructure will remain operational and not be impacted. Natural disasters can damage electrical distribution equipment in multiple ways (fire, vibration, water, etc.). In the U.S., flooding is the number one natural disaster. Since water and electricity do not mix, restoring power to water-damaged equipment can be a dangerous undertaking.
Businesses face increasing pressure to maximize profits and minimize downtime. Having a detailed emergency recovery plan can limit the financial and personal havoc that natural disasters, specifically flooding, can cause. A good starting point is to address the following key areas:
• Ensure electrical equipment is properly maintained
• Identify the electrical equipment that is critical to operations
• Be aware of the most current natural disaster recovery codes and standards
• Know the effects of water damage to electrical equipment
• Develop a safety plan that incorporates emergency procedures
• Develop an electrical emergency action plan
The National Fire Protection Association (NFPA) and the U.S. Occupational Safety and Health Administration (OSHA) provide guidelines to develop disaster recovery, emergency response, and safety plans. This article incorporates those guidelines to help in the creation of both short- and long-term restoration plans. The number one priority for both plans is to safely restore power.
RELEVANT CODES & STANDARDS
NFPA 1600 is the overarching standard and primary document on disaster recovery, emergency management, and business continuity. For workplace safety and planning, OSHA references NFPA 70E, Standard for Electrical Safety in the Workplace. NFPA 70B, Recommended Practice for Electrical Equipment Maintenance now includes a chapter on electrical disaster recovery in the 2013 edition. In addition, the National Electrical Manufacturer’s Association (NEMA) has published “Evaluating Water-Damaged Electrical Equipment” and “Evaluating Fire- and Heat-Damaged Electrical Equipment.”
THREE STEPS TO ELECTRICAL DISASTER RECOVERY PLANNING
Step 1:Knowledge of electrical system. As part of an electrical disaster recovery plan, businesses should:
• Have a current single-line drawing of their electrical distribution system
• Identify which electrical equipment is critical to the electrical infrastructure
• Understand what equipment must be replaced and that which can be reconditioned, as shown in Figure 1.
Sidebar 1: Flooding risks
Anywhere it rains, it can flood. The risk of flooding isn't just based on history; it's also based on a number of factors: rainfall, river-flow and tidal-surge data, topography, flood-control measures, and changes due to building and development. Source: www.floodsmart.gov
Electrical equipment exposure to water can be extremely dangerous if re-energized without proper reconditioning or replacement. Assessing the damage goes beyond visual water indications. Moisture and weather exposure can affect the equipment’s integrity. Water can be contaminated with sewage, chemicals, salt, debris, and other substances. These contaminates may cause loss of dielectric spacing and can be a hazard upon re-energization. It is also possible that foreign debris remains inside the circuit breaker or switch enclosure.
Whether the equipment is to be replaced or reconditioned, all services should be performed by qualified personnel familiar with the equipment’s operation and construction. It is important to note that the ability to recondition equipment will vary; it may include the repair or replacement of internal components. Reconditioned equipment should be tested per the manufacturer’s instructions prior to re-energization.
Step 2:Develop (or update) an electrical safe work practices policy. An electrical safe work practices (ESWP) policy is a written document created by the employer that covers all areas of the company’s electrical safety practices. While having a safety policy has been a requirement of NFPA 70E for several revision cycles, the 2012 edition further refines the requirements for developing and auditing the policy.
The ESWP policy is not a one-size-fits-all policy to cover a company in multiple locations. For example, voltage, energy level, circuit conditions, and hazard levels are different in each facility. The policy should identify detailed information specific to that location. This may include the location of the safety policy for employee (or contractor) access; who is the onsite authority having jurisdiction (AHJ) for decisions, etc.
Developing and auditing an ESWP policy is critical to business continuity and disaster recovery. Since creating a comprehensive program may seem overwhelming, it may be helpful to use the following guidelines to help get started.
Facility – Includes company policies and systems regarding:
• Equipment maintenance
• Clearance requirements
• Safe working conditions
Personnel – Focuses on actual work practices and addresses:
• Qualified and unqualified personnel
• Proper care and use of PPE
• Job preparedness
• Training and continuing education
Procedures – References on the job procedures and includes (but not limited to):
• Performing energized work
• De-energizing and re-energizing
• Lock-out / tag-out
• Job planning
• Arc flash hazard analysis
• Equipment labeling
• Reporting safety concerns
The employer must verify on a regular basis that each worker is complying with the safety-related work practices required by NFPA 70E. Managers and supervisors must strictly adhere to the ESWP, attend electrical safety training classes, and ensure all employees practice safe work practices every day. Finally, should a natural disaster strike, employers and employees should be very familiar with emergency procedures and work practices.
Sidebar 2: EEAP programs
In some cases, EEAP programs should be signed off by customers and facility management companies.
Consider this example: A customer leases multiple floors (one floor being a data center) from a large building management company. The customer’s IT manager may have no control over the equipment feeding power to office space and data center. In this type of situation, clear roles and responsibilities, work scopes, and agreed upon emergency actions are vital to a long-term building owner and customer relationship.
Step 3:Electrical emergency action plan. The purpose of the electrical emergency action plan (EEAP) is to understand the electrical assets, critical operational infrastructure, risks, and short- and long-term power restoration execution plans. Some of the benefits include:
• Reduces time to restore short and long-term power quickly and safely
• Reduces uncertainties when a disaster occurs
• Increases understanding of electrical assets, available emergencies services, replacement market availability
• Being able to know when a disaster occurs, the immediate financial implications
NFPA 1600 provides a framework for a site-wide disaster recovery program regardless of the risk type. Although it does not include specific guidelines for any specific system or hazard, its provisions help users manage potential disasters and associated plans in a common structure. The inclusion of electrical disaster recovery in the 2013 edition of NFPA 70B provides a foundation for the electrical infrastructure recovery. Prior to this update, electrical disaster recovery planning was based on experiences of the individual or team who has electrical responsibilities. One purpose of this article is to provide guidelines on developing an electrical emergency action plan and examples how it saved or could have saved facilities from excessive downtime.
Lessons learned from an actual event follows. Downtime, loss of revenue, and headaches could have been reduced from a robust EEAP program. (Note: The facility name remains anonymous.)
GENERATOR DISASTER RECOVERY - COMMERCIAL BUILDING WITH BUSINESSES AND DATA CENTER
Background: On May 1-2, 2010, Nashville, TN received a record 13 in. of rain, which caused a devastating flood. Thirty people were killed and damages exceeded $2 billion. Also known as “Music City USA,” Nashville is a key tourist area for music lovers around the world. Due to its proximity to the Cumberland River, downtown Nashville was hit hard and floodwaters damaged many commercial buildings. This example focuses on only one Class A commercial high-rise in downtown Nashville housing multiple businesses and a data center.
What happened? Following timeline of events depicts how the flooding affected downtown Nashville. Times shown serve as general guidelines.
Sunday, May 2
9 p.m.: Municipal power was no longer able to serve power to downtown Nashville. The commercial building auto switched to emergency power and only served vital data center operations. Teams were deployed to the site to closely monitor data center and weather conditions.
Monday, May 3
12 a.m.: Metropolitan police ordered a mandatory evacuation in downtown, which left data center operations and security vulnerable.
4:30 a.m.: Selected personnel were allowed back into downtown to perform key operations including refueling of generators. Although rain had stopped, the Cumberland River was steadily rising.
10 a.m.: A decision was made to start soft shutdown actions, but it was too late. Generators flooded, which began a 31-hour data center outage. Through a key relationship, a temporary generator was located 20 miles south the city.
Tuesday, May 4
4:30 p.m.: The data center was brought back online.
Wednesday, May 5
Still with no lights, air conditioning, or restroom facilities, a 24-hr staff rotation was set up to monitor the data center and the condition of the temporary generator. The water level was still too high to assess damage to the building’s generator.
Thursday, May 6
The building’s generator was drained and the damage was assessed. Spare parts were not immediately available. Five days later the generator was repaired and power switched back to the building generator. However, the temporary generator was retained as a back-up for a few additional weeks.
Thursday, May 13
Nashville Electric Services (NES) restored power to downtown Nashville and the building switched from generator to municipal power.
Three key lessons were learned from the aftermath of this historical flood.
Determine priorities.The EEAP plan must have a clear definition of what constitutes an emergency and when to execute the EEAP. In this case, only three hours were available before a mandatory evacuation order was issued. Prioritizing critical functions is essential to efficiently and safely restoring power.
Emergency service contracts.Searching for temporary equipment after a disaster occurs slows down the power restoration process and can be very expensive. The EEAP should include emergency service contracts to guarantee disaster recovery team response time, critical equipment pricing, lead times, and a deployment strategy with details on setting up and operating a command center to meet an organization’s needs.
Critical spare parts.Having a current single-line electrical diagram of the power distribution system is crucial to efficiently restoring power. Use the drawing to pinpoint the electrical equipment critical to business operations. Identify the equipment’s critical spares parts availability, pricing, and lead times for custom-made parts.
DEVELOPING AN ELECTRICAL EMERGENCY ACTION PLAN
The development of an EEAP may seem overwhelming. Below is a step-by-step guide created by Schneider Electric Services to help customers be prepared in the event of an emergency.
Define the criteria of an emergency.Lighting strikes on service entrance transformers and flooding of backup generators clearly indicate operational “states of emergency.” Are nuisance tripping on circuit breakers or power quality issues considered emergencies? When productivity is impacted, it can be confusing as to when a state of emergency should be declared. The EEAP should clearly define what constitutes an electrical emergency.
Identify electrical equipment that is critical to business operations.On the single-line diagram, trace the power from the incoming utility source to every piece of electrical equipment feeding critical business operations. Perform an analysis on each of these critical assets to include availability in the market, lead times, and a plan of action if and when the equipment is no longer functional. For most critical assets, it is necessary to understand cost of temporary rentals, logistics, contacts, and testing requirements for both temporary and permanent power restoration. For example, a special transformer’s spare parts may not be available in the primary market, i.e., the local electric municipality. Therefore, it is a good practice to determine available suppliers, acquire budgetary pricing, and lead times on the secondary market.
Selection of outside vendors and pre-negotiated commercial terms and conditions.In the aftermath of a disaster, it may be too late to negotiate pricing or lead times for the required resources to restore electrical power. Without pre-negotiated emergency service contracts, companies may suffer from overpricing and insufficient support. In addition to pre-negotiated normal and emergency rates, due diligence should include estimated response time and procedures for large-scale project coordination (see Step 4). The selected vendor(s) should have the depth and experience to handle major disasters.
Define internal and external responsibilities.The EEAP should clearly define “who has responsibility for what” in restoring power to the facility. They also have details on customer’s responsibility such as providing electrical one line diagrams, energization procedures, and coordination and communication activities between multiple vendors through a central contact. Third-party vendor(s) should also provide a clear procedure on how they will approach an emergency at a facility including assessing damage, mobilizing resources, appointing a project manager, and establishing a command center. All parties involved should fully understand the safety plan that is put in place.
Define the equipment and service scope.This section will further define the equipment and associated work scope. For example, an electrical distribution service company can define the equipment scope to be from the utility service entrance (13.8 kV) to low voltage switchgear (480 V). The associated work scopes include equipment installation and commissioning in both temporary and permanent scenarios.
Emergency contract terms.Emergency contracts should be written for a specific time period, with an expiration date. It is also recommended to add expiration dates to the EEAP plan for self-auditing purposes. A suggested best practice is to audit the EEAP when the ESWP is being audited which is every three years according to NFPA 70E – 2012.
Contact information. This section of the EEAP should include the latest contact information for anyone (internal or external) who has a defined responsibility in restoring power due to an emergency.
When a natural disaster strikes, its impact on individuals, communities, and businesses can be devastating. Restoring electrical power is a crucial part of the recovery process. Regardless of the industry or facility type, having a detailed ESWP and EEAP can help recovery efforts. Multiple standards and recommendations exist from NFPA, OSHA, and NEMA to serve as a guide for businesses to help them understand and develop a contingency plan in the event of an emergency or natural disaster.
1. ServePro, http://www.servpro.com/ready.
2. “Managing the Escalating Risks of Natural Catastrophes in the United States,” Lloyd’s Insurance Report, October 2011.
3. “Evaluating Water-Damaged Electrical Equipment,” National Electrical Manufacturers Association, 2011.