Data centers house our most critical information, from hospital records to personal financial data and from social-media profiles to classified government intelligence.
The Ponemon Institute surveyed companies for the root causes and total costs of unplanned outages in 15 industry sectors, including the public sector, media, hospitality, services, transportation, education, retail, industrial, consumer products, research, colocation, eCommerce, health care, communications, and financial services.1
The research report listed the “root causes” and “total costs” of unplanned outages for the years 2010, 2013, and 2016. (The data set for 2016 is presented in Table 1 and Table 2.) The average total cost of an unplanned outage ranges from a high of $994,000 in the financial services sector to a low of $476,000 in the public sector.
Although the small sample segments limit the ability to draw inferences about industry differences, this data suggests that IT equipment failures account for the most expensive unplanned outages. Of course, equipment failures often can be traced to problems with the grounding systems of a data center, particularly when originating with lightning or other external events. These also include UPS, generator, and computer room air conditioning (CRAC) failures, although these latter failures are typically not as costly as IT equipment failures.
At the most recent Data Center World Global Conference, several cost-effective wiring and grounding techniques to ensure uptime and reliability were described to AFCOM members. The Copper Development Association’s (CDA) presentation covered topics especially relevant to data center infrastructure and operation professionals, electricians, engineers, and managers. It is important that these individuals are well-versed in correct wiring, grounding, equipment bonding, and basic system auditing procedures to maintain the reliability and safety of installations.
MISSION CRITICAL SPECIALISTS
Frequently, data users utilize the services of colocation data centers, which offer flexibility, security and economy, including IT consulting and management. They often demand the highest practical level of reliability to guarantee near 100% uptime.
A state-of-the-art colocation center is expected to provide redundancy of the electrical power supply as well as power quality afforded by the superior grounding system and an energy-efficient cooling system. These factors contribute to meeting or exceeding stringent Level or Tier standards as described by the Telecommunications Industry Association (TIA) and the Uptime Institute. The Tier system is based on criteria originated by and still administered by the Uptime Institute. The system ranks reliability in Tiers numbered I through IV, the latter being the highest. The Tier level attained by a data center is based, in general terms, on redundancy of mission critical systems. Tiers can be “officially” certified by the Uptime Institute but the criteria for Tiers are frequently claimed by facilities.2
REDUNDANCY REIGNS
Reliability means having built-in redundancy, including at least duplicate equipment and/or systems for any mission critical needs. Such redundancy typically includes independent dual power feed systems extending all the way from the power substation to the customer racks.
Data centers commonly utilize two independent power feeds. In one such data center we have examined, the substation supplies the center with up to 2.5 MW at 12.7 kV. Power is fed to two step-down utility transformers that are individually dedicated to one power room a piece, supplying 480 V to the main distribution switchgear. Additional transformers inside the power room to step power down to 208 V for distribution to the data hall and 120 V for the office. 3
Power enters the facility as two feeds (called Red and Blue by this particular operator). In the event of a utility failure, there is a tie-breaker system; if one feed is lost then the tie will close and, for example, the Red bus will be fed through the Blue switchgear.
The distribution from the power rooms to the customer occurs through two power modules. Customers provide their own dual power distribution units to support their IT equipment. In this example, Starline™ busways enable technicians to add or change equipment quickly without relying on traditional connections.
All-copper conductors in the busways supply three-phase power to customers at 208V/225A, which is sufficient for any expected requirements. Loads are split evenly and usually do not exceed 50% on either of the dual circuits, so in the event that there is a failure in one of those feeds, the alternate busway can support the IT load on that particular rack or row of racks.
Busways are also used in the carrier room, which houses servers for outside vendors such as internet service providers, telephone and communications networks, etc. Racked equipment is bonded to traditional copper busbars installed on a portion of the racks overhead. Double-screw lugs ensure the long-term reliability of equipment bonding connections.
TWO GENERATORS FOR DOUBLE REDUNDANCY
A mission critical data center often is equipped with two or more emergency generators serving the separate power feeds, respectively. Depending on the size of the data center, the generators may vary in size to carry the critical load. They are load-tested monthly and no-load tested weekly with the results carefully documented.
Should there be a utility failure and one generator goes down, the second backup generator can support the critical load. Two UPS units are used to backup the power feeds to the data halls; this facility has two 500 kW UPS units equipped with sufficient battery capacity to keep the critical load operating for 50 minutes based on the current critical load.
One beneficial feature of the dual system is that it enables personnel to shut down either feed if the other one requires maintenance or repair. Equipment on the system then remains completely operational and the center’s 100% uptime is preserved.
FACILITY GROUNDING AND LIGHTNING PROTECTION
Even in modern data centers, electrical grounding and lightning protection is too often treated as an afterthought. The most reliable data centers provide oversized branch circuits (exceeding code minimums) to limit voltage drop, and robust bonding and grounding throughout the facility.
In any facility, there should be only one grounding system with uninterrupted electrical continuity. All of the center’s electrical equipment, including IT equipment, will therefore operate at a single uniform ground potential. That very important feature is surprisingly easy to overlook.
Grounding can be accomplished using an AWG 2/0 or larger (often up to 500 kcmil) and buried bare copper ring ground completely surrounding the facility, augmented by vertical electrodes of varying depth and spacing, depending on soil composition and chemistry. Copper-clad grounding electrodes (10-ft, 20-ft, or longer) are bonded to the ring, again, depending on soil conditions. Resistance to earth of 5-ohms or less is desired. Access ports along the ring allow for inspection of such items as lug tightness and ground resistance. All electrical and lightning systems should be connected to this ring.
Mechanical equipment, water, gas, and telephone systems are properly grounded per National Electrical Code requirements to this same ring. Mechanical equipment and IT systems should not share feeder or branch circuits.
Also bonded to the ring are copper down-conductors from the rooftop lightning protection system, which would consist of interconnected, regularly spaced surge termination devices (lightning rods) mounted on the roof, in accordance with NFPA 780 guidelines.
IT EQUIPMENT GROUNDING
IT equipment bonding begins at the cabinets, where it is recommended that #6 AWG or larger green wires are firmly bonded to equipment racks (bonding equipment to the rack is the option of the customer). The racks are connected to a larger (3/0 AWG in this case) bonding conductor, which then connects to an intermediate signal ground bus (SGB).
The SGB is then run directly (“home-runned”) to the master ground bus (MGB) located in the lowest level of the structure. Typically, a larger conductor such as 500-kcmil would be used for this purpose.
INDUSTRY STANDARDS
Colocation facilities or data centers that meet Level 4 or Tier IV standard generally are intended to host the most mission critical computer systems.
Data centers demand above-code installations to prevent information loss, down-time, equipment damage, and other electrical disasters. Unfortunately, the necessary and proper powering, grounding, and bonding practices are often overlooked or not understood by most designers and contactors.
COPPER IS THE KEY
Data centers depend on reliability, and reliability depends on copper connections. The initial cost of copper cable is normally higher than aluminum cable of equivalent ampacity. But, what appears to be simple economics in aluminum’s favor is strongly mitigated by the fact that electrical grounding conductors usually comprise less than 1% of a major technical construction project such as a data center.
In addition, copper conductors require far less maintenance and lower labor costs than aluminum systems. The red metal doesn’t “creep” like the softer aluminum does; meaning that connections remain tight and don’t require frequent torqueing to keep systems intact. Copper is also naturally corrosion resistant and represents a far less long-term deterioration. Taken together, these factors easily make copper cost-comparative. All this means it’s more reliable, which is a vital consideration at data centers.
Table 1. Root causes of unplanned outages, as percentage of all incidents reported (2016 survey).
Table 2. Total cost by primary root causes of unplanned outages, in thousands of dollars (2016 survey).
For more information about maximizing uptime at mission critical data centers or to access a variety of CDA’s electrical case studies, please visit www.copper.org/PQ. CDA also provides a wide range of free electrical seminars on a variety of topics in compliance with the Registered Continuing Education Program (RCEP). Visit www.copper.org/electricalseminars to learn more.
WORKS CITED
- Ponemon Institute, “Cost of Data Center Outages,” January 2016. Available online at https://www.ponemon.org/library/2016-cost-of-data-center-outages
- See the Uptime website for a complete explanation. https://uptimeinstitute.com/tiers
- Copper Development Association, Power Quality Case Study, “Copper Grounding System Guarantees 100 Percent Uptime for Boise Data Center.” https://www.copper.org/applications/electrical/pq/
