Data centers serve as the memory of commercial, industrial, and government facilities. They store credit card information, confidential government material, and other sensitive information that could potentially put an individual or country at risk. Thus, the reliability of the electric power and the power quality must be exceptionally high. In many cases, the stability of data centers depends on the copper inside.

One Summer Street, a massive data center occupying the basement levels and multiple roof areas of a two-and-one-half-acre, city-block-size building, supplies high-quality electric power to an extensive customer base 24 hours a day, seven days a week. It includes eight utility feeds from two substations and service connections to no less than 40 national and international communication carriers. It is the largest and arguably the most significant data center facility in the entire New England region.

In order to meet the energy demands required of a data center housed inside a 70-year-old department store building, the Markley Group, a leading data center developer, had completely new electrical, grounding, IT, and HVAC systems installed.

GOING BEYOND THE CODE

The upgraded electrical systems installed at One Summer Street not only met code, they exceeded it. Code-mandated wire sizes and voltage drop limits involve safety issues but can also affect reliability and power quality. The National Fire Protection Association’s (NFPA) National Electrical Code® (NFPA 70: NEC) covers safety primarily, but it doesn’t fully address reliability or sensitive electronic equipment, which are dealt with in publications such as the Institute for Electrical and Electronics Engineers (IEEE) “Gold” and “Emerald” books, and the Telecommunications Industry Association standards, among others. Data centers could not guarantee the levels of reliability and power quality required if they were designed and built to the code’s minimum requirements.

“The NEC is a minimum standard and we supersede it every day in every project we work on,” said Don Esson, the Markley Group’s facilities manager at One Summer Street. “Working strictly to code practice, we would be running AWG #12 wire for a 20-A branch circuit. Our standard is to run #10 instead. It can handle 1.5 times the ampacity; it’s more robust and therefore more reliable. Also, with #10, the customer has the built-in flexibility to upsize to 30-A receptacles in the future if he wishes. There’s very little increase in materials and labor costs, but reliability improves and you usually don’t have to upsize the conduit when upsizing copper, as you would with aluminum.”

One Summer Street limited the number of outlets per branch circuit to between four and six, except for the IT circuits, which are two-on-one since all IT branch circuits are double-ended. Each receptacle contains two #10 phase conductors and a neutral conductor for the “A” circuit and two more for the “B” circuit, plus separate full-size grounding conductors.

“Voltage drop is another important consideration,” Esson said. “We always bump up one conductor size, so where we might run #4, we would probably run that with #2 wire. Voltage drop gets taken out of the equation at that point. The code permits a 2% drop for feeders and 3% for branches, but we hold that to just 2%.”

FULLY REDUNDANT ELECTRICAL SYSTEMS

The building’s utility vaults are designed for a 20-MW electrical service, which is expandable to 40 MW. The vaults are dedicated to the data center. Two entirely separate 13.8-kV, 10-MW utility feeds enter through the two vaults, each containing four liquid-cooled, 2,500-kVA, 13.8 kV-480/277 volt transformers. Three double-ended switchboards provide fully redundant services, referred to as the “A” and “B” circuits.

The center’s 480-volt power is distributed from the basement to server floors via ten 4,000-A copper bus risers. Four 5,000-A bypasses connected to the 6,000-A rated switchgear enable load-switching among the riser feeds in the event one service should fail. Power bus risers are physically separate from IT/data risers in order to avoid electrical noise on sensitive circuits. Approximately 100 480 V-208/120 V distribution transformers of various sizes feed servers and convenience outlets. Additional 480/277 volt transformers serve lighting and HVAC requirements. All distribution transformers are copper-wound, dry-type, K-rated (mostly K13 and K20) units meeting the National Electrical Manufacturers Association (NEMA) TP-1 efficiency standards.

Individually fed power distribution units (PDUs) feed branch circuit panels and remote power panels (RPPs). For redundancy, branch circuits are extended to each server cabinet in an “A+B” configuration from two unique PDUs. The redundancy at the branch circuit improves reliability and permits either half of the electrical system to be shut down for maintenance without disrupting power.

SINGLE-POINT GROUNDING

Prior to the Markley Group’s ownership, tenants were all serviced individually. Each had their own electrical service entrance equipment, which created multiple ground current loops throughout the building.

“Grounding is critical in a data center,” said Christopher McLean, P.E., LEED^® AP, Vanderweil’s electrical engineer at One Summer Street. “Computer equipment is very sensitive to harmonics and high currents, so you want to isolate the equipment from transients, high frequencies, and ground currents that might be placed on your grounding system at points of voltage and current transformation. We only have two levels of transformation, from 13.8-kV utility service to 480 volts and again from 480 volts to 208/120 volts, but they occur at hundreds of points where you’re connecting neutral to ground at transformers, switchboards, and other separately-derived systems. Any of those points could feed transients, stray currents, or harmonics back to the computer equipment. Our enforcement of a single-point grounding system prevents that.”

The single point is the center’s main grounding bar, a 1-ft-high by 4-ft-wide copper plate located in the building’s subbasement. To that plate alone, all ground leads from the building are bonded. There are actually two risers for the grounding system: one for the power distribution system, which includes generators, transformers and UPS equipment; the other for IT and communications circuits. Separating the two systems isolates IT/communication equipment from noise, harmonics, and floating ground currents that may be present at ground-to-neutral bonds. The two systems are bonded together only at the main grounding bar.

Extending earthward from the main grounding bar are multiple connections to building steel in walls and the subbasement floor, as well as to the water main on both sides of the meter. The building stands just 23 ft above sea level, making the floor and footings effective earthing points (Ufer grounds). Design ground resistance is 5 ohms.

Connections to building steel, including the basement floor, are made only from the main grounding bar.

“All of the 100-plus transformers throughout the building are connected to ground, plus we have many neutral-to-ground currents from unbalanced single-phase sources that ultimately put current to ground someplace,” said McLean. “Had we connected to building steel from several points on the grounding system, the entire building steel system could have become one giant ground loop. That could compromise the ground-fault settings in switchgear and cause problems everywhere in the building. By connecting from only one point — the main grounding bar in the basement — we’ve created a very robust grounding system that isolates those problems.”

Conduits are also bonded to the grounding system only at the main grounding bar to create a complete system. However — and despite it being permitted by code — conduit is never used as the only grounding conductor.

“In almost all instances, conduit would not be an effective grounding path,” McLean said. “We always run at least one separate grounding conductor for each feed. Where we run two parallel feeds in the same conduit, each gets its own green wire. One look at Article 250 in the code would show you that the ampacity of the required copper grounding conductor is much higher than that of either galvanized conduit or electrical metallic tubing (EMT). Also, if we used the conduits as grounds, thousands of fittings and interconnections would then have to be inspected, torqued, and tightened periodically, whereas a separate green wire only has to be inspected and torqued at the two end connections.”

Bus bar and the two grounding system branches are made through separate risers in the building’s corners — one
for power distribution and one for IT equipment. Plans called for adding two more ground risers in the near future. Conductors from the main bar to the building floors are sized at
750 and 1,000 kcmil. They terminate at 1-ft by 3-ft copper grounding bars at each floor. Smaller conductors lead from those bars to multiple 4-in. by 1-ft collector bars located high on walls throughout the floor space. Two sets of large and small grounding bars per floor serve power distribution and IT equipment, respectively.

GROUNDING OVERHEAD AND UNDERFOOT

Equipment racks are grounded by overhead and under-floor systems. For overhead systems, individual cabinets are grounded with AWG #6 or #4 copper leading to overhead ladder racks. Grounding conductors from all cabinets in a row are bonded to a collector bus cable run in the racks. It may range from #2 to 4/0 in size depending on need.

Collector bus cables are directed to the nearest collector bar. The ladder racks themselves are also grounded via the bars. Cable trays carrying 208-volt power to servers are grounded to separate collector bars serving the power distribution branch of the grounding system.

The raised-floor grounding system consists of grids of AWG #2 and 4/0 bare copper. For approximately every 3,000 sq ft of raised-floor area, one 4/0 grounding conductor is run from the grid to the wall mounted collector bar.

PLANNED MAINTENANCE IMPORTANT

One Summer Street has well thought out and executed electrical and grounding systems, but even these systems can degrade over time: connections loosen, contact resistances rise, and loads change producing unexpected effects. Grounding connections are checked for tightness on a fixed schedule and re-torqued as needed. Power quality is monitored to ensure clean waveforms and voltage stability. Emergency generators and UPS units are periodically tested. Systems are also regularly upgraded to meet changing needs.

Because of the steps taken to exceed code and conduct regular maintenance, the Boston data center has never experienced a utility outage but is fully protected should one occur.

For more information about the Boston data center project, please go to http://bit.ly/2eg2QkE. For more information about installing a grounding or lightning protection system, visit www.copper.org.