In operation since 1995, the company’s Tier IV mission critical facility needed to upgrade its aging equipment in order to maintain optimal reliability and flexibility for its 2,000-kilowatt critical load. As an educated owner-operator, UPS performed 10 months of evaluations to determine just the right solution for its conditioned power distribution needs. In an effort to lower the risk of interruption and raise reliability while reducing cost and maintaining Tier IV infrastructure, UPS chose to replace its single network legacy conditioned power distribution system with one that split the IT and mechanical loads onto separate conditioned power networks.
Syska Hennessy Group designed this new 2N solution and worked together with the project team to refresh the equipment while minimally impacting the 172,540-square- foot live data center. Mitigating this and other challenges, while building a flexible and reliable conditioned power network, new UPSS distribution was delivered from construction through two-phase systems migration.
MAINTAINING FLEXIBILITY ANDRELIABILITY
When it opened its doors 16 years ago, the IT equipment at UPS’ Georgia data center required a significant amount of power. In order to meet this demand, two sets of rooms were dedicated to the load’s associated conditioned power systems. An additional set of rooms intended to serve future load growth was built out at day one, with underground conduit connected to the data center space. Thanks to Moore’s Law, however, today’s IT equipment requires less electrical power while delivering more computing power, so the facility has yet to come close to using 50 percent of its total capacity.
The third set of rooms was utilized during the current project as swing space. This allowed the current conditioned power distribution systems to continue supporting the live load, while new equipment was simultaneously installed in the extra spaces. Deploying existing space rather than brick-and-mortar construction minimized project costs and enhanced project phasing and expansion of the 2N system.
Before the extra rooms could be utilized for installation and migration, the design team would develop a new plan for power distribution to connect the new UPSS equipment to its corresponding computer rooms. While the existing underground conduit had, so far, managed the current load, its re-use would have put the exiting IT load at greater risk. Therefore, UPS and Syska Hennessy Group created a new overhead distribution system.
In addition to furthering data center flexibility and future load growth, overhead distribution provides the facility with a secondary path for running feeders between the conditioned power distribution systems and the computer rooms they serve. An additional layer of distribution added to the system with an alternative main allows all feeders to endure future conditioned power refreshes as well. Power centers are fed directly from the new layered distribution and can easily be connected to another source, while provisions were made for future tie-ins at new data hall switchboards.
Built as one of the world’s first Tier IV data centers in 1995, the facility exhibits UPS’s strong commitment to reliability and service. Continuing this high Tier IV standard was imperative to UPS.
The new UPSS’ 2N design was created to provide ultimate reliability by separating the mechanical and IT loads. The 2N design includes two, five-module parallel UPSS for the IT total load capacity of 3.38 megawatts (MW) and another two two-module non-parallel UPSS for mechanical and BCP total load capacity of 1.35 MW. Redundant 125-volt direct current (Vdc) station batteries supply the UPSS switch-gear controls with supplementary back-up power. Additionally, 480 V of conditioned power with high resistance grounding was maintained in the refresh for enhanced fault tolerance.
BIM added a level of reliability to the project during the design phase by identifying potential conflicts and eliminating unsuccessful installation sequences and plans early on in the project. Holder Construction also used BIM during the construction phase to detail conduit routes, piping, power distribution system requirements, and more. As conflicts were identified, the design in question evolved to meet the appropriate parameters.
Planning the integration and migration of the old and new systems with minimal impact to a live data center with a 2,000 kW critical load was not a simple task for the project team. Working together as a team from day one, UPS, Syska Hennessy Group, and Holder Construction, Atlanta, met at regularly scheduled meetings and project milestones. Organized team collaboration provided a “reality check” that determined which designs would ultimately play out in the field and how they would be executed.
Running new overhead cables was one of the most significant challenges the team faced. Using a scaffolding system with netting around it, along with tethers on every tool to prevent even a screwdriver from dropping onto the computer equipment below, new conduits were placed on metal racks and suspended from the concrete slab above. Holes were drilled through concrete beams and used to attach the conduits to the steel framework for support. Vacuum cleaners with high-efficiency HEPA filters were used to capture the concrete dust during the drilling.
Planning and sequencing the migration, then developing the methods of procedure to track when, where, and how it would be done, and then ultimately moving data center loads from the old to the new UPSS took tremendous management and logistical skill that involved all disciplines. Integrating the new equipment into the data center’s existing comprehensive building automation system required similar planning as well.
Additionally, other procedures were developed to move wiring into the tight spots surrounding the static transfer switches (STS). In order to extend the life of the legacy STS equipment through the retrofit, high-flex cables with very fine strands were employed to turn in tight corners. Installers used Shoo-Pin adaptors to help terminate the cables in tight spaces. Legacy power distribution units were also retained through the UPSS retrofit and employed downstream of the STS to power computer equipment.
In June 2011, the UPSS supporting the data center’s IT load were installed and fully commissioned, moving the IT loads from the old to the new UPSS, completing Phase I of the project. Phase II, scheduled for completion by the end of 2011, will move the mechanical systems from the old UPSS onto the new.
Efficient distribution means more than just shipping packages across the globe for United Parcel Service, Inc. It also means promoting flexibility and reliability while optimizing conditioned power distribution systems back in their data centers at home.
Prior to the completion of this project, the 2011 NEC was released. Changes within Article 645 in the new NEC align with the direction taken at United Parcel Service’s Alpharetta, GA data center, utilizing the EPO relocation exception and working with the AHJ and current adopted codes.
The New EPO System
Once proper design of a data center’s mechanical and electrical systems has been established, potential outages related to human error pose the greatest threat to reliability. And the emergency power off (EPO) system is one of the usual suspects.
Typically controlled by a large red push button located adjacent to the computer room’s egress doors, the EPO system, when manually initiated, facilitates the automatic shut down of power and ventilation to the data center as defined by firewall separation. Smoke and fire dampers are required to close and maintain the integrity of the fire-rated envelope. Like most data center stakeholders, UPS was interested in mitigating this potential human error risk.
Under the 2008 edition of the NEC being enforced by the Authorities Having Jurisdiction (AHJs), Article 645 provides allowance for an “orderly shutdown” of the integrated electrical system or alternate installation practices.
Syska Hennessy Group conducted analysis and met with local fire marshals and other AHJ. The result was the creation of an EPO-like system with required shutdown, isolation, and monitoring. The traditional push button at the computer room door was replaced with a centralized EPO system in a controlled location. The system consists of an A-side panel and a B-side panel, each containing keyed switches specific to the zone they service. This new system was approved upon stipulation that the facility be manned 24x7x365, with qualified personnel present to work with the emergency responders and perform an orderly shutdown of power and ventilation.
IT Load Migration
One of the greatest challenges and successes of the data center refresh project was its IT load migration. Migrating the data center’s existing critical load to the new uninterruptible power supply systems (UPSS) was completed over a period of weeks in two phases. The following schematics represent the original configuration all the way to systems isolation and finally ultimate restoration.
STS-AUX-01 provides a means to isolate an STS with an alternate input path to the associated PDUs. This auxiliary STS configuration allows both A and B sources feeding dual-corded loads to be preserved during concurrent maintenance.
After connecting STS-AUX-01 to the new UPS source 3A, the PDU load temporarily migrates to the new UPSS source on STS-AUX-01. The transfer from STS-103 to STS-AUX-01 includes a closed transition manual operation using source 1B. Then a STS operation is initiated to move the critical load to the new UPSS source 3A.
After replacing the feeders to STS-103 and connecting to the new UPSS source 3A and 3B, the load transfer process is reversed to return the critical load back to STS-103.
The PDU alternate inputs are opened, and STS-103 returns to online mode. STS-AUX-01 output is a common cable bus connected to multiple PDU auxiliary inputs. Migration of other STS units occurs using the same process.
Figure 1 Click to enlarge
Figure 2 Click to enlarge
Figure 3 Click to enlarge
Figure 4 Click to enlarge
Figure 5a and b Click to enlarge