The fall of 2007 found Dranetz Technologies partnering on a number of high-profile projects in California, including one of the country’s leading health-care organizations. Having recently embarked on an ambitious energy-saving and efficiency-optimization program in data centers around the U.S., the provider deemed one particular 120,000 square-foot location (figure 1) in Riverside County, CA, to be especially critical to the reliable operation of the company’s entire health-care network.
A by-product of going paperless-a process undertaken for many operational reasons-resulted in the provider also becoming a more environmentally conscious health-care supplier. In operation, the organization’s paperless mandate requires critical power management of many elements, including patient records, patient insurance documentation, medical records, MRI scans, laboratory test results, and access to any of this information on a 24/7 basis by surgeons, doctors, other health care staff, and even patients.
REDUNDANCY MEASURES
A number of facility hardening and redundancy measures were implemented to protect this highly mission-critical facility, including:
- A fully base-isolated building foundation designed to help the facility ride through an 8.0 magnitude earthquake (figure 2);
- Five 2,520-horsepower (hp) diesel generators providing a total of 7-megawatts (MW) (N+1) of emergency back-up power (figure 3);
- 960 battery jars made up of 1920 by 2-volt cells provide 3.6 MW of back-up power to the data center’s four uninterruptible power supply (UPS) (figure 4).
The leader of the facilities team responsible for keeping everything running smoothly notes, “Although we only have one main utility feed, the data center has its own utility databank substation on campus, which is connected to the 66-kilovolt (kV) electrical grid.”
The chief engineer further explains how utility power is routed through transformers and switch gear to the equipment on the computer floors by means of dual-power sources made up of separate (A & B/C & D) double-conversion UPS systems. “Since the batteries are a component of the UPS system (online 24/7), the computer load is continuously supported by the batteries via the UPS inverter,” the chief engineer explains, “Therefore, any utility disruption or abnormality on the input of the UPS system is transparent to the computer load, which experiences no delay or kick-in time to use the stored energy from the batteries.”
The data center’s lead engineer notes that the electrical system switch gear provides sensing circuits that monitor utility power. “Once it senses a utility outage, five seconds later a start-up signal is sent to the emergency generator system (figure 3). After the generators start, power is routed throughout the facility via automatic transfer switches, including power to the input of the UPS systems.” Once normal utility power returns, he says, the electrical switch-gear sensing circuits send a signal to transfer the automatic switches back to normal and shut down the emergency generator system.
The facility manager notes that many of the mission-critical systems that the facility supports are actually health-care applications that enable physicians, nurses, and various health-care providers to deliver real-time, technology-enabled health care to its members and patients at the bedside. Knowing what’s at stake, he points out, adds a much greater sense of individual and collective accountability to what he and his team are responsible for. “Since there are so many components and variables that factor into maintaining continuous availability, it is essential to have the right building automation and management tools in place for the data center facilities engineering staff to utilize,” he said.
Without question, it is a job that requires constant vigilance. For this reason, facilities engineers routinely perform power quality troubleshooting using portable three-phase power monitors to determine where, when and why a power incident may have occurred.
PUE-CHARACTERIZING DATA CENTER POWER EFFICIENCY
Complementing this “reactive” power strategy, permanently installed power-monitoring instrumentation throughout the facility provides a greater degree of granularity in terms of power analysis capability and in so doing enables valuable additional insight to be gained into their complex data center operations, including the ability to:
- Trend and merge electrical energy data into the existing building management system (BMS)
- Monitor all UPS systems and emergency back-up generators
- Monitor the data center’s mechanical load
- Provide load profiling capability and a daily record of peak demand of data center loads
These necessary elements enable the provider to optimize the efficiency of its data center operations using known as PUE, or power usage effectiveness.
The data center’s objective in characterizing facility operations in terms of this metric is to reduce its PUE value as much as possible. In a real-world sense, this is typically a moving target due to its susceptibility to a number of variables, including load fluctuations, grid conditions, and even current weather conditions. However, it is possible to achieve significant reductions in the facility’s PUE through implementation of corrective or energy-efficient enhancements. In the case of this facility, the lead engineer estimates that they have achieved at least a 10 percent reduction through on-going optimization efforts and are now in the neighborhood of 1.66 PUE, with efforts constantly afoot to reduce that number even further.
POWER QUALITY
The facility manager notes that it is absolutely essential to have the right systems in place, ones that enhance the provider’s ability to monitor, control, and alert to potential adverse conditions. “It’s just as important to have equipment and systems in place that enable us to be proactive, and even predictive, in our ability to effectively and efficiently manage and maintain the environment. In a rapidly changing computer environment, the ability to monitor power quality is essential to ensure that environmental changes are performed properly and also to help perform diagnosis of identified anomalies.”
In high-availability, mission-critical applications like data centers, the value of power-quality instrumentation cannot be over emphasized for several reasons, not the least of which is evaluating the quality of the power supplied by the utility to ensure that it is delivering a reasonable level of service without an excessive number of disturbances.
Unlike this facility, many others are under the impression that the quality of the utility supply is of less concern if there is a UPS standing guard between the utility service entrance and the sensitive load. However, every utility disturbance puts a stress on the UPS system every time it temporarily switches to battery power to ride through an event. This not only reduces the life of the battery, but often unnecessarily stresses transfer switches and other electromechanical elements. Identifying such problems and having the utility correct them can dramatically improve system and facility uptime.
Power quality instrumentation is also used to evaluate the quality of power supplied from the UPS itself, which can only indicate when it is on or off line, on bypass, or when some other serious problem has occurred. It cannot indicate the quality of the voltage being supplied to the critical load. In one real-world scenario at another institution, for example, the UPS experienced an overvoltage condition at the end of a utility sag. The UPS never reported the event, but the power quality monitoring instrument did, which allowed the user to service the UPS for a potentially serious problem that would have otherwise gone undetected and possibly caused an outage at the data center.
Power quality instruments provide another benefit as well-accountability-as a key tool in the provider’s energy-efficiency initiative. According to the chief engineer, “When we make a change or improvement, we have to back it up with data. The energy-monitoring instruments give us the solid data we need to prove to management that the change we made to the system was effective, by showing the data before the improvement and the after-results that prove the change was warranted.”
“So in addition to power-quality analysis, it’s a validation tool,” he said. “For example, if we get a report from the IT department about a problem, we can go back through our historical data and determine the root cause of the problem.”
An operations analyst with facilities engineering explains that “occasionally there are down-stream problems and generally the first assumption is that it was caused by the UPS or power system. The power quality instruments help us to characterize the actual nature of the problem, which eliminates any finger-pointing and helps us to more quickly identify and solve the problem.”
In the initial discussion with the provider’s facilities operations staff it became apparent that only a powerful and flexible power-monitoring system could satisfy the operators of this mission-critical data center. To that end, in December 2008, they installed and commissioned 18 Dranetz Encore 61SG switch-gear-mounted three-phase permanent power-monitoring systems at key locations around the facility, including:
- Main data center raised floor
- Computer services
- UPS back-up power systems
- Main incoming utility service entrance
EMERGENCY BACK-UP GENERATORS
All monitoring instruments are available to the main control room through a secure Intranet communications system and their existing Modbus BMS. Within six months, the provider’s engineering control room was up and running, providing operating engineers with instant feedback and visibility on the process through means of a “dashboard” of performance data (figure 5).
One of the most important recorded parameters is the real-time kilowatts demanded through their four UPS systems, which provide a proactive approach to managing the electrical power needs of the data center. Should engineering need to distribute or shed some load across different circuits, the live feed of peak and average power demand allows them to perform vital and important energy decisions in near real-time.
LOOKING FORWARD
Over the last 10 years the power demand for a typical data center IT rack system has tripled, increasing by a factor of five to an average of 10 to 12 kW. This demand has generated a need to quantify data-center efficiency as the need for faster computer speed, greater memory storage, and a rapidly growing need for instant access to web server-based data.
The manager of this high-uptime facility puts the challenge this way, “My team and I are naturally focused on maintaining continuous availability. However, one major differentiating factor between our data center and most others is that ours supports the many health-care applications that our physicians, nurses, and professional health-care providers utilize to care for our patients and members.” The facility manager went on to say that “continuous availability is an essential element of our ability to provide real-time, technology-enabled health care to our members and patients. The data-center facilities staff takes great ownership of this accountability and we are always looking at means and methods, including available technology, to further enhance our ability to complete the mission.”
To that end, the fall of 2010 saw the kick-off of several new enhancements to the data center’s existing electrical infrastructure, a process of improvements that will hopefully reduce its PUE factor even lower. As this leading provider evolves the scope of its operations, additional pressure will be placed on the data center’s demand and load profiles. The Dranetz power quality monitoring system, for example, has proven over the last 18 months that it will continue to be a valuable and important part of that process by providing live data feeds and real-time access to energy studies, as well as the PUE metric.
As of this writing, the data center has operated for more than four consecutive years without any systemic infrastructure incidents impacting IT availability. Speaking to that accomplishment, several “Continuous Uptime Achievement Awards” proudly displayed in the front lobby put a public face on what is clearly one of the highest-availability, mission-critical facilities of its kind in the country.
Sidebar: PQ-Standing Guard Over Mission-Critical Apps
Increasingly, data centers and other mission-critical applications are using power quality and/or demand and energy monitoring systems like the Dranetz Encore Series to obtain important load data. Equally important, simultaneous measurements also allow facilities personnel to check for power quality anomalies that can affect the uptime and efficiencies of the building’s critical systems.
In addition to collecting the energy data used to calculate the facility’s power usage effectiveness (PUE) value, the modular design of the Encore Series, whether portable or permanently installed, provides a unique benefit by allowing users to monitor multiple three-phase circuits using a single instrument-thus saving significant cost without compromising performance requirements.
Simultaneously monitoring the input and output of a UPS (see the figure) is quite common in mission-critical applications, however, the ability of the Encore Series to monitor both using only one instrument-rather than requiring a separate instrument for input and output as with other monitoring solutions-results in lower cost, reduced installation space requirements and simplified IT administration.
Data presentment via the Encore Series’ web-enabled software provides facility personnel with an easy-to-use, web browser-based environment for any monitoring need. Further, information is easily accessible throughout the organization by any user with a web browser, valid password and authorized connectivity to the system.
