- ~6% first cost savings
- 230 metric tons annual CO2 avoidance
- Improved thermal performance
- No UPS room footprint requirements
- Redundancy/concurrent maintainability
Facility and project overview
The prototype close-coupled UPS cooling system was installed in an existing data center as part of the Phase II power expansion project.
• 2,320 m2 raised floor
• Additional 2,785 m2 future
• 9.6 MW end of day UPS capacity
• 2.4 MW UPS capacity installed day one
• 3.6 MW upon completion of Phase II
• Uptime Tier III certified (design and facility)
Key project points:
• Install additional 1,200 kW UPS system (2N redundant)
• Install cooling systems to support UPS
• Additional items not related to case study:
• One 3 MW generator and support systems
• Six chilled water computer room air handlers
• Eight power distribution units
UPS Cooling Design Requirements
As is typical in mission critical environments, the owner had high expectations when it came to the UPS cooling system. Phase I of the facility employed large chilled water air handlers to cool the UPS rooms. The owner was not confident in the ability of the existing air handlers to serve the additional UPS load, and charged the design team to find another way to cool the UPS system. The owner’s requirements were clear: Maintain the UPS inlet temperatures within the manufacturer’s specifications (UPS manufacturer’s specifications indicated an acceptable ambient range of 0°C to 40°C, with no rate of change limit specified), provide scalability, and provide “double-concurrent maintainability,” which is defined as the ability to maintain both the cooling system and UPS system simultaneously (or ability to maintain the cooling system with all critical load on a single UPS system) without impact to the IT environment. An additional key constraint was the fixed room size and basic layout, since the rooms were constructed in Phase I.
First Cost Savings
First cost data for the close-coupled cooling system and two comparative systems is shown in Table 1. This identifies a first cost savings of ~6% (day one loads) for the close-coupled cooling system. Note that this is equipment cost only. While installation of all systems is assumed to be similar, systems with outdoor equipment (condensing units) will have additional costs associated with protecting and screening equipment. While the cost of a system to support the end of day load is ~14% higher, it is important to note the energy savings from “cooling delivery” alone amounts to a savings of $21,000 per year, even at the lower day one loads, providing for a quick payback. This payback is even shorter when considering the alternative system (DX CRACs) has no method of economization, while the close-coupled cooling system utilizes the waterside economizer associated with the central chilled water plant.
Energy Savings/Environmental Impact
The primary way that the close-coupled cooling system provides energy savings (Table 2) is the fact that no additional fans are used to cool the UPS. Instead, the UPS module fans are utilized to move air across the cooling coils, and small pumps (1A power draw at 480 V) supply the refrigerant to the coils. The close-coupled cooling system uses 94% less energy than comparative systems to deliver the cooling to the UPS modules. This leads to an avoidance of 230 metric tons of CO2 emissions per year. Additional environmental impact reduction (when compared to a DX system) comes from the efficient chilled water plant, which utilizes a waterside economizer, a chemical-free condenser water treatment system, and advanced control logic to optimize plant performance.
Sink/Source ?T Imbalance
As part of this project, the concept of “sink/source ?T imbalance” was identified and documented. As the name implies, this phenomenon is related to the difference in airside ?T of the heat “sink” (cooling equipment) and the heat “source” (UPS equipment). Figure 1 shows the relationship of UPS module efficiency, module heat loss, and airside ?T as a function of UPS module load.
As can be seen from the upper one-third of the chart, the maximum airside ?T that can be expected from the UPS is approximately 5.5°C, while the maximum under normal operating conditions is approximately 2.8°C. When contrasting this to the typical 10°C airside ?T of precision cooling equipment, it becomes clear that this phenomenon will make for a challenging cooling scenario because of the difference in airflow requirements between the UPS modules and the cooling equipment.
There are several different ways to account for sink/source ?T imbalance, including over-ventilating (providing higher airflow at a lower ?T, which leads to decreased energy efficiency), controlled mixing (allow portions of the room to operate at elevated temperatures while mixing cool air to provide cooling), and close-coupled cooling, which is the focus of this case study.
UPS System Design and ROom Layout
The layout of the new UPS system is shown in Figure 2. Gray shaded equipment is the “A” side UPS, blue is the “B” side, and green is the refrigerant pumping units, which are described below. The dashed equipment is future. A summary of the UPS system design is outlined as follows:
• Two UPS modules per system
• Systems arranged as 2N redundant
• Systems compartmentalized
• Maximum design load per module = 600 kW
• 1,200 kW system load, scalable to 2,400 kW
Overview of Close-Coupled UPS COoling
Close-coupled cooling is so named because the heat load is removed close to its source. This “couples” the cooling in a close manner (Figure 3). Cool air is drawn into the UPS module by the UPS module fans. A key design feature is that this cooling system uses only the UPS module fans — no additional fans are used. As the air moves through the UPS it absorbs heat from the electrical components. Hot air is then exhausted vertically into a sheetmetal evase, which is designed to provide uniform airflow across the cooling coils above.
As the hot air moves through the cooling coils, it is cooled using a pumped refrigerant (R-134a), which is always maintained above the dewpoint in the room. This ensures that condensation will not form on the cooling coils or on the supply pipes which feed the coils.
The cooling coils are fed from three different refrigerant pumping modules (Figure 4), which reject heat from the refrigerant to the chilled water system (14.4°C supply temperature) via a heat exchanger. Each pumping unit has a controller that modulates the control valve as required to maintain the proper refrigerant supply temperature to the coils. Refrigerant is supplied to the coils as a liquid, and returns as a mixture of liquid and vapor. Because no compressor or expansion valve is in use, the refrigerant is not sub-cooled, which enables the refrigeration process to take place above the room dewpoint.
Design summary of UPS cooling system:
• Pumped refrigerant system
• Refrigerant maintained above the dewpoint eliminates opportunity for condensation in room (adjusts refrigerant temperature to be 2.2°C above the dewpoint)
• Redundant power sources (with automatic changeover) provided for each pumping unit
• Chilled water isolation valves provide concurrent maintainability for refrigerant pumping units
• Sheetmetal evase (Figure 5 and Figure 18)
• Designed for even airflow across cooling coils, critical for cooling unit redundancy
• Access doors allow serviceability of UPS module fans and underside of coils
• Cooling system is supported from overhead with a flexible duct connection at the UPS module; no additional weight supported by the UPS module
• Custom design coils for application
• Sized to be less than 25 Pa airside ?P (per UPS manufacturer’s requirements)
Commissioning Methods and Data acquisition Equipment
While commissioning is a requirement for any newly installed mission critical system, this project necessitated a more thorough review than normal, as this type of system had never been utilized in a similar manner.
The system was tested in every conceivable operating mode, including normal operation (various loads) as well as several different failure modes. For testing purposes, the UPS was loaded using resistive load banks to simulate the critical load. The commissioning test results are discussed in the following pages.
Understanding the thermal performance of the cooling system under various operating modes was critical. In order to collect this valuable information, a custom data acquisition system was installed. The system’s 48 thermocouples, National Instruments interface hardware, and a custom software/GUI programmed in NI LabVIEW collected over 2 million data points during the commissioning process. Each UPS module had 12 sensors installed: one on the discharge of each of the six cooling coils, four on the inlet to the UPS module, and two in the sheetmetal evase. The sampling rate (all sensors simultaneously) was 15 points per minute. All of the following figures utilize actual data points without interpolation or extrapolation.
Commissioning Results
The commissioning process validated that the designed and installed system performed as anticipated. While many tests were performed, the key items and their results are summarized below.
Normal operation: operation at various loads
Loading summary:
• Two UPS systems operating (balanced loading)
• Stepped UPS load
• Three pumps operating
Results summary:
System responded very well as load was increased. Stabilization occurred relatively quickly after each step load (Figure 7).
Normal operation: Operation at 100% of load
Loading summary:
• Two UPS systems operating (balanced loading)
• 1,200 kW UPS load
• Three pumps operating
Results summary:
System stabilized at a maximum module average inlet temperature of approximately 24°C. This is well below the manufacturer’s upper limit of 40°C (Figure 8).
Normal operation: Failover to a single UPS system
Loading summary:
• 1,200 kW UPS load
• UPS “failover”
• Three pumps operating
Results summary:
UPS “failover” was simulated by step loading one system (600 to 1,200 kW) while simultaneously unloading the other system (600 kW). The cooling system performed well to this dramatic change in load. This is a condition that would occur due to a failure of a UPS system or transfer for maintenance. Maximum average module inlet temperature of 25.5°C is well below the upper limit of 40°C (Figure 9).
Double-concurrent maintainability
Loading summary:
• 1,200 kW UPS load
• One UPS system operating
• Pump “failure” at 00:30
Results summary:
Cooling system met performance objectives during the “failure” of one pumping unit with UPS load on a single UPS system. Note that two of the six coils on each UPS module (supplied by the “failed” pump) were discharging air as warm as 35°C, yet the coil discharge mixed to a maximum average module inlet temperature of ~ 29°C; which is well below the upper limit of 40°C.
Even though two of the six coils on each module were failed for this scenario, Figure 11 shows that there was no negative impact to the UPS module inlet temperatures, as the inlet temperature stayed well below the 40°C allowable (Figures 10 and 11).
Maximum allowable refrigerant temperature
Loading summary:
• 1,200 kW UPS load
• One UPS system operating
• Two pumps operating
• Refrigerant supply temp. = 26.7°C
Description/results summary:
This test determined how humid the room could be before the cooling system would fail due to a) condensation occurring, or b) the refrigerant temperature being raised so high that proper cooling was not provided.
The results show that even at the maximum refrigerant temperature to which the pumping unit could control, the maximum average module inlet temperature of approximately 34.5°C is well below the upper limit of 40°C (Figures 12 and 13).
Open Transition Test
Loading summary:
• 1,200 kW UPS load
• One UPS system operating (simulates UPS system failure or full system transfer for preventative maintenance)
• Two pumps operating
• Interrupt all power to cooling system for 25 seconds
Description/results summary:
This test simulated the response of the cooling system during an “open transition” electrical transfer event. The facility utilizes closed transition transfer switches for planned transfers to generator and return to utility. An unplanned loss of utility interrupts power to the UPS cooling system while generators are started, paralleled, and transferred. Disabling both redundant electrical sources to each pumping unit for 25 seconds simulated this transfer. Note: The pumping system restart time after power restoration is approximately 60 seconds. The system responded well to this test, as noted by the maximum average module temperature of 31.5°C which is well below the upper limit of 40°C. See Figure 14.
Additional Testing Notes
The tests described above, along with other similar testing, gave the design team, and more importantly the owner, a high level of confidence in the performance of the close-coupled UPS cooling system.
After observing the testing, and reviewing the results, the owner is satisfied that the new system meets (and exceeds) their operational, environmental, and sustainability requirements.
Summary
Overall project results often overshadow the journey that was required to achieve them. In this case, the journey is also an important part of the story. When the owner expressed a concern with the existing cooling system, and a desire to implement a different solution on this phase, the design team rolled up their sleeves and went to work. After preliminary design calculations and room layouts, the design engineers met with the manufacturer who would ultimately provide the refrigerant pumps and the cooling coils.
As the design progressed, there were numerous conference calls, meetings, and reports issued by the design team to accurately convey the cooling concept. Additionally, a custom sized coil was designed to better meet the needs of the project. While this is an excellent example of design/consulting engineer driving a manufacturer’s design process, it is even more incredible considering that the total project timeline, from concept to commissioning, was nine months. This is an amazing achievement, especially when considering that a prototype concept was used in a mission critical system.
Another important plot line in the story is the rapport built between the owner and the engineers. When the owner expressed concerns, the engineers heard and understood the voice of the customer, and reacted accordingly. When the engineers presented a solution, the owner trusted their calculations, experience, and advice. With each party shouldering both the risks and rewards of the project, this is an excellent example of working together to provide a unique win-win solution to a difficult problem.
Both the owner and design team see applicability of this system in other data centers and beyond. Virtually any facility with a UPS can utilize this cooling technology. If chilled water is not available, the refrigerant pumping units can be cooled with air cooled condensers. Additionally, this technology is applicable to cooling other types of equipment — any equipment with high airflow and low airside ?T.
While it is often said that “form follows function,” this is not actually the case. Rather, it can be said that “form follows failure” (Henry Petroski, professor of civil engineering at Duke University presents this concept in his book titled, To Engineer is Human: The Role of Failure in Successful Design). The close?coupled cooling system did not arrive at its current form because this was the only way to cool a UPS. Instead, the current form reflects the attempt by the design team to eliminate as many of the known failure modes as possible. Failures can take on many forms, as noted in Table 3. No engineer should consider their designs to be perfect, and while it is noted that there is always room for improvement, this design is considered to be the most appropriate solution to the problem at hand given the many constraints presented.
