The enterprise and colocation data center operators are infused with a risk adverse culture. In most aspects, this is a prudent philosophy, especially when it comes to ensuring IT power, which essentially is a binary condition; you have it or you don’t. Assuming you have made the decision to have whatever level of redundancy or seeking to maintain power availability, the second concern is energy efficiency. Clearly, UPS technology has made great strides and now are available with operating energy efficiencies in the 95% and above ranges at full load, and still remaining relatively efficient operation profiles even when under low per unit loads encountered in highly redundant designs such as N+1, 2N, or even 2(N+1). So while there is always room for improvement, it seems we are only going to be able to squeeze out a few more points of electrical efficiency.
As you may have guessed by now, this article is about cooling. In most cases, if we’re going to discuss energy efficiency, conventional cooling systems (i.e., closed loop CRAC/CRAH units in the whitespace) have traditionally used a substantial portion of energy consumed by the facility infrastructure. This is especially true in older data centers, resulting in a PUE of 2 or higher. And while newer cooling equipment and better practices regarding airflow management have improved this situation, cooling still represents a substantial portion of facility energy usage.
The ASHRAE Thermal Guidelines have been the data center de-facto standard since the Technical Committee 9.9 (TC 9.9) published the first Thermal Guidelines for Datacom Equipment in 2004 with a “recommended” temperature range of 68° to 75°F (20° to 25°C). They expanded the recommended temperature and humidity ranges starting back in 2008 to 64.4° to 80.6°F (18° to 27°C), and introduced the “allowable’ ranges A1 to A4 in 2011 and updated them in 2015. In point of fact, virtually every commodity server can operate in the A2 range of 50° to 95°F.
I will not belabor this too much further other than to say that here we are in 2017 and finally after nine years and three updates to the ASHRAE guidelines, more operators are slowly getting more “comfortable” with the concept of raising the supply temperature beyond 68°F and finally monitoring IT intake temperature. However, when operators finally decide to raise their temperature to the upper end of the “recommended” range, this is not as dramatic as might seem.
Nonetheless, when operators decide to raise the temperature, they typically just change the setpoint on their traditional cooling units, which in most cases normally regulate temperature based on sensing the return airflow. This means that by setting a typical CRAC or CRAH control panel to 80°F to save energy, still results in supply air temperature of about 60°F (based on the typical 20° Delta-T of a cooling system). The issue of airflow management is really the key to ensuring that the temperature of the air actually entering the IT equipment (measured inside the front of the cabinet), is within the intended temperature range, rather than the “room” temperature (wherever that is), or simply measuring the temperature in the middle of the cold aisle.
Nonetheless, regardless of how high the IT intake temperature gets, the servers still do not sweat, so why do we still need to have de-humidification systems in all the cooling units?
While I have been talking about temperature, the other major aspect of the first edition of ASHRAE guidelines also recommended that the data center maintain the humidity range to 40% to 55% relative humidity (RH), when it was published in 2004. Based on this, most cooling system controls were tightly set to 45% to 50% RH. This has become an embedded memory which has caused numerous data center operators many years to change their habits, despite updated versions issued by ASHRAE which have expanded the humidity ranges as IT equipment improved.
This all tracks back to the site humidity control which was strictly enforced in all the original data center designs and operations. This strict control was based on several factors such as older IT equipment being sensitive to (humidity either too dry or too high), as well as the fact that paper and paper handling devices like printers and punch-card readers originally inside the whitespace were also sensitive to humidity.
The inherent issue with conventional cooling units is that the coils which are typically below dewpoint when the unit is cooling, results in condensation, extracting moisture, thus effectively dehumidifying the air while cooling. To counteract this fact, cooling units also contained humidification systems which constantly replenish the moisture extracted by the cooling process. This typically involves electrically heating water into steam to humidify, which also adds heat to the airflow and also raises the latent cooling system energy required to cool the air when moisture condenses on the cooling coils. This ongoing process reduces the energy efficiency.
In addition, the cooling units also contain a dehumidification control system which involves a “re-heat” function that activates a heating coil while running the cooling function. These electrically heated steam humidifiers can draw 3 to 15 kW per unit and the re-heat coils can range from 5 to 30 kW per unit, depending on the size of the cooling unit.
As you can imagine this increases energy usage. Moreover, since it was common practice to outfit all the cooling units with a humidity control system, it also significantly impacts the size and rating of all upstream electrical components of the power chain. This includes items such as the distribution panels feeding the cooling units, the backup generator and ATS, the main switchgear, as well as the size of utility feed, since they have to be sized based on the possibility of any or even all the cooling units drawing the maximum power when trying to control humidity (in addition to the compressor power).
While this is well known to the designers of cooling equipment, only more recently has any changes in the viewpoint of tight humidity control become part of the mindset of the data center operators, despite the fact that in 2008 the “recommended” humidity range was in reference to dew point (15.8°F DP to 59°F DP and 60% RH). The latest 4th edition of environmental guidelines (released in 2015) is virtually unchanged from the prior 2011 version except for the fact that the “allowable” humidity range is now 8% to 80% RH. This was a result of a 2014-15 ASHRAE study which showed that low humidity ranges had very little practical impact on IT failure rates due to electrostatic discharge (ESD).
Despite this, many operators and IT managers still fear low humidity conditions. There are still some concerns amongst some in the industry regarding operation in the higher ends of the humidity ranges causing long-term issues with IT reliability, especially if there are pollutants in the atmosphere. To address this concern, ASHRAE developed a test “coupon” which is a printed-circuit card that is left in an operating data center for 30 days and then sent to labs to see if there are any signs of corrosion.
For the past few years more and more cooling equipment manufacturers and data center designers have moved toward reducing or eliminating the inherent dehumidification effect by dewpoint control of the cooling coils. For chilled water cooling units this can be accomplished by raising the chilled water temperature at the chiller. For DX cooling units, more recent advances utilize variable speed internal compressors and variable refrigerant-flow systems. This has multiple positive effects; increasing the efficiency of mechanical cooling process itself (i.e., compressor efficiency), as well as reducing the need to add moisture and the energy used by the humidification system. One of the common problems in earlier systems, and even today, is a phenomenon known as battling CRACs. This occurs when the individual cooling units would simultaneously try to dehumidify in response to other units trying to humidify based on individual unit control schemes. Most new dew point control schemes also involve centralized control, instead of individual unit based control of humidity.
THE BOTTOM LINE
Even if you have an older design using conventional cooling units with individual humidification control systems, there is an easy fix for the battling CRAC phenomenon without the need for installing a centralized control system. By changing the set-points to a broader range, such as 30% to 70% RH, it will minimize the duration of humidification/dehumidification cycles, and still be within the ASHRAE “allowable” range. Also consider disabling the control system on half of the cooling units in the room and relying on the remaining units to control humidity within the new broader settings. Of course make sure that your sensors are properly calibrated, since this is often a predominant cause of battling CRACs.
You will be surprised how much energy you will save in your existing facility using this simple no cost solution. Moreover, if you decide to build a new data center, consider specifying that only a fraction of the cooling units being equipped with humidity controls. This will save not only the cost of the cooling units, it will also decrease the maximum capacity rating and cost of all the upstream electrical components as well. Or more strategically, utilize the extra utility power capacity and energy for more IT equipment, rather than the power required to maintain outdated tight humidity control ranges.
To be clear, I do not recommend operating near the 95°F upper end of the A2 range under normal condition, since this leaves no margin for any temperature excursions should there be any cooling system issues, such as failing cooling unit or during a utility power interruption and the restart delay of cooling once on back-up generator.
One other caveat that should not be overlooked, this does not apply to tape back-up systems, which are sensitive to temperature and humidity ranges as well as rate of change. Consider placing them in a separate area with a dedicated cooling system. This will save energy and allow the rest of the data center to safely operate at the wider temperature and humidity ranges that the ASHRAE guidelines encompass and the IT equipment manufacturers’ specifications cover. The hyperscalers have proven that even using direct outside air for free cooling works and the millions of the servers they operate don’t all suddenly fail at 81°F or 20% RH. Traditional approaches are no longer a strategy, so stop trying to control humidity and temperature so rigidly. Your servers won’t sweat over it, so neither should you.
