Row-level testing method


Room-level testing method

Once a data center is designed and built, how does the organization know if the on-site cooling infrastructure will adequately support the power-hungry IT equipment? No amount of floor plan or specification analysis will provide a definitive answer to this question, especially for traditional data center designs. Too many systems are interacting and are influencing each other’s performance. To ensure proper data center operation under both normal and failure conditions, power and cooling systems must be tested during the commissioning process. The commissioning process can also ensure that power and cooling system redundancy actually functions as designed and that the systems function at all load levels including 100% load.

As part of the commissioning process, power and cooling systems should be tested at two levels: the bulk level and the distribution level. Bulk-level testing determines the ability of the power and cooling components to handle anticipated load. For example, attaching one large load bank to a UPS output can help test the bulk-power capacity of a the unit. Distribution-level testing measures the ability of the power and cooling distribution system to handle the IT loads at each individual rack. Plugging individual load banks into each rack via rack-level power strip can test the capacity of the power distribution system.

Testing Methods

Power and cooling systems can be tested at the room, row, or rack levels. Each test method utilizes simulated loads.

•    Room-level testing. Load banks of are applied to power distribution units (PDU) in an open data center.

•    Row-level testing. Load banks are placed in the hot aisle of every row of racks.

•    Rack-level testing. Rack-level testing where individual load banks are mounted directly into every rack.

Room-level testing is most commonly used today and consists of plugging large load banks directly into PDUs that are located across an empty data center. No racks or cages exist at the time of the commissioning. A number of large, side-discharge load banks are distributed on the raised floor to simulate the computer load and all the floor tiles are perforated for air distribution. This method effectively tests the bulk power and cooling system capacity but does not effectively test the distribution capacity of either system.

Row-level testing is similar to room-level testing, except that the data center is populated with empty racks. The load banks are placed between the rows of racks and simulate more realistically an operating data center. This method effectively tests the bulk power and cooling system capacity. The power distribution capacity is not generally tested down to the rack-level. The testing of the cooling distribution capacity is more effective than the room-level method.

The third method, rack-level testing, has only come into existence over the last couple of years and consists of mounting load banks directly into the racks themselves and plugging them into the power strips within each rack instead of directly into the PDUs. This method effectively tests the bulk power and cooling system capacity. The power and cooling distribution capacity is also effectively tested because each individual rack is run as if it had real IT equipment loads. These server simulators can be set to produce the amount of heat equivalent to the amount that will be generated by actual loads in the rack. The rack-level simulators plug into individual power strips and exhaust hot air to the hot aisle. In addition to variable power density/heat settings, some of these server simulators have variable speed fans that can be tuned to the same air flow as the planned IT equipment.

Row-level testing method using small load banks

Commissioning Issues

The idea of testing both the power and cooling systems became popular with the advent of client/server networks about ten years ago. These servers were large in size compared to today’s 1U servers or blades and therefore had a very low power density of about 1 kilowatt (kW) per rack. At such a low power density, the power and cooling distribution capacity of traditional data center designs had no problem handling this load. However, average IT equipment power densities have increased dramatically over the last few years. This presents serious problems for managing heat at the rack level.

At higher IT power densities, proper cooling distribution (i.e. amount of air flow available to every rack) becomes less predictable due to various factors such as floor geometry, under-floor piping, and perimeter cooling unit orientation. Therefore, poor airflow predictability makes it imperative to test the cooling distribution at the rack level. Given this requirement, the following problems have been identified with both room and rack-level testing.

The room-level method does not test at the rack-level because no racks are present to test. The side-discharge of air from the load banks and the large fans required to cool the load bank are not representative of the real world environment. Therefore, the test does not detect the possibility of development of hot spots and disruption of normal raised floor air patterns. The room-level method is capable of determining whether the total power applied to the room can be removed by the mechanical system, but it is not possible to determine the effectiveness of individual rack cooling in any operational mode (i.e. normal, maintenance). Therefore, the room-level method should never be used to test a modern data center.

The row-level method of testing data center heat and power does a better job of simulating realistic data center conditions but fails to fully capture the airflow dynamics that occur at the rack level. This is because the load bank fans are placed in the aisles instead of inside the rack. In a production data center, the fans inside the IT equipment are what drive the cool air through the rack. Even in cases where smaller load banks are placed outside of every rack the air flow patterns are still distorted compared to those of IT equipment. This lowers the level of confidence as to whether the distribution system will effectively cool the production data center.

Given these issues, the industry should consider the rack-level method of simulating data center heat and power as the standard for commissioning power and cooling systems. Because these server simulators are individually mounted inside racks, they can be set to specific power densities (heat levels), and can be tuned to specific airflows. These simulators provide the best prediction of how the power and cooling system will function in a production data center. This includes failure mode testing which will test that the cooling system really does have the necessary redundancy to continue operating under abnormal conditions. Results from commissioning projects confirm how predictable this method is.

CFD

In some cases server simulators can be used to validate the results of a computational fluid dynamic (CFD) model of the data center. This adds to operator confidence in the model and allows it to be used for accurate prediction of any future modifications (i.e. the impact on power consumption and heat when new data center components are added). As long as the CFD model accurately represents the physical space, it has been found that the correlation between the temperatures predicted in the CFD model and those actually measured during rack-level testing is accurate, usually within 2-3 degrees F. Therefore, rack-level server simulators hold a huge advantage over the traditional methods of testing data center heat loads.