As part of the commissioning process, power and cooling systems should be tested at two levels: the bulk level and the distribution level. Bulk-level testing determines the ability of the power and cooling components to handle anticipated load. For example, attaching one large load bank to a UPS output can help test the bulk-power capacity of a the unit. Distribution-level testing measures the ability of the power and cooling distribution system to handle the IT loads at each individual rack. Plugging individual load banks into each rack via rack-level power strip can test the capacity of the power distribution system.
Power and cooling systems can be tested at the room, row, or rack levels. Each test method utilizes simulated loads.
• Room-level testing. Load banks of are applied to power distribution units (PDU) in an open data center.
• Row-level testing. Load banks are placed in the hot aisle of every row of racks.
• Rack-level testing. Rack-level testing where individual load banks are mounted directly into every rack.
Room-level testing is most commonly used today and consists of plugging large load banks directly into PDUs that are located across an empty data center. No racks or cages exist at the time of the commissioning. A number of large, side-discharge load banks are distributed on the raised floor to simulate the computer load and all the floor tiles are perforated for air distribution. This method effectively tests the bulk power and cooling system capacity but does not effectively test the distribution capacity of either system.
Row-level testing is similar to room-level testing, except that the data center is populated with empty racks. The load banks are placed between the rows of racks and simulate more realistically an operating data center. This method effectively tests the bulk power and cooling system capacity. The power distribution capacity is not generally tested down to the rack-level. The testing of the cooling distribution capacity is more effective than the room-level method.
The third method, rack-level testing, has only come into existence over the last couple of years and consists of mounting load banks directly into the racks themselves and plugging them into the power strips within each rack instead of directly into the PDUs. This method effectively tests the bulk power and cooling system capacity. The power and cooling distribution capacity is also effectively tested because each individual rack is run as if it had real IT equipment loads. These server simulators can be set to produce the amount of heat equivalent to the amount that will be generated by actual loads in the rack. The rack-level simulators plug into individual power strips and exhaust hot air to the hot aisle. In addition to variable power density/heat settings, some of these server simulators have variable speed fans that can be tuned to the same air flow as the planned IT equipment.
Commissioning IssuesThe idea of testing both the power and cooling systems became popular with the advent of client/server networks about ten years ago. These servers were large in size compared to today’s 1U servers or blades and therefore had a very low power density of about 1 kilowatt (kW) per rack. At such a low power density, the power and cooling distribution capacity of traditional data center designs had no problem handling this load. However, average IT equipment power densities have increased dramatically over the last few years. This presents serious problems for managing heat at the rack level.
At higher IT power densities, proper cooling distribution (i.e. amount of air flow available to every rack) becomes less predictable due to various factors such as floor geometry, under-floor piping, and perimeter cooling unit orientation. Therefore, poor airflow predictability makes it imperative to test the cooling distribution at the rack level. Given this requirement, the following problems have been identified with both room and rack-level testing.
The room-level method does not test at the rack-level because no racks are present to test. The side-discharge of air from the load banks and the large fans required to cool the load bank are not representative of the real world environment. Therefore, the test does not detect the possibility of development of hot spots and disruption of normal raised floor air patterns. The room-level method is capable of determining whether the total power applied to the room can be removed by the mechanical system, but it is not possible to determine the effectiveness of individual rack cooling in any operational mode (i.e. normal, maintenance). Therefore, the room-level method should never be used to test a modern data center.
Given these issues, the industry should consider the rack-level method of simulating data center heat and power as the standard for commissioning power and cooling systems. Because these server simulators are individually mounted inside racks, they can be set to specific power densities (heat levels), and can be tuned to specific airflows. These simulators provide the best prediction of how the power and cooling system will function in a production data center. This includes failure mode testing which will test that the cooling system really does have the necessary redundancy to continue operating under abnormal conditions. Results from commissioning projects confirm how predictable this method is.
In some cases server simulators can be used to validate the results of a computational fluid dynamic (CFD) model of the data center. This adds to operator confidence in the model and allows it to be used for accurate prediction of any future modifications (i.e. the impact on power consumption and heat when new data center components are added). As long as the CFD model accurately represents the physical space, it has been found that the correlation between the temperatures predicted in the CFD model and those actually measured during rack-level testing is accurate, usually within 2-3 degrees F. Therefore, rack-level server simulators hold a huge advantage over the traditional methods of testing data center heat loads.