Figure 1. 208 Vac three-phase after the PDU transformer. 120 Vac line-to-neutral


Numerous papers and articles have been published by The Green Grid, The Uptime Institute, PG&E, Lawrence Berkeley Laboratories, and others discussing ways to measure, monitor, and increase power efficiency in the enterprise data center because power is the biggest cost within enterprise data centers.

Figure 2. 480 Vac three-phase to 208 Vac power path

The power path from the building entrance to the IT loads contains several power converters and transformers, with each conversion, power is lost. Reducing the number of transformers and operating at a higher voltage will result in greater efficiency and reduced electrical costs. An alternative approach to power distribution, bringing 415 Vac to the cabinet level, increases efficiencies and savings by reducing upfront capital costs, power consumption, and floor space. Several North American data centers have implemented this technique with success.

Figure 3. 480 VAC 3-phase power at the building entrance

Power Distribution in North American

In North America, utilities deliver 277/480-Vac, three-phase power to most large commercial buildings and IT facilities. The voltages are measured from line-to-neutral (277 Vac) and line-to-line (480 Vac) (see figure 1). Automatic switching power supplies inside IT equipment typically operate within a range from 100 to 240 Vac, single-phase, so isolation transformers must step down the utility power before it may be used, as outlined in figures 2 and 3. In data centers, electricity flows through a power distribution unit (PDU) transformer that steps it down from 480 Vac three-phase to 208 Vac three-phase.

From the PDU, remote power panels (RPP) or cabinet-level power distribution units (CDU) typically distribute the 208 three-phase power in three ways

1. 120 Vac single-phase (measured line-to-neutral)

2. 208 Vac single-phase (measured line-to-line)

3. 208 Vac three-phase (in a Delta or Wye configuration).

Figure 4. CDU power distribution within the rack

Most IT devices have automatic switching power supplies that will accept low-line (100-120 Vac) and high-line voltages 200-240 Vac. Running devices at the higher voltages increases efficiencies approximately 2 to 3.5 percent, so the first step for data center managers is to review the nameplate ratings to determine the operating voltage range and ensure devices operate at 208 Vac where possible.

IT devices in a 208-Vac distribution system are fed by powering a line-to-line configuration versus line-to-neutral for 120 Vac (see figure 3). This system can deliver 120 Vac for legacy devices; however, newer devices with universal power supplies can operate at higher voltages.

Within the IT device's internal power supply, power then goes through one more transformation and conversion to rectify 120 Vac or 208 Vac to direct current (dc) transformers (figure 4).

Demands for higher densities and computing power have led more facilities to use the 415-Vac three-phase system more commonly used in the rest of the world instead of the familiar North American distribution system.

Table 1. HP ProLiant DL380 Generation 5 (G5)

415 Vac

Reducing the number of power transformations and operating at higher voltages can reduce power and energy costs in the data center. Converting the UPS output from 277/480 Vac to 240/415 Vac through an autotransformer and eliminating the PDU transformer is one alternative. Or a PDU autotransformer can replace the PDU transformer (see figure 5). The line-to-neutral voltage is 240 Vac in a 415-Vac distribution system. According to various UPS manufacturers, eliminating the PDU transformer will result in a 2 percent efficiency gain.

This is a significant difference from the typical 208- Vac system in which the line-to-neutral voltage after the PDU transformer is 120 Vac. The 415-Vac approach doubles the power delivered to the devices while increasing efficiencies, and it reduces installation costs by eliminating components and using smaller diameter cables for distribution.

Figure 5. Difference between 415 Vac and 208 Vac power distribution systems

Additionally, operating rack-level equipment at 240 Vac vs. 208 Vac provides an additional 1 to 1.5 percent efficiency gain as shown in Table 1 for the HP ProLiant DL380.

To gain a better understanding of the benefits of a 415-Vac system, consider what is necessary to deliver redundant power to a high-density rack of blade servers (see figure 6). Using the power configuration calculator from a major blade system manufacturer, the power requirement for four fully populated blade systems is 15.1 kW. Table 2 shows four different approaches to delivering this amount of power to the rack, along with the number of power drops needed to provide redundancy.

Figure 6. Comparison between the baseline power path and the 415 Vac power path with the elimination of the PDU transformer

Additionally, operating rack-level equipment at 240 Vac vs. 208 Vac provides an additional 1 to 1.5 percent efficiency gain as shown in Table 1 for the HP ProLiant DL380.

To gain a better understanding of the benefits of a 415-Vac system, consider what is necessary to deliver redundant power to a high-density rack of blade servers (see figure 6). Using the power configuration calculator from a major blade system manufacturer, the power requirement for four fully populated blade systems is 15.1 kW. Table 2 shows four different approaches to delivering this amount of power to the rack, along with the number of power drops needed to provide redundancy.

Table 2. Power Capacity Comparison

Efficiency gains by implementing a 415-Vac design are typically 4 to 5 percent (see figure 7). Upfront cost savings are also realized by eliminating the PDU transformer, using smaller gauge power cords and less expensive plugs, taking up fewer poles at the breaker panel, and requiring fewer power drops to the cabinets. Additional savings come from reductions in cooling costs with the elimination of the PDU transformers and at the cabinet-level by having fewer CDUs and cable drops that can impede airflow. Additionally, though a 208 V, 60-A system provides the same power as a 415 V, 30-A system, lower current flow allows use of smaller conductors, which also lowers initial costs and reduces line losses due to cable resistance.

Table 3 demonstrates the annual cost savings by implementing a more efficient power distribution system based on a $0.10 per kWh rate. In general, power distribution equipment is more efficient and runs cooler when it operates in the middle of its designed range; therefore additional cost savings can be achieved by understanding the capacity requirements for meeting immediate needs with the ability to expand to accommodate future growth.

Table 3. Annual energy savings from efficiency gains

The Cabinet Level

At the cabinet level, a cabinet power distribution unit (CDU) distributes power to the IT loads. CDUs designed for 208-Vac three-phase systems are not compatible with 415-Vac systems because they are wired line-to-line. In order to deliver 240 Vac single-phase, outlets must be wired line-to-neutral. Server Technology offers designs that meet these requirements such as the Sentry Switched CDU (see figure 8) that offers a combination of IEC 60320 C13 and C19 outlets. Server Technology's CDU is a CDU configured for 415-Vac service. This particular CDU has three outlet sections, each split into two branches protected by 20-amp (A) fuses allowing it to deliver 17.2 kW, the full capacity of the circuit. With this design, if the current in Branch 1 exceeds 20 A, it will blow the fuse and remove power to the branch while leaving the rest of the CDU unaffected.

Power Monitoring

In order to manage power usage, the capability to measure and monitor it must be present. Since the CDU is the most intelligent device closest to the IT load, it is an ideal point to monitor power consumption.

Figure 7. Efficiency gains by eliminating the transformer and running equipment at 240 V rather than 120 V

Intelligent, smart, or switched CDUs are equipped with local LEDs display, provide useful information at the cabinet when powering up and load testing; however, they provide little useful information over the course of a day, week, or month. Since power draw varies, CDUs that can be networked and allow power information to be queried, polled, and trended at both a circuit and at the device level will provide data center and facilities managers a better understanding of actual equipment power usage and identify underutilized equipment. Information can also be rolled up so power usage can be viewed by device, at an application level, or at a cabinet row or floor level. Additionally, the accumulated data may be used to calculate the power usage effectiveness (PUE) and other metrics established by The Green Grid and other organizations.

Figure 8. Server Technology cabinet power distribution unit (CDU)

Environmental Monitoring

Maintaining the proper temperature and humidity range within the cabinet has become even more critical as densities within the cabinet and higher server utilization due to virtualization increase. Server Technology CDU's have the capability to monitor and send alerts from up to four combination temperature and humidity probe inputs. Additionally, when used with Server Technology's Sentry Power Manager, temperature, humidity, and power may be logged and trended.

Figure 9. IT Equipment with IEC C14 inlet

120-V IT Devices

The 120-Vac NEMA power cord that often feeds IT equipment will not be compatible with the CDU in a 415/240- Vac distribution system. Fortunately, most IT equipment manufacturers use power supplies that accept 100 to 240 V and standardized on IEC connectors on their products such as the Dell 2161 DS KVM Switch, which is designed with an IEC 60320 C14 inlet. This allows the power cord to be changed out to an IEC type cord. For those devices that do not have universal power supplies that will accept 240 Vac, rack-mounted transformers are available from a number of vendors.

Figure 10. IEC 60320 outlets and plugs to connect IT equipment to the CDU

Summary

With small data centers operating in the range of tens of kW and large data centers running in tens of MW, incremental increases in efficiencies translate into large savings. There are several other proposed power distribution architectures being discussed, such as rack-level and facilities-level direct current power distribution and 277 -Vac distribution. The Data Center Pulse group, an organization of data center owners, operators, and users, recently held their inaugural conference in February 2009. They compiled a list of their top ten goals, one of which is to eliminate all transformers and deliver 277/480 Vac at the rack level.