One of the key objectives of hyperscale data centers is to reduce the cost per MW, as well as designing for reliability and sustainability. A strategy that we have seen among the hyperscale community is to bring your electrical loads as close to the ratings of the equipment as possible. While this shouldn’t affect day to day operations, it does affect the commissioning process. Typically, the risk only applies to Level 5 IST commissioning under 100% load. Here are a few tips when commissioning for hyperscale.
GENERAL CX TIPS
- Conduct Level 4 commissioning on each individual component at 100% load.
- Conduct Level 5 commissioning during IST at 90% load.
- When burning in your generators, use 12 hours, instead of 24, of burn-in. At 24 hours of burn-in, the failure rate is approximately 6%, whereas a 12-hour burn-in is much less.
CX FOR ELECTRICAL
From an electrical standpoint, there are primarily three different electrical designs that hyperscale clients recognize to get to 99.999% reliability. Each design has different weak points within the system that need to be accounted for during the commissioning process. The three designs that are accepted within the hyperscale community are catcher block, distributed redundant, and distributed redundant with static switches installed upstream.
Commissioning distributed redundant design. A distributed redundant design is based upon dividing your load equally among multiple UPS and PDU distribution. Each UPS (on a four-module block) is loaded at 75% of the capacity of the unit, leaving 25% capacity for redundancy in the event of maintenance or outage. The challenge is that the load is transferred at the server level. This creates a couple of problems. One, the load transfer is now the responsibility of the IT operations director since the active ATS is located within the server. Due to this configuration, commissioning from an IST perspective is limited only to the PDU. Second, statistics have proven that when transferring load at the server, large data centers typically lose up to 4% to 6% of the load due to the fact that the ATS units within servers are not very reliable.
Commissioning catcher block systems. In order to make 99.999% reliability, the catcher block system needs to be configured at no higher than seven blocks to make six blocks. Additionally, a second utility feed is required to support the 99.999% reliability requirements. The difficult aspect of this design is that the inrush to a block that is idling can damage the UPS module. In most cases, an STS design with required step load functions (i.e., STS2) to manage the inrush current is desired to provide greater control of the inrush.
Commissioning distributed redundant w/ static switch. This type of design is the most accommodating of the three electrical designs when it comes to commissioning. The full system can experience complete commissioning during the IST, and only one utility feed is required to the site in order to make 99.999% reliability. Additionally, this design is far easier to manage your electrical distribution loads than the distributed redundant design without a static switch upstream.
Commissioning for mechanical for hyperscale. Since hyperscale clients typically operate above and beyond ASHRAE limitations 72° to 78°F at the inlet, hyperscale clients often operate at 80° to 90°F at the server inlet. Most of the mechanical designs for hyperscale use an indirect evaporative or direct evaporative mechanical system to achieve 1.185 PUE or lower. Response rates and system tuning for evaporative cooling systems can increase the amount time required to commission and optimize the mechanical system.
UL cloud certification Cx. One of the aspects of getting UL Cloud Certified is the participation during the commissioning process. While UL does not create scripts or require minimum testing, it does identify the documentation stage (Level 6) which often becomes fragmented. Often, there is limited Cx sign off for Level 6 Cx, and documentation/training is not emphasized as it should be. While the UL team has certified Cx agents to witness the IST, the final level of commissioning must be completed in order to get a UL Cloud Certification. Having that third-party observation is highly important to the hyperscaler, especially in areas of network and security which are typically integrated with mechanical/electrical.
THE TIMES THEY ARE A-CHANGIN’
As Bob Dylan once wrote, “The times they are a-changin.” Whereas Uptime was once the design standard, today’s hyperscalers are dictating the new standard in both design and commissioning. Instead of the Cx agent dictating 100% Cx across the board, the Cx agent needs to understand why the hyperscale process may be different than the traditional Cx process, and adapt to the needs of this new data center purpose.
