Within a short time frame, hyperscale data centers have gone from being designed with a typical wholesale data center layout to industrial-level design. 

The design aspects of high-end interiors and concrete hardening are features long gone, while the hyperscale client now focuses exclusively on speed-to-market and cost. Over the last decade, wholesale data center providers have focused on supporting the enterprise user. But today, the same wholesale providers are trying to cater to the cloud–hyperscale customers. 

While their design philosophies remain to drive to five nines of reliability, the frills of previous designs reflect much more of a super-industrial design.  Metal buildings, trailer-type offices, and cost per MW take precedence in this industrial age of hyperscale. While each hyperscale client will have different design requirements, common elements remain the same for cloud providers.

For this reason and more, hyperscale owners will want to examine their total cost of ownership (TCO). Consider the following three critical components to developing a meaningful hyperscale TCO.

Leg One: Site Selection and the Economic Development Impact of Hyperscale

The power consumption of the selected sites requires 180 to 300 MW (or more) of power per regional campus. This has two downfalls. One, just being able to get reliable power of that magnitude creates timing delays and increases cost. Secondly, the consumption of power and water can create a negative impact within local communities. And while they do create construction jobs, the operating employment numbers are minimal once the sites are built out.

When selecting the site, the Economic Development Authority (EDA) within each hyperscale company prepares themselves to address the community through multiple programs. While the programs are similar in nature, each hyperscale company addresses the community differently. Some include:

• Community Outreach — Building out schools, day cares, and community centers;
• Local Sponsorship — Concerts, events, and promotional support;
• Green Energy Programs — Purchasing credits as well as building out windfarms, solar power plans, and leveraging micro grid designs; and
• Hiring Local Veterans — Veterans are well-suited for data center operating jobs due to their stringent processes used on the battlefield.

While some communities may be hesitant to see the eyesore of today’s hyperscale, other communities welcome cloud providers by offering large tax-incentive programs. The construction trades within these communities benefit from large contracts and constant churn within the data center.

However, many local communities have a hard time with construction trades. In many cases, these campuses require 600 to 800 electricians on-site during construction, which may be difficult for the local trades to meet — especially in remote areas of the country, where many of these campuses are built.

While the EDA challenges can change from community to community, there is baseline selection criteria that remains the same. The site selection checklist covers a variety of topics, including weather threats, crime, economic stability, tax incentives, cost of utility power, security, flood zones, and more. But these are just the items that can be quantified and remediated.  Additional items, including land, soft cost, and taxes, are easier to predict than the EDA side of site selection. These items are considered fixed costs when developing the TCO.

With the above challenges within the EDA, developing a financial program to address the items is very challenging due to so many moving targets. It is difficult to predict public protest and the cost impact associated with it. Construction delays have a cost impact, and depending on the protest or dispute, these delays can become permanent if the community can’t cut a successful deal with the cloud provider.

The combined site selection elements of the EDA and fixed site selection criteria are the first step in developing the TCO Program.

Leg Two: Developing the Hyperscale Prototype

Today’s hyperscale data center reflects the design of super-industrial plants, but they are not manufacturing anything tangible; they are manufacturing data.

While prototypes in the past consider cost, the hyperscalers have been able to get below a $6 million per MW when totally built out. Here are some of the prototype design features that are trending:

1. In the past, several of the shell-and-core prototypes utilized tilt-up precast or factory precast. Hyperscale data centers are designing the shell and core as metal buildings. The roof systems are still designed for 90-mph wind loads (unless other criteria was identified during site selection). The impact to the TCO is $74 per square foot for metal versus $123 per square foot for precast. 
2. The interior office design is baseline minimum. The interior design of yesterday typically cost $85-plus per square foot versus today’s design of $45 per square foot (not including furniture, fixtures, or equipment). 
3. Over the years, there have been multiple design topologies for hyperscale. The majority of hyperscale owners select distributed redundant electrical designs. Of the opinions that drive the distributed redundant design, the most important one is cost. 
   A distributed design only requires one electrical feed to achieve five nines, whereas a catcher system requires dual utility feeds. This saves the business the cost of the second feeder. This is especially true when considering the purchase of over 180 MW of power.
   There are, however, some downfalls to this design. For one, the load transfer relies on the ATS at the sever. The components are not as reliable as for, say, a static switch. Typically, it’s estimated that during a load transfer at the server level, as many as 4% of the servers drop out, but they are backed up in the network. Additionally, the distributed redundant design is more difficult to manage where your load resides. In most cases, generators are still being utilized for emergency power. 
4. A majority of the hyperscalers are achieving less than 1.2 power usage effectiveness (PUE) via direct evaporative or indirect evaporative cooling. These units are stackable for multifloor applications, with three stories being the highest they will typically go. (See Figure 1.)
   Additional design requirements for hyperscale data centers can be found in UL 3223.
5. Peer Review is critical, including safety, commissioning, and design. This will ensure that the prototype is achievable and safe. Peer reviews are conducted by a third-party design team and often include internal team members evaluating the operational aspects of the prototype.
   During the development of the prototype, alternative system designs come into play as well as detailed estimating to support the new concept. Equipment placement plays a key role in the new prototype, concentrating on the reduction of feeder lengths. Additionally, procurement processes will play a key role in determining the cost per MW for your TCO. Of course, bulk purchasing, skidding, and procurement methods all contribute to the final MW cost.
   There will be two different dollar amounts per MW cost: Day One, and Total Cost per MW, when the site is fully built out. Day One costs typically come in around $8 million to $9 million per MW due to the land development price. Total Cost per MW after fully being built should come in around $6 million per MW. 

Leg Three: Development and Operations

There are really two parts to Leg Three: development, which is the project management team for design and construction, and ongoing operations. 

The cost for development will include all of the required design disciplines, contractor cost, and internal project management. Typically, a hyperscale team will consist of project management, architecture, electrical/mechanical leads, construction leads, BAS, procurement, safety, security, technology, and (sometimes) risk management.

Typically, the engineering/architect professional fees should come in at around 3% of the total construction cost. However, due to the prototype already being developed at 70% completion, the additional 30% of work to be completed is allocated to site adaptation of the prototype. Therefore, design fees should drop down to around 1.5% of construction, plus construction administration. Not a good sign for engineers.

Commissioning also is expected to follow the prototype process. Once the script is created, it will be expected to be used again within each project.  The question becomes: At what level is commissioning engaged?

New Trend: During cost of MW, the load ratings are designed all the way up to the equipment ratings (sometime even surpassing them). Because of power management, this is only an issue at full-load commissioning. Many hyperscalers commission Level 4 at 100%, but Level 5 IST is commissioned at 90% of rating to protect the equipment.

The construction side of the development has been stretched thin. Never before has the data center industry seen such a surge in construction activity. In the top five domestic construction companies, the average construction contract within a hyperscale campus is in the hundreds of millions of dollars. This creates two things: fierce competition and underqualified staff. 

To calculate the TCO for construction, allocate 1.5% construction fee, plus 5% general conditions.

The second part of Leg Three is operations. The biggest challenge that we have in the industry is finding qualified people to operate data centers. There are few colleges that offer a data center operations curriculum.  Within our industry, the military backgrounds work well due to the process-orientated mindset. For this reason, businesses like Salute are doing well within the industry.

As for operations, remember to include facility personal, maintenance, and previously identified elements of the EDA in the TCO. When developing pro formas in the past, best practice was to calculate 24% of the wholesale provider revenue to be contributed toward operations. As the facility fills up, that number drops down to 17%. Hyperscale calculations are exact and include all aspects of operating the data center. 

Conclusion

The Industrial Data Center Age is here, and it’s going to move quickly.  Between the combined power of cloud, AI, edge, and 5G, new technologies will emerge that we can’t even comprehend at this point. After all, no one could have predicted today’s hyperscale campus six years ago.  While we can predict the total cost of ownership, we have no way of knowing what data it will manufacture.