Much data center infrastructure investment goes to large-scale, facility-wide systems like chillers and air handling units, which run at reduced efficiency (and high average cost per kW) until their optimum load is reached. At the same time, data center owners have to make significant investment in energy generation, UPS, and backup systems, which may never be used.
Facility and energy infrastructure components are typically designed with a 20-year horizon in mind, yet IT systems (networks, servers, and storage) typically have a one- to three-year lifespan before they are upgraded or obsolete. As the cost of servers (and associated processing power) has declined, the cost of the infrastructure (facilities, power, and cooling) needed to house and operate those servers keeps rising dramatically. The annual cost of data center infrastructure is now running at more than twice the cost of the IT equipment itself, and this proportion is increasing.
In a February 2007 article available on the Electronics Cooling website, Christian Belady identified the year 2004 as the point when infrastructure costs first reached approximately equality with server costs.
Facing these economic pressures, the traditional data center expansion model is becoming less sustainable. A new, truly modular approach is needed to move data center infrastructure and operations into the next generation.
“The mismatch of infrastructure and utilization is at the crux of the problem,” explains Jakob Carnemark, senior vice president at Skanska, “leading to overinvestment, overcapacity, and misalignment with the business mission and economics.”
“Traditional data center facilities are vastly inefficient,” he continues. “It can take years to get to the point where IT utilization reaches anywhere near full mechanical and infrastructure system load. Skanska saw the need for more efficient solutions that could modularize infrastructure to match IT expansion with just-in-time delivery of additional capacity.”
Skanska’s Mission Critical Center of Excellence has partnered with innovators Inertech, LLC and Energy Metrics, LLC to design a new scalable data center solution to meet these industry challenges. Working with TELUS, one of Canada’s leading telecommunications providers, the system has been refined and deployed at a new facility in Rimouski, PQ.
TELUS serves more than 12.7 million wireless and wire line customer connections for the consumer, large enterprise, and small-to-medium business segments. It also provides transaction support for many of the country’s largest health-care organizations. Facing rapid growth over the coming years in its consumer base and platforms, TELUS realized that legacy infrastructure expansion models did not meet the needs of its business and economic realities.
The design of the new TELUS facility represents a significant shift in data center design philosophy. Core objectives that underpinned the months-long collaborative evaluation and design process included:
- Offering the lowest total cost of ownership (TCO) and optimal capital deployment over the life of a facility without compromising security and reliability
- Supporting long-term customer growth and enabling end-to-end service capabilities for cloud, network, and devices
- Taking advantage of changes in technology to optimize density and efficiency over time
- Supporting a forward-looking environmental sustainability and minimize climate change impact
Taking a holistic view of the IT infrastructure, TELUS and Skanska incorporated all elements of design, build, and operations into the strategy. Together, they developed a detailed capital spending analysis for the life of the project to ensure maximum flexibility as facility capacity increases. What emerged was a six-phase infrastructure plan to meet the company’s evolving load multi-year load growth profile (as shown in figure 1).
The Skanska/Inertech modular solution allows for rapid deployment of new infrastructure in units as small as 300 kilowatts, providing maximum flexibility to scale investment with utilization growth.
Phase 1 construction of the Rimouski facility has been completed. It comprises 60,000 square feet (sq ft), with planned expansion to 125,000 sq ft by the end of Phase 6. The facility is completely modular, with a design that connects individual server modules to a central access spline. As the facility expands to meet TELUS’s growing needs, the spline can be extended in each phase and the next module or set of modules is added to the facility. TELUS is also taking a performance-based procurement approach, enabled by Skanska’s vendor-neutral modules, preserving the greatest flexibility for future IT component configuration (see figure 2).
A TRULY MODULAR SOLUTION
Current modular data center solutions attempt to address the misalignment between mechanical and IT investment timelines by packaging data center components (for example in shipping container-sized units) that allow owners to expand operations more incrementally. However, these solutions still fall short, in some cases serving to simply “miniaturize” the same overcapacity issues.
What makes the cost-efficient TELUS deployment possible are the unique modules that closely couple the mechanical and IT elements. The modules use patent-pending cooling technology, eOPTI-TRAX from Skanska partner Inertech, integrated side-by-side with each IT unit. The fully modular system, called eComb, includes plug-and-play components that contain both customizable IT components and the hyper-efficient cooling units arranged along a centralized, expandable “backbone.”
The eComb system was designed to provide traditional aisle environments while giving clients the ability to rapidly scale up their density requirements. Using the Inertech technology, eComb’s modular configuration eliminates the traditional raised floor data center space, instead relying on a hot and cold aisle design, and requiring just a concrete slab to support the high-density modules. As shown in figure 3, no excess capital was expended to build structures around systems that do not require it (e.g., fluid coolers).
All of the eComb components are manufactured to TELUS specifications with a short 16-week lead time, allowing the company to respond quickly to changing business needs and anticipated customer demands. The modular pod configurations are manufactured offsite at Inertech’s facility (see figure 4).
The eComb design allowed TELUS to minimize investment in mechanical “overhead” that does not directly contribute value to the end services, and to maximize the proportion of facility footprint used for computing, network, and storage. “The final solution delivered a $60 million savings on day one,” reports Lloyd Switzer, senior vice president of Network Transformation at TELUS, “thanks to the Inertech technologies and Skanska design that allowed us to make minimal investment in mechanical infrastructure such as chiller and UPS components.”
Each eComb module can be customized to meet specific tier and density requirements. This on-demand build strategy allows TELUS to continue refining individual module configuration to take advantage of the best in evolving technologies and to design for any “density sweet spot” during all project phases. Each of the currently installed modules has an average density of 14 kW per cabinet, with up to 50 kW possible. The current design of the system includes 2.7 megawatts (MW) in Phase 1 and a planned 16 MW total, although the design of the entire system would allow for up to 32 megavolt-ampere (MVA), power capacity according to the Skanska product specifications (see figure 5).
BREAKTHROUGH ENERGY EFFICIENCY
The key to this efficient, modular deployment is the eOPTI-TRAX technology: individual cooling distribution units (CDUs) that are installed alongside the IT components allowing the cooling capacity of the TELUS data center to expand in increments that match 1:1 with the expansion of computing capacity. Housed within vendor neutral rack technology, the CDU has the same footprint as a computer room air conditioning (CRAC) unit, yet the units use a water-cooling technology and “require virtually no maintenance over a 20-year lifespan,” according to Inertech’s CEO, Earl Keisling. “Customers like TELUS can have a virtual ‘lights out’ facility, minimizing the changes, costs, and error rate typically associated with each hands-on maintenance touch point.”
In rigorous repeat testing by the University of Maryland, the eOPTI-TRAX technology demonstrated a mechanical PUE of just 0.01 compared to typical data center mechanical PUE readings of 1.4 to 1.6. As Dr. Reinhard Radermacher at the University of Maryland explains, “Mechanical PUE refers to the power effectiveness unit measurement of only the mechanical and cooling systems, distinct from the energy usage attributed to the total system including IT components.
“When mechanical and energy PUE are combined,” Radermacher continues, “our research results show the eOPTI-TRAX technology has a net PUE of 1.15 on average (not just a best day sample), versus industry standard PUEs of 1.8 to 2.0.”
Compared to traditional CRAC and air-cooled facilities, eOPTI-TRAX provides significant efficiency gains, using less energy to maintain a lower temperature. Gerry McDonnell, distinguished engineer at Energy Metrics, provides data (shown in figure 6) that demonstrates the lower energy requirements of the eOPTI-TRAX water-cooled technology vs. other cooling approaches.
McDonnell’s data reveals an additional performance attribute of the technology. The energy efficiency of the CDU modules is nearly the same at 30 percent capacity as it is at full (90 percent) capacity (figure 7). This enables the system to achieve a low PUE under lightly loaded IT conditions as well as under extremely dense loads.
“Traditional systems don’t deliver those attractive PUEs on day one. It can take months or years before density load reaches optimal PUE efficiency,” explains Keisling. “By contrast, the Skanska modules with eOPTI-TRAX deliver high energy efficiency early on in the lifespan of the data center.” This spread in efficiency at low loads is further magnified in warmer climates. “Moreover,” Keisling adds, “we can handle densities up to 50 kW per rack and also eliminate server fans, saving more than double the system’s energy use to achieve net-zero added energy.”
PERFORMANCE AND SUSTAINABILITY
The efficiency of the Rimouski facility is enhanced by design that takes advantage of free cooling for 99 percent of the year. In addition to the dramatic drop in energy usage (82 percent less than conventional data center installations), the modular systems in use at TELUS demonstrate a significant drop in water usage: a WUE of just 0.58, representing an average 80 percent reduction.
These energy and water saving attributes are the most notable of many environmentally responsible strategies in place at Rimouski. For example, using a direct rotary UPS system for power backup eliminates the need for batteries, thus eliminating the ecological impact from battery disposal. TELUS is committed to implementing sustainability in the design, construction and operation of the facility. To support this goal, the project has been registered with the Canadian Green Building Council (CaGBC) with the goal of LEED Gold.
The decreased ecological footprint of the facility does not, however, correspond to any decrease in the performance or reliability of its systems. In fact, the Uptime Institute has certified the design of the Rimouski facility as a Tier III data center.
THE NEXT GENERATION DATA CENTER
The solution at TELUS provides truly modular, just-in-time deployment. The facility can grow incrementally with efficient use of capital at each stage, adapting as technology evolves. TCO is reduced through lower initial investment and the ability to defer capital expenditures, lower operating costs from reduced energy and water usage, and lower ongoing cost of maintenance with fewer large-scale and hands-on mechanical systems.
Dr. Greg McRae, Bayer professor of chemical engineering at MIT explains, “The Skanska/Inertech solution goes beyond the obvious. The genius behind their approach is that they came from a system (holistic) perspective rather than a component level approach. This allowed them to solve many interrelated issues simultaneously and dramatically change the performance expectations of the data center industry.”
The TELUS facility—the first of two planned in the next two years—exemplifies a data center strategy that looks at end-to-end capabilities across building infrastructure, mechanical components, IT systems, networks, and devices, supporting the TELUS vision of world class data centers and national computing grid design. n