By utilizing the same design and construction techniques as the primary market, while taking advantage of the lower cost of land and power in the secondary market, developer Data Realty, South Bend, IN; MEP firm, Environmental Systems Design, Inc., Chicago; and Turner Construction, Chicago, have joined forces to create a data center in Northwest Indiana.
Located in South Bend, the new 47,000-square-foot (sq ft) co-location data center, including 21,000 sq ft. of raised floor space and an additional 5,000 sq ft of office space, opened its doors November 1st, 2012. Just 90 miles from Chicago, on the grounds of a former industrial campus whose power infrastructure has been impeccably maintained over the years, including functioning transcontinental fibers and two substation feeds, the Data Realty campus is the newest high-performance technology park in the Midwest.
“The symbolism of Data Realty’s physical location is remarkable—it sits right on acreage that used to be a major auto industry production center,” said Pete Buttigieg, mayor of South Bend. “The cleaned-up Studebaker factory campus, which we now call Ignition Park, is coming back to life. It’s a great meeting point of past and future: the site is once again powered by an innovative business, but now it’s from an industry that didn’t even exist a few years ago.”
In addition to the lower cost of land and power, the secondary market can offer enhanced tax incentives and improved disaster recovery over big cities because its power comes from a separate, local grid. The secondary market in South Bend offers these and something more—the cutting-edge scholarship and market leadership that only academics can provide.
The University of Notre Dame is not only the first tenant in the building, but also a partner of Data Realty, which sets it apart from other data center providers.
“We are building a team of data scientists to take the data colocated in our facility and develop complex algorithms that help companies interpret and develop solutions to business problems as they’ve never done before,” said Rich Carlton, president and COO of Data Realty. “Our access to talent both from a faculty level and a student level at nearby Notre Dame has us excited about our opportunity to fill a niche in the marketplace.”
With a power usage effectiveness (PUE) of 1.28, Data Realty is one of the most efficient mission-critical facilities ever designed by ESD’s veteran team, featuring water-based cooling and the newest in chiller equipment technology; efficient interior rack design; uniquely reliable and redundant power to support the highest performance computing; and a non-conventional, wireless, state-of-the-art monitoring system.
COOLING
Efficient data center design begins with efficient mechanical systems. Data Realty utilizes a free-cooling refrigerant economizer cycle, which takes advantage of the cooler Northwest Indiana climate. For South Bend, IN, this amounts to almost 4,100 hours of free cooling, or 47 percent of the year, while maintaining space temperatures well within the ASHRAE TC 9.9 recommended window. Utilizing evaporative-cooled condensers also reduces the electrical demand on the system, thereby reducing the load that is backed up by standby generators.
Multiple chiller technologies were evaluated to determine various levels of mechanical cooling efficiency and their ability to facilitate free cooling. ESD designed an innovative modular refrigeration system that would produce and circulate chilled water throughout the data center, relying on an industrial refrigeration concept for heat rejection and economizer operation. As the colo data center will begin with fewer racks and build up to full capacity over time, it was important to specify a refrigeration system that can begin with a smaller output and grow with the facility. The refrigeration system design provides additional benefits, including remote evaporative condensers that eliminate the need for condenser water pumps and/or heat exchangers for free cooling as well as Turbocor compressors using magnetic bearings to reduce friction loss.
The partial PUE of the SMARDT chiller coupled with its integrated refrigerant economizer is as low as any in the industry. While the system’s first cost is about 20 percent more than a conventional commercial chilled-water system, the payback, including efficiency and operations, is typically within a few years.
POWER
As the centerpiece of any data center’s MEP infrastructure, electric power must be both available in large quantities and reliable. Data Realty was designed with this in mind to meet a baseline of 150-watts(W)/sq ft, per typical 5,000 sq ft colo module, with the additional capability of fulfilling localized high-performance computing (HPC) needs reaching upwards of 300 to 400 W/sq ft
The power behind this comes from two separate substations, each supported by three feeds, each from its own independent generating station. Both substations are on a single grid, segregated from the Chicago marketplace. These separate feeds bring fully redundant power to the site, with capacity for future expansion of the facility.
“For the data business, there are big advantages associated with our inexpensive power, high fiber bandwidth and even our cold weather,” said Mayor Buttigieg. “The Internet really is a physical thing—and it runs right through South Bend, with fiber conduit passing along the old railway right-of-ways. Even the ‘cloud’ needs to be on the ground somewhere, and we’re glad to see it grow right here in South Bend.”
Redundancy and concurrent maintainability is provided through a catcher distribution system consisting of a dedicated primary switchgear and generator for each data center module, and a reserve “catcher” switch gear and generator for redundancy. Automatic transfer switches allow the redundant 500-kVA UPS modules and computer room air handling units to be supported from any one of four power sources: Utility 1, Utility 2, the dedicated generator, or the “catcher” generator. The data center currently contains two main switch gear and 1,500-kW generators, expandable to four.
Additional efficiency comes from the data center’s physical site design and floor plan. Building a greenfield data center from the ground up allowed the electrical rooms to be strategically placed along each side of the data center suites so the power could run directly into the data center, reaching its destination more efficiently and therefore reducing costs dramatically. Owning its own transformers, Data Realty is able to buy power at a higher voltage and step it down, minimizing cost per kilowatt hour as well.
Together, the new efficient SMARDT chiller and the UPS system drive the building’s efficient PUE of 1.28, while individual colo tenants can attain even higher PUE efficiencies based upon how they build or strip their servers.
MONITORING SYSTEM
Data Realty has a two-tired monitoring system. Tridium Monitoring System (www.tridium.com) is the staple BAS used first and foremost to control and monitor the facility’s infrastructure and its systems' interconnectivity. This monitoring system was chosen for its strong industry name and ability to promote operational efficiency, providing Data Realty with the ability to react quickly to any adjustments that need to be made in efficiency or quality of service.
Secondly, a Packet Power wireless power monitoring system (www.packetpower.com) is employed at Data Realty to monitor and database all of the power quality and activity in the data center at any given time, down to the individual circuit level for each customer. Typically a costly venture for data centers, power monitoring has previously required running multiple connections to each rack and a completely separate networking infrastructure for support. However, Packet Power’s wireless solution was specified for its ability to allow Data Realty to do the power monitoring without all the infrastructure.
“We looked at a number of solutions and this is a new company with new ideas, doing it in a less conventional way than everyone else. Their solutions solved a lot of problems,” said Tom Panozzo, chief technology officer for Data Realty.
Data Realty monitors every rack for every customer with the wireless system, pulling information from circuits every 30 seconds regarding rack temperature, humidity, differential pressures, potential power overloading, and more, notifying the system administrator of any issues immediately. System administrators use the power monitoring to generate the number of kilowatt hours that each client is using, carbon emissions, and redundancy status, while providing a portal for tenants to plug into and access their own reports historically as well.
“The system allows us to have an unprecedented amount of visibility into power quality for our clients, and we can provide them with information that will allow them to react to potential issues before they become problems,” said Panozzo.
INTERIOR DESIGN
Using a 36-in. plenum raised floor produced in South Bend by Swedish floor manufacturer Bergvik Flooring (www.bergvik.com), Data Realty’s interior design features a hot-aisle/cold-aisle configuration with hot-aisle containment using a panel system that runs from the floor to the ceiling in the hot aisle to create a “hot isle chimney.”
The warmer the air is when delivered to the racks, the lower the work and therefore, the less energy the chiller system consumes. Air intake in Data Realty’s cold aisle is maintained by controlling the constant discharge air temperature off the computer room air handing (CRAH) units at 60°F. This results in a constant temperature at the middle of the rack between 70°F and 75°F, rendering the hot aisle return through the plenum as hot as 90°F. These system set points have reduced operational costs, resulted in a better PUE and will help ensure optimal equipment efficiency for the facility’s MEP systems.
CONCLUSION
By employing water-based cooling, the newest chiller equipment, efficient interior design, reliable and redundant power, and a wireless monitoring system, Data Realty has created a secondary market data center that is first rate in both design and efficiency.
“Data Realty brings significant investment to our city at a time when we are pivoting into a new economic future,” said Mayor Buttigieg. “We’re excited, not only about the new jobs created here, but about the step forward into new economic territory. As the first tenant in Ignition Park, Data Realty is not only a technological innovator but an economic pioneer for our community.”
