Colorado State University is a leading university with world-class research in infectious disease, atmospheric science, veterinary medicine, clean energy technologies, and environmental science. The university has a long history of advanced technology, and its data center also has a storied history: the original CSU data center dates back to the 1960s with IBM supercomputers that were protected 24/7/365 by an armed guard.
The water-cooled system was replaced with airside-cooled computers in the 1970s, and that system remained in place in large part well into the new millennium. However, as crucial campus infrastructure elements became dependent on the data center, school officials needed to expand and upgrade the system.
“The data center, serves as a hub for all CSU campuses, and it is definitely a mission-critical facility because of the university infrastructure it serves,” explained Gene Ellis, mechanical engineer for CSU Campus Facilities Management.
The roughly 3,600-square-foot (sq ft) data center is located in the unassuming Engineering Building, which was built in the 1950s. Specifically, the facility sits in the basement of the Mathematics Wing, and it was in dire need of an upgrade.
“Everything, including the floor, was worn out. The room was at end of its serviceable life. We had kept the HVAC system and the floor limping along, but we also needed more capacity and it was time,” said Ellis.
MAGAZINE ARTICLE PROVIDES KEY TIP
However, since the university couldn’t afford to shut the data center down, Ellis and his staff brainstormed and researched underfloor piping systems that would be flameless, fumeless, and release zero particulates into the air. He had coincidentally seen an article in an industry trade magazine about a German manufactured polypropylene-random (PP-R) pipe system called Aquatherm and considered that as an option.
Ellis contacted Aquatherm’s local manufacturer’s representative, Priest-Zimmerman, Inc. (Denver), received some information including a detailed catalog, and talked the PP-R system over with Chuck Fox, CSU’s Remodels and Construction Services Plumbing supervisor. Since they have considerable experience with high-density polyethylene (HDPE), which, like Aquatherm, is connected via heat-fusion welding, both were relatively comfortable with the concept.
“Also, the product line is pretty famous in Europe, and it is high quality. But while it looked good on paper, it wasn’t until Chuck started working with it and gave it his seal of approval that I was fully onboard with it,” recalled Ellis.
Fox worked with Priest-Zimmerman, a specification-driven PHVAC, waterworks, and industrial manufacturer's representative serving six western states and the local office of Ferguson Enterprises, the local Aquatherm distributor, to sort out the logistics, set up training, and arrange procurement of the necessary fusion welding equipment.
NO SHUT DOWNS ALLOWED
“The number one reason we opted to use PP-R on this project was the application inside the room. We couldn’t shut down the system and needed a product that could go under the floor, be reliable, and that didn’t require soldering or welding,” said Ellis. He added that a second benefit Aquatherm offered was that PP-R’s natural insulation value delivered considerable space savings.
“It didn’t require a lot of insulation. We put a vapor wrap on the pipe, just to be conservative. But the floor is not very deep so saving those two to four inches of space was critical,” he said. Additionally, Aquatherm’s warranty helped sell both Ellis and Fox on the system. When installed by Aquatherm-trained and certified technicians, the pipe and fittings carry a 10-year, multimillion-dollar warranty covering product liability, personal injury, and property damage.
Early in 2010, eight CSU in-house maintenance staff plumbers and assistants were trained onsite by Aquatherm, Ferguson, and Priest-Zimmerman. According to Fox, the training was effective: the staff quickly and easily mastered the process.
They began tearing out the existing steel piping and flooring and installed Aquatherm Climatherm, which is designed specifically for HVAC applications, for all the condensate and chilled water supply lines to the cooling units. Eight-inch Climatherm was used for the mains, and butterfly branches with stubs were installed in several areas for future expansion.
Aquatherm was installed under the entirety of the computer room floor. In the “air-cooled” section of the room, supply and re-return piping connects to seven Liebert chilled-water computer room air conditioning (CRAC) units. A distribution system was also installed under the future high-density computing area, which will serve in-row cooling units. Aquatherm was also used in the equipment yard outside, connecting the room to two Carrier Aquasnap chillers. The chilled water mains were up to 8-in. in size, with branch distribution lines down to 2-in.
TRICKS TO BE LEARNED
Fox explained that while the installers were quickly comfortable with Aquatherm’s heat-fusion connection process, there was a learning curve.
“Once you learn all the tricks, it goes even easier. For instance, when you are using the fabrication machines [welding jigs], you can hand push the 4 in. [pipe together], but we didn’t get a fabrication machine until the end of the job, and that is key for connecting the smaller pipe.”
“You do have to be creative with the welding machines sometimes, like taking it off of the stand and things like that, which shows that there is some craft to working with fusion welding and especially with butt fusion,” Fox added. Butt fusion is the welding process used for Aquatherm pipe sizes over 4-in. diameter, and it turns the pipe and fitting into a single material—creating long-lasting, monolithic connections that eliminate potential leak paths. While CSU purchased the fusion welding equipment necessary for 4-in. and smaller pipe, it rented the butt-fusion equipment from Ferguson.
Even allowing for the learning curve, Ellis and Fox were sold on using Aquatherm. “I love the product. It does not leak if you put it together right, it just will not leak,” Fox said.
NO CRANE NECESSARY
Aquatherm is also considerably lighter than steel. “We would have needed a crane for 8-in. steel, and the crane would have been very difficult to maneuver into some of those spaces, and we’d have had to open up the roof to get it into the building,” Fox said.
“When we renovated the computer room there were no shutdowns. We moved in the equipment, and working in stages, Chuck prefabricated and put in 30- and 40-foot sections as we went. The facility was operational the whole time,” explained Ellis. Someone on the project team likened the difficulty of the project to “completely rebuilding a 747 during a trans-Atlantic flight,” which Ellis called an apt assessment.
The system runs a 30 percent glycol mixture (plus water treatment chemicals) at between 45°F to 55°F fluid temperature and maintains a 75°F airside environment in the facility. While Climatherm’s inherent R-1 insulation value was helpful on this job, a vapor wrap was applied in the plenum spaces to ensure the supply piping would not sweat. Left uncovered in the plenum for a short while, the PP-R pipe “does condense a bit on the bottom side, but it’s not condensing like copper or steel would,” explained Fox. In non-plenum under-floor spaces, the piping was left uncovered.
TYING INTO CAMPUS COOLING
While the pipe system was a large part of the retrofit, the new HVAC system was also a huge consideration. The new system features 400 tons of cooling capacity. The primary source of cooling is the campus chilled water utility via a flat plate heat exchanger.
The utility plants operate at an average of less than 0.7 kilowatt (kW)/ton during the cooling season, with free cooling during winter months at around 0.3 kW/ton. Two Carrier 30 Series air-cooled chillers provide on-site back-up cooling, and seven Liebert chilled water CRAC units with VFD provide 140 tons of cooling to the airside portion of the room.
The remaining 260 tons is dedicated to the high-density area, which will begin to see load soon. One cooling system failure occurred during start-up, and since then, CSU has had 18 months of uninterrupted operation.
With the new HVAC system, a new floor, and new piping, the CSU data center now has considerable— roughly 10 times—more capacity, but just as importantly, it is well positioned for sustained growth over the next decade, including the planned high-density area. The system delivers 400 tons of total cooling capacity, with a power density of roughly 200 W/sq ft in the air-cooled area and roughly 1,000 W/sq ft in the high density area. It also has a 1.4 MW build-out computing capacity.
“The system is running great with zero leaks, and it is more efficient even though we have more capacity, which is ideal,” Ellis said. The data center installation went so well that campus officials have already deployed Aquatherm pipe systems on the cooling and heating water at a nearby early childhood development building, a chilled water main at another building, and on other projects.