Supercomputer keeps its cool with piping
The Lawrence Livermore National Laboratory (LLNL), in Livermore, CA is at the forefront of innovation, strengthening U.S. security through development and application of world-class science and technology. Recently, LLNL’s new Vulcan supercomputer has been grabbing headlines. The IBM Blue Gene/Q system is being used for collaborative projects to develop new technologies in applied energy, green energy, manufacturing, data management, and other fields.
Ranked no. 8 on the list of the world’s fastest high-performance computing (HPC) systems, Vulcan is considered one of the world’s most powerful computing resources available for collaborative projects.
“The Vulcan machine is designed for an open environment to encourage collaboration and innovation, and greater interaction between academia, industry, and national labs. The idea is to make high-end supercomputing available to industry for technological and business innovation, giving those companies a competitive edge in the global market,” said Anna Maria Bailey, Livermore computing program facility engineer.
A recent example of this kind of collaboration is the HPC4energy Incubator program which provided supercomputing resources and expertise to six selected energy companies developing new technologies and management capabilities. These included projects to improve the efficiency of auto and jet engines and ways of better managing the electric grid. “So it’s a new area for us wanting to vector our capability for wider exposure and uses,” said Bailey.
A 5 petaflop/s (quadrillion floating operations per second) machine, Vulcan is a smaller version of Sequoia, the 20 petaflop/s system that was ranked the world’s fastest supercomputer in June 2012. The Vulcan supercomputer at LLNL is now available for collaborative work with industry and research universities to advance science and accelerate technological innovation through the High Performance Computing Innovation Center.
At one point in time during its shakeout period, through collaboration with the Rensselaer Polytechnic Institute (RPI), Vulcan was combined with the larger Sequoia system. The combination produced breakthrough computations, notably setting a world speed record of 504 billion events per second for a discrete event simulation.
Both supercomputers reside in the same building, but Sequoia’s 96 racks are a totally separate system and the two are not mechanically or electronically connected. Vulcan, with its 24 racks, can remain operational if Sequoia has to be taken offline. Although Sequoia and Vulcan are not currently integrated, they have some common threads. Based on the success of the design and supporting infrastructure of Sequoia, LLNL once again turned to Aquatherm polypropylene-random (PP-R) as the pipe system for the Vulcan supercomputer cooling system.
Keeping It Cool
The Vulcan installation mimicked the original Sequoia design, including the cooling system and the piping design. Akima Construction Services (ACS) applied similar processes using up to 12-in. Aquatherm Blue Pipe® (formerly Climatherm) and because the team was under a three-month turnaround time constraint, Aquatherm’s reliability and installation time and labor savings were key to the project.
Because of the ease of application and experience with Sequoia, the decision to use Aquatherm made perfect sense, and LLNL already had some of the heat fusion equipment prior to the project start date, forecasting future PP-R piping work. “We purchased a heat fusion machine because we were planning on using the polypropylene pipe in other facilities,” said Bailey.
Also, similar to the Sequoia installation, the computer manufacturer had established strict water treatment and water quality requirements for the cooling system, and Aquatherm’s chemical inertness played a key role in meeting that requirement.
“We had already seen polypropylene being used on things like the Sequoia project and nuclear plants so we were familiar with it. The cost savings and the type of material used were the big reasons for using Aquatherm on this particular job, the same as it was for Sequoia,” said Bailey.
Due to the chemical purity of PP-R, the system was not exposed to any foreign oil or material from the pipe. If steel had been used, the cooling water would not have met the specification due to oil residue. Also, had PVC been used, there was concern that potential glue overflow could have clogged the computer coil.
“By using polypropylene, we didn’t introduce metals into the system, and it also let us meet our ISO 14001 goals since there are a lot less chemicals in the system. It’s easier to maintain and there isn’t waste that comes from treating the system with chemicals,” said Bailey.
The valves, fittings, and parts of the heat exchanger are the only metal components in the system. Two Bell & Gossett plate and frame heat exchangers and two 150-hp Bell & Gossett pumps form the backbone of the mechanical system, with an Automated Logic digital control system running it. The heat exchangers are sized for 75% of the load at 30 million Btuh each.
Aquatherm’s PP-R piping systems are connected via heat fusion, which is a big factor in its reliability. Heat fusion bonds both sides of a joint into a single, homogenous material without the use of chemicals or mechanical connections, eliminating systematic weaknesses and fail points in the pipe.
“We liked the heat fusion connections because there’s no welding. You can make the fusion bonds in the same space as the computers,” said Bailey.
LLNL could do the mechanical work in a controlled environment — in the same space — instead of having to do the welding outside where you have to worry varying temperatures and humidity, and precipitation would have been a concern. The installation team built a “doghouse” or dirty room where the pipe could be connected in the same space without impacting the temperature in the computer room, which is crucial. “It was critical to keep the temperature in the facility constant,” explained Bailey.
Installing Aquatherm in lieu of copper and steel provided an estimated $1.5 million in savings on the project, after LLNL had saved roughly $2 million using Aquatherm on the Sequoia project. Additionally, using the PP-R pipe system allowed the designers and installers to comply with the specifications of plastic type piping and deliver a good product to the lab in addition to helping LLNL achieve Environmental Management Systems (EMS) goals and standards.
Aquatherm piping systems are also expected to indirectly contribute to LLNL’s LEED™ recertification. Livermore was LEED gold in 2009, and the decision to use Aquatherm pipe is also expected to assist in the facility’s 2015 LEED point re-certification.