Lawrence Livermore National Laboratory (LLNL), a pre-eminent scientific resource for U.S. defense, science, and industry, is the institution that applies advanced inter-disciplinary science and technologies to ensure that the nation’s nuclear weapons remain safe, secure, and reliable. This is known as the Stockpile Stewardship Program, which entails authentication of the U.S. nuclear weapons stockpile (a responsibility of LLNL as part of the National Nuclear Security Commission) in support of the comprehensive nuclear test ban treaty.
Scientists and engineers at LLNL use supercomputers to certify weapon performance via simulation rather than actual testing. Because these simulations involve many trillions of computations that perform at ultra-high speeds, a new mission was undertaken in the mid-1990s to upgrade these supercomputers to “monster” computer systems that perform at tera-scale levels (trillions of calculations per second). The mission resulted in LLNL’s recently operational tera-scale Simulation Facility (TSF).
The TSF facility consists of two 24,000-square foot computer rooms enveloping tens of thousands of processors in hundreds of cabinets.
Figure 1. Digital relays backup the TSF's send/receive operations.
The TSF facility has a capacity of 25 megawatts (MW) to support the computers, and a robust mechanical system includes a large air handling system with cooling towers, fire protection, and alarm systems.
Bailey explains that among the operational priorities of the TSF are flexibility, scalability, and reliability. The latter would be greatly reliant on power system protection and the ability to switch power sources if necessary. Power protection and source transfer, as well as the communications technologies supporting them, would have to be advanced, simple to operate, and above all-reliable.
“This was one of the first projects I’ve been involved with where the electrical system was one of the first design considerations,” Bailey says. “In many instances, the requirements for the electric power distribution are determined at the end, but it was critical for this project. We had to make certain that the availability of power was a priority.”
Typical concerns were that an upstream glitch might cause a fault and that there would not be a safe way to shut down in the event that the cooling system was lost at the 24/7 facility. “We were very concerned that if we have a glitch, how do we safely shut down the chillers. The computers will usually ride through a glitch, but a chiller takes 20 minutes to restart, and the computational calculations are at risk. So then there is redundancy built into the mechanical system as well as the electrical,” says Bailey. To further ensure the quality of computations, mechanical loads were separated from computer loads.
TSF’s large mechanical infrastructure includes thirty 80,000-cubic foot per minute air handlers, a 10-MW cooling tower, four 1200-ton chillers, and one 675-ton chiller. The electrical infrastructure includes a 25-MW switching station, a 3-mile duct bank system, and elaborate fire alarm and communications systems.
To further support the overall power system, Bailey wanted an automatic transfer scheme that would seamlessly switch sources between two 13.8-kilovolt primary sources should there be a loss of power to an incoming feeder or any under voltage condition.
“We didn’t have the budget to provide uninterruptible power,” Bailey says, “and with a total projected load of 23 MW, there would be no way for us to do that.” The Schweitzer Engineering Laboratories, Inc. (SEL) solution met the budgetary and operational- reliability requirements.
TSF power system monitoring, protection, communications, and source transfer requirements, outlined in the specifications, led to the installation of multiple SEL-351S microprocessor-based relays for state-of-the-art protection and control technology that assures the mandated flexibility, scalability, and reliability.
Bailey says, “We had used a lot of individual SEL relays at various locations, and they had a good track record. But this was the first integrated project where all the relays are SEL. They offered the best combination of product and technology for what we wanted to accomplish. When it came to relay-to-relay digital communications [SEL MIRRORED BITS communications], we were impressed by the speed of operations.”
Figure 2. The specification of SEL-351S multifunctional relays involved an array of advanced capabilities and features.
Automatic power source transfer is facilitated by MIRRORED BITS communications between relays that are located on the main breakers and act to close the tiebreaker with voltage and synchronism-check supervision.
“Schweitzer systems support is also important to us,” Bailey says. The SEL Systems and Services Division in Pullman, WA, was contracted to implement the initial settings for the relays. “I consider the educational support important to program management,” adds Bailey. “Robin Jenkins, an SEL integration engineer who specializes in SCADA-type applications, came to the site and provided training on the communications processors.” In addition to onsite training, several of the LLNL engineers and technicians attended SEL University courses for additional training.
