ipsumWhat will an emergency backup power system look like and be capable of 30 years from now? While no one knows for certain, it may look a lot like the system at Rex Hospital in North Carolina. The hospital’s recently upgraded backup power system was designed to ensure the seamless delivery of both normal and emergency power to all loads… today and for the next 30 years.

Rated one of the top 100 U.S. hospitals by Becker’s Hospital Review in 2012, the 660-bed flagship of the not-for-profit Rex Healthcare treats nearly 34,000 inpatients every year. The staff includes more than 1,100 physicians and 1,700 nurses, who also provide services at 13 affiliated clinics and other facilities throughout Raleigh and the surrounding Wake County.

The new comprehensive power system provides the hospital with more reliability, more redundancy, and more flexibility. Taking into account anticipated growth, the system has enough emergency capacity (8.25 megawatt [MW]) for a seven-story heart center currently on the drawing board as well as for a future cancer center addition. At present, Rex has what is called an N+1 arrangement — it could lose or take out of service any one generator and have adequate capacity remaining. Three 1.25 MW generators have been replaced with two new Caterpillar 3 MW generators, and an existing Caterpillar 2.25 MW generator has been retained. There is room to add more switchgear and circuit breakers. An automatic transfer switch and an uninterruptible power system have been added to protect the hospital’s data center.

There are two 40,000 gallon (gal) underground fuel tanks (totaling 80,000 gal), and the system maintains fuel in each generator’s emergency 150-gal “day tank” at all times. With all tanks full, the hospital could meet its own peak demand (about 5,200 kW) for almost six days. However, since that peak is reached only for short periods on the warmest of summer days and peak demand in winter months ranges from about 3,800 to 4,500 kW, the hospital could probably operate under its own power for more than nine days, depending on the time of year. (Fuel capacity for the previous system was 60,000 gal— one-third less.)

BOTH SIDES HOT

The hospital’s new substation consists of four utility-owned, pad-mounted 2,500 kVa paralleled transformers that serve the hospital at 4,160 Volts. This provides a total utility capacity of 10,000 kVa (10 MVa). The hospital assumes ownership at the transformer secondaries, which are connected to the hospital’s outdoor switchgear. When an outage occurs, the switchgear automatically disconnects from the utility by opening four 1,200 Amp circuit breakers, and simultaneously sends a signal to start the generators.

Based on its present peak load (about 5,200 kW), the hospital can continue to operate without interruption should there be a loss of one transformer, since there would be 7,500 kVa of remaining transformer capacity. This design follows the same N+1 policy as that for generator capacity. However, should there be a loss of two or more utility transformers, the hospital’s generators will start and parallel while the outdoor switchgear disconnects from the utility system. The hospital will then remain on the generator source until the utility source is restored, at which time the generators will parallel with the recovered source and, once the utility voltage has stabilized, reconnect the hospital load to it without interruption.

The utility’s transformer primaries are served by two 25 kV utility feeders from separate distribution systems. Though both are energized, the hospital can draw from only one at a time. Should the active feeder be lost, the utility can manually switch the hospital to the backup 25 kV source at the hospital’s substation. For faster switching, the hospital’s electrical staff has lobbied the utility to install remote-controlled source-transfer motor operators on these feeder switches.

The hospital’s system employs a “both sides hot” strategy that feeds utility power to both sides (normal and emergency) of every automatic transfer switch. If a breaker trips and any downstream feed is lost, power is immediately transferred to the alternate side of the switch. This arrangement allows the automatic transfer switch to quickly transfer, within cycles, to the alternate source without starting the generators. This strategy also lets the power plant staff test transfer switches without starting the generators.

Another new feature is a comprehensive supervisory control and data acquisition (SCADA) system. Technicians can now fully monitor and control the entire power system from the control room at the central energy plant. A new simulator that uses the same control logic software as the switchgear’s programmable logic controllers facilitates training. Another special feature of the system allows the scheduling of tests and automatically generates regular reports for the Joint Commission on the Accreditation of Healthcare Organizations (JCAHO).

In the event of an internal failure, the SCADA system can rapidly and automatically configure a path to bypass the failure and re-energize the system without starting the generators.

“It is unlike any other system I’ve ever seen,” says Mike Doggett, territory manager for Robert W. Chapman & Co., sales representatives for Russelectric Inc., the supplier of design assistance, power control switchgear, transfer switches, the SCADA system, and implementation support. Doggett was Russelectric’s chief contact for the hospital’s upgrade team.

The new system, which came online in the spring of 2013, was designed by a dedicated team who knew what they wanted and fought for it with the hospital’s cost-conscious management, engineering firm, and building contractor. Mike Raynor, facility services director; Tony Powell, senior utilities manager; John Stanford, facility services coordinator; and Travis Jackson, P.E., consulting engineer, convinced management that the extra cost of a reliable system with superior equipment would be worth it in the long run. Before the dust had settled, Raynor had added director of construction services to his job title — a testament to his persuasive prowess.

DON’T BLINK

The previous backup power system at Rex Hospital was the closed-transition type — no one wanted a “blink” (a brief interruption of service) when the feed had to be switched from utility power to generator power when both sources were available. Yet the engineering firm the hospital first hired for the upgrade in 2011 recommended a modified open-transition system that would have meant numerous outages throughout the hospital, not only during construction but also thereafter. Furthermore, this proposed cut-rate system relied on generators and fuel tanks on flatbed trucks to provide additional capacity during construction or when adequate power could not be delivered to the hospital load. In addition, it would have provided no paralleling, load curtailment, peak shaving, or redundant energized feeders. Raynor and his team argued against the proposal and in favor of a fail-safe, forward-looking system they had envisioned.

Raynor, who remembers “a lot of jumping up and down at meetings, a lot of talk, a lot of phone calls, and a lot of bumps to get over,” is candid about the plan he opposed: “People would have noticed a difference when the power went out — BANG! — or came back on — BANG! — like when there is an outage at your house. There is just no need for a hospital to go through that in this day and age.”

“It would have taken us back many years,” agrees Raynor’s longtime engineering consultant Jackson. “We like closed transition, and we already had the capability to do paralleling and load curtailment. We certainly didn’t want to give those up.”

Jackson continues, “We also knew that Russelectric equipment has welded construction and is sturdy, durable, and extremely reliable. The engineers in the other firm were not familiar with our configuration or with the equipment required to meet the hospital’s standards. We knew what we had and we knew what was being proposed, so we saw the limitations. We also knew where we wanted to wind up in the future. As a team, we persisted in questioning the proposal. Eventually, even with budget constraints, the administration came around, and we were able to make it all work out.”

The men presented their case to all stakeholders, including the hospital’s executives and medical staff and a constellation of officials at various regulatory agencies.

“Finally,” Jackson recalls, “support for our point of view began to spread, and eventually everyone got on track. I think what swung the decision in our favor was that the administration realized the hospital’s needs and realized what was at stake.”

The Jackson/Raynor/Powell/Russelectric design that was implemented meant replacing the utility substation and making it more reliable as well as relocating the switches and switchgear from cramped quarters in the main hospital building to a newly constructed central energy plant. The entire project and system switchover was completed with only a single, planned 10-second outage.

UP AND RUNNING

Now that the system is up and running, Raynor’s crew can do peak shaving, supplying some of the hospital’s power while the utility is supplying the rest, thereby saving on utility demand charges. Although the system does not contribute power to the grid, through its load curtailment capabilities it can quickly respond when the hospital is asked by the utility to reduce demand on the grid by a specified amount. The resulting contractual rebates lower the hospital’s overall energy costs.

“Load curtailment is a huge relief to the utility if, for example, they’ve got a reactor down or they are experiencing an unusually high demand for power [for air conditioning] during a heat wave,” says Raynor. “For whatever reason, they can ask us under the contract we have with them to generate our own power for a specified amount of time. Usually they give us about 30 minutes’ notice, and we run the generators no more than 8 hours at a time. One year, we did this five or six times, but in an average year it happens only once or twice.”

Very important to the hospital’s power control system upgrade was the installation of the new state-of-the-art SCADA system that includes software and screen displays customized for the hospital’s needs. It provides interactive monitoring, real-time and historical trending, distributed networking, alarm management, and comprehensive reports around the clock for every detail of the whole power system, not just for the backup components.

The SCADA operator uses full-color “point and click” computer-screen displays at the system console, where he or she can access and change the system’s PLC setpoints, display any of the analog or digital readouts on switchgear front panels, run a system test, or view the alarm history. A dynamic one-line diagram display uses color to indicate the status of the system, including the positions of all power switching devices. Operating parameters are displayed and updated in real time; flashing lights on the switchgear annunciator panel also flash on the SCADA screen. Event logging, alarm locking, and help screens are standard.

GEEKS AND TWEAKS

The SCADA system includes a simulator, which shows trainees what to expect when they lose a feed, open or close a breaker, add or remove load, etc. The crew also uses the simulator during startup and for troubleshooting, system improvements, preview testing, and tours.

Jackson, who has spent many hours learning the simulator, sometimes late into the night, praises it for its “many, many great features,” adding, “It’s almost like a really nice video game; you can look at any situation that might occur, including any kind of outage. It’s like real life, but without the consequences. It’s totally safe.”

“It’s a great system to begin with, but Travis has improved it many times,” says Raynor. “Beyond the basics of the simulator, there are nuances built into it that I don’t think anybody but Travis would have found.”

Yet another selling point is full manual backup. If the touchscreen on the control panel fails, operating personnel can manually open and close breakers, synchronize and parallel the generators onto the bus, and add or shed load. Other manufacturers’ systems do not provide for full manual operation.

Bringing a new power system online is a major undertaking that entails months of minor decisions and follow-up service. Doggett and Nate Mattson of Russelectric continue to work with Jackson and Stanford to tweak the system program (adjusting timers, time delays, setpoints, etc.), but only after consulting the others at regular meetings.

“We enjoy working together,” says Raynor. “Mike [Doggett] has been a huge help and a steadfast supporter of this project from the beginning. He provided much expertise to make sure we got the right gear and that it would all work together as a coordinated system. He also was a terrific liaison between Rex and Russelectric concerning pricing, dates, shipping, etc., which helped us coordinate the project. And he continues to help us with the fine-tuning.”

AS GOOD AS IT GETS

Looking back, Raynor has the luxury of summing up a job well done.

“The hospital needed a new and modern system that built on what we had already,” he begins. “The hospital was just not getting that from the engineers management hired, so the facilities staff had to take on more responsibility in ironing out these issues. Working closely with Russelectric, we came up with a very sophisticated system, and we’re at a point now where the system is functioning as we expected — all the hospital’s electrical needs are covered.”

Raynor continues, “If you envision your destination, you need a company like Russelectric to keep up with you and help you over the speed bumps. It’s a hospital, after all — everything has to work right. This system is as good as it gets.”