Averting A Medical Center Meltdown
The story of how one medical center used a management system to simplify its testing and reporting capability.
Master electrician Mike Wiruth at the Bryan Medical Center in Lincoln, NE, saw the “handwriting on the wall”... and on spreadsheets, and he wanted to do something about it.
He saw that the center's monitoring and control requirements for its power distribution system had outgrown its current capabilities. Too much of the data gathered during monthly tests of the center's emergency system power for Joint Commission and local fire marshal inspections had to be entered on spreadsheets and collected from multiple sources. The process took too much time and was an opportunity for human error to creep into results.
He knew Joint Commission inspectors and the local fire marshal prefer to review automated reports. He also wanted to simplify testing procedures and cut the time required to produce the test reports. Bottom line, he recognized that standard monitoring and control software would no longer “cut it” for the center's very decentralized system, which isn't paralleled. He needed a full-fledged, critical power management system (CPMS).
By any measure it was a tall order. He had to accomplish it, though, to continue supporting the medical center's commitment to patient safety and helping maintain the Joint Commission Gold Seal of Approval. The gold seal sets the highest standards for health care quality and safety.
"Patient safety is why we invested in an emergency power system having layers of redundancy that help ensure reliable operation during power outages and equipment failure," Wiruth said.
Bryan Medical Center's East Campus relies on its emergency system to power nearly all operations spanning the center's 1,075,000 sq ft of floor space. Facilities include a 382-bed hospital building serving patients throughout Nebraska, as well as parts of Kansas, Iowa, South Dakota, and other regional states. Other East Campus facilities include the Bryan Medical Plaza, Edward's office building, Faulkner Medical Plaza, obstetrics tower, and a 700-student college. Premier services include cardiology, neuroscience, women's and children's health, orthopedics, vascular, intensive care, imaging and diagnostics, gastroenterology, oncology, urology, nephrology, pulmonary, general medicine, and surgery.
More than 2,100 employees depend on the center's emergency power system during power outages.
Bryan Medical Center is part of Bryan Health, a not-for-profit, locally owned health care organization. It's named in honor of William Jennings Bryan, the 41st U.S. Secretary of State, one of the great orators of his era and widely known as “The Great Commoner.”
Wiruth decided the CPMS for the power distribution system, which included six generators totaling more than 4,000 kW in rated capacity, required highly automated and flexible monitoring, control, and reporting functionalities. The system needed to capture data from the utility feeds coming into the center to 37 power transfer switches and loads.
"We bring in 12,470 volts from two different feeds, one each from two different utility substations," Wiruth said. "If one substation goes down, they can throw a vacuum breaker to transfer us to the other substation."
The main switchroom distribution feeds 10 substations located throughout the medical center. One is dedicated to five chillers and the remaining nine feed all other loads.
Interestingly, the six gen-sets are not paralleled.
"We elected not to parallel them because they're located throughout the campus — from a ninth floor penthouse to a basement — rather than in one centralized plant," he said. To call the power distribution system “decentralized” is an understatement.
"The gen-sets and transfer switches were added over time as the center needed additional emergency power." As a result, the system comprises multiple busways.
"To parallel the gen-sets, we would need to run a lot of common bus, making changes to existing busways costly. To be honest, we like the multiple busways because we consider them 'veins' of power that provide considerable redundancy," he said. If one fails for whatever reason, it doesn't affect power throughout the campus.
Redundancy also is built in with the sixth gen-set. Should normal sources fail, all six gen-sets start simultaneously. Five bear the 4,000 kW of load. Two are rated at 620 kW, and one each at 750 kW, 800 kW, and 1,000 kW.
One feeds five transfer switches as the second power source. In fact, the gen-sets all branch to different areas of the campus, so if an event disables part of the power system, the rest of the system continues to thrive with the multiple buses mentioned earlier. A 2,000 kW gen-set assumes the load of any generator(s) that might not start or fail.
"It wouldn't be able to take all the load during the day, but it could cover the loads of three gen-sets," he said. "The reason it's so large is so it can back up three of the five chillers and other loads." The $300,000 that medical center administrators invested in the gen-set and additional transfer switches reflects their determination to maintain a state of the art emergency power system,” according to Wiruth. “That two megawatt unit is awesome," he said.
Though the gen-sets start at the same time and are online in less than 10 seconds, they accept non-critical loads in a staggered, timed sequence.
"I considered each generator and each equipment branch transfer switch," Wiruth explained. "I then prioritized each generator's loads."
For example, an air handler for surgery comes on before one that supplies office space. Essentially, life safety and critical loads come on in 10 seconds or less. Using the power transfer switch controller timer, priority two loads come on in 20 seconds, priority three in 30 seconds, and the next in 40 seconds. "That way, we're not dumping all the equipment loads on the generators at once," Wiruth said.
Before the gen-sets accept their loads a series of UPSs (uninterruptible power system), bridge the gap between normal source failure and the time that prioritized loads transfer to gen-sets.
"UPS powers computers throughout the center, and a 300 kVa UPS powers the data center, which serves not only the Bryan Medical Center campus, but also sister units of Bryan Health on another campus," said Wiruth.
The data center manages patient information, billing, admin, and operations. The UPSs input is on one generator and bypass input is on another generator, each with their own transfer switches connected to different distribution panels.
The 37 power transfer switches, five of them generator to generator, manage which source feeds the variety of loads, ranging from surgery, intensive care, emergency lighting, and X-ray and CT equipment to the data center, chillers, food service, and other loads. Transfer switches managing the most critical loads always have live power on both sides. In the event of an internal outage, such as a breaker trip, they immediately switch to the second live source, meaning the center is protecting itself from both external and internal outages.
"In comparison with a paralleled system using a single bus and five generators feeding that bus, if that bus shorts because a short occurs in all the phases, then you're pretty much done." Wiruth said. "Visitors who've seen the system are amazed at its redundancy. Before we installed the sixth gen-set, we would be nervous while even changing the oil, concerned that a thunderstorm or a car hitting a power line might cause a power outage. Having said that, we've been blessed to have very stable utility power. In the 27 years that I've been here, we've lost normal source perhaps five times ... and all have been minor outages," he added.
All the power transfer switches range from 260 amperes to 3,000 and operate in the open transition mode, which means they “break” with one power source before “making” with another. Because some are 20 or more years old, they were retrofitted with modern, digital controllers and power managers.
The retrofit also included upgrading software, replacing control panels, removing existing serial communications cards, and changing communications medium from twisted pair RS485 to Ethernet. The six gen-sets also were outfitted with the power managers and Ethernet cards. This helped ensure that everything would “play nice” with the type of highly automated critical power management system that Wiruth wanted. "The upgrade went really well, minimizing interruptions to operations," he said.
Wiruth wanted to build on his experience with the power transfer switch manufacturer's automated monitoring and control capabilities, so he had very specific requirements that needed to be met for conducting tests and reporting results.
Wiruth's list of requirements was unabashedly long and demanding. He refused to settle for anything less than capabilities that, first and foremost, would help him produce monthly Joint Commission-quality testing reports more efficiently, faster, and more user friendly. He needed to be able to conduct a test of the entire, decentralized emergency power system from a single computer screen — no easy feat. Conversely, he also wanted to test a single gen-set with a load bank and get an automated report.
Other capabilities that had to be part of an overall critical power management system included the ability to operate the system from up to 10 client computers, view historical event logs, power readings, and overall system operation at a glance, and change power transfer switch settings from the software. He particularly wanted to get emails automatically on a variety of events and reports.
Ideally, the system should piggyback on the center's existing VLAN Ethernet network to minimize installation time and avoid the labor and materials costs of running a secondary, RS485 twisted pair communications structure.
First, however, he considered another manufacturer's monitoring and control offering used in other units of Bryan Health. After evaluating both capabilities, he selected the critical power management system offered by his current power transfer switch partner. He realized he could get more of the information he wanted from its capabilities than the generic capabilities offered by the other manufacturer, among other reasons.
"It was more economic to standardize with the ASCO PowerQuest 5750 Critical Power Management System," he said, "so the overall system will 'play better.' It's great."
It also will stay secure. Strong data security employs AES 128-bit Encryption to protect the center's emergency power system against unauthorized data access and control. It's the same advanced encryption standard used by the National Security Agency to protect top secret information.
"I personally have a high comfort level with their quality and service," he said. "Realistically, if I have problems, I know where the buck stops. There's no blaming."
Also, the flexible CPMS was, indeed, able to connect to the center's existing VLAN network. The center's IT department quickly connected cable to the closest IT closets and provided IP addresses that were assigned to emergency power equipment through facility routers. As an added plus, CPMS is designed with the capability to remotely connect to the system through the internet to seamlessly perform software upgrades and diagnostics throughout the start-up process. The CPMS System gives the hospital the capability to view the status and test of the emergency power system from a single computer screen.
"Now, we highlight multiple transfer switches on one computer screen and give one command to 'go,'" he said. "Before, we had to go from screen to screen to screen to tell each of the 37 ATSs (automatic transfer switches) to go separately."
He doesn't even want to think about how much time it would take to conduct tests manually at each transfer switch.
"If we had to walk to each ATS, it would take hours," he said. "If they had half-hour timers, they would have to be transferred back before someone completed all the initial transfers. It's invaluable to do from one location, from one screen."
The automatic email alerts and reports he wanted have proven to be one of the biggest plusses of the CPMS because generators and transfer switches are in parts of buildings where no one is around.
"When a generator starts or a load transfers anywhere on campus, I know it," he said. "When I was on vacation, my partner did a 'no-load' test and I knew it got done because I got an email telling me it was done.”
Then, Wiruth began customizing the CPMS — highly.
"As we gained experience with the management system, it occurred to me that it would be helpful to monitor actual load on the generators during tests," he explained. "To see an actual number showing the exact percentage of a gen-set's nameplate kW rating that is under load, in addition to the slide scale rule that's part of the 24 in. generator screen display, would tell us at a glance when a test met Joint Commission's 30% of nameplate load requirement for a given amount of time."
He got it.
"The customized capability also helps me trend gen-set load growth so I know how much load I can add to a given gen-set during a facility remodeling or expansion."
Other available information for gen-sets that he monitors is maximum, minimum and average volts, amps, and Hertz.
Even relatively little things help simplify system monitoring, testing, and reporting. One example is a "Notes" capability that Wiruth requested for each transfer switch and generator. It allows him to indicate equipment location, respective loads, and other aspects that he finds useful.
"It eliminated the need for a paper spreadsheet and since it's now electronic, I can access it remotely off campus from a computer or my phone, so it's always available," he said.
Because of their high utility, the custom changes that have been made so far have become standard CPMS features.
Finally, the critical power management system helps train technicians and keeps them sharp on power distribution operation, event diagnosis and servicing.
Wiruth knew he had to travel the road to more efficient, faster, and more user-friendly monitoring, control, and reporting for the center's emergency power system. But what was the journey like?
Although there were inevitable potholes that required patience and persistence, he said, "Yes, it's been a good experience. The manufacturer bent over backwards to make the custom changes we wanted and the system is all that we need it to be. It helps us produce test reports that Joint Commission and the fire marshal accept. And we continue to fine tune the system."