Rush University Medical Center Protects Against Fire Loss
Information technology is driving more effective, lower-cost health care, and better patient outcomes at hospitals nationwide. Information in the form of patient records, diagnostic images, prescriptions, etc., as well as the need to handle and process data, is expanding exponentially. At Rush University Medical Center in Chicago, the capacity to handle this explosion of data has increased 20-fold over the last two decades, evidenced by the massive ongoing investment its administrators have made in data handling, storage, and network management technology during this period. This critical IT infrastructure, key to day-to-day hospital operations at every level, is managed at the hospital's data center erected in 1986 adjacent to the hospital campus west of the Loop in downtown Chicago.
Wil Raske, director of Data Center Operations at Rush offers these details on his facility: "We support over 600 production and over 100 test servers for our medical center's computing needs. We run and support approximately 500 different end-user software applications over our campus-wide wired and wireless networks to over 10,000 PC workstations. These workstations are located throughout the medical facilities, research labs. and the medical school operated here, as well as the other hospital locations in our network.
Rush Medical has three operational modes for physical redundancy. The first is instantaneous back-up of data from all server hard drives to a mainframe. The second is six-second back-up to a tape drive for storage onsite at Rush in an environmentally-sealed room. The third is remote storage at a confidential site in the Chicago suburbs.
Raske said, "All our systems are backed up either daily or every other day, using an incremental scheme, via the Veritas Netbackup product. We have approximately 200 terabytes of on-line Hitachi SAN stored information available in our on-campus data centers." Rush's on-campus tape arm consists of more than 100,000 tapes, averaging 10 GBs per tape or 1 petabyte of off-line data stored on campus. Raske's department also supports the medical center's corporate financial applications on an IBM 2066 mainframe that processes over 1.86 million instructions per second.
To help the hospital IT staff protect its myriad servers, data storage drives, and backup tape system, Siemens Building Technologies, Inc. was tapped to apply its considerable systems and solutions acumen to replace the existing Halon-based system, which after 20-plus years of service, was well past its technical and serviceable prime.
In response to this situation, Raske submitted a proposal for vendor review. Subsequently, hospital administrators received the budget to replace the obsolete fire detection and suppression system, which had become a drag on operations and maintenance budgets-not to mention a serious financial and legal liability if it were to be triggered by a false alarm and accidentally discharged.
The ExposureIn their search for an optimal system, Rush managers required a solution that would protect every bit and byte of back-up data stored onsite and drastically limit collateral damage to servers, network systems, and other hardware-without imposing lengthy or costly outages if the system activated. Furthermore, as part of the entire Rush Medical Center's commitment to employ "green" standards where possible, any solution considered would have to be as environmentally friendly as possible and not create health risks if an emergency discharge occurred.
Rush project managers determined that Siemens Building Technologies' SINORIX fire suppression system delivered as a comprehensive, tailored solution by its local team of company personnel would meet all the project requirements. The Sinorix-based solution is now on-site behind the scenes at Rush protecting the data and network systems required to deliver patient information to caregivers when and where they need it, plus full financial information, all to keep the 600-bed hospital and educational/research facility running smoothly 24/7, 365 days a year. Once the installation was completed, Sinorix Solutions and Sinorix Service also offered Rush the Sinorix Limited Discharge Warranty that guarantees replacement of the agent should the equipment, installation, or the technical support fail in its performance.
According to Raske, one of the system's highlights is its environmentally friendly agent that suppresses fire and leaves no residue, a marked improvement over the previous system. The system at Rush includes the Fireprint early warning smoke detection hardware and a clean agent, HFC-227ea, which has a zero Ozone Depletion Potential (ODP), making it safer for the environment and posing no health risks to humans. HFC-227ea is used in medical inhalers as the propellant to deliver medicine to an asthmatic's lungs. Because of its benign nature, the use of this agent is eligible for a credit toward LEED Certification (Leadership in Energy and Environmental Design) from the U.S. Green Building Council.
Past Its PrimeRush's existing systems were long overdue for replacement, Raske explains. "Our legacy systems were well past their anticipated service life and we were very concerned about them becoming prone to false discharges. Our service contractors were having trouble finding spare parts, plus the environmental impact of Halon was an obvious consideration. We also knew the consequences of an accidental discharge." Raske offered a cautionary tale that started innocently but ended badly, with an accidental discharge that cost his department $20,000 in replacement suppressant at the time. The price tag on such an incident would be much higher now.
As part of the overall turnkey solution, Siemens began the installation of system by decommissioning the environmentally unfriendly and obsolete Halon systems.
Raske describes Rush Medical's data center as a Tier 2 facility with some infrastructure subsystems approaching Tier-3 level, including the power subsystem. The power configuration comprises two completely redundant subsystems. Almost every computer device in the data center has redundant power supplies, each plugged into different sides of dual power subsystems. Each side has a 150-kVA fully redundant capacity (300 kVA total) provided by parallel/redundant UPS. A diesel generator provides further emergency backup. Raske says, "This rather unique double redundant configuration insures we will maintain uninterrupted power for our Medical Center computer systems in the event of any loss of power feeds from ComEd.
The cooling subsystem provides more than 100 tons of cooling to air, configured through a N+2 cooling tower and chiller subsystem. Although Rush Medical has three chillers, Raske estimates its current capacity usage to be 85 percent of one chiller. "We also have chilled water cooling capacity piped into our computer room, but it's not in use at this time, " he said. This can be used to bring actual liquid cooling directly to equipment cabinets to meet future additional equipment-cooling capacity needs as cooling requirements continue to grow exponentially. Raske says, "We average 99.98 percent scheduled systems availability across all of our supported computer systems."
So Much DataRush University Medical Center data is handled by recent vintage Unix-based servers in tandem with some 400 Intel-based blade and non-blade servers that manage an average of 26 terabytes of data. Routers and other network controls reside alongside these servers as well. An automated, robotic system manages backup data storage routines and readies data storage tapes for interim onsite and then eventual offsite storage. Raske takes considerable pride in his robots. "At one time, our data mounting took 2-1/2 minutes; now that task is done in six seconds-with a substantial reduction in human interaction. We have a triple-redundant system whereby, every day, every bit of data from all systems in our network is redundantly recorded. With over 14,000 workstations across the campus, that's a considerable task, but no patient's care is ever worth being put at risk by losing data."
Covering approximately 11,000 square feet. which includes the data center, test center, and data storage vault, the Sinorix system comprises a tightly integrated array of Siemens Fireprint cross-zone smoke detectors and Sinorix discharge nozzles strategically located below the ceiling space and within the utilities space beneath floor level. The Sinorix dispensing tanks are easily accessible and occupy a dedicated space just off the main floor of the data center, tape vault, and test room. Two data rooms and a phone room in another building on the hospital campus are also protected by this system, which comprises a total of 4,480 pounds of clean agent.
Siemens provides total flooding in the data center, tape vault, and test room that includes the three-level floor/sub-floor configuration, where low- and high-voltage wiring and network cabling are separated by an 18-inch air-cooled space.
Two Fireprint smoke detectors must activate before a discharge countdown is activated, explains Raske. "The Fireprint smoke detectors are designed to operate only when smoke particles are present, representing a true fire-causing event." The innovative, advanced Fireprint detectors are programmable and have eleven application-specific settings. A Fireprint smoke detector analyzes the particles in the air and measures them against stored algorithms located in the detector's software. The detector makes the decision to alarm or continue monitoring the situation. Fireprint smoke detectors perform a complete self-diagnostic check every four seconds. According to Siemens, these detectors have 100 times greater sensitivity than conventional flame/heat/smoke detectors and are at the heart of the company's warranty-something that Raske and his staff must trust-and for good reason.
Another unique aspect of the solution Siemens delivered are the key-actuated fail-safe switches that physically lock out the Sinorix suppressant discharge system during detector network testing and maintenance routines mandated by regulatory code. A stand-alone keyed switch interrupts the Emergency Power Off (EPO) actuating circuit, which sends a signal to the main power distribution circuit breakers, causing them to shut data center power off in the event of a major fire. This lockout is manual only and not tied to the Siemens system. The lockout system was engineered by Siemens to ensure that under no circumstances can the system activate and prompt an emergency power shutdown-an event that has extremely expensive consequences.
"We have zero tolerance for false alarms," Raske explains, "Our former conventional systems would engage an EPO condition via relays in the panel in the event of an alarm. Every 20 minutes of EPO time translates into 48-72 hours of downtime to recover the facility's operational status. Thus, the reduction of false alarm risk was paramount in the vendor selection process," said Raske.
No Fuel, No FireBecause the HFC-227ea suppressing agent interacts at the molecular level, it mitigates the fuel's ability to combust rather than eliminating oxygen or extensively damping heat like the water from a sprinkler head. Whatever is burning (insulation on a short-circuiting wire, for example) will extinguish, once the suppressant is discharged. And, because Siemens Fireprint smoke detectors are equipped with algorithms to discern smoke particles from a fire-causing event at its absolutely earliest stages, heat and smoke have little time to generate to levels that can harm adjacent equipment or be toxic to humans. "The system is so sensitive," says Raske, "that it could detect and suppress a fire in one blade server and nip it in the bud before it affects the other servers in the rack."
Coupled with the environmental upsides, Raske notes, the Sinorix solution was deemed to bring the most value at all levels for Rush. Installation was completed in just three months, including three other remote locations used for redundant data storage and phone communications.