A new data center came to life recently on Delta Dental of Michigan’s property in Okemos, MI. The new 22,500 square foot data center will house hundreds of computer servers and the technology needed to keep them operating without interruption.
In fact, the new data center is one of only 10 in the world to have earned a Tier 3 rating for its design from the Uptime Institute and is on track to set the new standard for electrical power handling.
“We should never have to power down the data center,” said data center manager Christian Briggs. If they do, Briggs added, it’s because something so catastrophic occurred that it wiped out several levels of redundancy.
Delta Dental of Michigan, with its affiliates in Indiana, Kentucky, New Mexico, Ohio, and Tennessee, is one of the largest dental plan administrators in the nation. In 2008, the enterprise paid out more than $2.1 billion for dental care for more than 6.6 million enrollees. (Delta Dental of Michigan provides coverage to employees of BNP Media, the publisher of Mission Critical.)
Claims processing, email systems, document storage, records retention, and online activities for enrollees, groups, providers, employees, and the public are all handled at the company’s new data center.
“We’ve put a design in place to accommodate technology needs for at least the next 10 years plus we built in capabilities for expansion as needed,” said Brenda Laird, senior vice president and chief information officer for Delta Dental of Michigan. “This significant investment serves and protects our company and customers while showing our commitment to supporting the economic future of mid-Michigan.”
The Uptime Institute Tier system evaluates data centers based on their ability to remain up and running regardless of power failures, natural disasters, or other potential power-robbing catastrophes.
While designing Delta Dental’s new data center, ensuring electrical power was always available to keep the servers running was a key factor. The Tier 3 design rating means that maintenance tasks, such as testing breakers, can be performed while the data center is still up and running, a concept known as concurrent maintainability. To meet the Tier 3 requirements, Delta Dental added a second set of paralleling switch gear into its electrical system. Paralleling switch gear provides clean power by helping to get the generator output in sync before feeding power to the computers. Delta Dental collaborated with the generator manufacturer and ASCO, the manufacturer of the automatic transfer switch gear, to devise an industry-first system for automatic failover between the two paralleling switch gear.
“Delta Dental has a one-of-a-kind emergency power distribution system,” said Nathan Cronenwett, an electrical engineer with ThermalTech Engineering, a mechanical, electrical, and plumbing consulting engineering firm that helped commission the new data center. “This type of design is superior to those with no emergency redundancy or those that require twice the number of generators to accomplish the same level of reliability.”
The data center’s three Cummins 1-megawatt generators are able to begin providing clean electricity within seconds of the electrical grid failing, preventing loss of power to the computer servers.
“Companies are now running 24/7 through online resources and they need to be able to do so reliably. With this new data center, we have the capability to meet the growing needs of customers from around the world,” Laird said.
To put the demands of the servers into context, only about 5,000 square feet of the 22,500 square foot building is dedicated to computer hardware. The rest of the space is utilized by the electrical and mechanical support systems Delta Dental put into place.
“The technology we’re using has been around for a while, but we chose, based on our business model, to have a level of redundancy most businesses don’t,” said Briggs, whose team of nine shares the responsibility of continuously monitoring and maintaining the data center.
Should the power stream from the utility company diminish or fail, the new data center’s dual Eaton 500-kVA 9315 UPS systems will kick in automatically, smoothing out any dips in the power stream or keeping the power flowing until the diesel-powered generators fire up or emergency backup battery systems engage. In these instances, flywheels should carry the load 95 percent of the time, until the generators start. Enough diesel fuel will be stored in an underground tank to run the generators for three days.
Briggs describes flywheel technology as a sophisticated spinning top capable of generating enough energy to run the data center while backup electrical systems kick in. “While you have power coming in, it’s making the top spin. Remove the power and the top’s inertia will keep it spinning for another 30 seconds. The inertia is used to generate power for the servers,” he said.
Those 30 seconds are enough for a safe transfer of the power feed from the electric company to the center’s two 500-kW UPS units. The units can keep the servers up and running for 15 minutes under full load and for as long as 45 minutes if the load is reduced. In addition, the UPS units back each other up so if one fails or is down for maintenance, the other can handle the load.
If there is a problem with the power delivery from the generators (generators not starting; they go into overload or overtemp; paralleling gear not functioning; transfer gear fault etc.) Delta also has battery backup, which will provide ample time to gracefully shutdown the IT load and allow electricians to troubleshoot and resolve the issue. Each UPS has two battery strings rated for 15 minutes of up time each. (de-rated by 80% at end of battery life to meet Uptime Institute requirements)
The data center only needs two generators to operate, but when the enterprise’s management team decided to build in redundancies, it took that concept seriously. “We have room for four and plan on getting there eventually,” Briggs said. “The idea is that you always have one more generator than you need so you have one on standby in case a generator doesn’t start or it fails in some way.”
The success of the redundant systems will rely in part on Briggs’ team, which divides the day into three shifts, constantly monitoring what is occurring on the business systems and in the data center. The data center team will be housed in the company’s main office building, watching the activities remotely via computer monitors.
“We will monitor the entire electric supply chain--we will always know what power system is being used and how much,” Briggs said. “Even if some kind of catastrophic failure occurs and we go through all the redundancies, we will have enough warning to have a safe, controlled shutdown of the servers.”
ServerTech PDUs will allow Briggs’ team to see the draw on each outlet on the power strips installed in server cabinets. Briggs team hopes to use this visibility to detect when equipment has been powered off and is no longer in use. They can also use this information to determine which kinds of hardware are most efficient. We were one of the first shipments for the that monitor to this level.
“That is cutting-edge technology,” he said. “We’ll eventually be at a point where I can monitor and manage individual power strip outlets right from my desktop. We’ll be able to see trending in power usage and set thresholds for use for early warnings of any trouble.”
All the computing power in the new center also means a lot of heat, so the design team took on the task of providing effective and efficient cooling to the building.
“Whatever you can do to improve cooling cuts down your operating costs,” Christian said. “Cooling is responsible for 60 percent of the energy costs in a data center.”
A 36-inch deep floor, much deeper than the company’s former data center had, will be used to move air more efficiently around the server room. Teraframe passive cooling cabinets from Chatsworth (CPI) direct waste heat from the cabinets into the ceiling plenum. The heat exhausts into the utility corridor where Liebert DX units cool it back down. Delta’s facility includes six 30-ton DX downflow units. Briggs said that he hadn’t noticed the compressors on these units running from July to January, which he sees as evidence of the effectiveness of the cooling system.
“Normally the hot air is simply ejected into the room where it mixes with cold air and then that mixed air gets fed into the coolers,” Christian said. “Sending higher temperature air into the cooler unit actually is better from an efficiency standpoint.”
The new data center also is designed so that when the outside temperature is below 60 degrees, the cooling units will be switched over to pull cold air in from the outside, allowing Michigan’s natural climate to help the company save on electric costs as well.
Protection from overheating wasn’t the only security provision considered during the design process. The cooling systems and generators are in walled-off areas, something else not usually seen at other businesses. Neither are tornado-proof heavy steel doors or a layered roof that covers the data center with metal, rubberized roofing and concrete sandwiched together to create a sturdy barrier. In addition to the standard card-swipe technology there also will be an inner access door protected with a biometrics system.
The data center will be well protected from intruders, debris, and most anything nature can throw at it.
“These systems are often left exposed to the elements or unprotected from passers-by who can just walk up to them,” Briggs said. “Vandals could damage those systems and bring down the building by cutting power or cooling if those systems are left exposed.”
In addition to being housed within the data center’s structure, the generators, cooling, and primary electrical gear are in utility corridors separate from the main computing area. Gaining access to the main computing area will require two separate authentications. The building also has cameras inside and out, including within the main computing area.
Briggs is proud of the company’s new data center because of how far it advanced Delta Dental’s capabilities over the previous data center. Nevertheless, as is the norm these days, getting something great accomplished doesn’t mean you can rest on your laurels.
Initially, the Data Center is housing approximately 250 servers and there is room for about 1,000. When that’s not enough, it will be time to blow out a wall and keep growing.
“We planned it all out with the idea that we will need to expand and we can expand,” Briggs said. “We built the data center at the back of the property for a reason. We can build a mirror image of it right next door and join the two buildings together for an even larger facility.”
