OnePartner's Advanced Technology Applications Center (ATAC) in Southwest Virginia began with OnePartner's search for a suitable Tier III or Tier IV data center to provide disaster recovery services for regional medical records. Simple research (using Google) seemingly identified hundreds of data centers claiming Tier levels, although there were none in the immediate area. Given the apparent abundance of Tier III and IV data centers, the design team set out to design a Tier III data center with resiliency (or uptime) suitable for health care and financial operations.

While virtually everyone in the industry has an intuitive understanding of the Tier classification system designed by The Uptime Institute, fewer have actually read the classification descriptions. They are surprisingly simple; however, this simplicity drives very complex facility design requirements.

Next, the project team recruited a designer and data center manager. Mac Scofield spent nearly 20 years with Eastman Chemical Company managing world-class data center operations. Eastman's technology infrastructure was truly world class, with enterprise-level operations in many countries. A devotion to process, meticulous planning, and attention to detail are hallmarks of Eastman, where the smallest mistake or disruption of service has substantial consequences.

Scofield also oversaw the design of Eastman's Business Continuity center in Northeast Tennessee. He also managed operations there on a daily basis after the center went on-line, and this experience highlighted many small, but important, improvements that could make a good data center even better. ATAC thought Scofield's hands-on management experience, combined with decades of industry knowledge and expertise qualified him to be the ideal designer.

Scofield's experience saved significant vendor research time. He knew, for instance what equipment he wanted at the center, ranging from Caterpillar generators to the two-factor biometrics systems that ensured security.

When basic architectural plans for the ATAC were complete, the design team contacted the Uptime Institute to begin the Tier certification process. The certification process required architectural drawings with a deeper level of detail than would normally be expected.

After completing additional electrical plans, the ATAC design team submitted them to the Institute. The Institute's engineering team spent about two weeks with the plans and then provided a list of modifications required to meet ATAC's desired Tier. The Institute noted 42 exceptions, which is impressive given the professionalism and experience of the design team. Thirty-two of the 42 were classified as "sustainability" issues that did not impact the Tier rating; there were 10 Tier-impacting exceptions.

ATAC's design team updated its plans to incorporate the Uptime Institute's recommendations. There were some painful moments. Achieving Tier III or IV is a real gut-check and can be expensive. Every switch and connection from the server cabinets all the way back to the generators has to be duplicated in a perfect, separated A/B circuit. The cooling design caused another expensive problem. OnePartner purchased Liebert DS105 units, which are Emerson's largest computer-room air conditioners (CRACs). The nameplate rating for this unit would allow it to cover the ATAC server room's projected heat load, but the team realized that two units would be required to achieve the no-single-points-of-failure scenario required to be certified Tier III.

However, the Uptime Institute restricts the expected actual capacity of such equipment to 75 percent of the nameplate rating, requiring the purchase of not just one extra CRAC, but two. Budgeting a new data center that requires two additional units that would be needed only if the first system failed requires an unwavering commitment to Tier III Certification.

Lessons Learned

About halfway through the Certification review, the team asked Julian Kudritzki, the certification manager for Uptime Institute Professional Services for the names of other companies in the region that had a Tier III or IV Certification. Drawing on the initial Google searches, the team assumed there were certified commercial data centers in Roanoke or Knoxville and surely several in Charlotte and Atlanta. Rather Kudritzki said there weren't any such facilities nearby. Kudritzki indicated that the nearest certified commercial facility was in the United Arab Emirates. For a complete list visit: http://professionalservices.uptimeinstitute.com/TierServiceProvidersMap.htm. As of May 27, 2009, OnePartner's ATAC in Duffield, VA, is currently the sole U.S. company providing outsourced commercial data center services (including colocation and disaster recovery) that has been awarded a Tier III Design Certification by the Uptime Institute.

How is it possible that their there are so few commercial Tier III (or higher) data centers in the U.S., given the number of facilities that seem to claim such high levels. Why didn't the other data centers claiming Tier III actually get Certified?

Certainly it can't be that these owners and operators aren't aware of the Tier standards. These standards are widely accepted in the industry and familiar to anyone considering colocation service. In addition these data centers realized the value of claiming and striving for a Tier level.

Review by the Uptime Institute isn't expensive. The ATAC review cost slightly less than 1 percent of its cost of the ATAC. Kudritzki shared a story that sheds some light into the real reason there aren't more certifications. He said that in December two data centers that had claimed for years to be Tier IV or 'close to it' completed the certification process. Both were certified as Tier I.

Neither had decided to publish their actual certifications. Kudritzki added that other data centers had also been certified but not published their results for similar reasons.

Touring data centers is a reassuring experience. Data center operators are eager to showcase their facilities. The biometric security scanners, uninterruptible power supplies, spotlessly clean server rooms, and of course the massive generators, create a very reassuring impression. But how close is this impression to reality?

But both private and commercial data centers do fail. Years of uptime come to an abrupt end with the sudden discovery of the "weakest link" in the facility's infrastructure that previously escaped the notice of the data center operator and hundreds of client tours.

Certification is designed to confirm that a data center is as reliable as it appears during a tour or after years of successful operation.

Certification Matters

When ATAC's prospective colocation clients learn that the facility is the only commercial Tier III (or higher) data center in the U.S. they immediately think it will be more expensive. They are weighing a perceived higher cost against the benefits of greater uptime. When they find that the ATAC is not more expensive than other commercial sites, they are relieved.

Imagine the operator of a hospital or a health care provider's electronic medical record (EMR) systems who has leveraged a professional colocation facility and avoided the common pitfall of creating an improvised backup data center in an office locations The facility claims Tier IV or 'close to it' status.

Then disaster strikes. A tornado or ice storm hits. Everyone is frantic, people panic, and the injuries pile up. As the physicians help the long lines of patients, an air unit in the primary data center goes out. Facilities personnel follow the disaster plan and immediately switch systems over to the disaster recovery site.