Data Center Facility Management Practices
Over the last few years I have been fortunate to have toured many critical facilities including performing in depth reliability assessments of over 45 sites in 20 countries across five continents. I have inspected literally millions of square feet of computer room spaces and supporting infrastructure. I have interviewed and evaluated facilities management staff and their processes and compared their performances against their corporate standards and industry best practices. What I have seen is a broad cross-section of compliance ranging from marginal to awesome.
What I have also noticed is that in almost every case my first impressions based on a familiarization tour and initial staff interviews pan out to be accurate in the long run. There are obvious telltale signs that quickly reveal what the culture is for any given site. General housekeeping and cleanliness, organization, institutional knowledge, and availability of accurate site specific documentation are just a few aspects that are indicative of how well the site is managed.
In the first sentence of my first column for this magazine I wrote, “Discipline, rigor, experience, training, process driven procedures, and a culture of excellence; that’s what it takes to deliver continuous operations over the life of a critical facility.” Everything I have seen over the last few years reinforces this statement. What follows are some characteristics that are common to the sites I have visited that have a culture of excellence.
General housekeeping is one of the first and most obvious indicators of how much pride and attention the staff has in their site. Some sites are relatively clean, especially in areas where people are most likely to traverse, and some are, well, less so. As you move through the site and inspect the less traveled spaces such as mechanical and electrical closets, tank rooms, roofs, etc., the level of cleanliness and housekeeping tend to drop off. When instead you find even the most remote and least accessible spaces to be clean and clear of debris, dirt, stains, etc., it is obvious that the staff enforce a high standard of care.
I’ve also noted that in many instances excellent lighting promotes excellent housekeeping, and the opposite is also true. Dimly lit spaces tend to get less attention. Good housekeeping is not only superficial, but also substantive in that a leak, stain, debris, or other discrepancy stands out and begs to be corrected.
Another obvious characteristic follows the old saying “a place for everything and everything in its place.” In a recently visited site, this practice was followed to perfection. Upon entering every mechanical or electrical room there would be a first aid kit, emergency flashlight, and floorplan. At least one laminated and framed single-line diagram would be posted in the room, with the portions that reside in the room annotated by dotted line borders and color coded. These were hung by string and wall hooks so the diagram could be removed and used by the staff while standing in front of the respective gear and equipment, but they were always returned to their rightful place. Ladders, tools, and portable equipment were stored in designated places identified by color-coded tape on the floor, and the only items allowed to be stored in the room were those that were applicable to the room’s purpose. Any parts or materials in the space were directly related to the systems and equipment in the room and otherwise the standing policy was that these spaces were not for general or unrelated storage.
Signage, labeling, and color-coding combined with intuitive conventions are also indicative of how standards are employed. The best sites typically have comprehensive use of color coded infrastructure and standardized labels such that upon entering a room everything is easily understood. Conduit and piping systems are simple to trace when they are painted or otherwise color coded. Labels that not only identify the system and/or equipment, but also conform to logical identification conventions, can provide lots of critical information at a glance. An example is electric panels with labels indicating what system, switchgear, and breaker the panel is fed from and with color codes that indicate whether the service is utility only, backed up by generator, or on UPS. This becomes even more important for sites with rooms and redundant systems that look similar if not almost identical such as “A” and “B” switchgear, UPS, and other infrastructure that otherwise could lead to human error due to misidentification of equipment especially during emergency or anomaly responses.
Easy access to site specific and accurate documentation is another characteristic of a culture of excellence. How a request for a drawing, manual, procedure, or other critical document is responded to is a clear indication of how well the site manages documentation. When the document is produced with ease and the staff is confident it is current and accurate, there is likely a formal document control system in place and enforced. When it takes several tries to locate the document, and then it is provided with the caveat that it may not be accurate, then there either is no formal document control process or it is not enforced. Regardless, the value of the information is reduced since it isn’t readily available and can’t be trusted.
HIGH STANDARDS A MUST
I could continue with an almost endless number of other aspects and indicators of what constitutes a culture of excellence. What is consistent is that in all cases there is a very high standard of what is considered acceptable and expectations that all staff will not only comply, but will collectively enforce compliance by others including teammates, contractors, visitors, and everyone else. This means when something falls below the standard, it gets resolved immediately. Messes are cleaned up, missing labels get replaced, leaks get repaired, documents get updated, and obsolete versions get archived. As parts and materials get used, the stock gets replenished. Tools, materials, and equipment get returned to their proper place. Staff get trained, drilled, and recertified whenever systems are modified or the site infrastructure changes. Contractors are supervised and their work inspected before they are allowed to depart or their work accepted.
As I also stated in that first article, the key is to do three simple, but very difficult things:
- Do everything
- Do everything right
- Do everything right every time
There is one other very important characteristic that is required to foster a culture of excellence. There must be a properly staffed and resourced facilities management organization. Effective leadership champions the mission, purpose, and needs to executive management to garner the required budget, resources, and support necessary to succeed. The leadership must also establish the standards that define what is acceptable. There must be good management that can establish both organization and processes that provide the order and structure needed to operate and maintain the facility. Management must also direct and supervise the staff in the execution of its duties and responsibilities, schedule tasks and activities, and enforce compliance through discipline and rigor. And last but not least, there must be sufficient technicians, operators, and staff to do everything right every time. This means qualified staff with site specific knowledge, the tools and resources required, and the skills to perform the tasks and activities assigned.
Insufficient staffing and/or resources inevitably results in a reactionary culture where staff constantly has to prioritize tasks and activities and compromise on performance. At first it is the superficial tasks that get deferred (housekeeping, storage and inventory control, document management, non-critical preventive maintenance, etc.), but eventually the standards aren’t met, morale degrades, and pride and ownership dissipate. Basically, the staff no longer can do everything, much less do it right every time.