During the first weekend of 2017, much of the United States was hit by a winter storm that dropped snow and ice over portions of the U.S. from the Rocky Mountains across the plains to the East Coast. My home near Charlotte, NC, got about three-quarter and inch of ice with two inches of snow on top. The roads that were treated with brine were clear throughout, but the side streets and rural roads didn’t fare so well. And down here on the border between North and South Carolina there are a lot of untreated roads.
So, like pretty much everyone else, I visited the local grocery store to purchase some staples to get me through what was predicted to be a couple of days of being “snowed in.” Of course, I waited until the last minute and the grocery store was pretty much empty of both products and customers. There were lots of empty shelves and plenty of parking spots, and as the rain was literally turning to freezing rain, sleet, and snow, I paid for my stuff and hurried back home. And I wondered who will make the decision to close the store and let all the employees finally go home, and how dangerous a drive would some have in store for them.
Not too many people will consider a grocery store a critical facility. But when it comes to people’s individual and family safety, what can be more critical than getting stocked up for a major storm? Sure, there is plenty of redundancy since there are lots of grocery stores, convenience stores, and other places to get those “must have” staples, but that doesn’t lessen the importance of having them. And each store must have employees on-site to be open for business.
I still remember Y2K and all the preparations and effort the IT organizations went through to get all software and applications Y2K compliant. This spread down into the facilities organizations as similar efforts were made to ensure critical monitoring and control systems, equipment firmware, and facilities management software was compliant. Despite all the preparations it was still a tense day on New Year’s Eve in 1999 and everyone waited for the clocks to turn over and 2000 to arrive.
New Year’s Eve 1999 was perhaps the biggest and most anticipated new year of our lifetimes. Most people were celebrating with friends and families. Parties were held all over. The executives that I worked for were all at plush hotels overlooking Times Square in NYC. But across not only America, but across the world, there were hundreds of thousands of employees at facilities everywhere waiting to act if anything went wrong and to restore impacted critical operations. And from everything I heard — nothing happened. It was business as usual.
I got home that night just after 2:00 a.m. and in time for my wife and I to watch the new year arrive in Las Vegas and the West Coast on TV. There would be no second chances, do-overs, or maybe next year. I believe I received a generic “thank you” card from the company for my sacrifice like everyone else who put the company’s interests first. But also like most everyone else, it was accepted as being an integral part of our jobs. We even took some pride that we were considered “essential personnel” and that our skills, knowledge, duties, and responsibilities were necessary to ensure continuous operations in support of the “mission.”
That spring, I had the opportunity to develop and implement a computerized maintenance management system (CMMS). After setting up an asset identification convention, location codes, and all the other working parts for a CMMS, it came time to start developing preventive maintenance procedures. The first one I wrote was to have the main building cupola cleaned and inspected at 9:00 p.m. on July 4th of each year. The cupola was an unoccupied, unused space accessed by a ship ladder and roof hatch, but it had the best views of the Washington skyline of any place around. And from that location you could see at least a half dozen fireworks displays around Washington. My thought was since the site required 7x24 continuous shift coverage, at least one building engineer would be working shift each 4th of July and this was my way of ensuring that he/she would at least get to watch some fireworks each year.
There have been many articles, papers, and presentations made about preparing for natural disasters, storms, floods, and even earthquakes. We develop disaster recovery and business continuity plans. We figure out what we will need to keep critical operations running and store canned foods, water, cots, and other essentials to keep critical facilities occupied throughout the events. But how much consideration do we give to the personal sacrifices we routinely ask of our 7x24 critical staff? Anyone who has worked the graveyard shift or rotating shifts for any length of time understands the inherent inconveniences. Anyone who has been required to be “on-call” or who must respond day or night to computer generated alarms from the NOC or BMS understands the latent demands. Anyone who has been a service technician who must respond to customer emergencies by phone within minutes and be on-site within hours understands the stress that just “goes with the job.”
Some organizations understand the human aspect to critical operations better than others. They reward staff who are required to work holidays with extra pay and/or additional time off. They ensure on-call assignments are spread across multiple staff so everyone carries a portion of the burden and everyone gets a break. They categorize computer generated alarms into various criticalities so off-site staff are not inundated with nuisance alarms, etc. I know of at least one large colocation data center provider who has standardized their facility designs such that any operating engineer from any site can reliably operate a different site in the portfolio. When a hurricane threatened a site recently in Florida, they sent the local site staff home to ride out the storm with their families in their own homes and brought in operating staff from other sites such as Kentucky and Tennessee to staff the Florida facility.
So, this article is in tribute to and to celebrate the dedication and loyalty of critical support staff everywhere. From the grocery store cashier who can’t go home until the rest of us have our extra water and toilet paper to the facilities staff, command center operators, IT staff, police, firemen, and everyone else who routinely sacrifice for the better good. Let’s not take them for granted. And Happy New Year to all!