Spending millions of dollars on business resiliency and reliability is a good start, especially when an organization can lose a great deal of money if downtime occurs at the precise wrong processing moment. Investing in and installing new equipment does not guarantee reliability and facility infrastructure health. To maintain the edge in business resiliency in today’s mission critical environment, the reliability model needs to include capital dollars for critical infrastructure plus ongoing funding for documentation and training that addresses the operational risks associated with critical environments. Since human error accounts for at least half of all downtime, prudence suggests that it make sense/cents to fund documentation and training. These programs are essential to achieving and maintaining optimum levels of reliability. Why bother with major investments in a robust facility without implementing a comprehensive documentation and training program to complement it? 

As equipment reliability increases and best practices mainstream, an increasing percentage of downtime results from actions taken by inadequately trained personnel or staff that lacks access to accurate, up-to-date comprehensive data during critical events.  Keeping on-site staff motivated, trained, and ready to respond to emergencies is a challenge, made even greater challenge without an appropriate program in place.

Years ago the typical organizations relied heavily on its workforce to retain much of the information regarding mission critical systems. Many of these employees had similar levels of expertise, and they remained with one company for decades. Therefore, creating and maintaining a fluid and living document repository for critical infrastructure took a low priority.

Today it takes three months or more to find qualified engineering staff to operate a critical facility, and at least another three to six months before the new engineer becomes familiar with the corporate culture and complex infrastructure. Without relevant documentation, how does an organization plan to teach this engineer the critical system essentials? Furthermore, what are the operational risks associated with the lack of knowledge?

Diverse mission critical system configurations and applications severely hinder the ability of employees to fully understand and master all necessary equipment and learn all the information required to keep that equipment running. Engineers and operators can no longer be asked to acquire their knowledge of increasingly sophisticated power supply and distribution technology from “on the job training.” Such training proves woefully inadequate in time of crisis, because of gaps in the knowledge of the employee who learns about a system so haphazardly. Instead, a clear plan must be put into place to develop a critical document repository and to continually educate and train employees while enhancing and sharing “lesson learned” experiences.

A comprehensive training program could include: 
  • Providing fundamental training on facility electrical, mechanical, and life safety systems
  • Determining staff qualification criteria
  • Identifying training topics and developing specific modules
  • Creating testing content and certification methods
  • Maintaining employee training records and ongoing training requirements
Prudent business practice recognizes the need to plan for employee succession, unexpected staff departure, orientation/training of new employees, as well as education of seasoned employees.  An education and training program, coupled with a document management system, can address these concerns. Site-specific training courses can be developed to target subjects such as UPS switching procedures, emergency generator operation and testing, company policies and procedures, safety, and critical environment work rules.

In the financial services industry, education and training that is standardized, comprehensive, and focused on the job at hand will create a pool of talented individuals possessing the knowledge and information necessary to solve problems during emergencies. This will lead to shorter and less frequent unplanned downtime. Documentation is essential not only to facilitate the ongoing education and training requirements of a company’s personnel, but also to maintain safety and to minimize risk to the company, assuring the integrity of a robust mission critical infrastructure and the institution’s bottom line.

A living document system can produce a “database” of perpetually refreshed knowledge, providing the level of granularity necessary to operate a mission critical infrastructure. Such a system should be supplemented with a staff training and development program. Keeping the ever-changing document and training programs current can then be addressed each time a capital project is completed or an infrastructure change is made. Accurate and up-to-date information provides first responders with the intelligence and support necessary to make informed decisions during critical events, as well as allow for employee succession planning. The short-term dollars saved today by eliminating documentation and education/training program will result in an exponential monetary loss tomorrow.