Because data centers and other high-performance computing (HPC) facilities can use up to 50 times the electricity of equivalent-size office buildings, they are prime candidates for energy-efficient design and retrofit measures to slash energy consumption and operating costs. By definition, mission critical data center loads require extremely high availability, and for that reason pose rigorous challenges in defining and implementing effective mitigation strategies to not only protect the computer equipment itself, but also factor in the facility’s power infrastructure, among other considerations, to ensure uninterrupted facility uptime.

The fact that most utilities only log outages that last longer than 1 to 5 minutes tends to ignore the many momentary interruptions that every facility experiences, and which annually result in millions of dollars in lost productivity. Given the potential cost ramifications if left uncorrected, it is easy to see how understanding what power quality (PQ) problems are, how to find them, and how to solve them will only increase in importance for both facilities and utility personnel. According to a 2001 study by the Electric Power Research Institute (EPRI), power quality disturbances manifest themselves in various forms, typically according to the following relative percentages:                   

• Power sags - 46%
• Power swells - 10%
• Spikes and switching transients - 10%
• Undervoltage - 7%
• Overvoltage - 7%
• Harmonics - 5%
• Line noise - 5%
• Frequency variations - 5%

High-reliability applications typically employ several mitigation strategies in order to maximize uptime to better than “six nines”— or 99.9999% — of availability. Although six nines reduces downtime to under 30 seconds per year on a 24/7 basis, it still cannot guarantee 100% availability, as even highly redundant systems are susceptible to failures or unanticipated problems. Permanently installed PQ monitoring equipment at branch circuit panels, in critical equipment cabinets, and at other strategic locations is useful for determining what happened and, equally important, what steps are needed to prevent a repeat performance. Monitoring PQ typically allows facility operators to:

• Acquire data relative to problematic conditions
• Obtain historical indication of trends
• Predict future PQ issues

DATA CENTER METERING OPPORTUNITIES

As a tool for monitoring energy and PQ, submeters are an excellent way for facility professionals to gain a deeper understanding of, and better handle on, their facility’s power infrastructure performance through:

• Benchmarking and baselining of current operational levels
• Helping to identify performance improvement projects and technologies   
• Tracking performance and efficiency improvements over time
• Monitoring, recording, and alarming PQ issues and events
• Integrating HVAC, lighting, and other systems and equipment into the building management system (BMS)
• Reliable measurement and verification (M&V) of expected energy improvement projects
• Diagnosing and ongoing monitoring of system performance to ensure long-term operational efficiency
• Determining system benchmarks for comparison with other facilities
• Setting future performance goals and determining that specific improvement targets were met
• Analysis of metered data to improve planning for future facilities

Whatever the market focus, HPC data centers share a common need to process large amounts of data, an electrical infrastructure-intensive operation that requires careful implementation and equally insightful management. In order to characterize the facility’s electrical performance, metering various loads behind the main electrical meter is especially useful for profiling the energy footprint of any HPC facility which, for the sake of this discussion, may be broadly categorized in terms of:

• Corporate-owned facilities
• Server farms

From an operations standpoint, financial institutions, company headquarters, and other large corporate-owned entities operating their own data centers are especially concerned with energy monitoring and management, identifying and mitigating PQ issues, and optimizing their existing electrical distribution network to maximize data center performance output.

Independent data centers, commonly called server farms, play host to several, perhaps hundreds of clients that, for various operational reasons, have outsourced those requirements to third-party facilities specializing in providing the necessary infrastructure, maintenance, and other services to support HPC operations. In addition to energy management, power quality, and optimization, there is also the need to meter the load in order to fairly allocate energy usage costs to each using tenant.

Doing so requires a substantial level of resolution, or granularity, since a single rack may contain servers for multiple clients. In response, advanced metering products, including DIN rail configurations, are now entering the market that are specifically designed for installation in server racks and other confined spaces. Other “black box” metering solutions capable of monitoring up to three dozen individual loads simultaneously are also entering the HPC data center space and will be especially useful for meeting cost allocation requirements.

At this point, it may be useful to remind that meters themselves do not directly save energy; however intelligent integration of meters facilitates improved building performance which in turn reveals previously hidden energy savings opportunities through ongoing monitoring of a wide variety of facility measurements, including those shown in Table 1.

Submetering at the component level provides the greatest degree of granularity. Table 2 shows the various levels of facility metering from the macro (site / building) to the micro (system / component). Capturing more granular energy data typically requires a greater up-front metering investment and, due to the volume of data, a higher level of data storage and analysis capability.

DATA CENTER PERFORMANCE BENCHMARKING

The several data center productivity (DCP) metrics in use today vary in the way they characterize data center efficiency in terms of the IT (information technology) load compared to the total facility power load. Table 3 briefly lists the most commonly encountered data center performance metrics as they relate to:

• IT load — equipment used to manage, process, store or route data within the center includes servers, racks, KVM (keyboard/video/mouse) switches, monitors, workstations, and other equipment for monitoring and controlling the data center.

• Total facility power — incoming power measured at the utility service entrance. In addition to the IT load, this includes support loads such as chillers, computer room air conditioners (CRAC) and handlers (CRAH), UPS, switchgear, power distribution units (PDU), backup generators and battery strings, data center lighting, and all other facility power requirements.

ADVANCED METERING FOR DATA CENTERS

As data center operators look for new ways to cut costs, increase efficiency, and promote green building initiatives, they’re discovering the value of “smart” electric submeters for integrating HVAC, lighting, and other systems into the BMS. Other uses include predictive maintenance, energy efficiency studies, and cost reduction through accurate, real-time performance feedback from pumps, compressors, chillers, and more. Each metered element is critical for learning when, where, and how energy is being used in the facility, as the first step to achieving potential savings of thousands of dollars, enabled by, among others:

• Usage analysis and peak demand identification
• Time-of-use metering of electricity, gas, water, steam, Btuh, and other energy sources
• Cost allocation for tenant billing
• Measurement, verification, and benchmarking for energy initiatives, including LEED Energy & Atmosphere (EA) and Water Efficiency (WE) credits (Table 4)
• Net metering capability for use with renewable energy sources
• Load comparisons
• Threshold alarming and notification
• Multi-site load aggregation and real-time historical monitoring of energy consumption patterns for negotiating lower energy rates, and more

SMART METER FUNCTIONALITY

Recent developments in commercially available smart metering technology include built-in communications, advanced energy and PQ information, expanded communications, and other functions. In addition to meeting ANSI C12.20 national accuracy standards, some meters even provide simultaneous dual-protocol communications for integration with multiple BMS systems. This enables one meter to talk to two systems at the same time over industry-standard protocols, greatly enhancing the value of the meter investment by allowing the user to obtain granular meter data from a single device acting as a separate billing meter and also as an independent building automation system meter.

Other developments in smart metering include:

• Scrolling LCD indications of kWh; demand with peak date and time; power factor (%) per phase, real-time load in kW, amps per phase, and more. 
• Built-in RS-485 communications.
• NEMA 4X enclosures for both indoor and outdoor use.
• Communication media choices include Modbus RTU, Modbus TCP/IP, BACnet MS/TP, BACnet IP, or LonWorks (TP). 
• External inputs for gas, water, Btuh or other pulse-output meters, and a phase-loss alarm.
• Load control and pulse outputs for kWh and kilo volt amps reactive hours (kVARh).
• kWh delivered, received, and net kWh. 

Submeter communications are typically accomplished via proprietary energy analysis software and protocols or via pulse output into an energy management system. In both cases, the software resides on the user’s PC and communications are accomplished through a “hard-wired” system or a phone modem.

Hard-wired systems work through dedicated RS-485 cabling or through an Ethernet connection that uses an existing network. Ethernet communications do require an optional module and an internet protocol (IP) address. Using the RS-485 approach allows up to 4,000 ft of cabling to be run in the building. Available software is able to use all of these methods simultaneously and is easily set up to do this. One important thing to remember is that pulse-output electric, water, gas, steam, and other similar meter types have to be used with an interval data recorder (IDR)  to provide communications.

CONVERTING ENERGY DATA TO USEFUL INFORMATION

Once this interval data is collected, it can be imported into energy intelligence software and displayed pictorially on a meter “dashboard” to show how the facility load varies over time. Dashboards provide a graphical interpretation of real-time and historical meter data gathered at user-defined time intervals. This helps facility professionals gain a quicker, more intuitive grasp of their energy consumption and demand patterns for real-time decision making at the facility level, relative to load shedding, carbon footprint reduction, and other energy conservation and cost cutting measures of value to the bottom line.

When integrated into the BMS to monitor a broad range of facility performance parameters, the raw meter data is converted by software into real-time and historical snapshots of delivered energy (kWh), real power (kW), apparent power (kVA), power factor (%), current load (A), and line-to-line voltage (V) for:

• Heating
• Lighting
• Air handling
• Domestic hot water
• Back-up power (UPS)
• Plug load and more

The ability to view usage whenever desired, either on-site or remotely, allows facilities professionals to stay current with energy conservation programs, green building initiatives or overall efficiency analysis programs, by means of some or all of the following typical meter dashboard functions:

• Modbus TCP/IP communications.
• Carbon footprint dashboard displays of CO₂, SO₂, NOx, and other calculations by meter.
• User email alerts when daily kW/kWh usage alarm set points are exceeded.
• Storage and display of data from current day, previous seven days, and previous 30 days.
• Internal memory storage of up to 36 days of 15-minute interval data.
• Readability of electric, gas, water, steam, BTU, or other pulse output meters.
• Display of current weather conditions and short-term forecasts from the National Oceanic and Atmospheric Administration (NOAA).
• Comparative usage by square foot for the user’s own facility vs. regional averages around the country.
• Capability to drag and drop data for specified meters and time periods to create charts and display comparisons of all recorded parameters by meter.
• Data exportable to Excel or pdf formats.
• Support for static IP addresses.
• UL, CE, C-UL, CSA, and FCC agency listings.

IN SUMMARY

Visualization of granular energy data is the fastest growing area of metering technology. The Internet has introduced trendy visualization tools such as “energy dashboards” from a browser and complete energy graphical and statistical analysis in the cloud. PC-based packages still dominate the market for intense data crunching and security. Lobby displays are showing up more and more as facility managers want to show their customers, tenants, and employees that they are good corporate citizens and care about the environment. At the same time, these various visualization tools promote behavioral change regarding energy usage among the building’s occupants.

As first-level energy data acquisition tools, electric submeters — in conjunction with automatic meter reading (AMR) software solutions, web-enabled meter dashboards, and other energy monitoring and management solutions — can help facility operators to measure, verify, and report compliance with whatever requirements they may encounter. The old energy cliché — “you can’t manage what you don’t measure” and its corollary, “you can’t save what you don’t manage” — was never more critically important than in today’s increasingly energy-minded facility environment, especially HPC facilities like data centers. To that end, smart submeters offer facility managers an accurate, cost-effective tool for doing exactly that, while providing the communications flexibility and scalability to respond to evolving operational requirements.