In my last column, I used my quantum computing crystal ball to help prognosticate on 2015 developments. In thinking further about this year’s data center trends, every week seems to bring a new crop of announcements of hyperscale projects with supersized facilities that are measured in hundreds of thousands of sq ft and budgets that are expressed in hundreds of millions of dollars (in some cases near or over a billion dollars). There is no doubt that this trend is not new and is here to stay, since the primary forces driving these mega-centers are fed by seemingly endless market demand for more capacity and greater cost-effectiveness based on scale. Moreover, the move toward colocation providers and cloud computing has become a means of escaping from the capital cost and operational responsibility of a data center, especially if it is not a core business (and in some case even if it is), and has focused even more attention on total cost of ownership (TCO).
At the moment the ongoing price wars to gain market share by the giant cloud service providers are artificially suppressing or masking the true underlying cost their offerings. However, how actual build costs and operational cost-effectiveness of colocation and traditional enterprise data centers are calculated is another matter. When done properly TCO should include everything, starting with the relatively easy to quantify direct capital costs such as the land, building shell, and its basic power and cooling infrastructure and related systems, as well as the upfront cost of capital and its net future value. Then there are the recurring costs such as personnel, maintenance costs, equipment depreciation, network bandwidth, etc. Moreover, there is the proverbial elephant in the whitespace — energy costs — which can range from 2.5 to 25 cents per kWh in the United States, as well as the site PUE (which makes me wonder — what is the floor tile rating to support an elephant?).
We have seen different methods of how to weigh each component in TCO calculations (typically based on 10 years). However, in the past, the most commonly derived results have been traditionally stated in square footage — which essentially became nearly meaningless as power densities rose. There are a variety of case studies projecting the cost of building and operating a data center from different organizations, ranging from $2,500 to $25,000 per kilowatt of critical power and depending on many factors, but in virtually all cases they are related to the level of infrastructure redundancy.
Although each project is different, a 2008 white paper, “Cost Model: Dollars per kW plus Dollars per Square Foot of the Computer Floor,” by Uptime Institute, is a commonly cited reference. It estimates that costs range between $12,500 (Tier II) to $25,000 (Tier IV) per kilowatt (kW) for the power and cooling infrastructure. In addition, it allows for $300 per sq ft for the computer room whitespace and $190 per sq ft for the related supporting infrastructure space. The white paper states that these numbers were established using 2007 costs, which were based on actual projects costs (with an average design of 3 kW per rack and 15,000 sq ft of whitespace). It also reflected that these build costs would be affected by many factors such as the cost of labor in cities like New York or Chicago and therefore could vary by as much as plus or minus 30%. Moreover, the white paper postulates that in some instances reducing the whitespace by 50% would only result in a 5% reduction in cost. The white paper rightfully concluded that making budget estimates based on a traditional cost per square foot model alone was no longer effective and could result in “career limiting” decisions.
Typically, the rule-of-thumb for enterprise data centers using conventional raised floor with perimeter cooling (at 100 to 125 W/sq ft), generally results in 250 to 400 racks per 10,000 sq ft of whitespace. This equates to approximately 25 to 40 sq ft per rack. Using the Uptime Institute’s 2007 constructions costs above (and adjusting costs by 20% for 2015). This can be simplified to a blended number of approximately $450 per sq ft, ($360 plus 40% of $228) or $11,000 to $18,000, just for the floor space cost per rack, before adding the above mentioned cost per kilowatt.
Okay, so not everyone builds data centers based on Uptime Institute’s design requirements and costs will vary widely depending on many factors. Nonetheless, I believe that there is one basic, very essential, but overlooked cost benchmark that in my opinion is masked by the focus on these commonly accepted cost comparisons, that I refer to as “Rackonomics.”
SO WHAT IS RACKONOMICS?
Ultimately, the most basic denominator (and reason we build data centers), is the amount of IT equipment we can install and support, which generally is a direct function of number of racks and rack power density. I believe the number of racks, as well as their power density, is an effective IT payload unit. Therefore, racks should be the logical basic element of the TCO calculation, hence the “rackonomics” concept, as a more meaningful yardstick. It should be noted that I am not claiming to be the first person to coin the term “Rackonomics.” It originated with Blade Network Technologies (now part of IBM) in 2008. At the time it referred to the economies of using lower cost network switches by locating them in the rack (which is now a commonplace scenario), instead of a centralized network using expensive core switches.
The typical purchase cost of most bare cabinets ranges from $1000 to $2000, however the “real cost” could be 50 to 500 times that amount when all the build-out and TCO recurring costs are considered. In effect, the basic build-out cost to provide mission critical power and cooling for a 10 kW rack can range from $25,000 to $250,000. That is just the capital cost — obviously the recurring costs could be two to three times that for a TCO amortized over 10 years. While that is a wide range, I took the liberty of using literary license to guesstimate this as a 10 year TCO of $50,000 to $500,000 per rack (note that is for the standard black rack, the gold plated version is higher).
THE RACKONOMICS OF ZOMBIES
According to the Uptime Institute they conservatively estimated up to 10% of enterprise servers are running obsolete or unused software, have no function at all, yet remain in operation. Decommissioning a single 1U rack server can result in $500 per year in energy savings, an additional $500 in operating system licenses, and $1,500 in hardware maintenance costs. While by now most people are aware that there are “zombie” servers lurking in data centers that waste energy, however, they are more expensive than you may realize. So if a rack can have a TCO of $500,000 what is 1U of rack space worth? Assuming that a majority of cabinets are 42U high, that comes to $12,000 for each 1U zombie (plus the other costs cited above).
Clearly there are many different factors that go into this calculating the cost per rack, which are difficult to formularize. However, the issue of space and power density has almost always been a tradeoff from the perspective of the original design capacity vs. the actual whitespace population occupancy rate, especially over the life of the data center. Essentially you need to calculate the TCO based on your initial lower usage in the first few years, as well as when the site is more fully populated in the fifth, seventh, and tenth years. Moreover, a data center is not operated at 100% of design capacity for a number of reasons, primarily for ensuring equipment reliability and avoiding downtime. Depending on the organization culture, typically systems are operated at no more than 80% to 85% of design ratings before it is considered “full.” All of these issues raise the effective rackonomic cost factor, and if your average occupancy is only 50%, you may even bump into some million dollar racks.
The Green Grid (TGG) has looked at this from a somewhat similar perspective. In October of 2014 they released a new metric: space usage effectiveness (SpUE), which is intended to quantify this issue in relation to the other TGG well known metrics. According to their website:
“The Green Grid proposes this new metric to address one of the most important challenges that data center owners face, space usage in data centers. The new Space Usage Effectiveness (SpUE) metric enables data center operators to quickly assess the space, energy, water, and carbon sustainability aspects of their data centers, compare the results, and determine if any resource efficiency and/or sustainability improvements can be made.” See more at: http://www.thegreengrid.org
While the new SpUE metric does not directly address cost per rack, it does create a framework for assessing how effectively the whitespace is utilized, which should help focus the attention on this issue.
THE BOTTOM LINE
The bottom line of Rackonomics is surprising to most people, even those very familiar with data centers. So while most racks will not really cost their owners $500,000 each, perhaps it makes sense to start thinking on what you put into each rack, since each 1U space is worth much more than you thought. Of course not all data centers are designed for 10 kW per rack and many sites still average 3 to 5 kW per rack, however regardless of your density, the net cost per rack is significant.
Nonetheless, IT equipment will continue getting denser, more powerful and energy efficient with every generation. Yet, many organizations are hesitant to earmark their budgets to refresh IT hardware in less than three years, which has become the generally expected life of a volume server (or a lease). So think effective use of each rack, in terms the energy efficiency of the IT equipment and the productivity of the application, not just the energy efficiency the PUE of the facility, and start making the most of every U in each rack.
The Internet giants offer low cost cloud computing by optimizing their use of space and power, since they can rack and stack for maximum IT payload, as well as every other component of their TCO. You can be sure that colocation providers have mastered their own formulas for space and power, since that is the underpinnings of their business model. While it is unlikely that major universities will offer a Masters of Rackonomics degree any time soon, you should at least consider a self-study course in your own enterprise data center, the results may shock you.