Many data centers, including most built before 2001, are at risk of outstripping their power and cooling capacities. Data centers already consume 10 to 30 times more energy per square foot than the typical office building — a figure that continues to grow. In fact, energy costs represent the single largest component of a data center’s operating expenses, and a potential barrier to future expansion. Compounding this challenge, today’s data centers are actively increasing agility and striving to maintain high standards of uptime while improving energy efficiency.

Several configurations are available for today’s topologies in the data center, and CIOs, data center professionals, and IT managers need to examine the pros and cons of each based on their specific needs and total cost of ownership (TCO).

There is no single end-all cabling configuration for every data center. Several switch manufacturers recommend a top of rack (ToR) configuration with access switches placed in each rack or cabinet. On the other hand, many data center environments can benefit from configurations like middle of row (MoR), end of row (EoR), or a traditional dedicated network row. These configurations use structured cabling with patch panels that serve as the connection point between switches and equipment.

Undertaking a study that looks at the impact of the various configurations on equipment, maintenance, and cabling costs; switch port utilization; manageability; scalability and upgrades; power consumption; and cooling considerations will help facilities and data center managers ultimately make the best educated decision.

COMMON CONFIGURATIONS

Popular for today’s pod-based designs where each row of server cabinets is dedicated to a particular purpose and upgrades are accomplished on a row-by-row basis, MoR and EoR configurations house access switches in cabinets located in the middle or at the end of the row of servers they support.

MoR and EoR use structured cabling where passive patch panels that mirror switch ports and server ports connect to corresponding panels in patching (or distribution) areas using permanent (or fixed) links. A dedicated network row uses the same concept but the switches and corresponding patch panels are typically housed in a separate row of cabinets that serve as the connection point between rows of core switches and rows of servers.

ToR configurations feature smaller (1 RU to 2 RU) access switches placed in each cabinet that connect directly to the servers via short preterminated small form-factor pluggable (e.g., SFP+ and QSFP) twinaxial cable assemblies, active optical cable assemblies, or RJ-45 modular patch cords, eliminating the use of patch panels and structured cabling for making connections. Figure 1 shows the four common configurations.

ToR configurations are often positioned as a replacement for structured cabling used with MoR, EoR, and dedicated network row options. It offers cabinet-at-a-time deployment, the ability to limit the use of copper cabling within cabinets, support for east-west (i.e., server-to-server) traffic, and cabinet-level management capabilities.

Structured cabling also offers several benefits, including reduced cost, maximum port utilization, improved manageability, scalability, reduced power consumption, and fewer cooling concerns. Let’s take a look at each of these factors that should be considered when evaluating ToR and structured cabling configurations in the data center.

EQUIPMENT, MAINTENANCE, AND  CABLING COSTS

In a ToR configuration with one switch in each cabinet, the total number of switch ports depends on the total number of cabinets, rather than on the actual number of switch ports needed to support the servers. In other words, if you have 39 server cabinets, you will need 39 ToR switches (or 78 if using dual primary and secondary networks for redundancy).

ToR configurations’ can significantly increase the amount of switches and power supplies required, compared to the use of structured cabling configurations that use patch panels to connect access switches to servers in multiple cabinets. All of these ToR switches also require ongoing maintenance, which further impacts cost.

As shown in Figure 2 and based on an actual 39-cabinet data center using separate dual networks, the cost for equipment, maintenance, and cabling for ToR ends up being more than twice that of structured cabling. The cost of the cabling itself is also more than double for ToR due to more expensive, and often proprietary, point-to-point cable assemblies.

SWITCH PORT UTILIZATION

Studies by DataCenter Dynamics and Gartner show that the average power supplied to a server cabinet ranges between 5 and 6 kW. Using this assumption, data center managers can typically fit 14 servers in a cabinet before running out of power. With just 14 servers in a cabinet, server switch port demand is typically lower than the 32 or 48 switch ports available on a ToR switch.

As shown in Figure 3, the same 39-cabinet example used in Figure 2 results in 1,404 unused ports with ToR. That equates to nearly 44 unnecessary switch purchases. Structured cabling allows virtually all active switch ports to be fully utilized because they are not confined to single cabinets. Using the patching area, switch ports can be divided up, on demand, to any of the server ports across several cabinets.

Of course, switch port utilization in a ToR configuration can be improved when a cabinet is able to support more servers. However, in high-density data centers reaching upwards of 20 kW per cabinet to support a full complement of servers, unused ports can still add up. For example, a 200-cabinet high-density server farm with 40 servers and a 48-port switch in each cabinet still results in 1,600 unused ports across the data center (i.e., eight unused ports per cabinet).

There are other configurations available that are outside the scope of this article that will have a different outcome on the number of unused ports and the amount of equipment and cabling. Regardless of the configuration being considered, it’s always best to calculate unused ports and ensure that they can be effectively managed.

MANAGEABILITY

With structured cabling, moves, adds, and changes (MACs) are accomplished at the patching area. Any switch port can be connected to any equipment port by simply repositioning patch cord or fiber jumper connections, creating an “any-to-all” configuration. The fixed portion of the channel remains unchanged, with the switches and severs left untouched and secure.

Because ToR switches connect directly to the servers in the same cabinet, all changes must be made within each individual cabinet rather than at a convenient patching area (Figure 4). Depending on the size of the data center, making changes in each cabinet can become complicated and time consuming. Imagine having to make changes in hundreds of server cabinets.

The SFP+ and QSFP twinaxial cable assemblies often used with ToR switches also limit the distance between the switches and the servers to a length of 7 meters in passive mode, which is more than sufficient for connections that reside in the cabinet. If needs change, these short lengths can restrict the location of equipment. The

cable lengths with structured cabling can be up to 100 meters, which will allow for flexible equipment placement throughout the life of the data center.

If there is a need to manage switches and servers separately, ToR configurations do not allow for physically segregating switches and servers into separate cabinets. However, a ToR configuration can be ideal when there is a need to manage a group of servers and their corresponding switch by application.

SCALABILITY AND UPGRADES

ToR configurations allow for cabinet-at-a-time deployments and upgrades, which can be a preference for some data centers, depending on the budget and business model in place. Once several cabinets are deployed, a widespread switch upgrade in a ToR configuration obviously will impact many more switches than with structured cabling. An upgrade to a single ToR switch improves connection speed to only the servers in that cabinet. With structured cabling, a single switch upgrade can increase connection speeds to multiple servers across several cabinets.

When using ToR switches with SFP+ and QSFP cable assemblies, these interfaces do not support auto negotiation — the ability for the switch to automatically and seamlessly switch between different speeds on individual ports depending on the connected equipment.

Supported by backwards-compatible twisted-pair cabling used with structured cabling, auto negotiation enables partial switch or server upgrades on an as-needed basis. Without it, a switch upgrade requires all the servers connected to that switch to also be upgraded, incurring full upgrade costs all at once.

Some equipment vendors may require the use of proprietary SFP+ and QSFP cable assemblies, which are not interoperable and can require cable upgrades to happen simultaneously with equipment upgrades. Data center managers value interoperability to leverage their existing cabling investment during upgrades regardless of which vendors’ equipment is selected (see sidebar). SFP+ and QSFP cable assemblies are also typically more expensive than twisted-pair patch cords, causing additional cost considerations during upgrades.

POWER CONSUMPTION

Power consumption is one of the top concerns among today’s data center managers as energy costs continue to rise, power becomes more costly, and green initiatives take center stage. Reducing the number of switches and power supplies helps reduce energy costs while contributing to “green” initiatives such as LEED®, BREEAM, or STEP. When there are fewer switches and power supplies in a data center, it is also easier and less of a capital expenditure to upgrade to more energy-efficient equipment hitting the market.

The ability to use all switch ports with structured cabling significantly lowers the number of switches and power supplies required. Plus, the patch panels used with structured cabling are passive and require no power. In Figure 2, the ToR configuration required a total of 90 switches and 90 power supplies, while structured cabling required only 42 switches and 42 power supplies.

When switches reach full utilization in a ToR configuration, adding a new server to the cabinet can require an additional switch and associated power supplies. This can take away from potential power that can be allocated for servers. Even with virtualization reducing the number of servers and related power and cooling, the increased number of power supplies required with ToR can potentially negate some virtualization savings.

It is important to remember that even in an idle state, unused switch ports can consume power. Furthermore, small access switches like those used in ToR configurations process more instructions in their central processing units. This can cause potential unanticipated power spikes. Regardless of the configuration, power consumption varies by switch model and manufacturer, and actual port consumption or power draw should always be examined across the entire data center.

COOLING CONSIDERATIONS

While a ToR switch can technically be placed in the middle or bottom of a cabinet, they are most often placed at the top for easier accessibility and manageability. According to the Uptime Institute, the failure rate for equipment placed in the top third of the cabinet is three times greater than that of equipment located in the lower two thirds. In a structured cabling configuration, the passive patch panels are generally placed in the upper position, leaving the cooler space for the equipment.

ToR designs can also land-lock equipment placement due to the short cabling lengths of SFP+ and QSFP cable assemblies and data center policies that do not allow patching from cabinet to cabinet. This can prevent placing equipment where it makes the most sense for power and cooling.

For example, if the networking budget does not allow for outfitting another cabinet with a ToR switch, placement of a new blade server may be limited to where network ports are available. This can lead to hot spots, which can adversely impact neighboring equipment within the same cooling zone. Structured cabling configurations avoid these problems.

CONCLUSION

Many factors impact the choice of which cabling configuration to deploy, and there is no single end-all solution.

ToR configurations reduce the amount of structured cabling and make a lot of sense for small server rooms when cabinet-at-a-time deployment or cabinet-level management is required, or where a higher number of unused ports can be managed. However, data center managers should carefully consider all the pros and cons as they relate to TCO.

Structured cabling configurations can offer lower equipment, maintenance, and cabling costs; maximum port utilization; easier manageability, scalability and upgrades; and reduced power consumption and cooling considerations.

ToR is often positioned as a replacement for structured cabling, but in many instances, structured cabling must coexist with ToR to support central switching for KVM or other in-band or out-of-band management and data center monitoring. 


SIDEBAR 1:

Switch Vendors’ SFP+ and QSFP Cable Assemblies Can Lock Data Center Managers into a Proprietary Solution

Some switch vendors require their proprietary cable assemblies for connecting ToR switches to servers when using SFP+ and QSFP twinaxial cable assemblies. Some ToR switches are even designed to check vendor security IDs on the cables connected to each port and either display errors or prevent ports from functioning when connected to an unsupported vendor ID.

While these requirements help ensure that vendor-approved cable assemblies are used with corresponding electronics, it can limit data center design options by locking data center managers into a proprietary solution. This is a substantial change from the standards-based interoperable and backwards-compatible fiber and copper connectivity successfully deployed in data centers for decades.

Despite these proprietary cable assemblies, third-party independent testing by University of New Hampshire's Interoperability Lab (UNH IOL) proved that passive SFP+ and QSFP twinaxial cable assemblies from cabling vendors pass interoperability testing with several vendors’ ToR switches that are designed to display errors. These tests demonstrate that more expensive proprietary cables are not necessarily required, which may not be the case for switches designed to actually prevent ports from functioning altogether.

In addition, many of the proprietary cable assemblies required by switch vendors come with an average 90-day warranty. Depending on the cable vendor, structured cabling typically carries a 15- to 25- year warranty.

SIDEBAR 2:

Structured Cabling vs. Top of Rack in Action

There have been several cases of enterprise data center managers carefully reviewing the impact of ToR configurations on TCO and saving significant cost, both now and in the future, by choosing structured cabling.

One large global retailer recently decided to deploy structured cabling for a critical upgrade project that encompassed 400 server cabinets. By deploying seven central patching (distribution) areas with structured cabling, the retailer was able to place access switches to support servers in multiple cabinets. This significantly reduced the number of switch ports required, saving more than $3 million at current gigabit speeds, with a projected savings of $18 million for future 10 gigabit speeds. The number of ports required on aggregation and core switches to support the access switches was also reduced. The configuration provided excellent functionality, manageability, and bandwidth.

When a leading insurance provider embarked on consolidating three mission-critical data centers into one 20,000-sq-ft space, they first chose to deploy a ToR configuration with dual primary and secondary networks across more than 300 cabinets. With a limited power capacity per cabinet, the insurance provider was ultimately faced with more than 24,000 unused switch ports across the data center, which equated to more than 500 switches that they did not need.

After carefully weighing the pros and cons and the $2 million more required for the ToR solution, the insurance provider worked with its designer and cabling vendor to deploy a traditional structured cabling approach. Using standards-based distribution areas, they were able to centralize the access switches and corresponding patch panels to support banks (pods) of server cabinets. They also realized that using a structured cabling configuration could save another $11 million in the event that they migrate to 10 gigabit speeds between access switches and servers.

While moving to a structured cabling approach meant more twisted-pair structured cabling and less fiber backbone cabling for the insurance provider, the cost of cabling was much less than the cost of the unused ports. In fact, the cost of cabling the new data center turned out to be only 10% to 15% of the enormous savings that they would reap by choosing structured cabling over ToR.