In figure 1, rack 1 depicts a ToR patching scenario between switch ports and servers without a structured cabling system. Rack 2 to rack 3 connections are indicative of point-to-point server-to-switch connections, also without a structured system. While proponents of these systems tout a decrease in cabling as a cost offset, further examination may negate such savings.
Efficient port utilization is often overlooked as a power and cooling inefficiency in these designs. In a point-to-point design, switch ports are dedicated to servers within a particular cabinet or within reach of short point-to-point connections, which in 10G SFP+ applications is 1, 3, or 5 meters (m) maximum in a passive system, which can cause an oversubscription of ports. If cabinet 1 needs only 26 server connections for the entire cabinet and a 48-port switch (ToR switching) or 48-port blade (point-to-point server to switch) is dedicated to the cabinet, 22 ports will be powered but unused. Most data centers do not allow direct patching between cabinets where active equipment is present by policy. So this would mean that any cord would need to be long enough to go out of the switch cabinet, into the pathways, and down into the next server cabinet. Short SFP+ style cables are not always long enough to cover this distance.
Another problem occurs when all full 48 ports are used. Adding even one new server would require the purchase of another 48-port switch. In this case, assuming two network connections for the new server, 46 unsubscribed ports would be added to the cabinet. Or in the case of dual networks for redundancy, 46 ports over two switches would be purchased but not used.
Further, many of these point-to-point technologies support limited channel length, ranging from 2 to 15 m. Short channel lengths may limit locations of equipment within the shorter cable range. With a structured cabling system, 10GBASE-T is supported over 100 meters of category 6A, 7, and 7A cabling—allowing more options for equipment placement within the data center.
One multinational retailer’s data center contains about 336 cabinets. The customer compared a structured system to a ToR system. In the ToR system, the purchase would have included two 48-port switches per cabinet, plus one for out of band management for a total of three network switches per cabinet. As the retailer had a significant investment in SAN directors and storage systems, this exercise was for the network components only. Due to power limitations, the number of servers per cabinet was capped at 12. The retailer chose the 48-port switches, which were less expensive than their 24-port counterparts due to software load. Due to regulative constraints, two physically disparate networks were required for communications.
The comparison for this customer was between ToR switches, quad attached to aggregate switches that were then attached to a pair of network core switches also via four connections each. The stated savings to customer B was $275,000 in copper cabling. The fiber cabling would be required in either scenario, although fewer strands and shorter runs are needed in a zoned approach, as they do not need to be run to every server cabinet but rather run to the zone.
ANY-TO-ALL STRUCTURED CABLING SYSTEM
The concept behind any-to-all is quite simple. Copper patch and fiber panels installed in each cabinet correspond to copper patch panels installed in patching zones or distribution areas. All fiber runs to one section of cabinets/racks in that same central patching zone. This allows any equipment to be installed and connected to any other piece of equipment via either a copper patch cord or a fiber jumper within the zone. The fixed portion of the channel remains unchanged. Pathways and spaces are planned upfront to properly accommodate the cabling.
These channels are passive and carry no recurring maintenance costs as realized with the addition of active electronics. If planned properly, structured cabling systems will last at least 10 years, supporting two or three generations of active electronics. The additional equipment needed for a point-to-point system will require replacement/upgrade multiple times before the structured cabling system needs to be replaced. The equipment replacement costs, not including ongoing maintenance fees, will negate any up front savings from using less/different cabling in a point-to-point system.
Figure 2 shows an example of an any-to-all structured cabling system. Fiber connections all arrive in the central patching/distribution zone in one location. This allows any piece of equipment requiring a fiber connection to be connected to any other fiber equipment port. For instance, if a server in a cabinet requires a fiber connection for a SAN on day one but needs to be changed to fiber switch connection at a later date, a fiber jumper change in the central patching zone is all that is required to connect the two ports.
The same is true for copper. As with fiber, any copper port can be connected to any other copper port in the central patching area within the zone. Switch ports are therefore not dedicated to cabinets that may not require them and active ports can be fully utilized. The ability to utilize all switch ports lowers the number of switches and power supplies and, therefore, decreases the networking investment, energy consumed, and ongoing maintenance. This depicts the distribution areas as outlined in TIA-942 and soon to be published draft 942A updates as well as the International standard ISO 24764.
In an any-to-all structured cabling scenario, the 48 ports dedicated to a single cabinet in a ToR design can now be divided to support any of several cabinets via the central patching area in each distribution zone. Where autonomous LAN segments are required, VLANs or address segmentation can be used to block visibility to other segments.
The table shows the equipment port allocations for both ToR and structured systems in the retailer’s 336- cabinet data center.
The limitation of the number of servers that can be utilized per cabinet is 12, based on power limitations of 4 kilowatt per cabinet, which is common in a passive cooling environment.
Moving to aggregate switches to support the edge (ToR) switches above would require 112 each 48-port switches in the point-to-point configuration, but that number would decrease to only 14 if the switches were zoned. This also decreases the need at the core where the aggregate switches uplink to two switch-blade cards as opposed to 12 in the top of rack scenario.
The retailer saved the cost of 504 ToR switches, 98 aggregate switches, and ten fiber-switch cards at the core. The savings paid for the cable plant and many of the servers that would later populate the cabinet. The savings in switch purchases also translates to lower power and, of course, lower active maintenance costs every year.
Virtualization is being implemented in many data centers to decrease the number of server power supplies and to increase the operating efficiency (kW/bytes processed or IT productivity per embedded watt [PEW] ratios within equipment. Increasing the number of power supplies (ToR) can negate virtualization savings and decrease the number of servers supported per cabinet.
Further, as servers are retired, the number of needed switch ports decreases. In a ToR configuration, this can increase the number of oversubscribed ports. In an any-to-all scenario, dark fiber or non-energized copper cables may exist, but these are passive, require no power, have no recurring maintenance/warranty costs, and can be reused for other equipment in the future.
Every port, network, storage, and management device contributes to the overall power requirements of a server. According to the U.S. Government Data Center Energy study from Public Law 109-431 signed December 20, 2006, approximately 50 percent of data center energy is for power and cooling, 29 percent is for server consumption, and only 5 percent is attributed to networking equipment. The remainder is divided into other systems.
From a networking standpoint, looking at port consumption or power draw varies greatly between various architectures (e.g., SFP+, 10GBASE-T, and fiber). Many of these reported power statistics do not show the entire consumption but rather make a particular architecture sound attractive by only reporting power based on consumption of an individual port, exclusive of the rest of the switch and the higher power server network interface card at the other end of the channel. The Tolly Group and The Uptime Institute, among others, provide end-to-end power and power efficiency matrices.
COOLING CONSIDERATIONS
Cooling requirements are critical considerations. Poor data center equipment layout choices can cut usability by 50 percent. Cooling requirements are often expressed as a function of power, but improperly placed equipment can wreak havoc on the best cooling plans. Point-to-point systems can freeze equipment positions. The ability to place equipment where it makes most sense for power and cooling can save having to purchase additional PDU whips, and in some cases, supplemental or in-row cooling for hot spots.
In point-to-point configurations, placement choices may be restricted to cabinets where open switch ports exist in order to avoid additional switch purchases rather than as part of the ecosystem decisions within the data center. Hot spots can be reduced with an any-to-all structured cabling system by allowing equipment to be placed where it makes the most sense for power and cooling.
In conclusion, while there are several instances where point-to-point ToR or EoR connections make sense, an overall study including total equipment cost, port utilization, maintenance, and power cost over time should be undertaken including both facilities and networking to make the best overall decision.
Reprints of this articleare available by contacting Jill DeVries at devriesj@bnpmedia.com or at 248-244-1726.
