This blog is the second in a two-part series. To read part one, click here or download the complete whitepaper below.
Space
High-density data centers emphasize the benefit of packing as much computing power into the smallest space, thereby lowering the cost to rent or buy additional physical space. Due to the cost of increasing a physical data center’s footprint, this may perpetuate the trend of moving towards higher-density cabinets.
In cities such as New York and London, the price of additional square footage would likely be a deterrent for maintaining low-density cabinets, but in major data center markets like Northern Virginia and Texas, this concern becomes moot. Where real estate does not come with an exorbitant price tag, the argument for lower-density data centers becomes stronger. The cost of additional floor space, however, is still partially responsible for the perception that it makes more sense to grow vertically rather than horizontally.
There is a commonly held belief that, with the exponential growth of data, it is necessary for data centers to grow physically. By this logic, data centers of low density will ultimately prove to be infeasible because the physical space that they require will become impractically large. It may seem logical to build vertically rather than horizontally, pack server racks to the brim and increase the power draw. However, this becomes an oversimplification when considering the nuanced complications of high-density cabinets, the potentials of virtualization, shared storage pools or disaggregation in a low-density facility.
Disaggregation efficiently separates servers and storage into smaller components to optimize resource consumption, and essentially delegates specific resources to designated components. Disaggregated servers and storage gear can be adjusted in accordance with changing storage requirements and computing workloads, removing the need to replace entire servers. Additionally, disaggregation can help to avoid crowding a data center with legacy hardware, which ultimately lowers data center density. Disaggregation has allowed Facebook to keep power density down. One example of this is found in the redesign of storage servers utilized for rarely accessed data. The environment can be optimized specifically for this rarely-touched content. Furthermore, with strides made in networking technology and achieving connectivity of 100-Gigabytes, it’s no longer necessary to have memory immediately next to a CPU. Again — through disaggregation — components are optimized independently to achieve greater efficiency, illustrating that high cabinet density is not necessarily a trend that holds weight.
Power Distribution
The distribution of power from upstream infrastructure to the cabinet level is the most significant one-time cost of support infrastructure for IT gear. The cost associated with the interior power distribution is driven by feeder size and length (breakers, conduits and cabling), distribution cables/busway tap boxes and the PDU/CDU within the cabinet. While the most common distribution feeders are 250A and 400A, we’ve seen an increase in demand for 800A and 1600A feeders. The quantity of cabinets served by a single 208A feeder varies.
It is possible to feed somewhere upward of 15 cabinets from a 400A/208V feeder if the average density per cabinet is 8kW. While 8kW is considered a low/mid-level density by the standards set by global cloud providers in today’s market, it remains a sweet spot for many, driven by the cost-per-cabinet footprint. Using the same 400A/208V feeder—but building to cabinet capacities of 17kW—you can only feed approximately 7-8 cabinets from a single feeder. This is another ROI sweet spot for higher capacities because 3 phase 60A distribution is still common and affordable. Both examples above are very common in data centers today. These distribution methods are widely adopted because of the relatively affordable cost of the distribution pathways and cable sizes, but more importantly because of the availability of many types of cabinet PDU’s with multiple outlet configurations and quantities.
The cost to distribute various capacity power circuits to a workload cabinet is shown in the table above. For densities over 17.2 kW, distribution becomes more expensive. At such high densities, components require customization. The availability of off-the-shelf products to support these densities makes us question why we often hear discussions of hyper-dense 20kW and above configurations, which will of course raise the cost of cooling. It is our belief that, in a practical world where power costs rule over real estate costs, 17kW and below configurations should be maintained. The components for these distribution densities are widely available , and the cost of real estate pales in comparison to the cost of the required power and cooling infrastructures.
Conclusion
Data centers often undergo changes and growth to keep pace with the speed of rapidly evolving technical advancements. Some providers have begun offering 415V delivered to the cabinet, which becomes convenient with low-density cabinets, as a higher number of cabinets can draw power from a single feeder. The constant search for efficiency with respect to both cost and compute power is perhaps the catalyst for the goal of increasing cabinet densities. In many cases, it becomes necessary to pack a cabinet to capacity, without having previously planned for it. This is the nature of technological growth, and often data centers must expect the unexpected. Competition for maximum cabinet density may ultimately prove to be inefficient and costly. It is our thought that it is cost-effective and operationally efficient to maintain cabinet densities under 17.2 kW, if possible. There are some converged configurations that may require power capacity over this threshold but we feel network, compute and storage designs should use these criteria as a starting point.
The Align Difference
Align brings over three decades of best-in-class service experience to every data center design and build. Our industry-certified technical professionals continuously improve and hone their skills to deliver cutting-edge expertise and sound advice at every step of the process.
Our data center designs offer flexible, budget-conscious plans, and Align’s experts are there every step of the way, from initial consultation to post-installation resources and documentation for your team. Align remains a leader in the field, enjoying an excellent rapport with a notable clientele and a reputation for going above and beyond. All this, in a turnkey, end-to-end solution with a single point of contact. We guarantee quality and keep your project management stress-free.
Whether you have questions about your cabinet density, or are just getting started in your search for a data center solution, Align’s engineers are here. Contact us today to speak with an Align data center design expert.
Quote Block. Image Credit: Align
