The following blog is an excerpt from our whitepaper, "Evaluating Data Center Power Density."
Over the past few years, IT equipment has continued to move towards increased power packaged in decreasing sizes. A similar tendency has emerged in data center design, using less and less white space while simultaneously packing cabinets to capacity. Bragging rights are granted to whoever has deployed the cabinet running at the highest kW value. It is consistently the talk of industry conferences, and raising the bar on cabinet capacity has become a common goal. We often see 48U+ cabinets delivered to data centers by the dozens, stacked high with powerful 1U servers. What’s the driving force behind this?Several application workloads might demand a higher capacity, but beyond sporadic use cases, there is a significant cost impact to distributing and supporting these hyper-dense cabinets that must be considered. We’ve explored the cost impact of densifying cabinets versus expanding into inexpensive data center space to provide for cabinets that are not stacked to physical capacity. Contrary to what the popularity of high-density designs suggests, it is frequently more cost efficient to maintain cabinet densities that can be supported with standard off-the-shelf electrical distribution products.
High-Density vs. Low-Density
To begin, it is necessary to define a high-density data center. Data center density is measured by the number of kilowatts consumed per cabinet. To increase density to meet computing resource requirements, it’s essential to increase the power draw per cabinet or square foot.
In today’s data centers, cabinets consuming an average upwards of 10kW are considered high-density. Data center density may also be defined by the number of watts consumed per square foot of white space; by this standard, over 150kW per square foot is high-density. The Association for Computer Operations Management (AFCOM), a leading association for data center professionals, segments data center density into four categories:
In the past, enterprise data center infrastructures were widely expected to increase rack consumption, targeted at 30 kW on average. Throughout the industry, this expectation has largely not been met, and power consumption per rack is typically kept at the medium to high range. A notably successful mid-density implementation can be found at Facebook’s data centers. Facebook targets only 5.5kW per rack on average, and they have been able to scale exponentially without increasing power density. Facebook was able to do this by rethinking their approach to how resources are consumed. The company’s Open Compute Project aims specifically to make computing hardware more efficient, scalable and modular using disaggregation.
Conversely, Intel has been successful in taking the opposite approach: a 5,000 square foot hyper-density data center. Racks can reach an average power draw of up to 43 kW. Intel reports that this data center has a total power capacity of three of their legacy data centers combined. The purpose of this facility is to enable significant growth in computing demand required by electronic design automation tools. While impressive, this is a particular use case, with specific workloads needed for complex silicon chip design and testing. Additionally, this facility required a custom rack design.
Despite mid-density successes like that of Facebook’s, and the evidence that high-density data centers like Intel’s are highly complex and customized, the belief that high-density cabinets are more efficient persists for a variety of reasons, some of which may include management viewing partially filled racks as wasteful due to unused space. Another view is that limiting the amount of cabinets necessary will reduce the cabling footprint. While true as far as lengths of cabling and total number of termination points, our view is that increasing the amount of cabling needed to support a single cabinet footprint may ultimately inhibit airflow and cooling efficiency, while also creating difficulty in day-to-day operations and maintenance of the cabling infrastructure.
There are many variables that must be factored into the design criteria for varying cabinet densities in a data center facility. For cost analysis, we will focus on the interior of the white space and power distribution. For the purposes of this article, consider a hypothetical 1MW facility to compare the costs of various power densities. This will allow us to examine how increasing densities affect the per-cabinet cost and the overall cost to fit out the space.