My first reaction when high density servers first came onto the market was that they were not really a problem as long as you managed their location in the data center and minimized the number of servers that you put in a rack. With power requirements of 3 to 5 kW per rack, or 80 to 100 watts per square foot, I felt like they could be managed without major redesigns of the data center. I still believe that’s true for the vast majority of users, due to the diversity of equipment in use in their data centers.

My first reaction when high density servers first came onto the market was that they were not really a problem as long as you managed their location in the data center and minimized the number of servers that you put in a rack.  With power requirements of 3 to 5 kW per rack, or 80 to 100 watts per square foot, I felt like they could be managed without major redesigns of the data center. I still believe that’s true for the vast majority of users, due to the diversity of equipment in use in their data centers.
 
However, for users that are planning to install only high density servers in their data centers, there are certain design requirements that I believe have to be met.  With new servers having power requirements approaching 15 to 25 kW per rack or 500 watts per square foot, they can only stand a very short interruption in cooling. My thoughts on how to design high density data centers have evolved as the density of the servers have evolved.  As a result, I don’t believe that you can continue to design data centers in the same manner as was done previously.

I was recently at a meeting with a chip manufacturer and was discussing one of the new chips they were working on.  It was smaller, faster, used more power and produced more heat, typical of the new chips that have come out recently.  What struck me was when I asked how long this chip would last without cooling.  The answer was 2 to 3 seconds before it would overheat and shutdown!  They were planning chips that basically required uninterrupted cooling.

How does this requirement impact the user?  Most data centers are designed so that during a power outage the cooling for the data centers drops off line.  While the data processing equipment is kept on line by the UPS, the cooling is not.  Once the generator starts and comes up to proper voltage and frequency, the cooling restarts.  This timeframe can range from 15 to 45 seconds depending on the design of the generator and cooling systems.  This 15 to 45 second timeframe without cooling would cause many of the new high density servers to overheat and shut down on thermal overload. 

So how do we design for uninterrupted cooling and why haven’t we done it in the past?    The primary reason we don’t put the cooling for the data center on the UPS systems are the size and type of loads as well as the cost.  UPS systems are not designed for motor loads such as chillers and large pumps.  Putting these loads on UPS systems would also double the cost of the electrical distribution systems.  We need a solution that provides uninterrupted cooling without doubling the size and quantity of the UPS systems.

One design solution would be to use chilled water cooling systems with a thermal storage tank.  Smaller circulating pumps could then be put on the UPS system to keep chilled water flowing for the 15 to 45 seconds until the chillers could be restarted.   The Computer Room Air Conditioning Units (CRAC) units would have to be put on the UPS as well, but these are basically just large fan coil units.  Having fans on UPS power would not be as significant an issue as having compressors, chillers and large chilled water or condensing water pumps.  The CRAC units would continue to run, they would continue to be supplied with chilled water and cooling in the data center would continue to operate in an uninterrupted manner.

The advantage of this type of design is its flexibility.  It would be adaptable for many of the cooling solutions being discussed today by the manufacturers.  These include liquid cooled servers and cabinets with heat exchangers built in.  While none of the servers currently on the market are liquid cooled, I would not be surprised to see them in the near future.  With densities continuing to increase, it is not inconceivable that they will exceed the capacity for air cooling before long.   While I do not look forward to data centers with a chilled water connection to every cabinet, I don’t discount the possibility either.  It is important that we design systems with the flexibility to adapt to potential changes.  Uninterrupted cooling is just one of the design changes that high density servers/data centers may require.  

Ron Hughes, President of California Data Center Design Group has been involved in the design, operation, construction testing and commissioning of data centers and technical office space for over twenty years. In the last 5 years alone, California Data Center Design Group, has managed the design of over 1,500,000 square feet of state-of-the-art data center/technical office space in the US, Europe, Asia and Mexico. He is currently designing several high density data centers and is working with the City of San Francisco and Stanford University on their long-term data center requirements.  A member of the High Density Data Center Design Committee, Ron recently moderated a panel on data centers of the future at the Data Center Dynamics Conference in San Francisco.