This guide walks you through evaluating the liquid cooling requirements for an AI data center, covering heat load estimation, cooling technology selection, flow rate and pressure drop calculations, and redundancy planning. You will need facility floor plans, server rack power densities, and ambient temperature data to get accurate numbers.
What You’ll Need
- Server and GPU thermal design power (TDP) specifications
- Rack layout and power distribution diagrams
- Facility water supply temperature and quality reports
- Chilled water or cooling tower performance curves
- Pump curves and piping material specs
- Ambient temperature and humidity historical data for your location
Step-by-Step
- Calculate total heat load. Sum the TDP of every GPU, CPU, and other heat-generating component in each rack. For example, a rack with 8 GPUs at 700W each plus 2 CPUs at 350W each gives 8×700 + 2×350 = 6,300W. Multiply by the number of racks to get the total IT heat load. Add losses from power distribution units (typically 2–5% of IT load) and any auxiliary equipment.
- Determine cooling technology based on rack density. For racks above 10 kW, air cooling becomes impractical. Use direct-to-chip cold plates for 10–40 kW per rack, and consider immersion cooling for densities above 40 kW per rack. For densities below 10 kW, rear-door heat exchangers or traditional air cooling may suffice.
- Define supply water temperature. For direct-to-chip cooling, typical supply temperatures are 25–35°C (77–95°F), which allows year-round free cooling in many climates. For immersion, supply temperatures can be 35–45°C (95–113°F) depending on the fluid. Higher supply temperatures reduce chiller energy but may require higher flow rates to maintain component temperatures.
- Calculate required flow rate. Use the formula: Flow (L/min) = Heat Load (kW) × 14.3 / (ΔT × 4.18), where ΔT is the temperature rise of the coolant (typically 5–10°C). For example, a 100 kW rack with a 7°C ΔT needs 100 × 14.3 / (7 × 4.18) ≈ 48.9 L/min. Convert to GPM by dividing by 3.785.
- Estimate pressure drop. Sum the pressure drops of all components in the loop: cold plates (typically 0.5–2 psi each), manifolds (1–3 psi), pipes (use Darcy–Weisbach or Hazen-Williams), and quick disconnects (0.5–1 psi). Ensure the pump can provide the total pressure at the required flow rate. Use pump curves to verify operating point.
- Design the coolant distribution unit (CDU). The CDU isolates the facility loop from the IT loop. Size it to handle the total heat load with a safety factor of 10–20%. Choose between water-to-water or water-to-refrigerant CDUs based on facility chilled water availability. For water-to-water, the facility side flow is typically 1.5–2 times the IT side flow.
- Plan for redundancy. Design with N+1 pumps and CDUs, and consider dual power feeds. For critical AI workloads, consider 2N redundancy for CDUs. Ensure that if one pump fails, the remaining pumps can still handle the full heat load at acceptable temperatures.
- Check water quality and corrosion control. The coolant should have low conductivity (< 1 µS/cm for direct-to-chip) and be treated with corrosion inhibitors. Use deionized water or a propylene glycol mixture (20–30%) for freeze protection. Plan for regular water testing and filtration.
- Calculate thermal performance. Determine the maximum allowable component temperature (e.g., GPU junction at 85°C). The coolant supply temperature must be low enough to keep the component below this limit, considering the thermal resistance of the cold plate and thermal interface material. Use vendor-provided thermal resistance values.
- Assess facility integration. Verify that your facility’s chilled water system can handle the additional heat rejection. For free cooling, check that the cooling tower or dry cooler can reject heat at the required supply temperature. Calculate the required cooling tower capacity in tons (1 ton = 3.517 kW).
- Document and validate. Create a detailed specification with all calculated values, then run a pilot test on a single rack to validate flow rates, pressure drops, and temperatures. Measure actual values and adjust the design accordingly.
Common Mistakes
- Underestimating heat load by ignoring power supply losses and auxiliary equipment.
- Assuming all racks have the same density; mixing high and low density racks can cause cooling imbalances.
- Oversizing flow rate, leading to unnecessary pump energy and erosion.
- Neglecting pressure drop in quick disconnects and flexible hoses, which can be significant.
- Using tap water without treatment, causing corrosion and biological growth.
- Ignoring the temperature rise across the facility loop, which reduces the effective cooling capacity.
Tips
- Use a simulation tool or spreadsheet to model the entire cooling loop before installation.
- Consider using a higher supply temperature (e.g., 35°C) to maximize free cooling hours.
- Install pressure and temperature sensors at key points for monitoring and optimization.
- Design piping with smooth bends and adequate diameter to minimize pressure drop.
- Plan for future expansion by leaving spare capacity in CDUs and pumps.
- Work with your server vendor to get precise thermal resistance data for cold plates.
Troubleshooting
- Insufficient cooling (component temperatures too high): Check flow rate and supply temperature. Increase pump speed or reduce supply temperature. Verify that all cold plates are properly seated and thermal paste is applied correctly.
- High pressure drop: Check for kinked hoses, clogged filters, or undersized pipes. Use a differential pressure gauge to isolate sections.
- Leaks: Inspect all connections and fittings. Use pressure testing with air or water before filling the loop. Tighten fittings to manufacturer torque specs.
- Pump cavitation: Ensure the pump inlet pressure is sufficient. Increase the reservoir height or reduce suction line length.
- Water quality degradation: Test conductivity and pH regularly. Replace filters and add corrosion inhibitors as needed.
- Free cooling not working: Verify that the cooling tower or dry cooler can reject heat at the current ambient temperature. Check that the bypass valves are operating correctly.
Frequently Asked Questions
What is the typical flow rate for a 40 kW rack?
For a 40 kW rack with a 7°C temperature rise, you need about 40 × 14.3 / (7 × 4.18) ≈ 19.6 L/min (5.2 GPM).
Can I use regular water in a liquid cooling loop?
No, regular tap water contains minerals and bacteria that can cause corrosion and fouling. Use deionized water with corrosion inhibitors or a propylene glycol mixture.
What is the difference between direct-to-chip and immersion cooling?
Direct-to-chip uses cold plates attached to the processor, while immersion submerges the entire server in a dielectric fluid. Immersion handles higher densities but requires more complex maintenance.
How do I calculate the cooling tower capacity needed?
Divide the total heat load in kW by 3.517 to get tons of refrigeration. Add a safety factor of 10–20% for ambient conditions and fouling.
What supply water temperature should I use?
For direct-to-chip, 25–35°C is common. Higher temperatures save energy but require higher flow rates. Check your server’s maximum allowable inlet temperature.
How often should I check water quality?
At least monthly, or more frequently if the system is new. Monitor conductivity, pH, and microbial growth.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
More cards in this drawer
- Microsoft Surface Duo: The Foldable Phone Story
- Samsung SM-T813 Hard Reset: Step-by-Step Guide
- Mafia Clash Codes: Real Meanings & Usage
- Remove Programs from Mac: Full Uninstall Guide
- Project Sentinel Morning Digest: Data Stack Failure Alerts
- Best KVM VPS Hosting: Top Picks & Guide