power budget

TDP (Thermal Design Power) is the maximum amount of heat a processor generates under sustained workload — measured in watts, this specification determines cooling requirements and power delivery, directly impacting system design for AI workloads where GPU TDP ranges from 75W to 700W.

What Is TDP?

• Definition: Maximum heat output under sustained load, in watts.
• Purpose: Specifies cooling system requirements.
• Measurement: Sustained power, not peak.
• Relation: Roughly equals power consumption under load.

Why TDP Matters for AI

• Cooling Design: Higher TDP needs larger/better coolers.
• Power Delivery: PSU must supply TDP + headroom.
• Data Center: Determines rack density and cooling capacity.
• Operating Costs: Higher TDP = higher electricity bills.
• Thermal Throttling: Inadequate cooling reduces performance.

GPU TDP Comparison

AI/ML GPUs:

GPU              | TDP (W) | Memory    | Use Case
-----------------|---------|-----------|------------------
NVIDIA H100 SXM  | 700     | 80GB HBM3 | Training/Inference
NVIDIA H100 PCIe | 350     | 80GB HBM3 | Inference, lower power
NVIDIA A100 SXM  | 400     | 80GB HBM2e| Training/Inference
NVIDIA A100 PCIe | 300     | 80GB HBM2e| Inference
NVIDIA L40S      | 350     | 48GB GDDR6| Inference
NVIDIA L4        | 72      | 24GB GDDR6| Edge inference
AMD MI300X       | 750     | 192GB HBM3| Training

Consumer GPUs:

GPU              | TDP (W) | Memory    | AI Use
-----------------|---------|-----------|------------------
RTX 4090         | 450     | 24GB      | Dev, small training
RTX 4080 Super   | 320     | 16GB      | Development
RTX 4070         | 200     | 12GB      | Inference
RTX 3090         | 350     | 24GB      | Budget training

TDP vs. Power Consumption

Understanding the Relationship:

TDP: Design thermal envelope (sustained)
Peak Power: Can exceed TDP briefly
Idle Power: Much lower than TDP
Actual Power: Depends on workload

Example (RTX 4090):
TDP: 450W
Peak: ~600W (transient)
Typical gaming: 300-400W
Idle: 20-30W
LLM inference: 250-350W

Power Modes:

Mode           | Power    | Performance
---------------|----------|-------------
Full TDP       | 100%     | 100%
Power limited  | 70-80%   | 95%
Eco mode       | 50-60%   | 80%
Undervolted    | 80-90%   | 100%

Cooling Requirements

Cooling Solutions by TDP:

TDP Range    | Cooling Type          | Noise
-------------|----------------------|-------
<100W        | Single fan           | Low
100-200W     | Dual fan             | Medium
200-350W     | Triple fan/AIO       | Medium-High
350-500W     | Custom loop/blower   | High
500W+        | Liquid (rack/water)  | Varies

Data Center Cooling:

Cooling Type     | Capacity    | Density
-----------------|-------------|-------------------
Air cooling      | <30kW/rack  | Standard
Rear-door heat   | 30-50kW/rack| Medium density
Direct liquid    | 50-100kW/rack| High density H100
Immersion        | 100kW+/rack | Extreme density

Power Budget Planning

System Power Calculation:

Component        | Power (W)
-----------------|----------
GPU (H100 SXM)   | 700
CPU              | 200-350
Memory           | 50-100
Storage          | 25-50
Networking       | 25-50
Misc             | 50-100
System total     | ~1100-1350W

PSU requirement: 1.5× total = 1650-2000W

Rack Planning:

8× H100 SXM system: ~10kW
Per-rack capacity: 30-100kW depending on cooling
H100 systems per rack: 3-10
Data center power: MW to hundreds of MW

Efficiency Considerations

Performance per Watt:

GPU         | TDP  | FP16 TFLOPS | TFLOPS/W
------------|------|-------------|----------
H100 SXM    | 700W | 1979        | 2.83
H100 PCIe   | 350W | 1513        | 4.32
A100 SXM    | 400W | 312         | 0.78
L4          | 72W  | 121         | 1.68

Optimization:

- Power limiting (90% power → 98% perf typical)
- Undervolting for efficiency
- Workload-appropriate GPU selection
- Batch scheduling to maximize utilization

TDP specification is fundamental to AI infrastructure planning — understanding thermal requirements determines cooling design, power delivery, operating costs, and ultimately the density and efficiency of AI compute deployments.