Buying GPUs for AI: Capacity, Power, Cost Guide

This guide helps you decide which GPU to buy for AI workloads by explaining how to match GPU memory (VRAM) to model size, how to estimate power and cooling needs, and how to calculate the true total cost of ownership. You input your model size, training or inference type, and electricity rate, and the calculator below gives you a minimum VRAM estimate and a rough annual power cost for a single GPU.

Formula

The calculator uses two main formulas. For VRAM:

Minimum VRAM (GB) = (Model Parameters × 2 bytes × Precision Factor × Training Multiplier) + 2 GB overhead

• Model Parameters – the number of parameters in billions (e.g., 7 for Llama 2 7B).
• 2 bytes – because a 16-bit (FP16) parameter uses 2 bytes.
• Precision Factor – precision bits ÷ 16. For 8-bit, it’s 0.5; for 32-bit, it’s 2.
• Training Multiplier – 2 for training (because you need to store gradients and optimizer states), 1 for inference.
• 2 GB overhead – a rough allowance for activations and CUDA context.

For power cost:

Annual Electricity Cost = (GPU Power Draw (W) ÷ 1000) × Hours per Day × 365 × Electricity Rate ($/kWh) ÷ (PSU Efficiency ÷ 100)

GPU power draw is a user input in the calculator (default 350 W, typical for a high-end AI GPU). PSU efficiency is the 80 Plus rating (e.g., 90% for Gold).

Worked Example

Let’s use the defaults: a 7B model, 16-bit precision, inference mode, 90% PSU efficiency, $0.15/kWh, 12 hours/day.

1. VRAM: 7 × 2 × (16/16) × 1 + 2 = 14 + 2 = 16 GB
2. Power draw: assume 350 W (default).
3. Annual cost: (350/1000) × 12 × 365 × 0.15 / 0.90 = 0.35 × 4380 × 0.15 / 0.90 = 229.95 / 0.90 = $255.50

The calculator shows 16 GB VRAM and $255.50 annual electricity cost for those inputs.

Why VRAM Capacity Is the First Constraint

GPU memory (VRAM) determines which models you can load. A model’s weights must fit in VRAM, and during training you also need memory for gradients and optimizer states. For example, a 7B parameter model in FP16 needs about 14 GB just for weights. Inference with a 13B model in 8-bit needs about 13 GB. If your VRAM is too small, the model won’t load, or it will spill to system RAM and become painfully slow. Use the calculator to estimate the minimum VRAM for your model size and precision.

Power and Cooling: The Hidden Costs

AI GPUs are power-hungry. A single high-end GPU can draw 300–450 W under load, and a multi-GPU system can exceed 1500 W. That means you need a power supply with enough wattage and headroom, plus adequate cooling. A typical 350 W GPU running 12 hours a day at $0.15/kWh costs about $230 per year in electricity alone (at 90% PSU efficiency). Add cooling – if your room gets hot, you may need air conditioning, which adds to the cost. For a home setup, a 1000 W PSU is often enough for one high-end GPU, but two GPUs may require 1600 W or more. Always check the GPU’s TDP (thermal design power) and add 20–30% headroom for the rest of the system.

How to Choose: Matching GPU to Your Workload

Your workload dictates the GPU class:

• Inference (running a model): You need enough VRAM for the model, but you don’t need the fastest interconnect. A mid-range GPU with 16–24 GB VRAM (like an RTX 4090 or RTX 4080) is often sufficient.
• Fine-tuning (training a model): Training requires significantly more VRAM (about 2–4× the model size) and benefits from faster memory bandwidth. High-end consumer GPUs (RTX 4090) work for small models, but for larger models you may need professional GPUs (e.g., A100, H100) with 40–80 GB VRAM.
• Large-scale training (multi-GPU): This demands high-bandwidth interconnects (NVLink, InfiniBand) and data-center GPUs. Not for home use.

Also consider precision: using 8-bit quantization can cut VRAM in half, making larger models possible on consumer GPUs. But training in 8-bit is trickier and may reduce accuracy.

Common Mistakes and Misconceptions

• Confusing VRAM with system RAM: Your GPU has its own memory; the model must fit in VRAM, not system RAM.
• Forgetting the training multiplier: Training needs 2–4× more VRAM than inference. A 7B model that fits in 8 GB for inference may need 24 GB for training.
• Underestimating power supply needs: A 350 W GPU does not run on a 400 W PSU; you need headroom for the CPU, motherboard, and fans. A 750–850 W PSU is the realistic minimum for a single high-end GPU.
• Ignoring cooling: A GPU that throttles due to heat will run slower. Ensure your case has good airflow and consider aftermarket cooling if needed.

Total Cost of Ownership: Beyond the Purchase Price

The purchase price is only part of the cost. To estimate total cost over 3 years, add:

• Electricity: Use the calculator to estimate annual cost. A 350 W GPU at 12 h/day costs ~$230/year (at $0.15/kWh).
• Cooling: If you need extra AC, add ~$50–100/year.
• PSU upgrade: If your current PSU is insufficient, add $150–300.
• Depreciation: GPUs lose value over time, but they also retain resale value. Depreciation is a real cost if you plan to sell later.

For a $1500 GPU with $300/year electricity and cooling, the 3-year total is $1500 + $900 = $2400. That’s $800/year – comparable to cloud GPU rentals for moderate usage, but you get the benefit of ownership.

Alternatives to Buying: Cloud and Rental

If you only need a GPU occasionally, renting cloud instances (e.g., AWS, Google Cloud, Lambda Labs) may be cheaper. For example, renting a single A100 for 100 hours at $2/hour costs $200 – less than the electricity cost of owning one for a year. However, cloud rental has its own costs: data transfer, storage, and setup time. For continuous use (more than a few hours a day), owning is often more economical. Also, cloud GPUs are not always available due to demand.

Frequently Asked Questions

How much VRAM do I need for a 7B model?

For inference in FP16, you need about 14 GB plus overhead, so 16 GB is safe. For 8-bit, about 8 GB. For training, you need at least 24 GB, but 32 GB is more comfortable.

Can I use a gaming GPU for AI?

Yes, for small models and light fine-tuning. RTX 30/40 series GPUs work well. The main limitation is VRAM – most gaming GPUs have 8–24 GB, which limits model size. Also, gaming GPUs lack ECC memory and NVLink, but for hobbyist use they are fine.

What is the difference between FP16 and INT8?

FP16 uses 16-bit floating point, giving higher precision but using twice the memory of INT8 (8-bit integer). INT8 quantization reduces memory and increases speed but can slightly reduce model accuracy. For inference, INT8 is often acceptable; for training, FP16 is safer.

How do I know if my power supply is enough?

Add the TDP of your GPU to the TDP of your CPU, then add 100–200 W for the rest of the system. Multiply by 1.2 for headroom. For a 350 W GPU and a 125 W CPU, that’s (350+125+150)*1.2 = 750 W. So a 750 W PSU is the minimum.

Is it cheaper to rent a GPU or buy one?

It depends on usage. If you use a GPU for more than about 10 hours a week, buying is usually cheaper over 3 years. For occasional use, renting is better. Use the calculator to estimate your annual electricity cost and compare with cloud rental rates.

Related

• How a Crude Oil Refinery Turns Crude Into Fuel
• Do Museum Visitors Leave Chemicals on Artwork? Study Findings
• Dim a 3-Way Switch From Both Locations?

in model_size | Model size (billions of params) | B | 7 | 0 | 1000
in precision | Precision (bits) | - | 16 | 8 | 32
sel mode | Workload type | Inference=1, Training=2
in psu_efficiency | PSU efficiency (%) | % | 90 | 50 | 99
in elec_rate | Electricity rate ($/kWh) | $ | 0.15 | 0 | 1
in hours_day | Hours per day | h | 12 | 0 | 24
out vram_needed | Minimum VRAM (GB) | GB | model_size * 2 * (precision / 16) * (mode == 2 ? 2 : 1) + 2 | 0
out power_watts | GPU power draw (W) | W | 350 | 0 | 2000
out annual_cost | Annual electricity cost ($) | $ | (power_watts / 1000) * hours_day * 365 * elec_rate / (psu_efficiency / 100) | 0