Fix CUDA Out of Memory Errors Running Local LLMs

If you run large language models (LLMs) on your own GPU, you’ve likely hit a “CUDA out of memory� error. It happens when the model, its context, and the overhead of your inference code together exceed the VRAM on your graphics card. This guide walks you through the most effective fixes, from simple settings changes to deeper optimizations, so you can get your model running smoothly again.

What You’ll Need

• A local LLM setup (e.g., Ollama, llama.cpp, Hugging Face Transformers, or a custom PyTorch script)
• Basic command-line comfort (you’ll run a few terminal commands)
• Access to your model’s configuration files or launch scripts
• Optional: GPU monitoring tools like nvidia-smi (comes with NVIDIA drivers)

Step-by-Step

1. Check your current VRAM usage. Open a terminal and run nvidia-smi. Look at the “Memory-Usageâ€� column for your GPU. If it’s already near 100% before you start, other processes are hogging memory. Close any other GPU-heavy apps (game, browser with hardware acceleration, another model) and try again.
2. Reduce the context length. The context window (how many tokens the model can “rememberâ€�) directly scales VRAM usage. If you’re using a Hugging Face pipeline, set max_new_tokens lower (e.g., 256 instead of 1024). In llama.cpp or Ollama, lower the --ctx-size (e.g., 2048 instead of 8192). A smaller context uses significantly less memory.
3. Lower the batch size. If you’re processing multiple prompts at once, reduce the batch size to 1. In Transformers, set batch_size=1 in your DataLoader or generation loop. In llama.cpp, use --batch-size 1. This cuts memory usage dramatically.
4. Use a smaller model or a quantized version. If your GPU has, say, 8GB VRAM and you’re trying to run a 13B parameter model, it won’t fit. Switch to a 7B model, or better, use a quantized version (e.g., GGUF with 4-bit or 5-bit quantization). Quantization reduces model size by 4–8x with minimal quality loss. In Ollama, you can pull a quantized model like llama3:8b-instruct-q4_0.
5. Enable memory-efficient attention (if available). For Transformers, use attn_implementation="flash_attention_2" (requires the FlashAttention library) or attn_implementation="sdpa" (built-in). This reduces memory usage for the attention layer, which is often the largest consumer. Example: AutoModelForCausalLM.from_pretrained(..., attn_implementation="sdpa").
6. Offload layers to CPU. If you have enough system RAM, you can offload some model layers to CPU. In Transformers, use device_map="auto" which automatically splits layers between GPU and CPU. In llama.cpp, use --n-gpu-layers to specify how many layers to keep on GPU (e.g., --n-gpu-layers 20). This slows down inference but lets you run larger models.
7. Clear GPU memory before running. Sometimes leftover memory from previous runs isn’t freed. In PyTorch, call torch.cuda.empty_cache() before loading the model. Also, ensure you don’t have multiple model instances loaded. Restart your Python kernel or terminal if necessary.
8. Use gradient checkpointing (for training/fine-tuning). If you’re fine-tuning, enable gradient_checkpointing=True in your training arguments. This trades compute for memory, reducing VRAM usage significantly.
9. Update your software. Sometimes the error is due to a bug in older versions. Update your CUDA toolkit, PyTorch, Transformers, and the LLM server you use. For example, pip install --upgrade torch transformers.
10. If all else fails, use CPU inference. For very large models, consider running entirely on CPU with a quantized GGUF file. It’s slower but avoids CUDA errors entirely. In Ollama, you can set OLLAMA_NUM_GPU=0 to force CPU.

Common Mistakes

• Not checking other processes. You might have two instances of your model running, or a browser with hardware acceleration eating VRAM.
• Ignoring the context length. Setting a huge context (e.g., 32k) on a small GPU is a common cause. Start low and increase only if you have headroom.
• Using a non-quantized model. Many people try to load a full 16-bit model when a 4-bit version would work just as well for most tasks.
• Not clearing cache. PyTorch sometimes keeps memory allocated even after a model is deleted. Always call torch.cuda.empty_cache().
• Assuming the error is about model size alone. The error can also come from the generation buffer, so reducing max_new_tokens helps even if the model fits.

Tips

• Monitor VRAM in real-time. Run watch -n 1 nvidia-smi to see memory usage while your model loads and runs. This tells you exactly where the spike happens.
• Use the smallest model that meets your needs. For simple tasks, a 3B or 7B quantized model is often enough and runs comfortably on 8GB VRAM.
• Prefer GGUF models. They are designed for efficient memory usage and work with llama.cpp, Ollama, and LM Studio.
• Set environment variables. For PyTorch, set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128 to reduce fragmentation (try values like 64 or 256).
• Consider using a cloud GPU for very large models. If your hardware simply can’t handle the model, renting a GPU for a few hours is cheaper than upgrading.

Troubleshooting

If you’ve tried the steps above and still get the error, here’s what to do next:

• Check the exact error message. It may say “CUDA out of memoryâ€� or “CUDA error: out of memoryâ€�. If it’s the latter, your GPU driver might be crashing. Update your NVIDIA driver.
• Try a different quantization. If 4-bit fails, try 3-bit or 2-bit (e.g., Q3_K_M, Q2_K). Quality drops, but it might run.
• Reduce the model’s max sequence length. Some models have a default max length that’s huge. Set max_length=512 in generation to force a smaller buffer.
• Check for memory fragmentation. Run nvidia-smi -l 1 to see if memory is fragmented. If so, restart your GPU driver (or reboot).
• Try a different inference backend. If you’re using Transformers, try llama.cpp or Ollama. They are often more memory-efficient.
• If nothing works, check your GPU’s total VRAM. You may be trying to run a model that is simply too large. Use the model’s documentation to see the minimum VRAM requirement.

Frequently Asked Questions

Why does the error happen even when the model fits in VRAM?

The model weights are only part of the memory usage. During generation, the model also allocates memory for the KV cache (key-value cache) for each token in the context. A long context or many tokens generated at once can push you over the limit.

Can I fix this by adding more system RAM?

Not directly. CUDA out of memory refers to VRAM, not system RAM. However, you can offload layers to CPU, which uses system RAM. That can help, but inference will be slower.

Does using a smaller context length always reduce memory?

Yes, the KV cache scales linearly with context length. Reducing the context from 4096 to 2048 roughly halves the cache memory, which can be enough to avoid the error.

What is quantization and why does it help?

Quantization reduces the precision of the model weights (e.g., from 16-bit to 4-bit). This shrinks the model size and memory footprint, at the cost of a small accuracy drop. It’s the most effective way to run large models on limited VRAM.

Is it safe to use CPU offloading?

Yes, it’s safe. The model will run slower because data transfers between CPU and GPU are slow, but it won’t damage anything. It’s a good fallback if you need to run a model that’s slightly too big.

How do I know how much VRAM my model needs?

As a rough rule: a 7B model in 16-bit needs about 14GB, in 8-bit about 7GB, and in 4-bit about 3.5GB. Add 1–2GB for the KV cache and overhead. Check the model card for exact numbers.

Why does restarting my computer sometimes fix it?

Rebooting clears all GPU memory and resets the driver, removing any memory leaks or fragmentation that may have built up from previous runs.

Related

• Fine-Tuning vs Prompting: A Practical Team Guide
• Open Source vs Closed Source AI: Which Is Safer?
• Project Zomboid Error 1: How to Fix It