keepcalculator

06 Numbers and Powers

LLM Inference on Older GPUs: A Practical Guide

Card No. 06·82817 · Updated · Standard formula, shown below

Running large language models (LLMs) on older or fewer GPUs is a practical way to keep working during chip shortages or when hardware budgets are tight. This guide shows you how to configure your environment, quantize models, and use CPU offloading to get usable inference speeds on hardware that would normally struggle. You’ll need a Linux machine with an NVIDIA GPU (compute capability 6.0 or newer, i.e., GTX 10-series or later), at least 16 GB of system RAM, and enough free disk space for the model files.

What You’ll Need

  • A GPU with at least 4 GB of VRAM (older cards like GTX 1080 Ti or RTX 2060 work)
  • Python 3.8 or newer
  • CUDA toolkit compatible with your GPU driver (check nvidia-smi)
  • Hugging Face Transformers library and its dependencies
  • Bitsandbytes library for quantization (supports CUDA)
  • Enough system RAM to hold the model weights (for offloading)
  • Internet access to download model weights

Step-by-Step

  1. Check your GPU capabilities. Run nvidia-smi to see your GPU model and VRAM. Compute capability 6.0+ is required for most modern inference libraries. If you have a Pascal or newer card, you’re good.
  2. Install the required software. Create a Python virtual environment and install PyTorch with CUDA support (matching your CUDA driver version), then install the Hugging Face Transformers library and bitsandbytes. For example: pip install torch --index-url https://download.pytorch.org/whl/cu118 (adjust the CUDA version to match your driver). Then pip install transformers bitsandbytes accelerate.
  3. Choose a model that fits your memory. For a 8 GB GPU, consider a 7B parameter model (e.g., Llama 2 7B, Mistral 7B) quantized to 4-bit. For 4 GB, a 3B model (e.g., Phi-3-mini) is safer. Use the Hugging Face Hub to find model cards with quantization details.
  4. Load the model with 4-bit quantization. Use the bitsandbytes config to load the model in 4-bit precision. This reduces memory usage by about 75% compared to 16-bit. Here’s a minimal code snippet:
    from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
    import torch
    
    quant_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
    model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", quantization_config=quant_config, device_map="auto")
    tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
        

    The device_map="auto" lets the library place layers on the GPU and CPU as needed.

  5. Enable CPU offloading for large models. If the model still doesn’t fit, set device_map="auto" with offload_folder="offload" to offload some layers to CPU RAM. This is slower but allows running models larger than VRAM.
  6. Optimize generation settings. Use max_new_tokens to limit output length, and set use_cache=True (default) to speed up decoding. Reduce the batch size to 1. You can also use torch.compile if you have a recent PyTorch and GPU (Ampere or newer) to gain speed.
  7. Test with a simple prompt. Run a quick generation to verify everything works. Time the response to gauge performance. If it’s too slow, try a smaller model or more aggressive quantization.
  8. Consider using a CPU-only fallback. If your GPU is too old (compute capability < 6.0), you can still run LLMs on CPU using the llama.cpp project, which is highly optimized for CPU inference. This is much slower but works on any x86 machine with enough RAM.

Common Mistakes

  • Installing a CUDA toolkit version that doesn’t match your driver. Always check nvidia-smi for the driver version and install a compatible CUDA.
  • Using a model that is too large for your VRAM without quantization. A 7B model in 16-bit needs ~14 GB; in 4-bit it’s ~3.5 GB.
  • Forgetting to set device_map="auto"; without it, the model tries to load entirely on the GPU and fails with out-of-memory.
  • Not having enough system RAM for offloading. If you offload layers, they reside in RAM, so you need at least as much RAM as the model size.
  • Using a very old GPU (e.g., GTX 9-series) that doesn’t support bfloat16, causing errors with some quantization paths.

Tips

  • Use 4-bit quantization as a default; it gives the best memory/speed trade-off on older GPUs.
  • Set max_new_tokens to the minimum you need; every token takes time.
  • If you have multiple GPUs (even old ones), use device_map="auto" to distribute layers across all of them.
  • Use the accelerate library’s inference_mode to reduce memory overhead during generation.
  • For CPU inference, use llama.cpp with quantized GGUF models; it’s often faster than PyTorch on CPU.
  • Keep the GPU drivers updated; sometimes performance improves significantly.

Troubleshooting

  • Out-of-memory errors: Reduce the model size (use a smaller variant), increase quantization (4-bit to 3-bit), or enable more CPU offloading. Also reduce max_new_tokens and batch size.
  • Slow generation: Check if the GPU is actually being used (nvidia-smi during inference). If GPU utilization is low, you may be bottlenecked by CPU offloading. Try a smaller model or more quantization.
  • CUDA errors on import: Ensure the PyTorch CUDA version matches your driver. Use python -c "import torch; print(torch.cuda.is_available())" to test.
  • Model loading hangs: This often happens due to network issues. Set HF_HUB_OFFLINE=1 if you have the model cached, or check your internet connection.
  • Quantization errors: Some models have compatibility issues with bitsandbytes. Try loading in 8-bit first, or check the model card for known issues.

Frequently Asked Questions

Can I run a 7B model on a 4 GB GPU?

Yes, with 4-bit quantization, a 7B model uses about 3.5 GB of VRAM. You’ll need to offload some layers to CPU if you have other memory usage, but it’s possible.

Is CPU inference ever usable?

For small models (1-3B) and short outputs, CPU inference can be acceptable (a few tokens per second). For larger models, it’s very slow but still works for occasional use.

What’s the difference between 8-bit and 4-bit quantization?

8-bit uses half the memory of 16-bit, while 4-bit uses a quarter. 4-bit is more memory-efficient but can slightly reduce model quality. For most applications, the quality loss is minimal.

Can I use multiple old GPUs together?

Yes, using device_map="auto" will split the model across all available GPUs. This works even with different GPU models, though speeds will be limited by the slowest one.

Do I need a specific NVIDIA driver version?

You need a driver that supports CUDA 11.0 or higher (most modern drivers do). Check nvidia-smi for the CUDA version; then install PyTorch that matches.

What if I have an AMD GPU?

Bitsandbytes officially supports only NVIDIA GPUs. For AMD, you can use ROCm with PyTorch, but support is less mature. CPU inference is a universal fallback.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →