If you’re running local LLMs on a GPU with 4-8GB of VRAM, the top pick is llama.cpp with a 4-bit quantized (Q4_K_M) model, using a context size of 2048-4096 tokens, and offloading as many layers as possible to the GPU. This combination gives you the best speed-to-quality trade-off, letting you run 7B-8B parameter models smoothly on modest hardware. The key is to keep the model and KV cache within VRAM while letting llama.cpp handle the rest on CPU.
What to Look For
When optimizing for limited VRAM, focus on these criteria:
- Quantization level – Lower bit quantizations (4-bit, 3-bit) drastically reduce VRAM usage with minimal quality loss. Q4_K_M is the sweet spot for most users.
- Context size – The KV cache scales linearly with context length. Reducing from 8192 to 2048 tokens can free up 1-2GB of VRAM.
- Layer offloading – With llama.cpp, you can offload some transformer layers to the GPU and keep the rest on CPU. More GPU layers = faster generation, but only if they fit.
- Model size – A 7B model at 4-bit uses ~4GB VRAM, while a 13B model at 4-bit needs ~8GB. Choose based on your VRAM.
- Backend/software – llama.cpp (with CUDA or Vulkan), Ollama, and LM Studio are the most VRAM-efficient options. Avoid full PyTorch-based loaders.
- Flash Attention – If supported, it reduces memory usage for the attention mechanism, allowing larger contexts.
Option 1: llama.cpp + Q4_K_M + 2048 Context
Strengths: This is the most flexible and VRAM-efficient setup. You control exactly how many layers go to GPU. For a 6GB card, you can offload 20-25 layers of a 7B model, keeping the rest on CPU, achieving ~10-15 tokens/sec.
Trade-offs: Requires command-line comfort. Setting up CUDA or Vulkan can be intimidating for beginners. Quality is slightly lower than 8-bit, but still very usable.
Best for: Users comfortable with terminal commands who want maximum control and performance.
Option 2: Ollama with 4-bit Models
Strengths: Ollama abstracts away all the complexity. You just run ollama run llama3:8b-q4_K_M and it handles quantization, offloading, and context. It automatically uses the best settings for your hardware. Also has a simple API for integration.
Trade-offs: Less control over layer offloading. You can’t fine-tune context size as easily (though you can set it in the Modelfile). May use slightly more VRAM than raw llama.cpp due to overhead.
Best for: Beginners and those who want a one-command solution without fiddling with settings.
Option 3: LM Studio with GPU Offload Slider
Strengths: LM Studio provides a graphical interface with a simple slider for GPU offload. You can visually see VRAM usage and adjust layers in real time. It also supports GGUF models and has a built-in chat UI. Great for testing different quantizations quickly.
Trade-offs: Slightly less efficient than raw llama.cpp because of the GUI layer. May have occasional bugs. The free version has some limitations on model downloads.
Best for: Users who want a visual, point-and-click experience without losing too much performance.
Option 4: 3-bit Quantization (Q3_K_M) for 13B Models
Strengths: If you have 8GB VRAM and want to run a 13B model, Q3_K_M uses ~5GB, leaving room for a decent context. This gives you better reasoning quality than a 7B model, even with lower quantization.
Trade-offs: Noticeable quality degradation compared to 4-bit. Some models become incoherent at 3-bit. You’ll need to test per-model.
Best for: Users with 8GB VRAM who prioritize model intelligence over speed and are willing to accept occasional gibberish.
Option 5: Reduce Context to 1024 for 4-bit 13B
Strengths: On an 8GB card, you can run a 13B model at Q4_K_M with a 1024-token context. This fits entirely in VRAM, giving you fast generation (20+ tokens/sec). Good for short Q&A tasks.
Trade-offs: Context is too short for long documents or multi-turn conversations. You’ll constantly hit the limit.
Best for: Single-turn queries or coding snippets where long context isn’t needed.
Option 6: Use CPU + GPU Hybrid with 8-bit Models
Strengths: If you have a lot of system RAM (16GB+) but limited VRAM, you can run 8-bit models with most layers on CPU. This gives better output quality than 4-bit, at the cost of speed (2-5 tokens/sec). llama.cpp supports this via --n-gpu-layers 0.
Trade-offs: Very slow. Only viable if you have patience or are doing batch processing. Not good for interactive chat.
Best for: Users who prioritize quality over speed and have plenty of system RAM.
How to Choose
First, check your VRAM. If you have 4GB, stick with 7B models at Q4_K_M and a 2048 context. If you have 6GB, you can push to 8B models or increase context to 4096. With 8GB, consider 13B at Q3 or 7B at Q8.
Next, decide your use case. For coding autocomplete, speed matters more than context. For long-form writing, you need more context. For general chat, balance both.
Common trade-offs: higher quantization = better quality but more VRAM. Longer context = more memory but slower. More GPU layers = faster but risk running out of memory (causing crashes).
Always test with a few models. Download a Q4_K_M and Q3_K_M of the same model and compare output quality on your specific tasks. Use nvidia-smi to monitor VRAM usage and adjust layers accordingly.
Frequently Asked Questions
What does Q4_K_M mean?
It’s a 4-bit quantization method that uses k-means clustering for weights. It offers a good balance between size and quality. The “K” stands for k-quants, a more advanced quantization scheme than older Q4_0.
Can I run a 7B model on 4GB VRAM?
Yes, with Q4_K_M quantization and a 2048 context, a 7B model uses ~4GB. You may need to offload some layers to CPU if it doesn’t fit exactly.
Why is my generation so slow?
You’re likely offloading too few layers to GPU. Check your VRAM usage and increase --n-gpu-layers until you hit the limit. Also, a smaller context size speeds up generation.
Should I use GGUF or GPTQ format?
For limited VRAM, GGUF (used by llama.cpp) is more flexible because it allows partial offloading. GPTQ (used by ExLlama) is faster but requires the whole model to fit in VRAM.
Does Flash Attention help?
Yes, if your GPU supports it (Ampere or newer), Flash Attention reduces memory usage for the attention layer, allowing longer contexts without extra VRAM.
What about 2-bit quantizations?
They exist (Q2_K) but quality degrades significantly. Only use if you’re desperate to fit a larger model. Most users find Q3_K the minimum acceptable.
Bottom Line
For most people with 4-8GB VRAM, the best settings are: llama.cpp (or Ollama) with a 7B-8B model quantized to Q4_K_M, context size 2048-4096, and as many GPU layers as fit. This gives you a balance of speed, quality, and memory usage. If you need more intelligence, try a 13B model at Q3_K_M with a short context. Always monitor your VRAM and adjust layers accordingly. Start with these defaults and tweak based on your specific hardware and tasks.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
More cards in this drawer
- Data Center Security: Encryption, Backups, Access
- Best Mini PCs for Gaming: Top Picks
- Phone Buttons: Clever Ways to Use Them
- Resonance Solstice Codes: What They Are
- Blue Protocol Star Resonance: Best Graphics Settings for Smooth FPS
- Jack Daniel’s Ransomware: Data Stolen?