An 8 GB GPU can run small, quantized local language models; 12 GB offers more room for 7B–8B models, 16 GB makes those models comfortable and can fit some 13B–14B models, and 24 GB can accommodate many 30B–32B models at low-bit quantization. These are practical estimates, not hard guarantees: model format, context length, software overhead, and whether the GPU also drives your display all affect what fits.
What VRAM actually limits
VRAM is the GPU’s fast memory. For local inference, the main demand is the model’s weights, plus memory for the conversation context (the KV cache), temporary computation, and the runtime itself. A model that barely fits on paper may fail to load or run out of memory once those extra allocations are needed.
Parameter counts such as 7B or 32B refer approximately to billions of learned weights, not a direct VRAM requirement. The weight data is often the largest component, but it is not the whole allocation. A model’s actual file size is a useful starting clue, not a promise that it will run in the same amount of free VRAM.
“Run” can also mean different things. You may load a model with some layers on the GPU and the rest on system RAM, but this is offloading, not a fully VRAM-resident model. It can work, though generation may be substantially slower and performance depends on the CPU, memory bandwidth, and software.
Rough model sizes by VRAM
The table assumes a quantized model, a moderate context, and enough spare memory for runtime overhead. Quantization reduces the bits used to store weights; Q4 is a common shorthand for roughly four-bit weight quantization. Exact file formats and quantization methods differ, so check the model’s download size and leave headroom.
| GPU VRAM | Typical starting point | What may fit with trade-offs |
|---|---|---|
| 8 GB | Small 3B–4B models, often at higher quantization quality | Some 7B–8B models at low-bit quantization and shorter context |
| 12 GB | 7B–8B models with more room for context | Some 13B–14B models at low-bit quantization or with reduced context |
| 16 GB | 7B–8B models at less aggressive quantization or longer context | Some 13B–14B models; selected larger models with heavy quantization or offloading |
| 24 GB | 13B–14B models with useful headroom | Many 30B–32B models at low-bit quantization, depending on context and implementation |
These bands are deliberately approximate. For example, a 7B model stored at four bits may have weights around 4 GB before metadata and other overhead, while a 32B model at four bits has a much larger weight allocation. The effective bits per weight are not always exactly the advertised number, and some formats add scales or other data.
Why context length and quantization matter
Context length is the amount of text the model can consider at once, including the prompt and prior conversation. Longer context generally increases KV-cache memory. The cache’s size depends on the model architecture, precision, batch settings, and runtime; therefore, there is no single reliable “VRAM per 1,000 tokens” figure that applies to every model. If a model loads at a short context but fails when you increase it, the cache is a likely reason.
Lower-bit quantization shrinks weight storage, often letting a larger model fit, but can affect output quality. The size and quality impact depend on the model and quantization method; a lower bit count is not automatically unusable, nor does it guarantee identical results. When choosing between a larger, more-quantized model and a smaller, less-quantized one, compare the task, model quality, context needs, and available memory rather than parameter count alone.
Also reserve memory for the desktop, other GPU applications, and inference buffers. A card advertised with 12 GB does not necessarily have all 12 GB available to a model. Leaving roughly a gigabyte or more free can be a sensible initial allowance, but actual overhead varies, so verify the runtime’s reported allocation.
How to choose a model for your GPU
- Check usable VRAM. Look at free GPU memory while your normal applications are open, not only the card’s listed capacity.
- Choose a model and quantization. Check the exact file size and supported format in the inference application. Treat the file size as a baseline, then allow space for cache and runtime use.
- Set a realistic context. Start with the context you actually need. Increase it gradually and watch memory use rather than selecting the maximum setting by default.
- Test a representative prompt. Loading successfully is not the same as handling your intended conversation length. Try a typical prompt and monitor whether the model stays on the GPU.
- Compare speed as well as fit. A model that barely fits may be less pleasant to use than a smaller model with room for a longer context and fewer memory constraints.
For a first local setup, the most reliable choice is usually a model whose weights take noticeably less than the available VRAM. That margin helps avoid out-of-memory errors when the cache grows or the application allocates additional buffers.
Common limits and misconceptions
- System RAM is not VRAM. Offloading can make a model technically runnable, but data moving between system memory and the GPU can reduce speed. Unified-memory systems behave differently from a discrete GPU and need their own measurements.
- More parameters do not always mean a better answer. Model training, architecture, and task fit matter too. A smaller model can be a better practical choice if it responds faster or handles your task well.
- Published context limits do not promise a fit. A model may support a large context in principle while your particular GPU cannot hold its cache at that setting.
- Two files with the same model size can differ. Quantization format, metadata, architecture, and runtime support affect both memory use and compatibility.
If you hit an out-of-memory error, reduce the context or batch size, close other GPU workloads, choose a smaller or more-quantized file, or enable partial CPU offloading if the software supports it. Restarting the application can clear allocations left by previous sessions, but it will not make a fundamentally oversized configuration fit.
Frequently Asked Questions
Can an 8 GB GPU run a 7B model?
Often, yes, with a suitable low-bit quantization and a moderate or short context. It is not guaranteed: available memory, model format, runtime overhead, and other GPU use matter.
Is 12 GB enough for a 13B model?
Sometimes, if the model is aggressively quantized and the context is limited. A 12 GB card may not leave comfortable room for the cache and runtime, so check the exact model file and test it.
Can 16 GB run a 30B model?
Not usually as a fully GPU-resident model at common quantizations. It may be possible with very low-bit weights or CPU offloading, but expect compromises in quality, context, or speed.
What can 24 GB run locally?
It can run many 7B–14B models with room to spare and often supports 30B–32B models using low-bit quantization. The context length and exact model implementation can still determine whether a configuration fits.
Does a longer context use more VRAM?
Generally, yes. The KV cache grows with the amount of context retained, although the rate depends on the model architecture and inference settings.
Can I use system RAM when VRAM runs out?
Many runtimes can place some model layers in system RAM, but performance may fall because the GPU no longer holds all the data it needs locally. The result depends on the CPU, memory, and software.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
More cards in this drawer
- Luxury V6 Sports Sedan: Can It Easily Top 100k Miles?
- Puppets in TV Shows: Classic Proof They’re Better
- Best Streaming Remote: Top Picks for Every Setup
- Claude Pro Hidden Features Power Users Try
- Tesla Rental Prices: Why They’re So Low255.00 USD
- Best Family Western Movies to Stream Right Now