Local LLM Inference: Match Runtime to Hardware

Local LLM inference lets you run large language models on your own hardware, and the key to good performance is matching your runtime—the software that executes the model—to your specific GPU, CPU, and workload. The right choice can double your tokens per second or allow a model that would otherwise crash to run at all. This article explains the main runtime options, how they use your hardware, and how to pick the one that fits your setup.

What Local LLM Inference Actually Does

When you run a local LLM, the software loads a model file—often several gigabytes—into memory and then performs billions of matrix multiplications to generate each token (a word fragment). The runtime is the program that orchestrates these calculations, manages memory, and exposes an API or command-line interface. Popular runtimes include llama.cpp, Ollama, LM Studio, vLLM, and Hugging Face’s Transformers with PyTorch. Each has different strengths depending on whether you have an NVIDIA GPU, an AMD GPU, an Apple Silicon Mac, or only a CPU.

The core bottleneck is memory bandwidth, not raw compute speed. Generating tokens requires moving the entire model through the memory system for every token. A GPU with 1 TB/s of memory bandwidth can process a 7-billion-parameter model at roughly 100 tokens per second, while a CPU with 50 GB/s might manage only 5 tokens per second. This is why the runtime’s memory management and quantization support matter as much as the hardware itself.

Key Hardware Factors: VRAM, Compute, and Bandwidth

Three hardware specs determine what runtime you should use:

• VRAM (video RAM): The model must fit in VRAM for GPU inference. A 7B model in 4-bit quantization needs about 4 GB; a 70B model needs about 40 GB. If it doesn’t fit, the runtime must offload layers to system RAM, which slows things down dramatically.
• Memory bandwidth: As noted, higher bandwidth means faster token generation. NVIDIA RTX 4090 has ~1 TB/s, Apple M2 Max has ~400 GB/s, and a typical desktop CPU has ~50 GB/s.
• Compute (TFLOPS): This matters more for prompt processing (the initial analysis of your input) than for token generation. A faster GPU processes long prompts faster, but generation speed is still bandwidth-bound.

For CPU-only inference, the runtime must use vector instructions (AVX2, AVX-512) and efficient threading. llama.cpp is the most optimized for this, while vLLM is GPU-only. Apple Silicon users benefit from Metal support in llama.cpp and MLX, which uses the unified memory architecture to run larger models than a discrete GPU of the same RAM.

Runtime Options Compared

Here is a practical breakdown of the main runtimes:

• llama.cpp: The most portable runtime. Supports CPU, NVIDIA (CUDA), AMD (ROCm/HIP), and Apple (Metal). Offers extensive quantization formats (GGUF). Best for single-user local use on mixed hardware.
• Ollama: A wrapper around llama.cpp that adds a simple CLI and API. Great for beginners, but less fine-grained control. Uses the same GGUF models.
• LM Studio: A graphical interface for llama.cpp on Windows/macOS. Good for testing models without command-line skills.
• vLLM: Built for serving many concurrent requests. Uses PagedAttention to manage memory efficiently. Requires NVIDIA GPU with CUDA. Overkill for a single user but excellent for a local API server.
• Transformers + PyTorch: The standard for research and fine-tuning. Supports many model architectures but is less memory-efficient and slower for pure inference unless you use additional optimizations like GPTQ or AWQ.
• MLX: Apple’s framework for Apple Silicon. Can be faster than llama.cpp on Macs for some models, but supports fewer model formats.

The choice often comes down to: if you have an NVIDIA GPU, vLLM or llama.cpp both work; if you have a CPU-only machine, llama.cpp is the only serious option; if you have a Mac, try both llama.cpp and MLX.

How to Choose the Right Runtime for Your Workload

Your workload—what you plan to do with the model—changes the best runtime. Consider these scenarios:

• Single-user chat or code completion: Ollama or LM Studio are easiest. They handle model downloading and memory management automatically.
• Batch processing or multiple users: vLLM is designed for high throughput. It can serve hundreds of requests per second on a single GPU, whereas llama.cpp would struggle with concurrent requests.
• Long-context tasks (e.g., analyzing a large document): Look for runtimes with efficient attention implementations. vLLM’s PagedAttention and llama.cpp’s recent flash-attention support help you fit longer contexts into memory.
• CPU-only inference: Use llama.cpp with a GGUF quantization like Q4_K_M. Avoid vLLM entirely. Expect 1-5 tokens per second for a 7B model on a modern desktop CPU.
• Apple Silicon: Try both llama.cpp and MLX. For a 13B model on an M2 MacBook Air, MLX can be 20-30% faster than llama.cpp, but llama.cpp supports more model types.

A common mistake is using a runtime that doesn’t support your GPU’s vendor. For example, AMD GPU users on Windows often struggle with ROCm; using the DirectML backend in llama.cpp or a Vulkan build can be more stable. Similarly, older NVIDIA cards without CUDA 12 support may need an older build of llama.cpp.

Quantization: The Biggest Performance Lever

Quantization reduces the precision of the model’s weights (from 16-bit to 4-bit or 8-bit), which shrinks memory usage and increases speed at the cost of some accuracy. The GGUF format used by llama.cpp supports many quantization levels:

• Q2_K: Very small, but noticeable quality loss.
• Q4_K_M: The sweet spot for most users. Good quality, roughly 4.5 bits per weight.
• Q5_K_M: Slightly better quality, larger file.
• Q8_0: Nearly lossless, but uses twice the memory of Q4.

For a 7B model, Q4_K_M uses about 4.2 GB of VRAM, while Q8_0 uses about 7.5 GB. If you have 8 GB of VRAM, you can run a 7B model at Q8 or a 13B model at Q4. The runtime must support the quantization format; GGUF is universal across llama.cpp-based runtimes, but GPTQ and AWQ are used by vLLM and Transformers and are not interchangeable.

Rule of thumb: choose the highest quantization that fits in your VRAM with room for the context window. A 4-bit model that fits entirely in VRAM will be many times faster than an 8-bit model that spills into system RAM.

Common Problems and How to Solve Them

Here are frequent issues when matching runtime to hardware:

• Out of memory errors: The model doesn’t fit in VRAM. Solution: use a smaller quantization, reduce the context length, or switch to CPU offloading (llama.cpp’s -ngl flag controls how many layers go to GPU).
• Slow generation on CPU: If you’re using llama.cpp, make sure you compiled with native CPU optimizations (e.g., -march=native). Also check that you’re using a GGUF model, not a full fp16 PyTorch model.
• AMD GPU not recognized: On Windows, use the Vulkan backend in llama.cpp or the DirectML build. On Linux, install ROCm and use the HIP build.
• Mac runs out of unified memory: Close other apps and reduce the context size. macOS uses swap aggressively, but that kills performance.
• vLLM fails to start: vLLM requires a recent CUDA version and a GPU with compute capability 7.0 or higher. Check your GPU’s compute capability and install the matching CUDA toolkit.

One misconception is that more VRAM always means faster. If your model fits entirely in VRAM, adding more VRAM doesn’t help speed—bandwidth and compute are fixed. Extra VRAM only lets you run larger models or longer contexts.

Costs and Practical Considerations

Running local LLMs has a hardware cost, but no per-token fees. A used NVIDIA RTX 3060 12GB can run a 13B model comfortably and costs much less than cloud API calls over time if you generate a lot of tokens. However, electricity and cooling matter for sustained use. A 300W GPU running at full load for an hour uses about 0.3 kWh, which is negligible for most users but adds up in a server context.

For CPU-only inference, the cost is mostly time. Generating a 500-word answer at 3 tokens per second takes about 3 minutes, versus 10 seconds on a GPU. If you only occasionally run models, CPU-only is fine; if you use it daily, a GPU is worth it.

Also consider the total cost of ownership: a Mac with unified memory can run larger models than a PC with a discrete GPU of the same price, because the CPU and GPU share the same RAM. An M1 Mac with 16 GB can run a 13B model at Q4, while a PC with 8 GB VRAM cannot.

Frequently Asked Questions

Can I run a 70B model on my 8 GB GPU?

Not entirely in VRAM. A 70B model at Q4 needs about 40 GB. You can run it with heavy CPU offloading, but you’ll get less than 1 token per second, which is impractical for chat. A better option is to use a smaller model like a 13B or 7B that fits fully in VRAM.

Is llama.cpp faster than Ollama?

Ollama uses llama.cpp under the hood, so the raw speed is nearly identical for the same model and settings. Ollama adds a small overhead for its API and model management, but it’s negligible. The difference is convenience, not speed.

Does quantization affect answer quality?

Yes, but less than you might think. At Q4_K_M, most models lose a small amount of coherence on complex tasks, but the difference is often imperceptible for casual chat. For code generation or math, Q5 or Q8 is safer. The quality drop is much smaller than the difference between a 7B and a 13B model.

Why is my CPU inference so slow?

CPU inference is limited by memory bandwidth. A 7B model has about 14 GB of weights (at 16-bit) or 4 GB (at 4-bit). Your CPU’s memory bandwidth is typically 20-50 GB/s, so even with perfect optimization, you’re looking at a few tokens per second. Using a 4-bit quantization and ensuring your CPU supports AVX2 will help.

Should I use vLLM for a single user?

Probably not. vLLM is optimized for serving many concurrent requests and uses extra memory for its paging system. For a single user, llama.cpp or Ollama will be simpler and use less RAM. vLLM only makes sense if you’re building an API for multiple users or need very high throughput.

Can I use my integrated GPU?

Integrated GPUs (like Intel UHD or AMD Radeon integrated) share system RAM and have low bandwidth. They are often slower than a decent CPU. Unless you have no other option, stick with CPU inference via llama.cpp.

Related

• Enterprise Web Hosting: What Large Companies Need
• Compare Two PDFs in Adobe Acrobat: Settings & Results
• Backup Internet Connection Setup Guide

in t | Finish time | min | 50 | 0
in d | Distance | km | 10 | 0 | 100000
out pace | Average pace | min/km | t / d | 2
out speed | Average speed | km/h | d / (t / 60) | 2