Self-Hosted LLMs: How to Choose and Deploy One

Choosing and deploying a self-hosted large language model (LLM) means matching a model and serving setup to your tasks, hardware, privacy requirements, and operating budget. Start by testing a few suitable models on representative work, then deploy the smallest one that meets your quality and latency needs on hardware you can reliably operate.

What self-hosting means

A self-hosted LLM runs on infrastructure you control, such as a workstation, an on-premises server, or a virtual machine in a cloud account. Your application sends prompts to that service rather than relying on a third-party hosted model API. “Self-hosted” does not necessarily mean disconnected from the internet: downloading model files, updates, or other services may still require network access.

Control can help with data residency, custom integration, predictable access, and experimentation. It also makes your organization responsible for the model-serving software, hardware capacity, updates, access controls, monitoring, and incident response. Hosting a model yourself does not automatically make prompts private: logs, backups, application traces, and downstream tools can still expose them.

How model serving works

Model weights are the learned numerical parameters used to generate text. An inference server loads those weights, accepts a prompt, and produces tokens—the small text units the model generates—one after another. Your application usually communicates with the server through an API, then displays, processes, or stores the response.

Inference has two distinct workload phases. Processing the prompt, often called prefill, depends substantially on prompt length and compute throughput. Generating the response depends on the model, hardware, and requested output length. A long context window can increase memory use, and a server that handles multiple requests may need extra capacity for concurrent sequences and their key-value cache.

Model files may be distributed in different numerical formats. Quantization reduces the precision used for weights, often lowering memory requirements and sometimes increasing speed, but it can affect output quality. A quantized model is not automatically suitable for every task; evaluate it on your own examples.

How to choose a model

Do not choose by parameter count or a benchmark headline alone. A larger model may perform better on some tasks but need more memory and deliver slower responses. A smaller model may be sufficient for classification, extraction, or routine drafting, while complex reasoning or specialized language tasks may call for a stronger model.

Define the job: List the task, languages, expected input length, output format, and whether the model must follow tools or structured-output instructions.
Check use rights: Read the model’s license and accompanying terms for your intended use, redistribution, and deployment. Openly available weights do not mean there are no restrictions.
Test real examples: Compare candidates on representative prompts, including difficult cases. Score factuality, completeness, formatting, latency, and failure behavior rather than relying on a single general benchmark.
Measure resource needs: Check the model’s documented memory requirements and test actual peak use with your chosen context length and concurrency.
Assess language and safety behavior: Evaluate the languages and content your users will encounter, including how the model responds when it lacks information.

For example, a team extracting fields from short internal documents should test whether a compact model can return valid, correct structured results. A team answering questions across lengthy manuals should also test long-context retrieval and whether the model accurately uses supplied passages. These are different workloads, even if both are described as “chat.”

Hardware and deployment choices

GPUs can accelerate inference, but a GPU is not mandatory: CPU inference is possible and may be adequate for light or latency-tolerant work. Performance depends on the model format, implementation, memory bandwidth, hardware, and workload. GPU memory must accommodate the model and runtime overhead; context and concurrent requests add further demand. Do not assume a model will fit simply because its file can be downloaded.

A local workstation is convenient for evaluation and individual use. A dedicated server is easier to operate as a shared service, while a cloud virtual machine can offer flexible access to suitable hardware but still requires configuration and capacity management. Compare options by total operating effort as well as compute: storage, electricity or cloud usage, backups, maintenance, and staff time all matter.

For deployment, use a maintained inference server that supports your model format and hardware. Pin model and software versions so upgrades are deliberate, and place the service behind an application or gateway rather than exposing an unauthenticated inference endpoint. Start with one model, one use case, and limited concurrency; record baseline response time, error rate, resource use, and output quality before expanding.

Common problems and practical fixes

Out-of-memory errors: Reduce context length or concurrent requests, use a smaller or quantized model, or add suitable memory. Check both model loading and peak use under real prompts.
Slow responses: Separate prompt-processing delay from token-generation speed. Shorten unnecessary context, review hardware utilization, and test a smaller model before assuming that more parallel requests will help.
Unreliable answers: Improve prompts and input data, test a different model, and make retrieval sources explicit when the answer must reflect private documents. Retrieval can provide relevant text, but it does not guarantee the model will use it correctly.
Inconsistent formatting: Use a supported structured-output feature if available, validate the result in your application, and handle invalid output instead of trusting it blindly.
Unexpected exposure: Inspect application and server logs, backups, monitoring systems, and connected tools. Restrict access and avoid sending sensitive data until the entire data path has been reviewed.

Privacy, safety, and operations

Self-hosting gives you more control over where inference runs, but privacy depends on the complete system. Document what prompts and outputs are retained, who can access them, and whether they appear in logs or backups. Apply authentication, network restrictions, least-privilege access, and encrypted connections where appropriate. Keep model files and serving components under change control.

Treat generated content as untrusted input. Models can produce false statements, mishandle instructions embedded in documents, or return unsafe text. For systems that use tools or take actions, validate model output, limit tool permissions, and require human review for consequential operations. Monitor quality and resource use after changes to models, prompts, or traffic; a technically healthy server can still produce poor answers.

Frequently Asked Questions

Can I run an LLM without a GPU?
Yes. CPU inference is possible, particularly for smaller models or low-volume tasks, but it may be slower. Test the response time and memory use on the exact hardware and workload you expect.

How much RAM or GPU memory do I need?
There is no single amount: requirements vary with model size and format, context length, runtime, and concurrency. Use the model’s documentation as a starting point, then measure peak usage during realistic tests.

Is a self-hosted LLM automatically private?
No. It can keep inference under your control, but applications, logs, backups, monitoring, and connected services may still store or transmit data. Review the full data path and configure retention and access deliberately.

Can I use a self-hosted model with my own documents?
Yes. A common pattern is retrieval-augmented generation: search a document collection, provide relevant passages to the model, and ask it to answer from them. You must still manage document permissions, retrieval quality, and answer verification.

Should I fine-tune a model or use retrieval?
Retrieval is often the more direct approach when answers need current or private reference material. Fine-tuning changes model behavior using training examples and may suit repeated style or task patterns, but it is not a dependable substitute for supplying up-to-date facts.

What is the easiest way to start?
Choose one narrow use case, select a model with suitable terms and documented hardware requirements, and run a small evaluation on representative examples. Move to a shared service only after quality, access controls, and operating needs are understood.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →