Open-Source AI Models You Can Run Locally

Yes. You can run many open-weight AI models on your own computer without connecting to a Chinese cloud service—or any model-hosting cloud—by downloading the model files and using local software such as Ollama, llama.cpp, or a desktop interface. The main trade-offs are hardware requirements, model quality, licensing, and the fact that “open source” is often used loosely for models whose weights are available but whose full training data or process is not.

What “local” and “open source” mean

A local model runs inference on your computer: your prompt is processed by software using model files stored on your device. Once the files and software are installed, ordinary use can work offline. This differs from a hosted chatbot, where prompts are sent to a provider’s servers for processing. Some local apps still use online services for downloads, updates, optional features, or telemetry, so check settings and network behavior if offline operation is important.

“Open-weight” is the more precise term for many downloadable models. It means the trained numerical weights are available, but it does not necessarily mean the training data, complete training code, or all development details are public. Licenses also differ: some permit broad commercial use, while others impose conditions or restrict certain uses. Read the license for the specific model and version rather than assuming every downloadable model is unrestricted.

Examples of models to consider

Several model families have releases that can be downloaded and run locally. Availability, size, terms, and capabilities vary between versions and distributors, so confirm the exact model card and license before installing.

Meta Llama: A widely used family of language models with multiple sizes and instruction-tuned variants. Llama releases are governed by Meta’s license terms, which are not identical to a simple permissive open-source license.
Mistral and Mixtral: Mistral publishes compact language models and mixture-of-experts models. Licensing varies by model; check the individual release rather than generalizing across the family.
Google Gemma: A family of models with downloadable weights and accompanying terms. Gemma’s terms should be reviewed for the particular release and intended use.
Qwen: Alibaba’s model family includes language and multimodal releases. It can be run locally; using it locally does not require sending prompts to Alibaba’s cloud. Check the license attached to the exact model.
DeepSeek: Some DeepSeek model weights are available for local inference. The name of a model family does not determine where it runs: a local installation processes prompts on your hardware, provided the software is configured for local inference.
Microsoft Phi: Smaller models in the Phi family are designed for a range of deployment settings, including local use. Check each model’s license and system requirements.

These are examples, not a ranking. Model quality depends on the task, the release, its size, the quantization, and the software used to run it.

How to run one without a cloud account

A typical setup has three parts: a model file, an inference engine, and an interface. The engine loads the weights and performs generation; the interface lets you chat, write prompts, or connect another application. Popular local options include Ollama, llama.cpp, and desktop programs such as LM Studio. They are distinct projects, and their supported formats, features, and download processes can change.

Choose a model based on the job, license, and hardware you have. Start with a smaller instruction-tuned model if you mainly want chat, summarization, or drafting.
Install a local inference tool from its official distribution channel. Avoid unfamiliar repackaged installers and verify that you are downloading the intended application.
Download a compatible model. The first download requires internet access unless you obtain the files another way; after installation, inference can be offline.
Test it with a harmless prompt, then disconnect the network to check whether the basic feature works without a connection. Disable any optional cloud, web-search, or remote API features you do not want.

Some tools offer a command-line interface, while others provide a graphical chat window. A model downloaded through one tool may not be in the format another tool expects. Follow the tool’s instructions for supported formats rather than renaming files or assuming compatibility.

Hardware, model size, and quantization

Model files can range from relatively small to many gigabytes or more. The computer needs enough memory to load the model and also needs extra room for the context—the conversation or document being processed. A graphics processor can speed up inference, but many tools can use a CPU, usually with slower responses. Exact performance depends on the model, hardware, software, and settings.

Quantization stores model weights at lower numerical precision to reduce memory use and often improve the chance that a model will fit on a consumer device. It can also affect output quality. A quantized model is not a different cloud service; it is a locally stored representation of the model. When choosing one, look for a clearly documented quantization and make sure the file is compatible with your inference software.

Do not judge hardware needs by parameter count alone. Context length, runtime overhead, image or audio inputs, and the way a tool divides work between memory and processor can all matter. If a model fails to load, try a smaller model or a more compact quantization before concluding that local AI is impossible on the machine.

Privacy, offline use, and practical limits

Local inference can keep prompts on your device, which is useful for private drafts or work that should not be sent to a hosted service. It is not an automatic privacy guarantee: another program could transmit data, a connected feature could call a remote service, or someone with access to the device could read saved chats. Review application settings, stored conversation locations, and permissions.

Local models can make mistakes, invent citations, or give outdated information. Their answers should not be treated as verified facts simply because no cloud provider is involved. They may also be less capable than a larger hosted model, especially on difficult reasoning, current events, or specialized tasks. If current information is needed, a local model alone may not have it unless you provide the source material or deliberately enable a separate retrieval or web feature.

For an offline setup, download the software and model files first, then test the application with the network disconnected. Keep a note of the model name, version, license, and file source. That makes it easier to reproduce the setup and distinguish a local model from optional online services.

Choosing a setup

For a straightforward chat interface: Choose a desktop app that explicitly supports local model files and explains whether any features use remote services.
For a simple local workflow or command line: Consider a local model manager such as Ollama, and verify the selected model and settings.
For more control over formats and performance: llama.cpp and related tools can suit users comfortable with configuration and model files.
For commercial or organizational use: Review the exact model license, acceptable-use terms, and any obligations before deployment.

There is no single best model for everyone. Pick a task, test a few models that fit your computer, and compare their accuracy and speed using the same prompts. For private or sensitive work, also assess the complete application—not just the model weights.

Frequently Asked Questions

Can I use local AI without an internet connection?
Yes, after downloading the inference software and model files. Disconnecting the network is a practical way to confirm that the feature you are using runs locally.

Does running Qwen or DeepSeek locally send my prompts to China?
Not by itself. A model’s origin does not determine where inference happens. If the weights are installed locally and the application uses its local engine, prompts are processed on your device; check for optional remote features.

Are open-weight models completely open source?
Not always. Downloadable weights do not necessarily include training data or the full training process, and licenses can impose conditions. Read the specific release’s terms.

Can I run a model on an ordinary laptop?
Often, if you choose a model and quantization that fit available memory. Smaller models may run on a CPU, though speed and output quality vary; larger models generally need more resources.

Is Ollama itself the AI model?
No. It is software for managing and running models locally. You still choose and download a model, and its capabilities and license depend on that model.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →