How to run an LLM locally?

Running an LLM locally means downloading compatible model weights and using an inference runtime to generate responses on your own computer. For most people, the fastest starting point is Ollama’s quickstart; for a graphical workflow, LM Studio provides a model browser and chat UI; for maximum control, llama.cpp exposes the underlying command-line and server tools. Local inference is also a useful building block for AI agents that need a private or offline model endpoint.

The shortest path: install Ollama, run ollama run gemma4, and test one real task.

The important constraint: your computer must hold the model weights, runtime memory and conversation context, so model choice matters more than the word “local.”

The practical rule: start with the smallest model that can answer your target task reliably, then move up only when evaluation shows that you need more capability.

What do you need to run an LLM locally?

A local LLM setup has four parts: a model, a runtime, memory and a task to test. The model provides the learned weights. The runtime loads those weights and performs inference. Your system RAM or GPU memory holds the weights and working data. Your task tells you whether the result is actually useful.

The model file is not the same thing as the app that runs it. A desktop application may download and configure a model for you, while a lower-level runtime expects you to choose the file, format and flags yourself. Keeping that distinction clear prevents a common mistake: downloading a large model because it looks impressive, then discovering that the computer cannot load it comfortably.

You also need enough storage for the model download and enough memory for inference. A quantized model uses reduced-precision weights to lower memory requirements, but quantization is a trade-off rather than magic compression: smaller files can be faster and easier to run while changing output quality for some tasks. Context length, batch size and runtime settings also affect memory use.

Which local LLM runner should you choose?

Ollama is the default recommendation when you want to get a local chat working quickly and point other tools at a local service. Its official quickstart says it runs on macOS, Windows and Linux and uses a simple ollama run command to start a model.

LM Studio is the default recommendation when you want to browse models visually. Its getting-started documentation describes a flow of downloading a model, loading it into memory from the Chat tab and then starting a conversation. It also makes the key hardware concept visible: loading a model means allocating memory for its weights and other parameters.

llama.cpp is the direct runtime choice. Its repository describes LLM inference in C/C++ and provides CLI and server commands, model downloads through Hugging Face, multiple hardware backends and GGUF support. Use it when a wrapper does not expose the setting you need or when you are building a local inference component into another application.

How do you run your first local model with Ollama?

1. Install Ollama

Download and install Ollama for your operating system from its official site. The documentation covers macOS, Windows and Linux. On Linux, the official instructions include a shell installer and a manual archive-based installation; follow your organization’s policy for reviewing remote installation scripts before executing them.

After installation, confirm that the command is available:

ollama --version

If the command is not found, restart the terminal or check that the installer added Ollama to your executable path. Do not troubleshoot the model before the runtime itself starts correctly.

2. Run a small first model

The official quickstart uses gemma4 as its first example. Start an interactive session with:

ollama run gemma4

The first run downloads the model if it is not already present, then loads it for a chat. Ask one question that matches your intended use—for example, summarizing a short document or producing structured JSON—and observe both the quality and the delay before the first token.

To leave the interactive session, use:

/bye

The exact model that is best for your computer depends on its memory, processor, GPU backend, context length and task. Our Local LLM Hardware Calculator answers the memory half of that directly: pick your GPU or Apple Silicon chip and it lists which open-weight models fit, at what context, and how fast they generate. The quickstart command is a way to verify the pipeline, not a claim that one model is ideal for every machine.

3. Test a repeatable task

A local chat can feel impressive while still being unsuitable for your workflow. Test a small set of representative prompts instead:

• one normal input;
• one long input near the context size you expect;
• one input with missing or ambiguous information;
• one output that must follow a strict format;
• one prompt that should be refused or escalated.

Record whether the model is accurate enough, how long responses take and whether the output format is stable. If it fails the task, changing the runtime will not automatically fix a model-selection problem.

How do you use a local model from another application?

A local runner becomes more useful when it exposes an API that your editor, script or agent can call. Ollama can run as a local service, and llama.cpp provides llama-server for an OpenAI-compatible API endpoint. The exact client configuration depends on the application, but the architecture is simple: your application sends a request to a local address instead of a hosted model provider.

Start with the runner’s own interface before adding an agent framework. Verify that the model responds correctly, then connect one client and inspect what data it sends. A local endpoint is still a powerful tool: an application can read files, call tools or write to a database while using a model that never leaves the machine. Keep model inference and tool permissions separate.

With llama.cpp, the official README shows both direct CLI inference and an API server:

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

llama-server -hf ggml-org/gemma-3-1b-it-GGUF

The -hf form downloads a compatible model from Hugging Face. You can also run a local GGUF file directly:

llama-cli -m my_model.gguf

Use a trusted model repository and verify the model’s license before incorporating it into a product. “Open weights” does not mean “no obligations.” Licensing, acceptable-use rules, training-data disclosures and redistribution terms vary by model.

How do you run an LLM with LM Studio?

LM Studio is the better first choice when you do not want to manage model downloads from a terminal. The basic workflow is:

1. Install the latest LM Studio release.
2. Open the Discover tab.
3. Search for a model and download a compatible file.
4. Open the Chat tab.
5. Use the model loader to select the downloaded model.
6. Start a conversation after the model loads into memory.

The application’s documentation warns that model openness varies: models may be distributed as GGUF, safetensors or other files, and licenses differ. Follow the model card and the app’s hardware guidance instead of treating every download as interchangeable.

LM Studio is especially useful for comparing several smaller models on the same prompts. Keep the evaluation fair: use the same task, context and output requirements, and note which model file and quantization you tested. A model that feels faster because it has a shorter context or produces shorter answers is not necessarily better for your real workload.

What is GGUF and why does it matter?

GGUF is a model-file format used widely in the llama.cpp ecosystem and supported by many local runners. It packages model weights and metadata in a form the runtime can load efficiently. The llama.cpp documentation shows both manual GGUF file usage and direct Hugging Face downloads.

You will often see several quantizations of the same base model. They represent different memory and quality trade-offs. A higher-precision file generally needs more memory; a more aggressively quantized file is easier to load but may alter accuracy, instruction following or tool behavior. The right choice depends on your task and hardware, not on the largest number in the model name.

Do not download a file solely because a filename looks familiar. Check the model’s instruction format, supported context, required runtime, license and provenance. A model that loads successfully can still be the wrong model for chat, coding, vision or structured output.

How do you estimate whether your computer can run a model?

Start with the runtime’s hardware guidance and the model’s own metadata, then test conservatively. Your computer needs memory for the model weights plus runtime overhead and the active context. If you use a GPU, the runtime may place some or all layers there; if the model exceeds available GPU memory, some runtimes can use CPU and GPU together, usually with a speed trade-off.

The most reliable process is incremental:

1. Check available system memory and GPU memory.
2. Choose a small quantized model supported by your runner.
3. Load it with a short context.
4. Run your representative prompts.
5. Increase model size or context only if the results justify the added memory and latency.

Avoid treating a model’s parameter count as a complete performance specification. Architecture, quantization, context length, hardware backend and prompt format all affect the experience. A smaller model that answers your task quickly can be more useful than a larger model that constantly swaps memory or produces an answer too slowly to use.

How do you keep local inference private?

Local inference can reduce the amount of prompt data sent to a hosted provider, but “local” is not a complete privacy guarantee. Review the runner’s network behavior, update checks, extensions, model registry, logs and integrations. A tool can run inference locally while still contacting the internet for downloads, updates or optional features.

Use a simple boundary checklist before loading sensitive material:

For sensitive workloads, isolate the runner, limit outbound network access, protect model and chat files, and keep tool permissions narrow. Local execution changes the data path; it does not remove the need for security engineering.

How do you connect a local LLM to an AI agent?

An agent adds planning, tool use or multi-step execution around a model. A local model can provide the reasoning component, but the agent’s permissions remain the higher-risk layer. Start with a read-only tool and a narrow task, then add write access only after you can inspect the agent’s decisions and recover from mistakes.

A safe progression looks like this:

1. Run the model in an interactive chat.
2. Call it through the local API.
3. Add one read-only tool.
4. Log the model request, tool call and returned result.
5. Add human confirmation before any write action.
6. Evaluate failures before expanding the tool set.

This separates model evaluation from agent evaluation. A model can be adequate for summarization while being unreliable for database updates. Conversely, a smaller model can be useful inside a tightly bounded workflow if the surrounding system validates its output.

What are the most common local LLM problems?

The model does not fit in memory. Choose a smaller model or more aggressive quantization, reduce context length and close other memory-heavy applications. Moving some work from GPU to CPU may make loading possible but can reduce speed.

Responses are too slow. Check whether the runtime is using the intended GPU backend, reduce unnecessary context, try a smaller model and avoid measuring only the first response while the model is still loading.

The model ignores the prompt format. Use the model’s recommended chat template and a runtime that supports the architecture. A compatible file is not enough if the application sends the wrong instruction format.

The output is unreliable. Test a different model, tighten the prompt and validate the output in code. Do not give the model access to consequential tools until it can meet the task’s accuracy and format requirements.

The local API works but the application fails. Check the endpoint, port, model identifier, request format and whether the client expects an OpenAI-compatible API. Start with a direct request to the local server, then add the application layer.

What should you run locally first?

For a first experiment, run a small instruction-tuned model through Ollama or LM Studio and give it one task you already understand well. Summarize a short text, extract a few fields or rewrite a paragraph. Compare the answer with a hosted model only after you know what “good enough” means for the task.

Choose Ollama if you want a clean command-line workflow and a local service. Choose LM Studio if you want to browse and compare models visually. Choose llama.cpp if you need direct control, a lightweight C/C++ runtime or an API server you can tune and embed.

The best local setup is not the one with the largest model or the most impressive benchmark. It is the smallest reproducible system that answers your real prompts, respects your data boundary and fails in a way you can detect. Start there, then add capability deliberately.