How much VRAM and GPU do you need for a local LLM?

A local LLM does not need a guess about “how many billion parameters” your GPU can handle; it needs a memory budget that includes the model weights, KV cache, runtime buffers, and whatever else shares the device. The Yowox Local LLM Hardware Calculator turns that budget into a concrete shortlist: choose a GPU, Mac, or custom memory size, then see which open-weight models fit, at what context length, and at what estimated generation speed.

Definition: Local LLM hardware sizing is the process of matching a model’s total runtime memory — not just its parameter count — to the usable memory and bandwidth of one device.

Example: A 24 GB configuration in the Yowox calculator shows Qwen3 30B A3B at Q4_K_M using 17.4 GB for weights and 19.5 GB total, with a maximum context of 35k and an estimated 103–309 tokens per second.

Key takeaway: The model file is only one part of the hardware requirement; context length and quantization can change what fits.

Business impact: Checking the exact device before buying or downloading a model reduces the risk of spending money on hardware that cannot run the intended workload at a useful context length.

What does the Yowox Local LLM Hardware Calculator measure?

The Yowox Local LLM Hardware Calculator measures whether an open-weight model fits on one selected device and reports the memory and speed assumptions behind that result. For each candidate, the calculator separates weights, KV cache, total memory, maximum context, and tokens per second, so the reader can see why a model fits rather than receiving a simple yes-or-no label.

The calculator starts with the hardware that actually matters for local inference: a GPU, Apple Silicon chip, AMD card, Intel Arc device, professional accelerator, or a custom memory value. It also lets the reader set the conversation length, model compression, and memory to leave free for the desktop, browser, and other processes. That means the result answers a specific configuration instead of pretending every 24 GB device behaves identically.

How much VRAM does a local LLM need?

A local LLM needs enough usable memory for the weight file, KV cache, runtime buffers, and the applications sharing the device; the required amount therefore depends on the model and the intended context. In the Yowox calculator, a 32B Q4_K_M model uses about 18.4–18.5 GB for weights and about 21.8–21.9 GB in total at an 8k-class context, so a 24 GB GPU is a tight but plausible fit while a 16 GB card is not. Background: How to choose the best local LLM for your hardware.

A simple parameter-count rule is useful only as a first filter because two models with similar parameter counts can have different weight files, cache costs, and supported contexts. The Yowox calculator reads the actual quantized file sizes and model configuration data, so the practical action is to select your device, set the free-memory reserve, and compare the resulting shortlist before choosing a model.

What fits on 12 GB of memory?

A 12 GB device is suited to smaller models and efficient quantization, not to every model marketed with a low parameter count. The Yowox calculator’s 12 GB reference section shows 70 of 116 catalogue models running, including Gemma 2 9B at Q4_K_M with 8.8 GB total memory and Qwen3.5 9B at Q4_K_M with 7.5 GB total, so a 12 GB owner should begin with 7B–9B candidates and verify context headroom.

The 12 GB result is not a promise that every 9B model will behave the same way. The table also shows granite 3.1 8B at Q8_0 using 10.2 GB total, leaving less room for the desktop and longer conversations; use the calculator’s free-memory setting rather than buying against the headline capacity alone.

What fits on 16 GB of memory?

A 16 GB device opens the door to larger 14B models and selected 20B-class models at compressed precision. In the Yowox reference table, gpt oss 20b at Q4_K_M uses 12.3 GB total, while several 14B models at Q4_K_M use 10.8 GB and reach a 28k maximum context, so 16 GB is a practical middle tier for local experimentation and coding workloads.

The 16 GB tier still requires a deliberate trade-off between model size and quality. A user who selects Q8_0 instead of Q4_K_M may preserve more precision but lose the memory needed for context, so compare both settings in the calculator with the same hardware and free-memory reserve.

What fits on 24 GB of memory?

A 24 GB GPU is the first consumer tier in the Yowox reference table that reaches 30B-class models at Q4_K_M. The calculator shows 96 of 116 models running on 24 GB, including Qwen3 30B A3B at 19.5 GB total and Qwen3.5 27B at 20.1 GB total, so this tier can support substantially larger local models while retaining useful context headroom.

A 24 GB card is not a general-purpose 70B solution. The Yowox calculator lists 70B Q4_K_M examples at about 44.6 GB total, so a 24 GB owner should compare a 30–32B model, a mixture-of-experts model with a lower total footprint, or a more aggressive quantization rather than assuming that CPU offload will preserve the same speed.

What fits on 48 GB and 64 GB of unified memory?

A 48 GB device moves the shortlist into 32B–36B models at higher quantization levels, while 64 GB or more of unified memory can hold models that do not fit on a single consumer GPU. The Yowox reference tables show 101 of 116 models on 48 GB, including Hermes 4.3 36B at Q8_0 with 40.1 GB total, and 115 of 116 models on 64 GB unified memory, including gpt oss 120b at 62.6 GB total.

Unified memory changes capacity and speed in opposite directions because the model shares memory with the operating system. The Yowox calculator therefore reports Apple Silicon as a separate hardware class and includes estimated memory bandwidth, so the practical choice is to decide whether your priority is holding a larger model or generating tokens faster. Related reading: How Much Agentic Memory Does Your Model Need?.

Why does context length use so much memory?

Context length increases the KV cache linearly because the runtime stores key and value data for every token in the active window. The Yowox calculator computes that cache from each model’s own configuration — 2 × layers × KV heads × head dimension × bytes — instead of applying one generic percentage to every model, so changing the context setting can change both the fit decision and the maximum context shown.

The effect is especially visible on models with long context windows. The Yowox calculator’s Qwen3 32B example gives a cost of 262,144 bytes per token at fp16, which puts the cache above 30 GiB at 128k context; if your GPU barely fits the weights, a long context can consume the remaining memory before a response is generated.

How does quantization change the answer?

Quantization changes how many bytes each model weight occupies, allowing a larger model to fit in the same memory budget. The Yowox calculator compares Q4_K_M, Q8_0, and F16 choices where the catalogue provides them, so the reader can see whether a device fits the best-quality option, a compressed option, or only a more aggressive memory trade-off.

Q4_K_M is useful when fitting the largest capable model matters most, while Q8_0 or F16 can be preferable when the task is sensitive to small errors and the device has room. The calculator makes that trade-off visible in one table: choose the compression mode, then compare total memory and context rather than comparing parameter counts alone.

How does the calculator estimate tokens per second?

The Yowox calculator estimates batch-one generation speed primarily from memory bandwidth and the number of bytes read per token, then presents a range instead of a single guaranteed benchmark. Its method reflects the fact that local LLM decoding is often constrained by moving model weights through memory, while real results also vary with the runtime, attention implementation, context length, thermal limits, and other active workloads.

The speed number is therefore a planning estimate, not a promise for every installation. Use it to compare two devices or two quantizations under the same assumptions, then run the exact model in your chosen runtime before making a production performance commitment.

How do you use the Yowox calculator before buying hardware?

Use the Yowox calculator as a decision sequence: choose the device, reserve memory for the system, set the target context, select the quality or compression mode, and inspect the models that remain. The result is most useful when it starts from a real workload — for example, a coding model at 16k context or a document model that must read long inputs — rather than from a vague desire to run “the biggest model.” More on this: How to run an LLM locally?.

The calculator also exposes the assumptions that rough VRAM charts hide: a 14B model may fit at Q4_K_M but not Q8_0, a 30B model may fit at short context but not at 128k, and two devices with the same memory capacity may produce different token rates because their bandwidth differs. Save or share the shortlist only after checking that the memory reserve matches your actual machine.

What the calculator does not model

The Yowox calculator models one device holding the workload; it does not model multi-GPU splitting or CPU offload as equivalent alternatives. Those configurations depend on interconnects, layer placement, system memory, and runtime behavior, so treat them as separate engineering designs rather than extending a single-GPU result by simple addition.

The calculator also estimates inference, not training. Fine-tuning and full training have additional memory requirements for gradients, optimizer state, and activations; use the calculator to answer the narrower question it is designed for: which open-weight models fit on the selected local hardware and under which inference settings.

Start with the hardware you already have

The fastest way to make a local LLM decision is to measure the machine you own before shopping for another GPU. Select the matching card or Mac in the Yowox Local LLM Hardware Calculator, leave realistic memory for the operating system, choose your required context, and let the shortlist show whether you should change the model, quantization, or hardware.

If the exact GPU is not obvious, the Yowox Chrome extension can detect the graphics adapter locally and fill in the calculator; the tool page states that detection stays on the machine. Once the hardware is identified, the calculator gives you a reproducible starting point for choosing a model instead of relying on a generic “B parameters equals GB” rule.

Want to run AI locally as part of a real business workflow? Explore Yowox services and use the calculator linked near the top to check the machine you already have.