Local LLM Hardware Calculator

Enough for three things at once: the weight file, the KV cache, and about half a gigabyte of runtime buffers. A 32B model at Q4_K_M is roughly 18 GiB of weights, and its KV cache adds about 0.25 GiB for every 1,024 tokens of context. On a 24 GiB card that leaves room for roughly 16k tokens of context. Doubling the context doubles the cache, not the weights.

At Q4_K_M a 70B model is close to 40 GiB of weights, so it does not fit a single 24 GiB consumer card. It fits comfortably on 48 GiB and above, or on Apple Silicon with 64 GB or more of unified memory, where the memory is shared with the operating system. Below that the practical options are a 32B model at Q4_K_M or a mixture-of-experts model with far fewer active parameters.

The cache holds a key and a value for every layer and every KV head, for every token in the window. That is 2 × layers × KV heads × head dimension × bytes per element. For Qwen3 32B, with 64 layers, 8 KV heads and a head dimension of 128, one token costs 262,144 bytes at fp16 — a quarter of a gibibyte per 1,024 tokens. At 128k context the cache alone is over 30 GiB, larger than the quantised weights.

Q4_K_M costs measurable quality, but the loss is proportionally smaller on larger models. In the same memory budget a 32B model at Q4_K_M usually beats a 14B model at Q8_0 on general tasks. Q8_0 is worth it when the model is small, when output is parsed by another program, or when the task is arithmetic or code where small errors compound.

Generation at batch 1 is limited by memory bandwidth: the runtime reads the weights once per token, so speed is roughly bandwidth divided by the bytes it must read. The figures here are a range rather than a point, because real throughput also depends on the runtime, flash attention, context length and thermal headroom. Prompt processing is a separate, compute-bound step and is not included.

It trades speed for capacity. An M-series chip with 64 GB or more can hold models no consumer graphics card can, because the memory is shared rather than soldered to one device. But its bandwidth is lower than a high-end discrete card, so the same model generates more slowly. Unified memory is also shared with the operating system, so a smaller share is available than the headline figure suggests.

Neither is modelled here. Splitting a model across two cards adds interconnect cost that depends heavily on the link between them, and offloading layers to system RAM slows generation by an order of magnitude because those layers are read across PCIe. Both make the answer depend on details this calculator does not ask for, so it reports what a single device holds.