Q4, Q8, FP16 and Long Context: Why a Local LLM Can Fail Despite Fitting on Disk

A 20 GB model file does not automatically fit on a 24 GB graphics card. The download describes stored weights and metadata. Execution adds context state, temporary buffers and other allocations. Available capacity also falls below advertised memory when other applications use the GPU.

Quantisation and context therefore belong in the same plan. The following comparison explains memory mechanisms and calculations without deriving measured quality or speed rankings.

Q4, Q8 and FP16 imply different budgets

These are theoretical decimal quantities, not file sizes for a particular model. Scales, blocks, mixed precision and unquantised components alter the result. GB and GiB also differ: 15 billion bytes are approximately 13.97 GiB. A Q4 file need not use exactly four bits for each parameter in the model.

Context consumes runtime memory

With conventional attention, the KV cache stores earlier keys and values. A simplified dense-cache calculation is: two × layers × KV heads × head dimension × tokens × bytes per value × sequences. This requires matching architecture and runtime assumptions.

An assumed example with 32 layers, eight KV heads, dimension 128, 8,192 tokens and two-byte values needs 1 GiB for one sequence. Four fully occupied equivalent sequences need 4 GiB. Weights and runtime buffers are additional. Paging, cache quantisation and other attention patterns change actual allocation.

Hybrid architectures need their own profiles

Qwen3.5 uses a hybrid structure with different layer types. A conventional-attention formula cannot be applied blindly to every layer. Recurrent state, attention cache and multimodal components require appropriate model configuration and runtime data.

MoE also separates active computation from total weight storage. A small active-parameter count does not promise a small model file. Planning solely from a model name misses these differences.

Offloading changes the workload

llama.cpp supports hybrid CPU/GPU execution. Partial offloading may make an otherwise oversized model runnable. This remains a different configuration from full GPU execution. System memory, CPU performance and transfers influence response behaviour.

A useful comparison records GPU allocation, system RAM, first-token latency and generation speed together. Successful startup is only the first check. Long documents and concurrent sessions can later expose a limit hidden by a short chat.

Check quality before optimising memory

A larger Q4 model does not automatically outperform a smaller Q8 model in quality. Training, architecture and task remain decisive. A shared dataset with expected answers reveals errors, omissions and format problems; file size alone does not establish usefulness.

The existing memory estimator provides an initial shortlist. Specific revision and backend, free headroom, context and concurrency belong in its inputs. Final configuration follows a real memory check and a small quality evaluation. An unknown architecture needs a visible estimation limit rather than a falsely exact result.

Memory planning for model selection: LLM VRAM estimator.

Sources

1. llama.cpp: Projekt und Backends
2. Qwen: Qwen3.5-9B, offizielle Model Card
3. Mistral: Ministral 3 14B Instruct, offizielle Model Card
4. vLLM: GPU installation

Sources checked: 5 October 2026.