Q4, Q8, FP16 and Long Context: Why a Local LLM Can Fail Despite Fitting on Disk

A 20 GB model file does not automatically fit into a graphics card with 24 GB of memory. The file mainly contains weights: numbers learned during training. While answering, the software also needs room for earlier text and intermediate calculations. Other programs may occupy graphics memory too. File size and memory used during execution are therefore different quantities.

Quantisation stores model numbers with fewer bits. This usually reduces storage, but may also affect quality. Context means the input and generated response considered by the model. Long documents and simultaneous requests can require extra memory. The calculations below are examples, not measurements of a particular model.

Q4, Q8 and FP16 imply different budgets

Q4 and Q8 here refer to quantisation methods using roughly four and eight bits per weight. FP16 stores numbers in a 16-bit floating-point format. The table covers only idealised weights. Extra scale values, parts stored differently and metadata change real file sizes. A GB is one billion bytes, whereas a GiB is 1,073,741,824 bytes. The table’s 15 GB therefore equal about 13.97 GiB.

Context consumes runtime memory

The KV cache keeps calculated keys and values from earlier text. This avoids repeating all that work for each new text piece. A token is such a piece, for example part of a word. For a simple cache with equally sized key and value vectors, the calculation is: 2 × layers × KV heads × head dimension × stored tokens × bytes per value × independent sequences. Layers are processing stages; heads and their dimension describe the number and length of stored number arrays. These settings come from the model configuration.

Example: 2 × 32 layers × 8 KV heads × 128 values × 8,192 tokens × 2 bytes × 1 sequence equals 1,073,741,824 bytes, or 1 GiB. Four independent, equally full sequences need 4 GiB of cache in this calculation. This does not necessarily load the weights four times. The calculation covers cache payload only, excluding bookkeeping and calculation buffers. Some programs reserve the configured maximum at startup; others grow the cache as needed. Shared prefixes, smaller number formats and limited attention windows change memory needs.

Hybrid architectures need their own profiles

Not every model stores every processing stage this way. Qwen3.5-9B, for example, combines 24 linear-attention layers with eight full-attention layers. The linear layers maintain a different running state. The simple KV formula therefore must not be applied to all 32 layers. Image processing also requires its own weights and work areas. The specific configuration and execution software determine the actual requirements.

MoE means Mixture of Experts: each calculation step uses only some of several expert groups. The weights of all groups may nevertheless need to remain in memory. An active-parameter count therefore does not automatically describe storage for all weights. This is another architecture case; Qwen3.5-9B is not an MoE model.

Offloading changes the workload

llama.cpp can split a model between the CPU and graphics card. Parts then stay in system RAM instead of everything occupying graphics memory. This can allow a model to start despite exceeding the graphics card’s capacity. The trade-off may be longer waits because CPU calculations and data transfers become part of execution. Available splits also depend on supported hardware and software versions.

A practical test therefore records graphics-memory use, system RAM, time until the first output token and subsequent tokens generated per second. The test includes the intended long input, room for the response and the expected number of simultaneous requests. A successful short chat does not establish that this larger load fits.

Quality and memory together

A larger Q4 model is not automatically better than a smaller Q8 model. Training and task also affect answers. A small test set could contain three questions about a document and the expected supporting passages. The comparison counts incorrect claims, omissions and unusable output formats. Previously unused cases help show whether the result extends beyond the initial test.

The linked memory calculator can support an initial shortlist. A reproducible plan records the model revision, number format, execution software, planned context, simultaneous requests and available memory. Without suitable architecture data, the result remains a rough estimate. Actual memory measurements under the intended load and a quality test are needed to confirm a useful configuration.

Memory planning for model selection: LLM VRAM estimator.

Sources

1. llama.cpp: project and hardware support
2. Qwen: Qwen3.5-9B official model card
3. Mistral: Ministral 3 14B Instruct official model card
4. vLLM: GPU installation
5. Hugging Face Transformers: How caching works
6. Hugging Face Transformers: Cache strategies
7. Qwen3.5-9B: config.json

Sources checked again: 25 October 2026. Calculated examples, not original hardware measurements.