Quantisation: fitting useful models into local hardware

A model is usually trained and released in a high-precision format such as FP32, BF16, or FP16. FP32 uses four bytes per weight; BF16 and FP16 use two. An 8B model therefore needs roughly 32 GB in FP32 or 16 GB in a 16-bit format for its weights alone. The runtime also needs memory for the context, KV cache, temporary buffers, and operating system. A 12 GB consumer GPU cannot hold all the weights of that 8B FP16 model, while a 70B model requires substantially more memory.

Quantisation replaces the original floating-point values with lower-bit representations, normally arranged in blocks with a scale and additional metadata. It is an approximation, not a deletion of decimal places. It reduces model size and can improve inference speed, but may reduce accuracy or change behaviour. Evaluation must measure that change on the intended workload.

Q8, usually encountered as Q8_0 in GGUF tooling, stores most quantised weights at approximately eight bits instead of sixteen or thirty-two. Scales, metadata, and tensors retained at other precisions increase the effective size. A well-made Q8 conversion is often close to FP16 or BF16 on general language tasks, which makes it a useful quality baseline when memory is available. Sensitive models and specialist tasks still need testing, and Q8 does not reduce the memory required by a long context unless the KV cache is quantised separately.

The official LFM2.5-2.6B GGUF repository shows the size difference without changing model family. Its Q8_0 file is about 2.9 GB, while Q4_K_M is about 1.7 GB. The difference can determine whether the model fits alongside other applications. Keeping a Q8 copy provides a direct comparison for evaluation on the target workload.

Q4_K_M is a common first test when a model needs to run on consumer hardware. It puts the main weights into the four-bit class, while the K-quant layout uses grouped scales and the M recipe keeps selected tensors at higher precision. The exact tensor recipe can vary with the model architecture and quantiser version, so the name does not mean that every value in the file uses four bits.

Q4_K_M appears frequently in local model libraries because it can make 7B, 12B, and larger models fit on consumer hardware. The local stack includes Gemma 4 12B through the Ollama Q4_K_M build at about 7.6 GB, as well as the LFM2.5 files above. These sizes do not imply a consistent quality change across models. A recent LFM2.5-2.6B quantisation report found severe degradation in several Q4_K_M conversions, while one conversion was unaffected. The specific file must be evaluated rather than selected by format name alone.

IQ2_XXS belongs to llama.cpp's i-quant family and uses approximately 2.06 bits per weight before other file contents are considered. It can use an importance matrix derived from calibration data to reduce error in more important weights. This is an aggressive format, and its suitability depends on the model and workload. It is relevant when a larger quantisation does not fit on the target hardware.

The quality of an IQ2_XXS file depends on the model, the calibration data, the quantiser version, and the runtime. Prefer a file made from a high-precision source with an importance matrix over a file that has been repeatedly requantised from another small quant. Treat IQ2_XXS as a measured operating point: test the prompts, tool calls, languages, and long-context cases that matter for the workload. A two-bit model that meets the workload's requirements is more useful than a four-bit model that does not fit on the target hardware.

Ternary weights take one of three values: -1, 0, or +1, with a shared scale applied to a group of weights. This differs from post-training quantisation of an ordinary FP16 model. The training or conversion process accounts for the restricted set of weight values, while the scale carries magnitude information. The theoretical weight representation is approximately 1.58 bits per weight, but practical encodings use more space for scales, packing, metadata, and tensors stored in other formats.

The local stack runs Ternary-Bonsai-27B-Q2_0.gguf through llama-swap. The model file is about 7.2 GB for a 27B-class model. Parameter count alone therefore does not specify memory use. This model-specific ternary representation is not interchangeable with an ordinary Q2 post-training quantisation, and its runtime currently has implementation-specific compatibility requirements.

A model's weight format is only one part of the memory budget. During generation, the runtime also holds the KV cache for the active context, and that cache grows with context length, batch size, and the model architecture. Quantising the weights to Q4_K_M does not automatically make a 128K context cheap. The cache may have its own format and quality trade-off, and the prompt-processing and generation paths may have different constraints.

This separation is visible in a practical local stack: the model weights vary between Q8, Q4_K_M, and the ternary Q2_0 file, while the Ollama and llama.cpp services configure the KV cache separately at Q8. A sensible deployment therefore measures both. If memory is tight, first decide whether the limit is the model, the cache, or the offload boundary before choosing a more aggressive weight quantisation.

A practical comparison can start with Q8 as a high-quality reference. Test Q4_K_M and other intermediate formats when memory is limited. Consider IQ2_XXS or a model-specific ternary checkpoint only when larger files do not fit and evaluation supports the lower-bit version. Keep the prompt set, tools, context length, sampling settings, and runtime constant when comparing formats; otherwise the result combines several changes.

The final decision depends on the workload. Compare factual accuracy, code execution, tool-call validity, refusal behaviour, latency, tokens per second, cold-start time, and peak memory. Quantisation is successful when it reduces deployment cost or memory use without making the required outcomes unacceptable. Select a model that fits the target hardware, runs reliably, and meets the workload's quality and risk thresholds.