A model file fits on disk, but the server runs out of memory when it receives a long document. Choosing a quantisation requires a budget for the running workload, followed by a comparison of quality and performance on the target hardware.
Establish the memory budget
Weight storage is a useful first estimate. Eight billion parameters at two bytes each occupy roughly 16 GB in decimal units before other data is included. Four-bit values would occupy roughly 4 GB if every parameter used exactly four bits, but practical files also contain scales, metadata, and tensors stored at other precisions.
Runtime memory includes more than the weights. Account for the active context, attention cache where applicable, temporary buffers, concurrent requests, and other applications using the device. Leave headroom and measure peak allocation under a representative load.
Distinguish system RAM from GPU memory. Offloading part of a model to the CPU changes where memory is used and can change performance. Record the offload configuration when comparing runs.
Select a small set of candidates
The following formats are relevant to GGUF-based tooling. They describe representations, not a model-independent quality ranking.
| Candidate | Representation | Role in a comparison |
|---|---|---|
| FP16 or BF16 | Two bytes per weight before other contents | Original or high-precision baseline where available |
| Q8_0 | Eight-bit blocks with scale overhead | A less compressed comparison when memory permits |
| Q4_K_M | Mixed tensor recipe based on K-quants | A candidate for reducing weight memory |
| IQ2_XXS | Approximately 2.06 bits per weight for this encoding | An aggressive option requiring close quality checks |
The llama.cpp quantiser lists supported formats and their requirements. A format name does not establish the exact size or quality of a particular model file. Record the source model, conversion provenance, calibration method where relevant, file digest, and runtime version.
Compare conversions of the same source model first. Changing both the model and its quantisation makes it harder to identify why a result differs. Include an intermediate format when the memory budget allows it.
Keep cache settings separate
Weight quantisation does not automatically change the attention cache. For example, the llama.cpp server exposes separate key and value cache type settings.
Measure memory with the intended context length and number of simultaneous requests. A short single-user test can miss the limit reached during normal service use. Cache layout and growth depend on the model architecture and runtime.
If the cache dominates, reducing weight precision may leave the actual constraint unresolved. Compare a shorter supported context, lower concurrency, or a supported cache format as separate changes. Test their effect on the workload as well.
Treat ternary models as a separate comparison
Ternary models such as BitNet b1.58 use weights from three values: -1, 0, and +1. They involve a model and training approach designed around that representation. Microsoft's BitNet framework documents supported models, hardware, and inference kernels.
A ternary checkpoint is not simply another quantisation of the ordinary model in the table. Include it as a different model candidate with its own compatibility and quality tests. The theoretical representation of three values is about 1.58 bits, but complete file and runtime sizes also depend on packing, scales, and other tensors.
Run the workload comparison
Keep prompts, tools, sampling settings, context length, and runtime constant for the initial weight-format comparison. Use the same task cases, including long inputs and known failures. Repeat cases where output variability could change the decision.
Measure outcomes the application needs: correct extraction, executable code, valid tool calls, or accurate answers with supporting sources. Report serious failures separately from an average score. A smaller file is of little use if it increases the correction work beyond an acceptable level.
Alongside quality, record prompt-processing time, generation speed, time to first response, cold-start time, and peak memory. A compact format does not guarantee a speed gain on every device; kernel support and data movement affect the result.
The decision record should name the selected file and runtime, supported context and concurrency, measured quality, peak memory, and known failures. Keep the comparison cases so later model or runtime upgrades can be assessed against the same requirements.