GPUFits
Quantization Explained: Q4 vs Q8 and What You Actually Lose
updated 2026-08-04 · verified 2026-08-04
Quantization is how a model that “needs” 140GB runs on a $700 graphics card. Here’s what you’re actually trading.
What quantization does
A model’s weights are stored as numbers. FP16 stores each weight in 16 bits; quantization repackages them into 8, 6, 5, 4, 3 or even 2 bits, using clever grouping (the “K” quants) so the most important weights keep more precision. You shrink the file — and lose a little fidelity.
The measured numbers (Llama-3.1-8B GGUF)
| Quant | File size | % of FP16 | Quality guidance |
|---|---|---|---|
| FP16 | ~16.1 GB | 100% | reference |
| Q8_0 | 8.5 GB | 53% | indistinguishable for almost all uses |
| Q6_K | 6.6 GB | 41% | the “if you have VRAM to spare” pick |
| Q5_K_M | 5.7 GB | 35% | excellent, minor losses on hard tasks |
| Q4_K_M | 4.9 GB | 30% | the default — best size/quality trade |
| Q3_K_M | 4.0 GB | 25% | noticeable degradation; emergency only |
| Q2_K | 3.2 GB | 20% | for curiosity, not for work |
How to choose
- Start at Q4_K_M. It’s the community default for a reason: ~70% VRAM savings, and in blind comparisons most people can’t reliably tell it from FP16 in conversation.
- Upgrade to Q6_K or Q8_0 when VRAM is free. If your card holds Q8 comfortably, take it — especially for math, code with long dependency chains, or tool-use agents where small errors compound.
- Drop below Q4 only to make something fit at all — and prefer a smaller model at Q4 over a larger one at Q3/Q2. A well-quantized 8B beats a mangled 13B.
The KV cache wildcard
Quantizing weights is only half the story. The KV cache (the model’s working memory for your conversation) defaults to fp16 and can exceed the weights at long contexts. Some runtimes support q8 KV cache, halving that cost with minimal quality impact. If you’re context-starved, enabling q8 KV is often a better move than dropping weight precision — see our KV cache guide.
Speed bonus
Lower precision isn’t just smaller — it’s faster. Token generation is bottlenecked by memory bandwidth: reading 4.9GB per token beats reading 8.5GB. Expect Q4_K_M to generate roughly 25–35% more tok/s than Q8_0 on the same hardware.
FAQ
Is Q4_K_M much worse than Q8_0?
For chat and coding, most users can't tell them apart in blind tests. Q8 shows measurable gains on math, long reasoning chains, and precise instruction following. If you have the VRAM, Q6_K is the sweet spot; Q4_K_M is the value pick.
Should I ever use Q2 or Q3?
Only when nothing else fits. Below Q4, quality drops noticeably — the model starts losing facts and coherence. A smaller model at Q4 usually beats a bigger model at Q2.
Does quantization affect speed or just size?
Both. Lower precision means fewer bytes to read from memory per token, so Q4 is roughly 30% faster than Q8 on the same GPU. Generation speed is memory-bandwidth bound.