Skip to content

Quantisation explained

Short answer

Quantisation stores a model's weights at lower precision — commonly 4 bits instead of 16 — cutting the file to roughly a quarter of its size for a modest and uneven loss of quality.

The arithmetic

A 9-billion-parameter model at 16-bit precision is about 18 GB. At 4 bits it is around 5 GB. That is the difference between needing datacentre hardware and running on a mid-range gaming card.

Nightjar's catalogue is quantised, which is why a 9B model is a 5.3 GB download.

What it costs

Not much, and not evenly. Heavily quantised models lose most on long chains of reasoning and on recalling obscure facts, and least on reading a document placed in front of them.

Since reading a document in front of it is exactly what a browser assistant does, quantisation is an unusually good trade in this specific application.

How to think about the labels

You will see names like Q4_K_M. The number is roughly the bits per weight; the letters describe the scheme. Lower numbers mean smaller and worse.

Nightjar picks for you based on what fits your memory, so the labels only matter if you are importing your own GGUF files.

Other ways people ask this

  • quantisation q4 vs q8 quality
  • What does quantisation cost me in answer quality?

Related