Skip to content

Quantisation

Short answer

Quantisation stores a model's weights at lower numerical precision — 4 or 5 bits per weight instead of 16 — which shrinks the file by roughly three quarters at a modest cost in answer quality.

A 9-billion-parameter model at full 16-bit precision is about 18 GB. The same model quantised to 4 bits is around 5 GB, which is the difference between needing a datacentre card and running on a mid-range gaming GPU.

The cost is real but smaller than people expect: heavily quantised models get slightly worse at long chains of reasoning and at recalling obscure facts, and barely worse at reading a document in front of them. Since reading the document in front of you is exactly what a browser assistant does, quantisation is a good trade here specifically.

Related terms

Other ways people ask this

  • what is quantisation

Related