Quantisation
Short answer
Quantisation stores a model's weights at lower numerical precision — 4 or 5 bits per weight instead of 16 — which shrinks the file by roughly three quarters at a modest cost in answer quality.
A 9-billion-parameter model at full 16-bit precision is about 18 GB. The same model quantised to 4 bits is around 5 GB, which is the difference between needing a datacentre card and running on a mid-range gaming GPU.
The cost is real but smaller than people expect: heavily quantised models get slightly worse at long chains of reasoning and at recalling obscure facts, and barely worse at reading a document in front of them. Since reading the document in front of you is exactly what a browser assistant does, quantisation is a good trade here specifically.
Related terms
Other ways people ask this
- what is quantisation