Tokens per second, and what the number actually means
Short answer
A token is roughly three-quarters of a word. Above about 15 tokens per second a local model reads faster than you do; below 5 it is a wait. Our test machine ranged from 69 down to 2.8 depending on model and backend.
Converting it to something human
Fifteen tokens per second is about eleven words a second, comfortably faster than reading speed. Twenty-seven, which is what Qwen 3.5 9B managed on our RTX 3060, feels immediate.
Three tokens per second — Qwen 3.5 9B with no GPU — is about two words a second, which is slower than typing.
The number nobody quotes
Time to first token: the pause before anything appears, which is dominated by how much document the model has to read first. A product can have an excellent tokens-per-second figure and still feel slow because every answer starts with a four-second silence.
Both numbers matter and most benchmarks publish one.
Why your figure will differ from ours
Every speed on this site came from one machine with a Vulkan backend. Different card, driver, thermal headroom and conversation length all move it.
That is also why pages about cards we have not tested say “not measured” rather than scaling our number to your hardware.
Other ways people ask this
- tokens per second how many is usable
- How many tokens per second do I need before a local model feels responsive?
Related
- Up a levelGuides: local AI, browser privacy and the hardware to run it
- Closely relatedRunning a local model on integrated graphics
- Closely relatedWhat a local model does to your battery
- Closely relatedWhat a local model actually costs to run
- Closely relatedRunning a local LLM without a graphics card
- Elsewhere on the siteCan your PC run a local AI model?
- Elsewhere on the siteHardware guide for local AI