Tokens per second
Short answer
Tokens per second is how fast a model produces text. A token is roughly three-quarters of a word, so 20 tokens per second is about 15 words per second — faster than most people read.
Anything above about 15 tokens per second feels like a conversation. Below 5 it feels like waiting. On Nightjar's test machine, an RTX 3060, the small models run at 65–69 and the large ones at 24–27; with no graphics card at all, the small models manage 13–15 and the 12B model drops to 2.8.
The figure also depends on how long the conversation already is, because every new token is produced with the whole history in attention. A speed quoted on a fresh prompt will not hold at message thirty.
Related terms
Other ways people ask this
- what is tokens per second