Skip to content

Mixture of experts

Short answer

A mixture-of-experts model contains many specialised sub-networks and activates only a few per token, so it has the knowledge of a large model at the compute cost of a small one.

The catch for local use is that all the experts have to be in memory even though only some run. An MoE model is cheap to compute and expensive to store, which is exactly the wrong way round for a consumer graphics card.

This is why the models that work well locally are mostly dense rather than sparse, and why efficient architectures aimed at devices are a distinct product line.

Related terms

Other ways people ask this

  • what is mixture of experts

Related