Skip to content

Edge inference

Short answer

Edge inference is running a model near where the data is — on the device or on local hardware — instead of in a central datacentre.

The usual arguments are latency and bandwidth. For personal software the stronger argument is jurisdictional: data that never moves cannot be subpoenaed from, retained by or breached at a third party.

The constraint is memory. Edge hardware has gigabytes where datacentre hardware has hundreds, which is the whole reason quantisation and small efficient models exist.

Related terms

Other ways people ask this

  • what is edge inference

Related