WHAT IS INFERENCE ENGINEERING? DEEPDIVE

Summary from the TLDR newsletter

Inference is when a model takes an input and generates an output, one token at a time. Inference engineering is becoming more widespread as open models become more capable. With closed models, inference engineering is only done by the AI engineers who build the model, but anyone can tinker with open models. This post discusses what inference engineering is and some interesting approaches to inference engineering worth knowing about.