UNDERSTANDING 100X SPEED UP IN LLM INFERENCE

Summary from the TLDR AI newsletter

With open models becoming useful for many enterprise tasks, people are starting to explore deployment optimizations. However, the landscape is complex and scattered. This post is a nice deep dive into many of the standard techniques used to speed up language model serving.