SPEEDING UP THE MPT 65K CONTEXT MODEL

Summary from the TLDR AI newsletter

The recent MosaicML MPT models are still gaining popularity. One of the big contributions was the “Storywriter” model which could use up to 65k tokens in context. Inference with this model was painfully slow. By adding a Key/Value cache we can speed up generation by almost 10x.