BLOCK TRANSFORMER FOR IMPROVED INFERENCE

Summary from the TLDR AI newsletter

This repository introduces a new type of Transformer with a dramatically reduced KV cache size. It hasn't been proven at scale, but it can - in theory - match regular Transformer performance.