LONG AWAITED RWKV PAPER

Summary from the TLDR AI newsletter

Transformers displaced recurrent neural networks as the primary tool for most text based tasks. However, these Transformers are expensive and limited in context due to the architecture. RWKV is an attention variant (vaguely reminiscent of linear attention) that claims to have the best of both Transformers and recurrent models. The Discord is quite active, but there hasn't been a formal write up of this hyped model. This paper is the first to explain the model in depth.