FEEDBACK IN TRANSFORMERS

Summary from the TLDR AI newsletter

TransformerFAM provides a feedback mechanism that allows Transformers to attend to their own latent representations. This can, in theory, introduce recurrence into the model for processing extremely long inputs in context.