MIXTURE‑OF‑DEPTHS ATTENTION

Summary from the TLDR AI newsletter

MoDA introduces a new attention mechanism that lets each head access both current‑layer and earlier‑layer key‑value pairs, helping preserve useful signals as models scale deeper.