MHC: MANIFOLD-CONSTRAINED HYPER-CONNECTIONS

Summary from the TLDR AI newsletter

Hyper-Connections (HC) compromise the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. Manifold-Constrained Hyper-Connections (mHC) is a general framework designed to address this issue. mHC projects the residual connection space of HC onto a specific manifold to restore the identity mapping property while incorporating rigorous infrastructure optimization to ensure efficiency. It is effective for training at scale, and it offers tangible performance improvements and superior scalability.