INTRODUCING NESTED LEARNING: A NEW ML PARADIGM FOR CONTINUAL LEARNING

Summary from the TLDR AI newsletter

Continual learning, the ability to acquire new knowledge without forgetting old information, remains a major obstacle to AGI since current models suffer from “catastrophic forgetting,” where learning new tasks diminishes proficiency on previous ones. A proof-of-concept architecture called Hope, a self-modifying model that optimizes its own memory, outperformed modern recurrent models and transformers on language modeling and excelled particularly at long-context needle-in-a-haystack tasks.