CONTEXT IS SOFTWARE, WEIGHTS ARE HARDWARE

Summary from the TLDR Dev newsletter

While context acts as temporary "software" for in-context learning, weight updates permanently modify a model's fundamental computational capabilities and internal representations. Effective continual learning requires both increasing context windows for working memory and persistent weight-space learning for accumulated, fundamental knowledge.