RAG IS DEAD, CONTEXT ENGINEERING IS KING

Summary from the TLDR Founders newsletter

Chroma's founder says RAG is dead because it bundles three concepts poorly - the real job is context engineering, figuring out what belongs in the context window for each LLM generation. The best teams now use a two-stage approach: first-stage retrieval (vector search, text search, and metadata) culls 10,000 candidates down to 300, then LLMs re-rank those to the final 20-30. This matters because context rot is real - despite perfect needle-in-haystack marketing, models degrade as you add tokens. The future looks radically different: models will stay in latent space without converting back to text and retrieve continuously during generation instead of once at the start. What we call "memory" is just context engineering with better branding.