RAG 2.0

Summary from the TLDR AI newsletter

Retrieval Augmented Generation (RAG) is a way to inject knowledge into a potentially stale language model. Unfortunately, the current paradigm of “frozen RAG”, where only part of the pipeline is trained and the actual model itself isn’t updated, performs poorly outside of demos. This blog outlines the next generation of RAG where everything is tuned specifically for the task. An open model like Mistral 7B can outperform traditional GPT-4 RAG in this system.