LANGUAGE MODEL SELF-TEACHING FOR DOMAIN ADAPTATION

Summary from the TLDR AI newsletter

When trying to use language models in domains that require niche knowledge, you can either fine-tune or use some retrieval techniques. Both have drawbacks. This novel method uses self-generated synthetic data to better learn the information at test time. It shows marked improvements in standard adaptation benchmarks over both fine-tuning and RAG.