SYNTHETIC PRETRAINING

Summary from the TLDR AI newsletter

Synthetic pretraining is the large-scale use of synthetic data sources throughout training. It is a practical response to the fact that it is difficult to collect data that reliably produces the capabilities we want. Synthetic pretraining seems to open up a new simultaneous space of data and model innovation. This post looks at what synthetic pretraining is, how it works in practice, and the stages of synthesis.