METADATA CONDITIONING ACCELERATES LANGUAGE MODEL PRE-TRAINING

Summary from the TLDR AI newsletter

Not a lot of research in generic pre training has been published recently. This work shows how including metadata in the first of training and annealing it out towards the end improves overall model performance.