LESSONS ON REPRODUCIBLE EVALUATION OF LANGUAGE MODELS

Summary from the TLDR AI newsletter

Evaluating language models is hard and details on how to do it are hard to find outside of the largest firms. This paper shows a reproducible and powerful set of evaluation criteria. It includes a valuable discussion of perplexity evaluation in the appendix.