PERPLEXITY CANNOT ALWAYS TELL RIGHT FROM WRONG

Summary from the TLDR AI newsletter

Perplexity is a function that measures models' overall level of surprise when encountering a particular output. It has gained significant traction in recent years as both a loss function and as a simple-to-compute metric of model quality. However, it may be an unsuitable metric for model selection. Perplexity will not always select for the most accurate model - any increase in model confidence must be accompanied by a commensurate rise in accuracy for the new model to be selected.