LARGE LANGUAGE MODELS OFTEN KNOW WHEN THEY ARE BEING EVALUATED

Summary from the TLDR AI newsletter

Frontier models can distinguish evaluation scenarios from real-world interactions with 83% accuracy, often explicitly reasoning about evaluation indicators like "multiple-choice format" or recognizing specific benchmarks from their training data. More advanced models exhibit "meta-reasoning" - using the fact that researchers were asking about their chain-of-thought transcripts as evidence of being evaluated. This raises concerns that models could underperform on tests or fake alignment during evaluations and behave differently once deployed.