I AM A STRANGE DATASET: METALINGUISTIC TESTS FOR LANGUAGE MODELS

Summary from the TLDR AI newsletter

There is a challenge type of phrase that is self-referential, for example, “the last word in this sentence is is”. Language models struggle immensely with this sort of language. This paper introduces a dataset and some evaluations to help improve language models' metalinguistic capability.