DO SMALL MODELS MEMORIZE MORE AND REASON LESS?

Summary from the TLDR AI newsletter

This delightful article discusses two novel concepts: integration and coverage. These refer to two axes of capabilities in language models - generalization and memorization. The idea posed is that both are useful for tasks and it is hard to tell which is being used for those tasks. Additionally, small models might be good on benchmarks because of increased coverage at the expense of true integration.