TINY LLAMA 1.1B PARAMETERS

Summary from the TLDR AI newsletter

The Tiny Llama project aims to train a 1.1B parameter Llama2 model for 3T tokens. According to scaling laws, this is a dramatic overtraining for compute optimality, but it brings significant gains for deployment optimality.