TINY STORIES, 1 TRANSFORMER BLOCK CAN GENERATE STORIES

Summary from the TLDR AI newsletter

With models getting larger and more expensive we generally get better performance. But what happens if we go the other way? It turns out that tiny models can learn to generate coherent, novel, and grammatically correct stories. If you carefully construct the training data set and restrict the vocabulary to common words, you get some compelling completions from these miniscule models.