TRAINING GREAT LLMS ENTIRELY FROM GROUND ZERO IN THE WILDERNESS AS A STARTUP

Summary from the TLDR newsletter

Training AI models requires compute, but there's a large amount of instability and huge variance in the quality of clusters, accelerators, and connectivity in compute providers. This article discusses the differences between training models at Google and training models 'in the wild' - outside of Google, where data scientists have to use GPUs instead of Google's TPUs. It makes comparisons between Google's infrastructure and tools to their equivalents in the outside world. With the right amount of brute technical strength, it is possible to train models from scratch that match Gemini Pro/GPT 3.5.