Parse notes
Recent progress in large language models has been dominated by large industry players. However, many of the key ideas still come from academia. Princeton is hoping to continue contributions to language and intelligence with this new center. They’re hiring for research scientists, engineers, and postdoc scholars.
Laion just released the Paella image generation model. It uses a discrete, token based, latent space and a cross entropy denoising objective. One goal of this project is to make an image synthetics model that almost anyone can understand. Their sampling code is only 12 lines long!
What if we didn’t have to tune the pile of hyperparameters we do to make deep learning work? This is a pretty technical paper that presents Automatic Gradient Descent which is a new optimizer that takes into account network architectures and eliminates a set of previously required hyperparameters. They work through all the math thoroughly, and the resulting optimizer seems to be fairly simple. They don’t have it working yet for all network architectures, but that feels like only a matter of time.
Recent breakthroughs in natural language processing have paved the way for foundation models in computer vision, producing all-purpose visual features through self-supervised methods and large, diverse datasets. By utilizing an automatic pipeline for curated data and training a 1-billion-parameter ViT model, researchers have created smaller models that surpass existing all-purpose features in most benchmarks at image and pixel levels.
We wrote recently about the new type of genitive model from OpenAI that is faster than diffusion and more stable than a GAN. They just released code and model checkpoints on a set of small experimental data sets. This model is a cool next step in generative image modeling, and now we can play around with it!
Combining spectral and multi-headed attention layers in the novel SpectFormer architecture improves transformer performance for image recognition tasks. This approach achieves state-of-the-art accuracy on ImageNet-1K and consistently strong results in transfer learning, object detection, and instance segmentation tasks across various datasets.
Anthropic recently announced plans to spend a billion dollars on training a massive language model. They announced this plan before their strategic partnership with Google, which may offer cheaper compute via their TPU pods. In any case it seems like one of the largest expenses of this endeavor is actually AI researcher salaries, followed closely by compute costs.
The European parliament is preparing tough new measures over the use of artificial intelligence, including forcing chatbot makers to reveal if they use copyrighted material.
Wolverine uses GPT-4 to edit your scripts and tell you what went wrong.