Whether it’s designing microchips or dreaming up new proteins, sometimes it seems like neural networks can do anything. Infamously, however, these brain-inspired AI systems work in mysterious ways, raising concerns that what they are doing might not make any sense. New research suggests that 200-year-old math could help shed light on how neural networks perform complex tasks such as predicting climate or modeling turbulence, a new study finds. This in turn could help boost the accuracy of neural networks and the speed at which they learn, researchers say.
Diffusion is currently one of the best generative methods in terms of sample quality. This comes from the nice properties of the loss and denoising process. In the community, different weighting schemes are used for noise per step which leads to different empirical results. It turns out that these non-uniform weighting schemes can be understood from the lens of likelihood maximization. This is a great step towards a deeper theoretical understanding of this modern workhorse.
Applying large language models to robots is challenging due to their lack of experience with the physical world. To overcome this, a guided decoding strategy is used to construct an action sequence that is both likely according to the language model and realizable in the environment. This strategy solves complex tasks in a robotic setting by leveraging the knowledge of both models. They compare their work to SayCan.
This paper discusses the use of diffusion models for zero-shot image restoration and proposes a method to handle images of arbitrary sizes while maintaining the excellent characteristics of zero-shot. The proposed method, called Mask-Shift Restoration, addresses local incoherence, and Hierarchical Restoration alleviates out-of-domain issues. These simple, parameter-free approaches can be used not only for image restoration but also for image generation of unlimited sizes.
The study shows that early dropout, applied only during the initial phases of training, can mitigate underfitting by reducing the directional variance of gradients and aligning them with the entire dataset’s gradient. The proposed method consistently improves generalization accuracy in various vision tasks and encourages further research on regularization in deep learning.
This work introduces Context Clusters (CoCs), a new paradigm for visual representation that views an image as a set of unorganized points and extracts features via a simplified clustering algorithm. CoCs are convolution- and attention-free and only rely on clustering for spatial interaction. Despite not targeting state-of-the-art performance, CoCs achieve comparable or even better results than ConvNets or ViTs on several benchmarks.
To get the inside story behind the ChatGPT — how it was made, how OpenAI has been updating it since release, and how its makers feel about its success — the author talked to four people who helped build it. They are Sandhini Agarwal (works on policy at OpenAI), Liam Fedus (scientist who worked on ChatGPT), John Schulman (cofounder of OpenAI), and Jan Leike (leader of OpenAI’s alignment team)
With the AI race heating up, Google is feeling the pressure. In a recent internal meeting the Google Cloud CEO said “The game is never over in the first minute” and urged employees to work as a team to continue to improve AI capabilities. He talked about what the history books would say about Google navigating this time period and expressed optimism about the future of search.
Another paper from the systems group at Stanford working on improving language modeling abilities for long context. They work to build a subquadratic attention replacement and build on much of their recent work in state space models and long convolutions work. They introduce the “Hyena Hierarchy” which is a drop in replacement for attention, while reducing FLOPs, and improving long context scaling. This is an exciting next step and worth a read.
The paper discusses the use of diffusion models for zero-shot image restoration and proposes solutions for handling images of arbitrary sizes. The current methods only deal with fixed-size images, but the proposed approach uses Mask-Shift Restoration to address local incoherence and Hierarchical Restoration to alleviate out-of-domain issues. These parameter-free approaches can be used not only for image restoration but also for image generation of unlimited sizes, making them a potential general tool for diffusion models.
The paper proposes a non-autoregressive (NAR) generative model called Stratified Image Transformer (StraIT) that outperforms existing autoregressive (AR) and diffusion models (DMs) in high-quality image synthesis. StraIT leverages the hierarchical nature of images to encode visual tokens into stratified levels, which alleviates modeling difficulty and lifts the generative power of NAR models. The experiments show that StraIT achieves FID scores of 3.96 at 256×256 resolution on ImageNet without leveraging any guidance in sampling or auxiliary image classifiers.
The paper proposes Collage Diffusion, a collage-conditional diffusion algorithm that enables precise control over the spatial arrangement and visual attributes of objects in generated images. By modifying the text-image cross-attention with alpha masks and learning specialized text representations per layer, users can edit individual components of generated images and control image harmonization on a layer-by-layer basis. Collage Diffusion generates globally harmonized images that maintain desired object locations and visual characteristics better than prior approaches.
One of the top open science collaboratives in ML, Eleuther has made many contributions to the field over the past year and a half. They have published 28 papers, 10 different models, and dozens of code bases. Many of the most exciting developments such as RWKV, GPTNeo, and open replications of AlphaFold are coming from this group. Read more here about what they’re working on and what’s coming next.
With a context length of 2048 tokens, this instruction tuned model is an excellent foundation model for text generation. With improved CoT, in context learning, and general performance (as much as 7.4% over FlanT5-xxl) this model is a great step forward in open source language models. This model uses the UL2 objective which is a mixture of denoisers and has shown impressive performance at general language modeling. Importantly, they also remove the need for cumbersome mode tokens by an additional 100k steps of training before the Flan instruction tuning process.
Text-conditional diffusion models can generate high-quality images, but they often require additional controls to ensure precise image output. In this paper, the authors introduce Collage Diffusion, a collage-conditional diffusion algorithm that allows users to control the spatial arrangement and visual attributes of objects in the scene. By defining a collage with an ordered sequence of layers, each with an RGBA image and corresponding text prompt, users can control image harmonization on a layer-by-layer basis and edit individual objects in generated images while keeping other objects fixed. Collage Diffusion modifies text-image cross-attention with alpha masks to ensure different parts of the input text correspond to the various locations specified in the input collage layers. This approach generates globally harmonized images that maintain desired object locations and visual characteristics better than prior approaches.
Aligning language models is hard. One method uses “Human Feedback” to train a reward model. However, if we can compute the reward model on the fly from each example, there is no need for expensive feedback. In this example, they use another language model to score the toxicity of a piece of text. They can use that as a reward to decrease the toxicity of a model’s output.
If allowed by the user, Bing Chat can see currently open websites. The authors show that an attacker can plant an injection in a website the user is visiting, which silently turns Bing Chat into a Social Engineer who seeks out and infiltrates personal information. The user doesn't have to ask about the website or do anything except interact with Bing Chat while the website is opened in the browser.
Meta’s efforts to overhaul its advertising technology in response to Apple’s privacy changes are leading to bumper results for brands but also fears from marketers they are being forced to relinquish too much control to the social media platform. The company has been pouring investment into applying machine learning and artificial intelligence to its advertising systems in recent months.
The paper describes a method for reconstructing a full 360◦ photographic model of an object using a single image of it. The approach involves fitting a neural radiance field to the image, but the problem is found to be severely ill-posed. To address this, an off-the-shelf conditional image generator based on diffusion is used, and a prompt is engineered to encourage it to "dream up" novel views of the object. The DreamFusion method is used to fuse the given input view, the conditional prior, and other regularizers in a final, consistent reconstruction.
MLLM is a multimodal transformer trained on internet scale text/image data. It uses a frozen CLIP image encoder and a Magneto decoder only transformer. It has 1.3B parameters and was trained for 360B tokens. It was trained with TorchScale, a new exciting framework, and finished off with language only instruction tuning. It got 26% on an IQ test, which is better than random chance. It has 73% zeroshot ImageNet classification and positive transfer between modalities. It has a 64k vocab size and no code yet but should be coming soon.
Georgi has been working on Whisper.cpp which dramatically speeds up the runtime of OpenAI’s state of the art speech to text model. He has included a WASM version for the web. This latest model uses 4-bit integer quantization. Interestingly, this type of quantization only seems to work on larger models and fails on smaller models.
The authors propose a framework for scaling robot learning in multi-task and multi-scene manipulation in kitchen environments, both in simulation and in the real world. The proposed framework, called CACTI, comprises four stages that handle data collection, data augmentation, visual representation learning, and imitation policy training. The approach uses state-of-the-art generative models for data augmentation and pre-trained visual representations to improve training efficiency. Experimental results show that CACTI enables efficient training of a single policy that can perform 10 manipulation tasks involving kitchen objects on a real robot setup, and 18 semantic tasks across 100 layout variations for each individual task in a simulated kitchen environment.
2023.03.01·
cacti-framework.github.io·GitHub repository·TLDR AI
Scene Diffuser is a conditional generative model for 3D scene understanding. This repository is the official implementation of paper "Diffusion-based Generation, Optimization, and Planning in 3D Scenes". SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning. In contrast to prior work, SceneDiffuser is intrinsically scene-aware, physics-based, and goal-oriented.