The paper proposes a new approach to Vision Transformers by introducing a mixed-resolution sequence of tokens to replace the standard uniform grid, where each token represents a patch of arbitrary size. This approach uses the Quadtree algorithm and a novel saliency scorer to construct a patch mosaic that processes important image regions with higher resolution while routing low-saliency areas in low resolution, resulting in significant accuracy gains on image classification while controlling for the computational budget.
VideoCrafter is an open-source video generation and editing toolbox for crafting video content, which contains three models: Generic Text-to-video Generation, Personalized Text-to-Video Generation with LoRA, and Video Generation with More Condition Controls
The paper proposes a new task that utilizes both music and text for 3D dance generation, which can guide humans to perform richer movements. To tackle the challenge of the existing datasets containing paired motion with a single modality, the authors utilize a 3D human motion VQ-VAE to project the motions of the two datasets into a latent space, and propose a cross-modal transformer architecture to integrate text instructions. The approach generates realistic and coherent dance motion conditioned on both music and text, while keeping comparable performance conditioned on two single modalities.
The Pythia paper from EleutherAI is finally here! They released hundreds of checkpoints of Transformer models over time as they train. This is useful because it is not at all clear what skills models gain over time, but now the community can inspect the dynamics of training and gain massive insights into potential areas for improvement and efficiency.
This paper discusses the evolution of language modeling for artificial intelligence algorithms, with a focus on recent advances in large language models (LLMs) with billions of parameters. The paper offers a comprehensive review of LLMs, including pre-training, adaptation tuning, utilization, and capacity evaluation, and provides a useful resource for researchers and engineers.
AUDIT is a new instruction-guided audio editing model based on latent diffusion models, which can automatically modify only the necessary audio segments and only requires edit instructions instead of full target audio descriptions as text input. The proposed model achieved state-of-the-art results in both objective and subjective metrics for several audio editing tasks, including adding, dropping, replacement, inpainting, and super-resolution.
OpenAI is leading the way in capabilities research, they also are a strong player in developing safety systems for their language models. They outline a few directions they are tackling such as factuality, child safety, privacy, and more. They have made progress on all of these aims and the fact that GPT-4 was finished training 6 months before its launch, but held back to focus on safety concerns is interesting. The biggest take away here is that GPT-4 is 40% less likely to generate factually incorrect output than gpt-3.5-turbo.
AI-generated videos are gaining popularity, often depicting surreal scenes or celebrity memes. The technology's rapid advancement raises questions about its potential in Hollywood, such as casting movies or modeling scenes. Although challenges remain, experts believe AI will have a significant impact on content creation, but it won't replace human talent in the film industry anytime soon.
Similar to Alpaca from Stanford, Koala was trained on dialog and instruction data. However, they didn’t use distillation from the OpenAI models, but sourced their data from freely available sources on the web. Their model actually seems comparable to strong closed source models and the code to fine-tune along with the dataset are available for further study.
This paper proposes a two-stage training scheme for generating text-editable and pose-controllable character videos, using easily obtainable datasets and a pre-trained text-to-image model. The method successfully generates pose-controllable character videos and retains the editing abilities of the pre-trained model.
2023.04.05·
follow-your-pose.github.io·GitHub repository·TLDR AI
ReBotNet is a video model that is designed to enhance live streams and video conferences in real-time. They use a dual-branch network that learns different types of video abstractions like location and time based ones. They then use a common decoder network to output the enhanced frames. It runs at 50 fps, uses about 5gb of memory and performs on par with models 8x larger.
Clearview AI built it’s facial recognition database using 30 billion images from Facebook. Because this database has been used by the police nearly a million times now, Clearview is now being accused of putting people in a perpetual police line-up.
VideoMAE pioneers a scalable and versatile self-supervised pre-training approach for constructing video foundation models using a dual masking strategy and progressive training paradigm. Achieving state-of-the-art performance on multiple datasets, VideoMAE proves its efficacy as a general video representation learner for a wide range of downstream tasks.
People are exploring how language models can use tools - Hugging Face has thousands of AI models hosted on their website, what if each of those was a potential tool? It turns out, you can use ChatGPT as the controller and each Hugging Face model as a specialized tool. Do you want to do a visual question answering task followed by summarization? This work is a first step towards potentially automating that task.
InceptionNeXt introduces a novel approach to enhance large-kernel-based CNN models' efficiency while maintaining performance by decomposing large-kernel depthwise convolutions into four parallel branches along the channel dimension. This innovative method results in higher training throughputs and improved top-1 accuracy, offering a more sustainable baseline for future architecture designs.
Auto-GPT is an experimental open-source application showcasing the capabilities of the GPT-4 language model. This program, driven by GPT-4, autonomously develops and manages businesses to increase net worth. As one of the first examples of GPT-4 running fully autonomously, Auto-GPT pushes the boundaries of what is possible with AI.