Researchers from Google propose a novel approach called SVDiff to address the limitations of existing text-to-image diffusion models for personalization, including overfitting and inefficient parameter storage. SVDiff involves fine-tuning the singular values of weight matrices, resulting in a compact and efficient parameter space that reduces the risk of overfitting, language-drifting, and has a significantly smaller model size, making it more practical for real-world applications.
Zero-1-to-3 is a framework for changing the camera viewpoint of an object from a single RGB image, using a conditional diffusion model that learns controls of relative camera viewpoint from a synthetic dataset. This approach shows strong zero-shot generalization abilities to out-of-distribution datasets and in-the-wild images, including impressionist paintings, and can be used for 3D reconstruction from a single image, outperforming state-of-the-art models by leveraging Internet-scale pre-training.
CG3D proposes a framework for zero-shot 3D geometric feature extraction by training a 3D encoder using point clouds, rendered 2D images, and text. Contrastive loss aligns features in a multimodal embedding space, and prompt tuning with trainable input parameters overcomes distribution shift issues. CG3D demonstrates impressive zero-shot, open-scene understanding, and retrieval capabilities, and serves as a strong starting weight for downstream 3D recognition tasks.
The paper proposes a solution to the challenge of robotic navigation in new environments by introducing CoDEPS, a continual learning approach that utilizes experience replay and domain mixing to mitigate catastrophic forgetting and adapt to changing conditions. The proposed method achieves state-of-the-art results and addresses the limited storage capacity of robotic systems.
The paper proposes LION, an efficient vision model that addresses the heavy computational costs of vision Transformers by using deep implicit models with stable memory costs. LION only inserts two equilibrium implicit layers in the pre-trained main backbone and prunes parameters according to the lottery hypothesis, reducing up to 11.5% of training parameter numbers while achieving higher performance than state-of-the-art baseline VPT on a wide range of datasets, especially under challenging scenes.
The paper introduces DiffusionRet, a diffusion-based text-video retrieval framework that models the correlation between text and video as their joint probability, p(candidates, query), from a generative viewpoint. DiffusionRet optimizes the generator through generation loss and the feature extractor through contrastive loss, leveraging both generative and discriminative methods to achieve superior performance on five commonly used text-video retrieval benchmarks, even in out-domain retrieval settings, and bringing fundamental insights into the related fields.
Instead of using an additive scalar score to determine how much influence a data instance has on final model performance, what if we used a full training simulator? This enables counter-factual exploration around dataset construction and long term training performance. This method tends to underestimate performance, but does a good job of capturing general dynamics for large language model training runs.
While GPT models may impact labor markets, the effect is not uniform across jobs. A preliminary look suggests physically active and mentally creative jobs are less likely to be affected by large language models. Not the most convincing methodology, but a good step at this early juncture to understand what our economic future might look like.
Duolingo has introduced two new features powered by OpenAI's GPT-4: Role Play, an AI conversation partner, and Explain my Answer, which provides contextual feedback on mistakes, in a new subscription tier called Duolingo Max. Duolingo turned to GPT-4 after finding that GPT-3 could not confidently handle the complex automated aspects of chats. The team sees the potential of GPT-4 to provide a more effective and engaging learning experience than ever before, which should improve learning outcomes, and it has simplified the entire engineering process, making it faster to bring new features to users.
The article presents COLT5, a new long-input Transformer model that uses conditional computation to allocate more resources to important tokens, achieving state-of-the-art results on the long-input SCROLLS benchmark with faster training and inference, and effective processing of extremely long inputs.
Low Rank Adaptation is a method to fine-tune language models by updating only a small subset of the parameters. This repo contains code that assists in tuning the popular Llama models via instruction tuning on consumer hardware due to the reduced computational requirements of Lora.
A list of extremely useful commands for ML research from a HuggingFace research scientist. It includes things like distributed training, environment set up, and configuration tasks. A useful reference for any practitioner.
The article proposes GlueGen, a method that aligns features from different encoders with the latent space of an existing text-to-image (T2I) model using the GlueNet model and a new training objective. This allows for efficient integration of new functionality, such as multilingual language models and multi-modal encoders, into existing T2I models, enabling sound-to-image generation and upgrading the current text encoder for challenging case generation.
Khan Academy is a nonprofit that offers thousands of lessons in math, science and the humanities for students of all ages. Khan Academy announced that it will use GPT-4 to power Khanmigo, an AI-powered assistant that functions as both a virtual tutor for students and a classroom assistant for teachers. The nonprofit began testing the newest version of OpenAI’s language model in 2022 and will initially make the Khanmigo pilot program available to a limited number of participants, though the public is invited to join the waitlist.
VALL-E X, a new AI model from Microsoft, can clone a voice from a short sample of 4-10 seconds and then use it to synthesize speech in a different language.
S4 models are a great new long range architecture. This work modifies them so they can initialize and update their internal state in parallel and then explores the modifications on a meta learning task. They found some promising results that point to S4 and other state space models potentially being used as the default RL architecture over LSTMS.
The article discusses the authors' reproduction and evaluation of the GANformer, a variation of the Generative Adversarial Networks (GANs) used for image generation. They successfully recreated both variations of the model and found differences between their results and those of the authors, as well as two undisclosed variations of the presented procedures.
This paper proposes Re-ReND, a method designed to achieve real-time performance of Neural Radiance Field (NeRF) rendering on resource-constrained devices by converting the NeRF into a representation that can be efficiently processed by standard graphics pipelines, distilling the NeRF by extracting the learned density into a mesh, and factorizing the learned color information into a set of matrices that represent the scene’s light field.
The repository contains code for reproducing a pre-trained T5 language model with limited resources in PyTorch. The model is pre-trained on the C4 dataset and fine-tuned on Super-Natural Instructions (SNI). Despite using only one GPU and limited resources, the model achieves ~40 RougeL on the SNI test set, compared to ~42 RougeL of the original HuggingFace model. The contribution is a user-friendly template for NLP research.
The article discusses the use of attention as a powerful tool for capturing long-range dependencies in vision transformers but highlights the challenge of its high computational burden and memory footprint. To address this issue, the authors propose a dynamic sparse attention via bi-level routing that allows for flexible allocation of computations with content awareness. The proposed BiFormer model, which utilizes the bi-level routing attention, shows good performance and high computational efficiency across various computer vision tasks.
The Vesuvius Challenge aims to read an unopened Herculaneum scroll for the first time and is offering $250,000 in prizes, including a $150,000 Grand Prize and $100,000 in Progress Prizes. Contestants receive 3D X-ray scans, papyrus fragments, tutorials, and tools to virtually unwrap the scrolls. They also have access to a Discord server to connect with other contestants and the Vesuvius Challenge team. The competition is open, and the organizers believe that it will help achieve the goal by 2023.