TLDR Index
TLDR AI

TLDR AI 2023-03-01

5 sections12 entriesinline urlcomplete parse

Parse notes

  • Subscription/account footer truncated before parsefooter_truncated

01HEADLINES & LAUNCHES

02RESEARCH & INNOVATION

REALFUSION: 360° RECONSTRUCTION OF ANY OBJECT FROM A SINGLE IMAGE

The paper describes a method for reconstructing a full 360◦ photographic model of an object using a single image of it. The approach involves fitting a neural radiance field to the image, but the problem is found to be severely ill-posed. To address this, an off-the-shelf conditional image generator based on diffusion is used, and a prompt is engineered to encourage it to "dream up" novel views of the object. The DreamFusion method is used to fuse the given input view, the conditional prior, and other regularizers in a final, consistent reconstruction.

arxiv.org15 min read

LANGUAGE IS NOT ALL YOU NEED

MLLM is a multimodal transformer trained on internet scale text/image data. It uses a frozen CLIP image encoder and a Magneto decoder only transformer. It has 1.3B parameters and was trained for 360B tokens. It was trained with TorchScale, a new exciting framework, and finished off with language only instruction tuning. It got 26% on an IQ test, which is better than random chance. It has 73% zeroshot ImageNet classification and positive transfer between modalities. It has a 64k vocab size and no code yet but should be coming soon.

arxiv.org23 min read

03ENGINEERING & RESOURCES

OPENAI’S WHISPER LARGE ON AN IPHONE

Georgi has been working on Whisper.cpp which dramatically speeds up the runtime of OpenAI’s state of the art speech to text model. He has included a WASM version for the web. This latest model uses 4-bit integer quantization. Interestingly, this type of quantization only seems to work on larger models and fails on smaller models.

github.comGitHub repository

CACTI: A FRAMEWORK FOR IMITATION LEARNING

The authors propose a framework for scaling robot learning in multi-task and multi-scene manipulation in kitchen environments, both in simulation and in the real world. The proposed framework, called CACTI, comprises four stages that handle data collection, data augmentation, visual representation learning, and imitation policy training. The approach uses state-of-the-art generative models for data augmentation and pre-trained visual representations to improve training efficiency. Experimental results show that CACTI enables efficient training of a single policy that can perform 10 manipulation tasks involving kitchen objects on a real robot setup, and 18 semantic tasks across 100 layout variations for each individual task in a simulated kitchen environment.

cacti-framework.github.ioGitHub repository

SCENE DIFFUSER

Scene Diffuser is a conditional generative model for 3D scene understanding. This repository is the official implementation of paper "Diffusion-based Generation, Optimization, and Planning in 3D Scenes". SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning. In contrast to prior work, SceneDiffuser is intrinsically scene-aware, physics-based, and goal-oriented.

github.comGitHub repository

T3-TURBO-AI

T3-Turbo-AI is a starter repo for building full-stack, type-safe, LLM-powered web apps.

github.comGitHub repository

04MISCELLANEOUS