Full-text search across article titles and summaries. Refine with sector, year, content type, and reading-time filters.
LoRA models can be plugged into a Stable Diffusion pipeline to change out the styles and quality of generated images. This model from Latent Labs allows you to generate panoramas and beautiful 360 scenes.
Voicemod has become the leading creator in real-time voice changing and soundboard technology. Their mission is to enable everyone to express themselves through sound. They’ve built expressive and immersive audio tools, making it easy to create unique sonic identities and enable interactions with personalized sounds. This will be huge as the podcast, voiceover, and audiobook spaces have blown up the last few years.
With code interfaces regularly changing and the limitation of in-context learning, there is a strong need to be able to update program synthesis performance without expensive data collection and model retraining. This work suggests that using documentation can improve the generated code of CodeT5. The results are good with ~3% improvement Pass@1. This isn’t a foundational change, but may be a useful trick for practitioners.
KOSMOS-12 is a Multimodal Large Language Model that can learn in context, follow instructions, and perceive general modalities. It achieves impressive performance on a range of tasks, including language understanding, perception-language, and vision tasks. The model was trained on web-scale multimodal corpora and benefits from cross-modal transfer. The authors also introduce a dataset for diagnosing the nonverbal reasoning capability of MLLMs.
Text-guided diffusion models struggle to compose scenes with multiple objects in specific positions, which is crucial in storytelling. To address this, the authors propose Directed Diffusion, a method that provides positional control over multiple objects by injecting "activation" at desired positions in cross-attention maps while attenuating the rest. This method can be used with an existing pre-trained model and requires only a few lines of implementation.
The authors propose voxformer, a framework proposed to enable AI systems to imagine the complete 3D geometry of occluded objects and scenes from 2D images. VoxFormer uses a two-stage design where a sparse set of visible and occupied voxel queries from depth estimation is followed by a densification stage that generates dense 3D voxels. The framework adopts a masked autoencoder design to propagate the information to all the voxels by self-attention. Experiments on SemanticKITTI show that VoxFormer outperforms the state of the art with a relative improvement of 20.0% in geometry and 18.1% in semantics and reduces GPU memory during training by ~45% to less than 16GB.
A nice compilation of various trends across the years in AI. Required computation doubles every 9 months while the number of parameters doubles every 18 months. Hardware improvements may improve through 2031. It doesn’t make sense to train models longer than 15 months. This is not exclusive to language but also includes vision and RL. Interactive charts included.
In this work we see additional evidence that alignment improves capabilities. Previous research seems to have mostly been focused on capabilities that are weakly aligned to human preference (e.g., improving FID or SOTA on a benchmark). Now, however, we find that if we train a reward model on human feedback data, our models become significantly more useful. They outline a “reward weighted likelihood maximization” scheme. Their experiments are constrained to categories of count, color, and background.
The goal of Civitai is to create a platform where people can share their stable diffusion models (textual inversions, hypernetworks, aesthetic gradients, VAEs, and any other crazy stuff people do to customize their AI generations), collaborate with others to improve them, and learn from each other's work. The platform allows users to create an account, upload their models, and browse models that have been shared by others. Users can also leave comments and feedback on each other's models to facilitate collaboration and knowledge sharing.
Composer is a large (5 billion parameters) controllable diffusion model trained on billions of (text, image) pairs. It can exponentially expand the control space through composition, leading to an enormous number of ways to generate and manipulate images, i.e., making the infinite use of finite means.
The generative AI chatbot can be spectacularly wrong, executives say, and is mostly good for rote tasks at present. Nonetheless, they say CIOs who fail to explore the technology’s possibility risk their companies falling behind.
Prompt engineers are being paid a healthy wage for their skill in getting AI systems to produce exactly what they want.
Of the 1,000 business leaders who use or plan to use ChatGPT surveyed by Resumebuilder.com, 25% have used ChatGPT to replace some employees.
A mix between powerpoint, figma, and docs - Tome is a creative tool powered by generative technologies. It has been growing quickly and looks like a potentially interesting combination of several ideas that have been in the community for the past several months. This brings their total money raised to just over $75m.
One of the main bottlenecks of generative video work is the inability of current methods to scale to the massive number of frames or maintain temporal consistency in generation. This work tries to solve both problems at the same time by working in a projective latent space. They obtain impressive FID scores on standard benchmarks and have fun videos on their project page.
By building on the fascinating “score based” interpretation of diffusion models, this work investigates why diffusion models fail at various tasks that require composition. They determine that the sampler, and not the underlying model, must be the source of error. They then propose a new energy-based parameterization and Metropolis-corrected sampler. They show strong improvement on a variety of tough tasks. Also, as a smaller win, they have a nice background section on all these topics!
Modular deep learning has emerged as a promising solution for developing models that specialize in multiple tasks without incurring negative interference and that generalize systematically to non-identically distributed tasks. This article provides a survey of modular architectures, explores various additional purposes of modularity, and reports various concrete applications where modularity has been successfully deployed.
The article discusses the use of parameter-efficient adaptation methods such as prompt tuning (PT), in-context learning (ICL), and instruction prompt tuning (IPT) for fine-tuning large language models. The paper empirically studies when and how in-context examples improve prompt tuning, and observes that IPT does not always outperform PT, PT is unstable, and prompts learned for a specific source task via PT exhibit positive transfer when paired with in-context examples of a different target task.
It could be argued that in addition to the powerful model, co-pilot’s success is because of the innovative user experience. In any case, this page provides a list of papers that study the effectiveness of different choices on users of programming assistant tools.
The article presents a method called Vid2Avatar, which learns human avatars from monocular in-the-wild videos without the need for ground truth supervision, external segmentation modules, or priors extracted from large datasets. The method solves the task of surface reconstruction and scene decomposition by jointly modeling the human and background in the scene using two separate neural fields.
This app brings stable diffusion to Macs. This app uses Apple's Core ML Stable Diffusion implementation to achieve maximum performance and speed on Apple Silicon based Macs while reducing memory requirements. It also runs on Intel based Macs too.
The U.S. Copyright Office has ruled that images in a graphic novel created by AI should not have been granted copyright protections, one of the first decisions by a U.S. Court or Agency on the topic of AI art and copyrights.
Sponsored
This 9th edition of the state of AI companies and growth gives great insight into the blistering pace of innovation. It is organized into four parts: Landscape (interactive), Financing, Data Infrastructure trends, ML/AI trends. The interactive landscape is a treat to go through to see various companies in different verticals.
This work uses off the shelf diffusion models to generate a prompt and “dream up” other 2D views for an object given a single image. They then fit a neural radiance field to generate a 3D representation of the object. This hierarchical chaining of models results in extremely compelling results. They also introduce some novel regularization for the final reconstruction step.
A cool collection of novel animations to help understand deep learning concepts like padding, stride, and normalization strategies! They are also permissively licensed if you want to use them for your teaching and presentations.
The high computational and memory requirements of large language model (LLM) inference traditionally make it feasible only with multiple high-end accelerators. FlexGen aims to lower the resource requirements of LLM inference down to a single commodity GPU (e.g., T4, 3090) and allow flexible deployment for various hardware setups. The key technique behind FlexGen is to trade off between latency and throughput by developing techniques to increase the effective batch size.