Parse notes
Another big funding announcement for an open source tool for AI. This time the fast growing embeddings database company raises cash. They’re hoping that folks will use their tools to add state and memory to different LLM applications. The goal is for this to reduce hallucinations and increase factuality. They’re growing quickly, it’ll be interesting to see how all the different LLMOps tools come together.
Training a model on an unsupervised dataset is so last year. Using human feedback data is all the rage and while it sounds easy in theory - it turns out to be tricky in practice. The HuggingFace folks have written another great blog showcasing StackLLaMA which answers StackExchange-like questions.
The paper proposes a new approach to Vision Transformers by introducing a mixed-resolution sequence of tokens to replace the standard uniform grid, where each token represents a patch of arbitrary size. This approach uses the Quadtree algorithm and a novel saliency scorer to construct a patch mosaic that processes important image regions with higher resolution while routing low-saliency areas in low resolution, resulting in significant accuracy gains on image classification while controlling for the computational budget.
VideoCrafter is an open-source video generation and editing toolbox for crafting video content, which contains three models: Generic Text-to-video Generation, Personalized Text-to-Video Generation with LoRA, and Video Generation with More Condition Controls
The paper proposes a new task that utilizes both music and text for 3D dance generation, which can guide humans to perform richer movements. To tackle the challenge of the existing datasets containing paired motion with a single modality, the authors utilize a 3D human motion VQ-VAE to project the motions of the two datasets into a latent space, and propose a cross-modal transformer architecture to integrate text instructions. The approach generates realistic and coherent dance motion conditioned on both music and text, while keeping comparable performance conditioned on two single modalities.
E2b is an open-source IDE powered by AI agents that do the work for you.