The Daily Index — Wednesday, April 17, 2024 · TLDR Index

RESEARCH & INNOVATION

TLDR AI

TLDR AI

COMPRESSION REPRESENTS INTELLIGENCE LINEARLY

Most modern AI is built around the idea of compressing a training dataset into a model. The better the compression, the better the model. This paper shows that relation rigorously and posits that scale benchmark scores correlate strongly to a model's ability to compress novel text.

arxiv.org · 18 min

TLDR AI

FEEDBACK IN TRANSFORMERS

TransformerFAM provides a feedback mechanism that allows Transformers to attend to their own latent representations. This can, in theory, introduce recurrence into the model for processing extremely long inputs in context.

arxiv.org · 24 min

TLDR AI

MEGALODON EFFICIENT TRANSFORMER PRETRAINING

Another long context paper - this time, a new architecture that uses two novel weight updating schemes. It outperforms Llama 2 on the same number of training tokens 2T. It also scales to infinite context length at inference time.

arxiv.org · 17 min

TLDR AI

ROAD LINE SEGMENTATION FOR AUTONOMOUS DRIVING

Accurately segmenting road lines and markings is crucial for autonomous driving but challenging due to occlusions caused by vehicles, shadows, and glare. The Homography Guided Fusion (HomoFusion) module uses video frames to identify and classify obscured road lines by leveraging a novel surface normal estimator and a pixel-to-pixel attention mechanism.

arxiv.org · 16 min

TLDR AI

ACCELERATING AI: HARNESSING INTEL(R) GAUDI(R) 3 WITH RAY 2.10

Anyscale's latest release of Ray, Ray 2.10, adds support for Intel Gaudi 3. Developers can now spin up and manage their own Ray Clusters, provisioning Ray Core Task and Actors on a Gaudi fleet directly through Ray Core APIs, tap into Ray Serve on Gaudi through Ray Serve APIs for a higher level experience, and configure Intel Gaudi accelerator infrastructure for use at the Ray Train layer.

anyscale.com · 5 min

TLDR AI

QWEN CODER

Code Qwen 1.5 is a new set of 7B models trained on 3T tokens of code related data. It performs well on HumanEval, with a non-zero score on SWE-bench. The chat variant specifically shows promise for long context retrieval tasks up to 64k tokens.

qwenlm.github.io · 12 min

TLDR AI

1-BIT QUANTIZATION

Extreme low-bit quantization for small pre-trained models, like Llama2-7B, is challenging, but fine-tuning just 0.65% of parameters significantly improves performance. Newly fine-tuned 1-bit models outperform 2-bit Quip# models, while 2-bit models with specialized data can exceed full-precision counterparts. This research suggests that proper fine-tuning and quantization may enhance efficiency without compromising model quality, potentially shifting focus from training smaller models to optimizing larger, quantized ones.

mobiusml.github.io · 7 min

TLDR AI

INTRODUCING ALOHA UNLEASHED (2 MINUTE VIDEO)

Google DeepMind's ALOHA Unleashed is a program that pushes the boundaries of dexterity with low-cost robots and AI.

twitter.com

TLDR AI

FOUR STEPS FOR FOUNDERS INTEGRATING AI

The pressure is immense right now to have a plan to factor AI into existing products. This short step-by-step guide will help you take the first step.

foundationcapital.com · 4 min

TLDR AI

LIMITLESS (PRODUCT)

Personalized AI app and wearable powered by what you've seen, said, and heard.

limitless.ai