TLDR Index
TLDR AI

TLDR AI 2025-01-13

6 sections19 entrieslinks blockcomplete parse

Parse notes

  • Subscription/account footer truncated before parsefooter_truncated

01GENERAL

02HEADLINES & LAUNCHES

BERKELEY'S NOVASKY RELEASED SKY-T1

UC Berkeley's NovaSky's Sky-T1-32B-Preview is an open-source reasoning model that rivals OpenAI's earlier models. Training costed under $450 with full replicability.

novasky-ai.github.io9 min read

OPENAI'S ROBOTICS PLANS

Caitlin Kalinowski, who joined OpenAI from Meta, has announced plans for OpenAI to create robots with custom sensors.

threadreaderapp.com3 min read

MARK ZUCKERBERG GAVE META'S LLAMA TEAM THE OK TO TRAIN ON COPYRIGHTED WORKS, FILING CLAIMS

Meta faces a copyright lawsuit that accuses Mark Zuckerberg of approving the use of pirated datasets from LibGen to train its Llama AI models. Plaintiffs argue that Meta knowingly stripped copyright information to conceal infringement while relying on a fair use defense. The case highlights the ongoing legal challenges tech companies face over AI training practices.

techcrunch.com5 min read

03RESEARCH & INNOVATION

KALM-EMBEDDINGS

A strong decoder only embeddings model based on Qwen2-0.5B that uses modern data filtering techniques to train an impressively powerful and open embedding model for retrieval tasks.

arxiv.org28 min read

GAUSSIAN MASKED AUTOENCODERS

Instead of training a masked autoencoder on pure reconstruction loss, these researchers introduce an intermediate 3D Gaussian representation which enables the model to learn 3D representations as part of the reconstruction process. The results are compelling for zero-shot transfer tasks.

brjathu.github.io25 min read

AN EMPIRICAL STUDY OF AUTOREGRESSIVE PRE-TRAINING FROM VIDEOS

Great paper from the same authors as the above GMAE paper. The researchers show that pre-training video models on 1T video tokens emits strong scaling laws over a wide range of design choices. Also, somewhat unsurprisingly, autoregressive training achieves similar benchmark scores to diffusion or flow based models.

brjathu.github.io31 min read

04ENGINEERING & RESOURCES

INTEGRATING ASCEND BACKEND WITH TORCHTUNE

Ascend is a series of AI computing products from Huawei that includes processors, hardware, software, and other frameworks. Torchtune introduced a device abstraction API. By specifying a config and a recipe, PyTorch works well on the Ascend NPU hardware.

pytorch.org17 min read

STABLE AUDIO CODEC

Stability AI has released a suite of state-of-the-art Transformer-based audio codecs for low-bitrate and high-quality audio encoding for things like speech generation and audio understanding.

github.comGitHub repository

ROTATED OBJECT DETECTION

The Unit Cycle Resolver (UCR) introduces a novel loss constraint to improve angle prediction accuracy in weakly supervised models for SAR object detection.

github.comGitHub repository

05MISCELLANEOUS

OPEN-SOURCE SAES FOR LLAMA 3.3 70B

Early last year, Anthropic had a great demo of its steerable models - Golden Gate Claude. This work is from a different group and uses similar techniques on the open weights Llama model. It can be used for interpretability and also steering.

goodfire.ai8 min read

CLAUDE FIGHTS BACK

Researchers explored if Anthropic's AI model, Claude, would resist if retrained for malicious tasks. Claude pretended to comply during training, undermining the malicious intent while maintaining a distinction between monitored and unmonitored user interactions. The study suggests AI might resist altering its core values, raising concerns about ensuring AI alignment and adaptability.

astralcodexten.com13 min read

WHY AI LANGUAGE MODELS CHOKE ON TOO MUCH TEXT

LLMs face efficiency challenges as context window sizes and compute costs increase with input size. Innovations like FlashAttention, ring attention, and the Mamba architecture aim to address these scalability issues. Future AI systems may need hybrid or new architectures to efficiently process larger datasets.

arstechnica.com15 min read