Daily Edition · Page 3

THE DAILY INDEX

Technology, AI, software, security, crypto and business

GENERAL

TLDR AI

UNDERSTANDING AND CODING THE KV CACHE IN LLMS

Key-value (KV) caches store intermediate attention computations during LLM inference to avoid redundant calculations. When generating "Time flies fast" token by token, the model normally recomputes attention for "Time" and "flies" at each step, but caching these values delivers 5x speedups. This tutorial progresses from modifying a 124M parameter GPT model with basic cache buffers and position tracking to production-ready optimizations like pre-allocated memory and sliding windows to address the linear memory growth that becomes prohibitive for long sequences.

OPENAI'S PRACTICAL GUIDE TO BUILDING AGENT

This guide emphasizes starting with single agents before multi-agent systems, using manager patterns where one agent coordinates others via tool calls or decentralized handoffs for peer-to-peer task delegation. Key insights include implementing guardrails as layered defenses (LLM-based classifiers, regex filters, and moderation APIs), designing tools for messy long-horizon tasks, and building human-in-the-loop mechanisms triggered by failure thresholds or high-risk actions.

REAL-TIME ACTION CHUNKING WITH LARGE MODELS

Unlike chatbots or image generators, robots must operate in real time - delays between inputs and outputs have a tangible effect on performance. While vision-language-action (VLA) models have achieved promising results in open-world generalization, they can be slow to run. This article presents an algorithm called real-time chunking that enables real-time execution without discontinuities. It works on any diffusion- or flow-based VLA with no training-time changes.

O3 TURNS PRO

o3-pro seems to provide better answers than o3, but it has a significantly longer wait time. Using the API at scale seems prohibitively expensive, so users should instead run parallel queries using the chat interface. o3-pro serves the same niche as o3, so users thinking of using Opus may prefer to use Opus rather than, or in addition to, o3-pro. The 80% price cut on o3 seems more impactful than o3-pro - o3-pro is still largely a 'special cases only' model.

WHAT WE LEARNED FROM BRIEFING 70+ LAWMAKERS ON THE THREAT FROM AI

Briefings on AI risks revealed most UK parliamentarians lack in-depth AI knowledge and face capacity constraints that limit research on AI issues. The briefings were well-received, with one in three lawmakers publicly supporting AI risk mitigation campaigns. Effective outreach included persistent follow-ups and leveraging statements from notable AI authorities to convey the seriousness of AI-related extinction risks.

TLDR IndexPage 3