Daily Edition · Page 3

THE DAILY INDEX

Technology, AI, software, security, crypto and business

HEADLINES & LAUNCHES

TLDR AI

WHAT ARE POPULAR AI CODING BENCHMARKS ACTUALLY MEASURING?

Popular coding benchmarks measure something narrower than what their names suggest. Claude scoring 80% on SWE-bench doesn't translate to it one-shotting 80% of the tasks thrown at it. This post looks at SWE-bench Verified, SWE-bench Pro, Aider Polyglot, LiveCodeBench, and other benchmarks to see what they are actually measuring. Designing a good benchmark is highly labor-intensive. Without human review and annotations, it's nearly impossible to make a good benchmark.

IS OPENAI'S REINFORCEMENT FINE-TUNING (RFT) WORTH IT?

OpenAI's reinforcement fine-tuning (RFT) for o4-mini is supposed to be able to train models to get better at specific tasks using reinforcement learning. It costs up to 700 times more than supervised fine-tuning but only seems to deliver clear wins on agentic coding tasks. The technology gives engineers flexibility in reward design through flexible grader configurations, and the performance gains can be substantial when it works, but it is expensive with limited applicability, making its use difficult to justify.

TOWARD COMPUTATIONAL TASTE: LLMS, AESTHETICS & JUDGMENT

LLMs are being used to model and optimize taste in diverse fields through personalized systems like Taste Engines, Aesthetic LLMs, and Taste Tribes. Methods like LoRe and models like TAPO enable LLMs to adapt to individual aesthetic preferences, potentially revolutionizing recommendation engines and social networks. This shift towards computational taste signifies a new era where machines not only reflect but also shape human preferences, offering both cultural and economic opportunities.

CONTINUING TO BRING YOU OUR LATEST MODELS, WITH AN IMPROVED GEMINI 2.5 FLASH AND FLASH-LITE RELEASE

Google DeepMind has released updated versions of Gemini 2.5 Flash and 2.5 Flash-Lite. The models, available on Google AI Studio and Vertex AI, feature improvements in quality and speed compared to the current stable models. They are also more efficient, with a 50% reduction in output tokens for Gemini 2.5 Flash-Lite and a 24% reduction for Gemini 2.5 Flash. The models are not intended to graduate to a new, stable version, but will help shape future stable releases.

GEMINI ROBOTICS 1.5 BRINGS AI AGENTS INTO THE PHYSICAL WORLD

Google DeepMind has released two models that unlock agentic experiences with advanced thinking. Gemini Robotics 1.5 is a vision-language-action model that turns visual information and instructions into motor commands for a robot to perform a task. The model thinks before taking actions and shows its process and also learns across embodiments. Gemini Robotics-ER 1.5 is a vision-language model that reasons about the physical world, natively calls digital tools, and creates detailed multi-step plans to complete a mission. These advances will help developers build more capable and versatile robots that can actively understand their environments and complete complex multi-step tasks in a general way.

AI ISN'T REPLACING RADIOLOGISTS

Improvements in image interpretation have run far ahead of their adoption. There are hundreds of models that could potentially help radiologists, but AI is often limited to assistive use on a small subset of scans in any given practice. The promise of AI in radiology is overstated by benchmarks alone. Models can lift productivity, but their implementation depends on behavior, institutions, and incentives. So far, radiologists have only become busier as machines have improved.

TLDR IndexPage 3