TLDR DATA 2026-10-05

It takes 5 minutes. GET YOUR AI TRUST SCORE

DIVING THROUGH DATA AT OPENAI: HOW A DATA AGENT NAVIGATES 70,000 DATASETS (27 MINUTE VIDEO)

OpenAI built an internal AI data agent that answers complex data questions by combining warehouse queries with rich context from schemas, lineage, code, company knowledge, memory, and live sources, while showing its assumptions, SQL, citations, and confidence. The big lesson is that reliable data agents depend less on raw model intelligence and more on giving them the right context, continuously learning from corrections, and testing them with strong evals.

HOW WE EXTENDED APACHE DATAFUSION TO EXECUTE ONE QUERY ACROSS MANY MACHINES

Datadog built Distributed DataFusion to let a single Apache DataFusion query run across multiple machines, giving heavy interactive queries much more compute while keeping lightweight queries on one machine to avoid unnecessary overhead. In production, some expensive queries ran up to 10× faster at the same total cost. Datadog released the system as an open-source framework that teams can adapt to their own data sources and infrastructure.

HOW LINKEDIN EXTENDED CLICKHOUSE FROM DISTRIBUTED TRACING TO METRIC DISCOVERY AND ANALYTICS

LinkedIn replaced three systems for finding and analyzing metric metadata with a single ClickHouse index. It now handles more than 150,000 queries a minute across 13 billion metrics, averaging 68 milliseconds per query. The team explains how it migrated behind an existing gateway and cut memory use to about a fifth of the old setup.

CONTEXT IS THE NEW CODE (50 MINUTE VIDEO)

Context is becoming a primary software artifact for AI coding agents: teams must generate, lint, version, and test context packages with the same rigor as application code. Managing context drift, establishing AI bills of materials, and instrumenting production retrospectives are now essential data engineering disciplines to prevent agent hallucinations and silent architectural degradation.

LINKEDIN BUILT KAFKA. WHAT DID IT CHANGE WHEN KAFKA WAS NO LONGER ENOUGH?

LinkedIn outgrew Kafka's partition model at extreme scale and built Northguard to separate ordering, replication, and placement, making data movement and recovery less painful. Xinfra then hid the migration behind a compatibility layer so applications could move gradually, showing how infrastructure rewrites become practical when clients are insulated from backend changes.

AMAZON AURORA POSTGRESQL NOW SUPPORTS DIRECT QUERYING OF APACHE ICEBERG AND PARQUET DATA IN YOUR DATA LAKE

Amazon Aurora PostgreSQL now queries Apache Iceberg and Parquet tables directly in Amazon S3 data lakes without requiring separate ETL or reverse-ETL pipelines. Powered by an embedded DuckDB engine, the feature supports standard PostgreSQL syntax, predicate pushdown, and column pruning across operational and historical analytical data with zero additional Aurora licensing fees.

INTRODUCING CLEF: OUR OPEN-SOURCE DECISION MODELS, AND NEW RL FINE-TUNING PLATFORM

Cloudflare has released Clef and Clef-flash, ‘Jev-like' open-source System 1 models that turn text or images into choices and probabilities. They can help route support tickets, classify websites, or decide when an agent should ask a human. Both run on Workers AI or locally, and Cloudflare is offering help fine-tuning them for specific tasks.

APACHE ICEBERG 1.12.0: WHAT'S NEW, BREAKING CHANGES, AND UPGRADE GUIDE

Apache Iceberg 1.12.0 broadens production support for major v3 capabilities, including Variant, geospatial types, row lineage, deletion vectors, encrypted metadata, and REST catalog operations. However, it also removes platform support and deprecated APIs, so treat this as a substantial upgrade and test compatibility carefully before rollout.

THE INFERENCE GAP

A six-week analysis of 43,261 Fable 5 calls found that the model name stayed the same while the delivered reasoning budget varied sharply, with many invocations showing no thinking at all. The broader warning is an inference gap: benchmark performance may depend on far more sequential reasoning than production users actually receive.

MICROSOFT, GOOGLE BACK APACHE OSSIE TO MAKE ENTERPRISE DATA AND AI PLATFORMS MORE INTEROPERABLE

Microsoft and Google are backing Apache Ossie, an open semantic-model interchange specification supported by 60+ data and AI vendors. Using standard JSON/YAML schemas, Ossie enables portable dataset, metric, and relationship definitions across disparate BI platforms, eliminating redundant semantic modeling and metric drift.

FASTER STRING AGGREGATIONS WITH DIMENSION TABLES

String-heavy aggregations waste time hashing and comparing the same text values repeatedly. DuckDB shows that replacing those columns with small integer surrogate keys through dimension tables can make grouping cheaper, reduce memory pressure, and lower spill risk. The broader takeaway: normalize low-cardinality strings before large analytical aggregations when performance matters.

WHY YOUR AI AGENT PIPELINE IS SLOW (AND HOW TO FIX IT WITHOUT CHANGING MODELS)

A production AWS Bedrock guardrail pipeline for AI agents reduced action latency from 13,874 ms to 1,824 ms without altering models or accuracy. The bottleneck was execution topology: independent safety checks ran sequentially. By parallelizing independent layers and short-circuiting cheap reject rules early, the team dramatically reduced execution latency and compute costs.

JEV PLAYS POKÉMON RED FOR $1.65 MODEL BILL

Jev completed Pokémon Red in 37 hours and 40 minutes, making 16,150 decisions with a reported model cost of about $1.65.

DUCKDB 2.0 DYNAMIC ATTACH STATEMENTS

An early look at dynamic ATTACH statements in DuckDB 2.0 Alpha.