TLDR DATA 2026-10-29

on AWS →

THE 2026 CLOUD DATA WAREHOUSE BENCHMARK

At 100 GB, MotherDuck led in single-user speed and cost (235 ms, $0.00124/query), Fabric served 100 concurrent users under five seconds for $5.76/hour, and ClickHouse failed 20.4% of single-user runs. Databricks had the smallest latency increase from 1 GB to 1 TB (3.7x versus MotherDuck's 30x), with tuning and autoscaling disabled throughout.

THE PAPER THAT SPLIT THE DATABASE IN FOUR: INSIDE AZURE SQL DATABASE HYPERSCALE, AKA SOCRATES

Azure SQL once kept four full copies of each database, which capped practical size at around 4 TB. The 2019 Socrates paper separated the log from storage, so durability and availability could scale apart. This change in approach let Hyperscale reach 100 TB, halve storage copies, and restore a 50 TB database in minutes.

HOW JD.COM BUILT A LAKESTREAM WITH APACHE FLUSS AND APACHE HUDI

JD replaced parallel streaming and batch training-data pipelines with a staged migration: write once to Fluss, tier to Hudi, and union fresh plus historical reads. The difficult part is consistency: workers coordinate a single Hudi commit per tiering round and report matched stream offsets only after it completes. That pattern enables rollback-safe catch-up, but leaves schema evolution and dynamic bucketing unresolved.

BEST PRACTICES FOR SINGLE ROW LOOKUPS (POINT QUERIES)

DataFusion can perform surprisingly well on point lookups when tuned for retrieval instead of analytics. The biggest gains came from storage layout and execution choices that minimize unnecessary work, such as sorted Parquet files, right-sized row groups, metadata caching, and prepared statements. The broader lesson is that analytical engines can serve low-latency reads if the workload is designed deliberately.

GETTING AN ANALYTICS AGENT FROM 50% TO 96% ACCURACY

Benchouse improved nao's accuracy from 49.6% to 88.2% by switching from Sonnet to Opus, then reached 96.5% with better context. The key fixes clarified business metrics, removed ambiguous terminology, and told the agent to use agreed calculations before suggesting alternatives.

THE BENCHMARKING CRISIS

AI code generation is helping database engines game ClickBench. One, pgrust, groups strings by hash value alone, caches predicates the benchmark repeats, and on one query skips most of the data. Any benchmark stops being useful once systems optimise for it, and vibe-coded engines have no customers to keep them honest.

Explore free assets

Start your free trial

PRIVACYLIB: PRIVACY CONTROLS AT THE DATA ACCESS BOUNDARY

Meta built PrivacyLib to enforce privacy rules at the moment data is accessed, replacing one-off integrations with shared controls that work across many systems. The key lesson is to normalize diverse data platforms into a small set of common operations, which makes policy enforcement reusable, scalable, and adaptable enough to extend from traditional infrastructure to AI agents.

EGRES: A POSTGRES-COMPATIBLE DATABASE FOR HEAVY, MULTI-TENANT WORKLOADS

What would Postgres look like if designed today? Egres is the answer: a new open-source database that speaks the Postgres protocol but shares almost no Postgres code. It runs thousands of tenants in one process, metering CPU, memory, and disk per query, and its benchmark used 46% less disk than Postgres 18.

GEMS OF DUCKDB 2.0.0

DuckDB 2.0 adds a Python appender for batching generated rows, CLI function manuals and queryable command history, plus secure views that hide underlying statistics and query plans. Other additions include custom partition paths, sorting within exported partitions, direct GeoJSON exports, and support for extension-defined window functions.

SLAYER

SLayer is an MIT-licensed, embeddable semantic layer for governed agent and human access to warehouse data through MCP, REST, SQL, or Python. It combines read-only connectivity, SQL translation, transformations, row-level security, semantic search, and memory, and imports dbt, Cube, or Ossie definitions. Its expression model supports ratios, time shifts, and multi-stage queries beyond fixed BI metrics.

MAKING FAILURE DIAGNOSIS RELIABLE WITH PENSIVE AI

Netflix made failure diagnosis more reliable by using LLMs to identify the exact log spans behind an error and provide on-demand root-cause analysis, instead of asking models to digest entire logs. That design improved coverage to all failures while reducing hallucination risk and latency, and early usage data suggests it sharply cuts manual log exploration during debugging.

DUCKDB UNIVERSE

The DuckDB Universe is a searchable map of 516 open-source projects and 59 companies and products using DuckDB, spanning extensions, dashboards, libraries, and database integrations. Entries include GitHub star counts and source links, with examples ranging from PyGWalker and Evidence to MotherDuck and DeepSeek's smallpond.

REDUCING BIGQUERY COSTS THROUGH ITS BIGGEST PROBLEM: THE USERS

An open-source BigQuery FinOps agent cuts waste from user-written SQL by detecting common cost mistakes, rewriting queries, and validating savings before execution.

HOW FLINK'S KAFKA CONNECTOR COMMITS OFFSETS TO KAFKA

Flink's Kafka connector now reconciles checkpoint state with Kafka's actual position, preventing misleading offsets on transactional topics and making its committed-offset metric match Kafka tooling without weakening exactly-once processing.