TLDR Index
TLDR Data

TLDR DATA 2026-09-10

6 sections15 entrieslinks blockcomplete parse

Parse notes

  • Subscription/account footer truncated before parsefooter_truncated

01GENERAL

Sponsored

COSTBENCH: 100 BILLION ROWS. UP TO 83% LOWER READ COSTS.

Fast queries are only part of your analytics bill. Ingesting and preparing fresh data costs money too. CostBench [5] measures ingest, preparation, and reads. We streamed 1 MILLION STOCK-MARKET QUOTES PER SECOND, reaching 100 BILLION ROWS, with dashboard and drill-down queries running throughout. Against traditional cloud data warehouses we tested, ClickHouse delivered: * Up to 82% LOWER INGEST COSTS * Up to 65% LOWER PREPARATION COSTS * Up to 83% LOWER READ COSTS

clickhouse.comSponsor

02DEEP DIVES

TOM LANE ON THE ARCHITECTURAL DECISIONS THAT SHAPED 30 YEARS OF POSTGRES

Tom Lane, a committer for 25 of Postgres' 30 years, walks through the architectural bets that held: process-per-connection for code simplicity, write-ahead logging in 8.0 that turned it from a plaything into a serious database, and MVCC pushing maintenance work into background vacuum. He credits Berkeley's liberal license for the project surviving where its contemporaries did not, names version 13 the buggiest release, and says he would delete partitioning given the chance.

snowflake.com41 min read

BUILT FOR RELIABILITY: HOW AMERICAN EXPRESS PROCESSES PAYMENTS AT SCALE

American Express uses cells as complete payment-processing failure domains: microservices, databases, DNS, and reference data are local, while dynamic transactions are routed to the cell that owns their state. The design trades global coordination for controlled failure scope and deliberately rejects transactions when consistency cannot be established.

blog.bytebytego.com12 min read

CLOUDFLOE'S QUERY ENGINE IN A CELLD CELL

Cloudfloe is a web interface for querying Iceberg data on S3 with DuckDB. This experiment tested whether a stateful, serverless-style cell could replace a per-query container: DuckDB WebAssembly read a 37,537-row Iceberg table in 148 ms, but cold or woken cells took 250–320 ms and retained most of their memory. Keeping the connection warm cut repeat queries to about 3 ms, making lifecycle policy, not raw startup, the core design tradeoff.

gordonmurray.ie9 min read

03OPINIONS & ADVICE

THE AI DATA SPRAWL

Non-technical staff now write billions of lines of analytics code. The problem is where it lives: random context, porous permissions, and output scattered across one-off chats, Slack pastes, and someone's laptop. With no record of which table an agent grabbed or which filter it dropped, the only feedback loop left is people disagreeing.

linkedin.com12 min read

5 CDC TOOLS AND THE TRADEOFFS YOU SHOULD KNOW

CDC evaluation should start with failure behavior, not connector counts. Compare capture method, initial snapshots and backfills, schema evolution, delivery semantics, observability, deployment model, and who owns recovery. A production proof needs representative data and a broken-destination test, especially for type fidelity and duplicate handling.

theseattledataguy.com14 min read

JITTER IS THE CHEAPEST RELIABILITY FIX YOU ARE NOT USING

Distributed systems synchronise themselves without being told to: shared events such as deploys, cache flushes, or connection loss turn independent workloads into recurring load spikes. Add jitter to initial retries, TTL refreshes, heartbeats, token renewals, and scheduled work. This preserves average load while preventing retry storms from repeatedly hitting a recovering dependency in lockstep.

ankit-rana.com5 min read

04LAUNCHES & TOOLS

KESTRA 2.0: A NEW ENGINE, WORKERS ANYWHERE, AND FLOWS AS AGENT TOOLS

Kestra is an open-source orchestration platform for scheduling and operating data, infrastructure, and business workflows. Its 2.0 LTS release separates control and data planes so workers can run across regions, clouds, or private networks over outbound-only connections. It also exposes flows as MCP tools for agents, decouples queue and storage choices, and includes migration tooling for 1.x users.

kestra.io16 min read

FILAMENT

Filament replicates data from sources to sinks with full, incremental, or CDC modes, verifying every bounded batch on both sides of the write and advancing checkpoints only after success. Sources, sinks, state storage, and event transport are all swappable.

github.comGitHub repository

JUPYTERGIS 0.16: COLLABORATIVE STORY MAPS AND REMOTE GEOSPATIAL WORKFLOWS

JupyterGIS 0.16 adds real-time collaborative story maps, lazy tile rendering for remote openEO workflows, and native GeoZarr and GeoPackage support. Declarative symbology and Xarray tiling keep visualizations reproducible while allowing large geospatial datasets to stay remote instead of being materialized locally.

blog.jupyter.org3 min read

05MISCELLANEOUS

WHY SPOTIFY IS NOT USING BAYESIAN A/B TESTING

Bayesian experimentation is not one method but a choice of prior, likelihood, and stopping rule. Spotify shows that common flat-prior posterior thresholds can reproduce frequentist peeking, while Bayes factors can control false positives under optional stopping. Choose guarantees for the experiment program first, then the inference configuration.

engineering.atspotify.com12 min read

PRETRAINING PROGRESS IS MOSTLY COMING FROM DATA

A controlled comparison of 2019-2025 open-model recipes and data corpora attributes 12.0x compute-efficiency gains to data improvements versus 3.7x to model recipes at a 1e19-FLOP budget. The result frames dataset extraction, curation, and filtering as a major systems lever, while cautioning that larger-scale and synthetic-data effects remain untested.

dwarkesh.com11 min read