TLDR DATA 2026-11-05

Start free on AWS Marketplace →

WHY WE'RE MOVING OUR WAREHOUSE PRODUCT OFF DUCKDB

Five months after rebuilding its warehouse product on DuckDB, PostHog is moving off it. Each instance had to be sized before the query arrived, and its DuckLake, holding about 1 trillion rows, could take over three minutes for a single-table write. PostHog is now rebuilding on Trino with its own table format, HogLake.

FRONTIER QUALITY AT 1% OF THE COST: A PRODUCTION BENCHMARK OF LLM CLASSIFIERS

Airbnb fine-tuned a model with 2 billion effective parameters to screen content for pricing-policy violations. Its F1 score rose from 0.563 to 0.902, matching the best frontier APIs, with a median latency of 35ms against their roughly 900ms. At thousands of requests a minute, the frontier models cost two orders of magnitude more than one self-hosted GPU.

F3 LIGHTNING: HOW META ACCELERATES ADS ML FEATURES TO PRODUCTION AT SCALE

Meta rebuilt its ads ML feature infrastructure to deploy features independently without losing the efficiency of shared computation. F3 Lightning uses stable computation IDs, a unified execution plan, and shared memory to avoid redundant processing across more than 20,000 features. Comparable features now reach production 3.6x faster without increasing infrastructure compute costs.

UPGRADING REDSHIFT FROM RA3 TO RG AND ACHIEVING FASTER QUERIES AT 32% LOWER COST

Halodoc migrated its Redshift warehouse from RA3 to Graviton-based RG instances after validating ingestion, ETL, dashboards, and governance compatibility. COPY operations ran roughly 6x faster, heavy reporting queries ran 2.1x to 5.2x faster, and queue times fell by around 70%. Overall Redshift costs dropped 32%, demonstrating the potential benefits of the new architecture for production analytics workloads.

AS HUMANS WRITE LESS CODE: WORKLOADS SHIFT FROM HUMAN TO AGENT VERIFICATION WITH ITERATIVE TEST SUITES

As AI agents generate more SQL, transformations, and dashboards, verifying their output becomes more important than writing the code itself. What dev teams really need are compiler-style validation, layered test suites, and unified lineage across data models and BI assets. Moving assertions earlier in development could make agent-generated analytics easier to audit and less prone to semantic drift.

THE GREAT DATA INTERREGNUM - MORBID SYMPTOMS OF AN INDUSTRY BETWEEN ERAS

The data industry is caught between vendor consolidation, AI-driven automation, and unresolved operational problems. Generating SQL, pipelines, and documentation is becoming cheaper, but broken data, inconsistent business definitions, and unclear accountability remain. The lasting value of data teams lies in semantics, governance, and understanding rather than producing more technical artifacts.

EXPLORATORY DATA ANALYSIS CAN'T BE AUTOMATED AWAY

Exploratory data analysis still needs human eyes on the data. What looks weird depends on what you intend to do, which no model can know in a vacuum. LLMs rarely make the work faster.

see significant gaps. Read the report

SURREALDB 4

SurrealDB 4 rebuilds the query engine with 240+ fixes. It reports 42% faster CRUD operations and projects 22x faster indexed queries, while adding PostgreSQL, MySQL, and MongoDB compatibility. The same engine powers Earth's serverless databases on Cloudflare and SurrealDS' distributed clusters, which confirm writes only after storing them across two zones.

APACHE HUDI-RS 0.6.0: NATIVE WRITES FROM RUST AND PYTHON

Apache Hudi-rs 0.6.0 introduces native write support for Rust and Python, allowing applications to create, append, update, upsert, and delete records without Spark or a JVM. Its record-level index enables sub-second small-batch updates on tables containing millions of rows. The release also improves Python read performance by up to 6x while maintaining compatibility with existing Hudi query engines.

HELIX FOUNDRY: TURNING ENTERPRISE DATA INTO AN AI-GENERATED KNOWLEDGE GRAPH

Helix Foundry connects databases, SaaS applications, and files, then uses an AI agent to propose an ontology and load the resulting entities and relationships into HelixDB. The open-source project supports local deployment and is adding consistent PostgreSQL snapshots and graph computation capabilities.

ROCKETRIDE: A C++ PIPELINE ENGINE FOR AI WORKFLOWS

RocketRide is an open-source AI pipeline engine built around a multithreaded C++ runtime with more than 50 Python-extensible processing nodes. It supports multiple LLM providers, vector databases, and agent frameworks, with debugging and deployment tools integrated into VS Code.

SCALING MODEL EVALUATION AT PINTEREST WITH RAY (PART 4)

Pinterest used to run inference with Ray, write predictions to S3, then start a Spark job to read them back and compute metrics. Its new evaluation platform computes partial metrics batch by batch inside the Ray inference job and combines them at the end. That cut end-to-end evaluation time by up to 32% for feature-importance runs.

MULTI-TENANT CDC: DATABASE-PER-TENANT SAAS ARCHITECTURE GUIDE

A practical guide to replicating tenant-isolated PostgreSQL databases into shared analytics tables while preserving primary keys, schema compatibility, and failure isolation.

TEXT-TO-SQL MODELS MAKE CHOICES WITHOUT SAYING SO

SpecBridge shows that text-to-SQL models often silently choose one interpretation of ambiguous questions, while answering each interpretation separately improves coverage by 29-63 percentage points without reducing precision on clear requests.