TLDR DATA 2026-07-30

GET STARTED →

EVAL-DRIVEN DEVELOPMENT: LESSONS FROM EVALUATING GENAI AT SCALE

Airbnb evaluates GenAI by reviewing 100 prototype outputs to find real failure modes before creating automated evals. Its stack uses programmatic checks, calibrated LLM judges with agreement in the high 80% range against human reviewers. Agentic systems are assessed across tool calls, reasoning steps, and final outputs.

INDEXING THE DATA LAKE FOR ONLINE POINT QUERIES

Spotify built an external index mapping keys directly to file offsets and row numbers in unmodified Parquet files, replacing distributed SQL scan chains with parallel ranged reads for lookups across petabyte-scale data lakes. Complementary write-time optimizations (per-key page flushing, ZSTD frame resets, column interleaving, covering index entries) reduce both bytes read and round trips further, making the same files serve both batch analytics and interactive queries.

PINNER PROGRESSION: BETTER USE-CASE REPRESENTATION DRIVING WEEKLY ACTIVE USER GROWTH AT PINTEREST

Pinterest replaced single-user embeddings with per-user agglomerative clusters over the last 500 engagements, with dynamic cluster count and lifecycle metadata tracking recency and frequency. It feeds retrieval, ranking, and blending layers, with lifecycle stages determining which utility signals get weighted during ranking. The retention benefit is nonlinear and accelerates sharply once users adopt enough distinct use cases.

SCALING STREAMHUB: TRANSITIONING FROM KINESIS TO KAFKA FOR 145 BILLION DAILY EVENTS

Atlassian moved from Kinesis to Kafka on AWS MSK as traffic rose from 22 billion to 150 billion events a day. Tiered Storage cut costs, but incidents exposed broker, S3, control-plane, and disk limits. Quotas, sharding, spare capacity, backups, and failover were critical.

YOUR AI AGENT DOESN'T KNOW YOUR BUSINESS | CONTEXT LAYERS EXPLAINED (31 MINUTE VIDEO)

AI agents need a context layer that captures business definitions, exceptions, institutional knowledge, and rules. Without it, they will guess when interpreting company data. MotherDuck's upcoming Guides feature stores this context alongside the data and automatically surfaces it to agents, with database-level, personal and organisation-wide scoping, SQL support, and version control.

THE NEW WAVE OF THE STREAMING LOG TECHNOLOGIES

New streaming platforms are dropping Kafka compatibility for different bets: Iggy uses Rust with TCP, QUIC, and HTTP; S2 uses HTTP and object storage for bottomless retention; and OpenData Log replaces partitions with key-based routing. Whether they complement or replace Kafka remains open.

HOW LANGCHAIN BUILT AN AGENT-FIRST DATA STACK

LangChain rebuilt its stack for agent-native analysis, migrating fully off its previous BI tool in six weeks. Hex combines dbt definitions, semantic models, trusted datasets, guides, and LangSmith traces so agents generate reliable SQL while the data team controls modeling and governance.

COMPOSABLE CANONICALS: FROM TRIBAL DATA KNOWLEDGE TO VERSIONED ARTIFACT

Composable canonicals encode domain knowledge in versioned models. Source layers define transformations and ontology, while business layers merge identities across systems. AI agents use stable concepts and identity rules, not simply more tables.

HOW MVCC AND TRANSACTIONS WORK IN ROCKSDB

RocksDB tags keys with increasing sequence numbers, so readers snapshot a consistent view while writers append without blocking. Pessimistic transactions lock on write - optimistic ones check conflicts at commit using pre-hashed mutexes. TiDB and CockroachDB skip native transactions and build their own layers.

FROM PYTHON SCRIPT HELL TO A MODERN DATA INTEGRATION FRAMEWORK

Apache SeaTunnel is presented as an alternative to fleets of hand-written ingestion scripts that duplicate connection handling, retries, checkpoints, idempotency, concurrency, and observability. Its Source-Transform-Sink model separates declarative pipeline definitions from execution and centralizes connector/runtime behavior.

THE REAL AI RISK IS INSIDE THE LABS

Frontier AI labs are more likely to cause the first serious AI incident than open models, since closed systems are vulnerable to insider leaks. Current open models also lack the domain depth to pose immediate risk in fields like biology, and restricting access helps malicious actors more than defenders.

THE 2026 GUIDE TO AGENT OBSERVABILITY TOOLS

Agent observability has moved from nice-to-have to a production requirement: Monte Carlo cites 73% of enterprises refusing to ship agents without monitoring and alerting, and 53% expecting major redesigns after deployment. The guide separates tracing-first, eval-first, gateway, and platform approaches, but highlights a persistent blind spot: most tools observe the agent while missing upstream data freshness, lineage, and quality failures.

HOW TO CONSOLIDATE YOUR DATABASE STACK FOR PRODUCTION AI

CockroachDB argues for consolidating transactional data, ephemeral state, and embeddings under serializable SQL, row-level TTL, changefeeds, and distributed vector indexing.

WIDE COLUMN STORES EXPLAINED: CASSANDRA, BIGTABLE, AND SCYLLADB

A clear refresher on when wide-column stores fit firehose workloads and predictable single-partition reads, and why they fail for ad hoc SQL or joins.