Sponsored
Put the duct tape away. You don't have to tack on a standalone search engine to your database to solve your AI-agent-scale search problem.
Lakebase Postgres now has a fast, scalable and serverless built-in search engine.
Separating storage from compute means you don't have to keep your entire search index in RAM just to keep queries fast. Compute scales with actual search activity, while your index stays in lower-cost storage.
Instead of chaining together multiple APIs for vector search, keyword search and filtering, you can do it all in one SQL query, against the same live operational data.
2026.10.15·links.tldrnewsletter.com·Sponsor·TLDR Data 2026.10.15·links.tldrnewsletter.com·TLDR Data 2026.10.15·links.tldrnewsletter.com·TLDR Data Meta uses idle computing capacity to improve ad recommendations by ranking more ads or routing requests to more powerful models. When that capacity disappears, the system quickly scales back this extra work while protecting normal service speed and reliability.
2026.10.15·engineering.fb.com·16 min read·TLDR Data GitHub built a star history API that returns daily counts without exposing users' identities, cutting React's full history from about 2,500 requests to 24. It reduces database work by filtering stars by date before counting them, rather than repeatedly processing every star.
Logical replication offers a practical path to near-zero-downtime PostgreSQL major upgrades, with live rollback enabled by reverse replication. A Docker-based demonstration ran old and new clusters in parallel, freezing writes and waiting for replication to catch up before cutover. It achieved 181 ms of write interruption during upgrade and 179 ms during rollback, without data loss.
Cold-start natural-language search often lacks both real queries and relevance labels. Airbnb's CASTLE jointly generates synthetic queries and labels from de-identified booking sessions. Its queries matched real guest query patterns much better than baselines, and human reviewers agreed with its relevance labels >90% of the time. Carefully governed behavioral data can bootstrap search training before meaningful traffic exists.
2026.10.15·links.tldrnewsletter.com·6 min read·TLDR Data This proof of concept shows how to add row-level security to Claude Dashboards using an MCP server to authenticate users and query DuckDB for only the data they can access. Access rules are enforced before data reaches Claude, allowing the same dashboard to show different results for each user.
2026.10.15·thedatavist.substack.com·11 min read·TLDR Data PostgreSQL micro-optimization alternatives often don't matter. For 1,000 IDs, IN, ANY, and VALUES joins perform similarly; prefer parameterized ANY for stable query text. Short text, enums, and smallints use near-identical storage; mostly-NULL columns are cheap. Don't blindly rewrite but EXPLAIN on your own data.
2026.10.15·explainmeasured.com·2 min read·TLDR Data Cloudflare held back exponential hardware growth by shrinking data earlier and moving duplication later: split monolithic schemas, switch to zstd, send each event once then replicate after sampling, sample at the edge, and pre-aggregate known queries. Optimize bytes, not just compute: prune fields, compress measured workloads, push reductions upstream, and replicate compacted data.
2026.10.15·blog.cloudflare.com·9 min read·TLDR Data Sponsored
Precisely AI Studio is a self-serve library of AI apps, agents, skills, prompts, and MCP connectors organized by industry and data domain. Connect to live, governed data and go from browsing to running in minutes. Try it free, no commitment, no strings.
SQLBuild is an open-source SQL and Python transformation framework for safely changing warehouses. It checks contracts and lineage before execution, runs graph-level tests and production diffs, and lets teams rename incremental models or columns while preserving history and compatibility views. A local DuckDB playground makes it easy to evaluate. Snowflake, Postgres, BigQuery, and Databricks adapters are available.
2026.10.15·github.com·GitHub repository·TLDR Data Phoenix is a self-hosted, open-source AI observability platform for tracing, evaluating, and debugging LLM applications. It captures OpenTelemetry traces; offers retrieval and response evaluations, versioned datasets, and experiments; and supports prompt testing and replay. Framework- and provider-agnostic instrumentation makes it useful for teams that need to diagnose agent behavior and measure changes before production.
2026.10.15·github.com·GitHub repository·TLDR Data XERJ is an Apache-2.0, Elasticsearch-compatible local search engine that automatically indexes code, logs, PDFs, SQLite, and CSVs into typed datasets for agent retrieval, incident analysis, and RAG. Its autoindex workflow avoids schema setup and returns ranked passages with file and line locations. The project defaults to local operation, although optional reranking can send text to an external provider.
2026.10.15·github.com·GitHub repository·TLDR Data Anthropic replaced Git with Scion, an in-house version control system built in five months, after daily code merges grew from 2,500 to over 30,000. By moving operations to servers and fetching only the files needed, it reports roughly 100 times faster checkouts and 200 times lower CPU and memory requirements than its previous Git mirror infrastructure.
2026.10.15·anthropic.com·17 min read·TLDR Data Daniel Lemire's benchmark processed 7.6 GB of JSON in about 0.12 seconds using fastsimdjson on free-threaded Python across 64 cores, reaching roughly 66 GB/s. The speed comes from running threads in parallel and creating Python objects only for the fields needed, with data already decompressed in memory.
Jev picked the right page first 88% of the time (vs. 70% for embedding search) across 168 test questions, and cut irrelevant results to the AI receptionist from 33% to 3% by answering "none" when no page fit.
Postgres's TOAST compresses large values or moves them to a separate table.
2026.10.15·planetscale.com·8 min read·TLDR Data Sponsored
As agents get more autonomous, they act on your numbers without waiting for a person. An LLM pointed at raw tables re-derives your metric logic on every prompt. Whatever it guesses becomes your source of truth. 🤦♂️
CUBE [5] is the agentic analytics and BI platform, built on a semantic layer. Your definitions and permissions are enforced deterministically — in dashboards, embedded apps, and Claude, ChatGPT or Cursor over MCP.
👀 Brex chose Cube over dbt Semantic Layer and LookML. Budget answers that took two days now take 2 seconds, for 35,000+ customers.
LinkedIn built a shared layer that tells people and AI agents which datasets to trust, which filters and joins are required, and what each row means. It combines profiles that an LLM writes from schemas, query logs, and code with notes that teams write and review in Git like code. After LinkedIn's analytics agent was connected to it, the share of acceptable answers in one test rose from 25.9% to 79.5%, and median time to answer fell from 10.7 to 4.7 minutes.
2026.10.12·linkedin.com·16 min read·TLDR Data Salesforce runs about 8 million Spark jobs a day and found some workloads left executors idle more than 90% of the time. Sizing targeted workloads to measured usage reduced weekly compute costs by 31%, while runtime, failures, and retries remained within acceptable levels.
2026.10.12·engineering.salesforce.com·7 min read·TLDR Data Pinterest replaces a fragile keyword-clustering pipeline with a fine-tuned 4B LLM that converts a user's activity history into ranked, localized “journeys” (active goals). Teacher-model distillation makes it affordable at scale. The system improves notification engagement while applying strict output validation, safety filters, and feedback loops.
2026.10.12·links.tldrnewsletter.com·14 min read·TLDR Data Apache Fluss' “Streamhouse” concept proposes one maintained logical table for streaming, serving, and analytics instead of separate synchronized systems. The key idea is to unify fresh stream data and historical lakehouse data behind shared metadata and tiering, so teams stop rebuilding state in multiple places. This can cut duplicate ingestion while still allowing specialized caches or replicas for specific access patterns.
2026.10.12·ipolyzos.substack.com·11 min read·TLDR Data Anthropic's new Claude Dashboards beta can build a dashboard without making a single chart manually. You can trace every number back to its source, but there is no row-level security, refresh, or audit log yet. The advice to BI developers: spend less time on tool tricks and more on data modelling, SQL, and design.
2026.10.12·thedatavist.substack.com·14 min read·TLDR Data Five PostgreSQL benchmarks show how unnecessary deduplication, CTE materialization, random sorting, extra column retrieval, and unindexed expressions can dramatically slow queries, with small SQL changes or a matching index delivering major speedups.
2026.10.12·explainmeasured.com·4 min read·TLDR Data Enterprise agents can enforce permissions yet make bad decisions when formal records omit the people whose judgment shapes outcomes. The Authority Resolution Framework separates documented rights, system permissions, recorded approvals, informal influence, and an agent's mandate, then treats disagreement among them as a governance signal. Audit authority gaps before automation turns organizational blind spots into confident actions.
2026.10.12·contextandchaos.substack.com·11 min read·TLDR Data Sponsored
Your AI agent's accuracy comes down to the data pipeline behind it. Data Pipelines and Lineage is a guide to engineering that pipeline: source-to-response lineage, data quality checks, and pipeline observability. Read the guide in AWS Marketplace [14].
Leviathan is a Rust tool that gives AI agents compact, cited search results from large datasets using SQLite full-text indexing, with optional persistent memory and CLI, MCP, or remote access.
2026.10.12·github.com·GitHub repository·TLDR Data A Databricks user found that Spark, Trino, and DuckDB all spent most of their time waiting on S3, with CPUs about 90% idle. After running four times as many read threads as CPU cores, one DuckDB machine beat Trino on the same hardware by 1.7 to 2.1 times on 10 TB queries. Against a Trino cluster with 5 to 13 nodes, it was 2 to 3 times slower but about half the cost.
2026.10.12·links.tldrnewsletter.com·TLDR Data The DuckDB 2.0 command line now detects when an AI agent is using it. It then switches to compact output, flags cut-off results, stops runaway queries, and returns errors as JSON. Across 132 TPC-H runs, agents read back about 60% fewer tokens and still got every answer right.
Netflix is using an ontology-driven knowledge graph to make observability data, incident chatter, and service topology queryable as one system. This delivers faster automated triage and root-cause analysis at large scale, with AI helping turn messy human signals into structured facts. The key takeaway: better incident response may come from modeling operational meaning, not just collecting more telemetry.
Postgres stores each relation as physical files identified by tablespace, database, and relfilenode, with extra forks and TOAST files appearing as needed. Logical objects and physical files are loosely coupled: operations like VACUUM FULL or REINDEX can rewrite storage and swap relfilenodes while the table's identity stays the same. Large relations also continue transparently across segmented files.
2026.10.12·boringsql.com·7 min read·TLDR Data Eight to ten major features have been pulled from Postgres 19 during beta, some very late.
2026.10.12·peter.eisentraut.org·5 min read·TLDR Data Agents waste most of their tokens working out what tables and fields mean, then do it again on the next question.
Sponsored
No one wants to use a tool they don't trust, and for data teams, trust in AI starts with the data underneath it. When no one can say where a model's inputs came from, whether they passed quality checks, or who can access them, projects stall in pilot. Collibra's free 12-question AI trust assessment [4] shows where your org stands:
* SCORE YOUR ORG across the 6 core capabilities that take AI from pilot to production * PINPOINT THE ROADBLOCKS holding back adoption, from data quality to governance gaps
OpenAI built an internal AI data agent that answers complex data questions by combining warehouse queries with rich context from schemas, lineage, code, company knowledge, memory, and live sources, while showing its assumptions, SQL, citations, and confidence. The big lesson is that reliable data agents depend less on raw model intelligence and more on giving them the right context, continuously learning from corrections, and testing them with strong evals.
2026.10.05·links.tldrnewsletter.com·TLDR Data Datadog built Distributed DataFusion to let a single Apache DataFusion query run across multiple machines, giving heavy interactive queries much more compute while keeping lightweight queries on one machine to avoid unnecessary overhead. In production, some expensive queries ran up to 10× faster at the same total cost. Datadog released the system as an open-source framework that teams can adapt to their own data sources and infrastructure.
2026.10.05·datadoghq.com·13 min read·TLDR Data LinkedIn replaced three systems for finding and analyzing metric metadata with a single ClickHouse index. It now handles more than 150,000 queries a minute across 13 billion metrics, averaging 68 milliseconds per query. The team explains how it migrated behind an existing gateway and cut memory use to about a fifth of the old setup.
2026.10.05·clickhouse.com·12 min read·TLDR Data Context is becoming a primary software artifact for AI coding agents: teams must generate, lint, version, and test context packages with the same rigor as application code. Managing context drift, establishing AI bills of materials, and instrumenting production retrospectives are now essential data engineering disciplines to prevent agent hallucinations and silent architectural degradation.
LinkedIn outgrew Kafka's partition model at extreme scale and built Northguard to separate ordering, replication, and placement, making data movement and recovery less painful. Xinfra then hid the migration behind a compatibility layer so applications could move gradually, showing how infrastructure rewrites become practical when clients are insulated from backend changes.
2026.10.05·softwaremill.com·9 min read·TLDR Data Sponsored
Learn how to analyze structured and unstructured data (like text, images, and PDFs) together in BigQuery, without moving data. Incorporate techniques like multimodal processing, automatic embedding generation, and hybrid search; then go further with generative AI functions and ObjectTables to extract deeper, unified insights. Watch the webinar session [10] or read the blog [11].
2026.10.05·cloudonair.withgoogle.com·Sponsor·TLDR Data Amazon Aurora PostgreSQL now queries Apache Iceberg and Parquet tables directly in Amazon S3 data lakes without requiring separate ETL or reverse-ETL pipelines. Powered by an embedded DuckDB engine, the feature supports standard PostgreSQL syntax, predicate pushdown, and column pruning across operational and historical analytical data with zero additional Aurora licensing fees.
2026.10.05·aws.amazon.com·5 min read·TLDR Data Cloudflare has released Clef and Clef-flash, ‘Jev-like' open-source System 1 models that turn text or images into choices and probabilities. They can help route support tickets, classify websites, or decide when an agent should ask a human. Both run on Workers AI or locally, and Cloudflare is offering help fine-tuning them for specific tasks.
2026.10.05·blog.cloudflare.com·10 min read·TLDR Data Apache Iceberg 1.12.0 broadens production support for major v3 capabilities, including Variant, geospatial types, row lineage, deletion vectors, encrypted metadata, and REST catalog operations. However, it also removes platform support and deprecated APIs, so treat this as a substantial upgrade and test compatibility carefully before rollout.
A six-week analysis of 43,261 Fable 5 calls found that the model name stayed the same while the delivered reasoning budget varied sharply, with many invocations showing no thinking at all. The broader warning is an inference gap: benchmark performance may depend on far more sequential reasoning than production users actually receive.
2026.10.05·links.tldrnewsletter.com·22 min read·TLDR Data Microsoft and Google are backing Apache Ossie, an open semantic-model interchange specification supported by 60+ data and AI vendors. Using standard JSON/YAML schemas, Ossie enables portable dataset, metric, and relationship definitions across disparate BI platforms, eliminating redundant semantic modeling and metric drift.
2026.10.05·infoworld.com·4 min read·TLDR Data