Full-text search across article titles and summaries. Refine with sector, year, content type, and reading-time filters.
ChatGPT Work's Data agent connects to approved company data and business definitions, letting employees investigate questions and build shareable dashboards using plain language. It respects existing permissions, works with major data and BI platforms, and can recommend or carry out approved follow-up actions.
DuckDB 2.0 speeds up S3 queries by 2-3x through asynchronous I/O, dramatically improves deep recursive queries, and makes semi-structured VARIANT data smaller and faster than JSON. It also adds practical SQL features including triggers, nested schemas, and data-modifying CTEs.
Comparing AI adopters with non-adopters usually exaggerates impact because more engaged customers are also more likely to adopt. Instead, estimate causality using variation that customers did not choose, such as an eligibility threshold, while clearly limiting conclusions to the population and decision it measures.
Large PostgreSQL CDC backfills fail less from historical reads than from WAL retention, delayed slot acknowledgments, long-running transactions, and stale snapshot rows racing live changes. Safer designs keep consuming WAL during chunked reads, checkpoint progress, reconcile overlaps with watermarks and fences, and size chunking around real write rates.
Four production Spark cases cut runtime, compute, S3 operations, and costs by tracing hidden waste beyond top-level CPU and memory metrics.
Tools like dbt turn fragile, undocumented analysis into reproducible systems, while AI often misses the constraints of real workplaces.
Jetpack adds a portable 1-RTT fast-commit path to existing consensus systems, cutting latency for write-heavy workloads by up to 60% while adding redundant work, delaying execution, and requiring careful recovery to remain safe during leader changes.
Sponsored
Fast queries are only part of your analytics bill. Ingesting and preparing fresh data costs money too. CostBench [5] measures ingest, preparation, and reads. We streamed 1 MILLION STOCK-MARKET QUOTES PER SECOND, reaching 100 BILLION ROWS, with dashboard and drill-down queries running throughout. Against traditional cloud data warehouses we tested, ClickHouse delivered: * Up to 82% LOWER INGEST COSTS * Up to 65% LOWER PREPARATION COSTS * Up to 83% LOWER READ COSTS
Try ClickHouse Cloud on your own data. Start with $300 IN FREE TRIAL CREDITS.
Tom Lane, a committer for 25 of Postgres' 30 years, walks through the architectural bets that held: process-per-connection for code simplicity, write-ahead logging in 8.0 that turned it from a plaything into a serious database, and MVCC pushing maintenance work into background vacuum. He credits Berkeley's liberal license for the project surviving where its contemporaries did not, names version 13 the buggiest release, and says he would delete partitioning given the chance.
American Express uses cells as complete payment-processing failure domains: microservices, databases, DNS, and reference data are local, while dynamic transactions are routed to the cell that owns their state. The design trades global coordination for controlled failure scope and deliberately rejects transactions when consistency cannot be established.
Cloudfloe is a web interface for querying Iceberg data on S3 with DuckDB. This experiment tested whether a stateful, serverless-style cell could replace a per-query container: DuckDB WebAssembly read a 37,537-row Iceberg table in 148 ms, but cold or woken cells took 250–320 ms and retained most of their memory. Keeping the connection warm cut repeat queries to about 3 ms, making lifecycle policy, not raw startup, the core design tradeoff.
Non-technical staff now write billions of lines of analytics code. The problem is where it lives: random context, porous permissions, and output scattered across one-off chats, Slack pastes, and someone's laptop. With no record of which table an agent grabbed or which filter it dropped, the only feedback loop left is people disagreeing.
CDC evaluation should start with failure behavior, not connector counts. Compare capture method, initial snapshots and backfills, schema evolution, delivery semantics, observability, deployment model, and who owns recovery. A production proof needs representative data and a broken-destination test, especially for type fidelity and duplicate handling.
Distributed systems synchronise themselves without being told to: shared events such as deploys, cache flushes, or connection loss turn independent workloads into recurring load spikes. Add jitter to initial retries, TTL refreshes, heartbeats, token renewals, and scheduled work. This preserves average load while preventing retry storms from repeatedly hitting a recovering dependency in lockstep.
Kestra is an open-source orchestration platform for scheduling and operating data, infrastructure, and business workflows. Its 2.0 LTS release separates control and data planes so workers can run across regions, clouds, or private networks over outbound-only connections. It also exposes flows as MCP tools for agents, decouples queue and storage choices, and includes migration tooling for 1.x users.
Filament replicates data from sources to sinks with full, incremental, or CDC modes, verifying every bounded batch on both sides of the write and advancing checkpoints only after success. Sources, sinks, state storage, and event transport are all swappable.
JupyterGIS 0.16 adds real-time collaborative story maps, lazy tile rendering for remote openEO workflows, and native GeoZarr and GeoPackage support. Declarative symbology and Xarray tiling keep visualizations reproducible while allowing large geospatial datasets to stay remote instead of being materialized locally.
Bayesian experimentation is not one method but a choice of prior, likelihood, and stopping rule. Spotify shows that common flat-prior posterior thresholds can reproduce frequentist peeking, while Bayes factors can control false positives under optional stopping. Choose guarantees for the experiment program first, then the inference configuration.
A controlled comparison of 2019-2025 open-model recipes and data corpora attributes 12.0x compute-efficiency gains to data improvements versus 3.7x to model recipes at a 1e19-FLOP budget. The result frames dataset extraction, curation, and filtering as a major systems lever, while cautioning that larger-scale and synthetic-data effects remain untested.
xlDuckDb adds a DuckDbQuery() function to Excel that runs DuckDB SQL and spills results into cells, querying worksheet ranges, tables, and local or remote JSON, CSV, and Parquet in one statement.
Referencing the same CTE twice tests whether the optimiser materialises the result.
Most of what a coding agent does is I/O rather than reasoning, such as reading five files to answer a question about one method. Spotify routed that work to two declarative Portal modes, bulk-reader and code-writer, both running Gemini 2.5 Flash on an ephemeral runtime, and reserved the frontier model for problems that need it. A quarter of engineering leaders already spend $200 to $500 per developer per month on tokens, and some are past $2,000.
This post explains why Meta put a managed proxy tier in front of ZippyDB, its massive key-value store. The real lesson is that proxies can turn client-fleet sprawl into a controllable infrastructure boundary: connection fan-in drops roughly 97–98%, batching and coalescing happen across clients, excess traffic from noisy tenants can be selectively rejected, and failover and load-balancing policy can change centrally without waiting for a million clients to update.
Vortex is an open-source columnar format designed to stream training data from S3 to GPUs with minimal copying. Cascading lightweight encodings, layout-aware pruning, and zero-copy memory paths target CPU and NVMe bottlenecks, giving ML platforms a way to keep expensive accelerators fed without converting everything upfront.
Semantic layers define business metrics, context layers provide supporting knowledge, and ontologies describe entities and their relationships. Together, they help AI agents interpret data, but people still need to keep that knowledge accurate and current.
Nao's founders argue that AI is shifting data teams toward maintaining business context and enabling self-service analytics. They recommend starting with one focused use case, measuring results, and improving data models before investing in a full semantic layer.
AI analytics can fail when agents generate free-form SQL against inconsistent warehouse metadata. Governed systems should route repeated questions through verified semantic-layer tools, parameterized calls, and explicit refusal paths, reserving open-ended SQL for cases humans can inspect. Maintaining definitions is the hard part regardless of model upgrades.
A 100,000-row Postgres load took 41 minutes because each row became its own statement and transaction; COPY cut the same work to 11 seconds. Real batching removes network round trips, parse overhead, and WAL flush barriers, while driver flags and transaction chunking prevent bulk APIs from silently behaving row by row.
Polars 2.0 makes the streaming engine the default for lazy queries, aiming for much lower memory use and faster execution while accepting that row order is not guaranteed unless requested. It also tightens type coercion and concatenation behavior, and removes deprecated APIs so data mismatches fail earlier instead of hiding in production pipelines.
Apache Iggy is a Rust streaming engine with Kafka-like logs and consumer groups, optimized around thread-per-core execution and modern Linux I/O. It looks promising for low-latency single-node or simpler deployments, but Kafka still has the deeper production story for replication, exactly-once semantics, compaction, and multi-region operations.
Sail 0.7 pushes Spark-style intermediate state toward object storage with blocking shuffle and checkpoint support. It persists shuffle outputs as compressed Arrow streams so downstream task regions retry without cascading upstream failures, while checkpointing keeps iterative jobs from rebuilding long query plans. The design targets stateless workers, smaller clusters, and better resilience under preemption.
Text-to-SQL assumes users ask questions that a single query can answer, when in practice they bring problems that need a sequence of queries with reasoning over the intermediate results. The paper proposes data investigations as the replacement paradigm and presents D2, a prototype that autonomously searches, reasons over, and collects data to reach evidence-backed conclusions, benchmarked on a Murder Mystery dataset.
Warner Music Group turned a hand-edited anti-counterfeit tracking spreadsheet into governed Databricks tables without taking the spreadsheet away from operators. The pipeline handles shifting tabs, renamed headers, late edits, row fingerprints, and full-overwrite loads so executive reporting can trust a messy business interface that people actually use.
PostgreSQL 19 Beta 3 adds temporal SQL, idempotent upsert-and-fetch, IGNORE NULLS window behavior, concurrent repacking, DDL extraction helpers, SQL/PGQ graph querying, and stricter NULL handling.
DuckFlight lets PostgreSQL tools and Arrow Flight SQL clients read and write the same live DuckDB database without a separate service or copying data.
Sponsored
They're all showing up to OPENSEARCHCON NORTH AMERICA [5] to share how they run search, observability, and data infrastructure at massive scale. WILL YOU BE THERE TOO? Get your ticket now [6] for three days of practical learning and direct connections with other technical leaders. Here's what to expect: * REAL-WORLD IMPLEMENTATION STRATEGIES: vector search, agentic AI workflows, observability pipelines * TECHNICAL SESSIONS: search relevance tuning, plugin development, and vector database optimization * HANDS-ON WORKSHOPS: build a full observability stack with OpenTelemetry, or a working agentic AI search application. * $250 CERTIFICATION VOUCHER toward the Linux Foundation's OpenSearch Certified Associate exam.
Netflix's member-sessionizer pipeline turns tens of billions of hourly client events into viewing sessions, but scale bottlenecks made ordinary tuning painfully slow. Instead of optimizing one Spark run at a time, the team built a parallel experiment loop that launched, monitored, and compared multiple production-scale tests to expose memory and shuffle bottlenecks faster.
Lyft retired its internally built Flink Kubernetes operator and migrated its streaming fleet to the open-source Apache Flink Kubernetes Operator. The useful lesson is the migration pattern: run both operators in parallel, preserve the user-facing deployment contract, patch gaps such as default restart behavior, and then use the move to simplify ownership and upgrades.
Databricks traced silent MCP tool failures that caused agents to repeatedly retry, revealing seven small bugs responsible for an estimated $499,000 in wasted tokens and $1.2 million in annual lost productivity. Fixing them took about an hour, highlighting the value of tracing tool calls and designing tools to accept the reasonable input variations generated by LLMs.
Modern apps can write successfully, invalidate cached state, and immediately read from a replica that has not replayed the change yet. PostgreSQL 19's WAIT FOR LSN lets applications route reads to replicas while waiting for a specific WAL position, turning read-your-writes consistency into a measured timeout budget instead of primary pinning or sleep hacks.
AWS bought DuckLabs to gain its core team and influence DuckDB's roadmap, positioning DuckDB, DuckLake and Quack as foundations for S3-based analytics and AI agent workloads. DuckDB remains independently governed and open source, while MotherDuck will likely differentiate through higher-level tools for pipelines, agents, and data applications.
NVIDIA demonstrates how modern CUDA tools can make GPU code safer and dramatically faster by combining better debugging, optimized CUB algorithms, pooled memory, pinned transfers, and asynchronous streams. These incremental changes reduced an image-processing pipeline from 6.8 seconds to 23 milliseconds, a roughly 300x speedup.
Named queries turn SPARQL from an expert-only endpoint into a governed API surface. By registering queries as RDF resources with parameters, output contracts, metadata, and ACLs, teams can expose graph data to applications and agents without giving every caller raw query access or forcing them to understand the whole ontology.
FrontierHarness found that coding-agent harnesses achieved relatively similar pass rates but varied by up to 17x in cost, showing that tooling, caching, and model compatibility strongly affect efficiency. Codex had the highest pass rate at 66.7% with moderate cost, while Exo was cheapest and DSH Minimal was fastest.
MLPerf Storage v3.0 adds benchmark tests for KV cache and vector database workloads, plus an S3 object-storage access layer alongside POSIX. That makes the suite more relevant to modern AI systems, where inference caches, vector indexes, checkpointing, and object storage can be the real bottlenecks behind model throughput and cost.
dbt doctor is an open-source static analysis tool that scans dbt projects for missing documentation and tests, schema drift, stale models, DAG complexity, and governance gaps. It can run locally, through coding agents, or in CI to score project health and block changes that introduce quality issues.
Large-scale operators often default to line charts even when they need to answer different questions about capacity, latency, fleet sizing, and distributions. A broader telemetry visualization toolbox is needed that matches the operational question instead of forcing every signal into time-series lines.
Cloudflare's Cache Transcoding prototype compresses eligible text assets with Zstandard before storing and transferring them, reducing their size to roughly one-third on average. This could unlock petabytes of additional cache capacity and reduce bandwidth usage for only a small increase in CPU consumption.