Full-text search across article titles and summaries. Refine with sector, year, content type, and reading-time filters.
Google's TimesFM-3 extends time-series foundation models from univariate forecasts to multiple interacting series with past and future covariates, aiming to make retail, observability, finance, and operations forecasting work without task-specific fine-tuning.
Keenable SELECT lets agents research the live web using read-only DuckDB queries that combine search, filtering, and AI-powered extraction.
Sponsored
Last year's Astronomer virtual conference drew 8,000+ data engineers. This year, _Orchestrate Everything_ [5]_ _will kick off with Marc Lamberti's free crash course on orchestrating AI with Airflow. You'll get a free certification exam code ($150 value) for joining.
Free to attend. Built for data engineers at every level.
Twilio's Hudi pipelines store checkpoints in S3, so ordinary Kafka consumer lag misses when lake data is actually stale. This design reads the latest valid Hudi checkpoint, seeks Kafka partitions, and computes freshness from the oldest uncommitted record, turning petabyte-scale SLA drift into a breach ratio operators can alert on.
Pinterest scaled candidate generation by conditioning retrieval embeddings on explicit contexts such as interests, boards, and pins. The system combines conditioned sequence modeling, semantic IDs, unified models, slot routing, request deduplication, and GPU serving so one home feed can represent multiple user intents without exploding serving cost.
Kafka remains a strong log abstraction, but many streaming workloads ultimately reconstruct tables in object storage anyway. This walkthrough contrasts log-first pipelines with table-first streaming designs such as Fluss, showing where update semantics, compaction, snapshots, and catalog integration may belong closer to hot storage instead of a downstream lakehouse job.
Target encoding, rolling averages, and shifted features can silently leak the answer when computed before a proper split or across mixed entities. The safe workflow is to split first, calculate training-set encodings out of fold, fit other statistics only on training data, use group-aware transforms, and give unseen categories honest fallbacks.
A company moving four large Salesforce instances into Snowflake wants to replace Fivetran after cheaper tools failed at scale. The strongest alternatives suggested were Salesforce Data Cloud's native zero-copy Snowflake integration or a custom dlt pipeline using Salesforce's Bulk API.
Grab operationalizes data reliability by treating contract breaches as "Data Production Issues" (DPIs) with a standardized incident lifecycle. The key design move is a stable "Data Health API" that decouples incident orchestration from platform-specific internals, categorizing failures (upstream, job, or data) to automate root-cause assignment. This transforms data quality from passive alerting into an operational governance model where incidents only close upon verified recovery.
Major cloud incidents repeatedly stem from unavoidable risks such as resource saturation, networking failures, security controls, and uncommon changes like migrations. Reliability measures add their own complexity, so outages cannot be eliminated and teams must invest in strong incident response.
Sponsored
Most AI tools don't understand your dbt project. They break lineage and miss dependent tests, creating more work. dbt Wizard CLI reads contracts and dependencies before editing, and coordinates changes across models and YAML configurations. It validates its own work. Engineers get dbt-native accuracy without a platform migration. Learn
Spark 4.1.0 adds declarative pipelines, real-time structured streaming mode, Arrow-native UDF and UDTF support, SQL scripting and VARIANT as GA features, and Spark ML on Connect for Python. The release targets lower-latency streaming and a cleaner PySpark and SQL experience while improving caching, memory management, and large-workload stability.
DataFusion 55.0.0 ships ten weeks of work from 175 contributors, with MERGE INTO support, range partitioning, runtime row-group pruning, and broad performance gains. Highlights include TopK pruning, faster IN-list evaluation, fewer object-store requests for CSV, nested-column Parquet read reduction, and lower memory use for fixed-size binary GROUP BY operations.
Ablation results in tabular foundation models can be misleading when redundancy looks like causal importance. Measurements of direct and total effects across four models found little evidence of downstream self-repair, suggesting data teams should treat stable outputs after ablation as a reason to examine causality more carefully, not as proof that a layer is irrelevant.
X's legal action against Nitter shows how closed platforms can revoke access at any time, limiting competition and innovation. AT Protocol offers an alternative by giving users portable identities and letting apps freely replicate, query, and build on shared live data.
Lorikeet sped up some Postgres queries by up to 1,000x with a simple typecast that fixed an index-breaking mismatch between timestamp and timestamptz under Row Level Security.
Better data, expert input, and iterative labeling often improve ML systems more than model tuning.
This geospatial modeling framework automatically selects and combines satellite, tabular, and foundation-model data for high-resolution predictions.
Sponsored
They're all showing up to OpenSearchCon North America [4] to share how they run AI search, observability, and data infrastructure at (massive) scale. Come for real production patterns and two hands-on workshops: * Build an OpenTelemetry observability stack * Build an agentic AI search app Additional speakers [4] come from Airbnb, Intel, IBM and more. Join in San Jose from September 22 to 24, to build the future of open source search. Come for the talks, leave with something you can actually use on Monday.
LinkedIn built a cognitive memory agent that personalizes its hiring assistant using four layers: conversational, episodic, semantic, and procedural memory. It replaced GraphRAG with a hierarchical tree that supports cheaper incremental updates, while dedicated ingestion, consolidation, and retrieval services keep preferences fresh, traceable, access-controlled, and available within tight latency limits.
DoorDash's SafeChat handles millions of chats, calls, and images per day with a hybrid moderation pipeline. Fast internal classifiers clear obvious safe cases, LLMs score the hardest cases across severity axes, and no-code workflows plus backtesting let teams tune actions before production. The system cut verbal-abuse incidents roughly in half.
In Kafka-style streaming systems, the shared log is the ordered event backbone that brokers and consumers build on. LogDrive separates that log into an AtomicLog sequencing layer and a durability layer built from flat single-value registers plus weakTail, making striping and quorum replication reusable across S3, DynamoDB, S3 Express, or other backends.
RocksDB amplification depends heavily on workload shape, compaction capacity, and key order. Random inserts produced 4.56x write amplification while sequential keys stayed the same, and delete-heavy workloads temporarily pushed space amplification to 1.58x. Bloom filters kept L0 read latency nearly flat as file counts grew.
GitHub's secret-scanning work shows why LLM evaluation should optimize the production decision, not generic benchmark scores. Prompts, models, input construction, and eval configs were versioned like code, then tested against production-like ambiguity. That discipline cut false positives by 95% offline while preserving recall within the safety guardrail.
Complex tasks can exceed a single AI agent's organizational capacity, regardless of better context or tools. Graph Engineering addresses this by mapping tasks, specialized agents, and system states to coordinate parallel work, verification, and persistent execution.
AI creates more pipelines and dependencies without matching growth in data teams, making engineers the operational bottleneck. Companies should prevent burnout with scalable infrastructure, governance, and automated operations matched to their actual needs and capacity.
Sponsored
Missed RFS dates, Phantom Clusters, and layers of vendors compound challenges that start with a volatile supply chain. Michael Francisco, Vice President of WhiteFiber Cloud US, breaks down what's driving these dynamics from the seller's side, and how to simplify your sourcing process.
DuckLabs is joining AWS to gain the infrastructure, resources, and reach needed to scale DuckDB and the wider Duck Stack. DuckDB, DuckLake, and Quack will remain MIT-licensed open-source projects stewarded by the independent DuckDB Foundation, while AWS supports faster development and broader adoption.
Khaos is an open-source Kafka load-testing and chaos-simulation tool that was recently rewritten in Go. It generates realistic producers and consumers, exercises broker failures, hot partitions, consumer lag, rebalances, and backpressure, and gives platform teams a repeatable way to test Kafka behavior before they encounter production incidents.
PostgreSQL regex search can vary wildly by index and extension. In a 1.6M-row, 33 GB plan table, pg_trgm cut a simple LIKE scan from about 40 seconds to 1.6 seconds, pg_tre added fuzzy matching but had painful index build time, and pg_re2 delivered faster regex execution with fewer language features.
PagedAttention brings virtual-memory-style paging to LLM KV caches by replacing contiguous per-request allocation with fixed-size blocks and a block table. That design eliminates most cache fragmentation, raises utilization to roughly 96%, supports copy-on-write for shared prompts, and can serve 2–4x more users per GPU without changing attention outputs.
Enterprise application software is still growing, but aggregate market numbers hide where AI is actually changing buyer behavior. Workflow-heavy and automation-friendly categories are pulling ahead while customer service and HCM lag. For buyers, AI packaging and pricing now reveal more about value capture than broad market-growth headlines.
Analyst Gym uses daily two-minute workplace drills to sharpen analytical judgment rather than coding skills.
Neocarta builds a Neo4j semantic layer that unifies database schemas, business definitions, metrics, and query history, then exposes this context to AI agents through MCP.
Model maintenance is not just retraining. When COVID-era demand patterns faded, Airbnb's forecasting team separated parameter drift from structural change, choosing among refit, respecify, or hold decisions instead of blindly updating thousands of markets. The result was a useful operating model for production forecasting under regime shifts.
Good SQL indexes come from real query patterns, not the table schema. Composite indexes need the right column order, usually equality filters first and sort or range columns after, with EXPLAIN ANALYZE used to verify the result. The right indexes cut example queries from 17ms to 0.04ms and 436ms to 0.5ms.
Netflix is moving 30,000+ Flink jobs from a homegrown autoscaler toward the Apache Flink Autoscaler. The old system saved 25–45% by scaling whole clusters from container metrics, but missed operator-level state and suffered from telemetry gaps. The OSS autoscaler reasons from inside each job and produced 58% annualized compute savings for one team.
Uber's Gulfstream payments platform now supports $217B in annualized gross bookings, nearly twice that in money movement, and ledger balances for 1.2B+ entities. The design relies on immutable zero-sum money orders, double-entry accounting, DynamoDB-backed balances, Kafka pipelines, Cadence for synchronous flows, and generic payment-instrument interfaces that let new business lines plug in with minimal core changes.
Bulk export APIs are giving way to shared table references. Iceberg REST catalogs, object storage, Parquet, and credential vending let producers publish metadata while consumers bring their own compute, avoiding pagination, rate limits, and duplicate pipelines. The hard parts move to contracts, compaction, semantics, privacy, and table maintenance promises.
Dashboards are not neutral: the metrics and visuals they prioritize shape what leaders see, fund, and act on, while harder-to-measure realities disappear. In African public-sector and NGO settings, donor-facing dashboards can obscure frontline needs and should be balanced with local knowledge.
Production AI systems work best when each component earns its complexity. Recommenders, moderation, warehouse queries, and search cold starts need different mixes of deterministic logic, classic ML, semantic layers, and LLMs. The strongest pattern is often hybrid: use agents for reasoning, but keep business logic, metrics, routing, and guardrails deterministic where possible.
DuckDB v2.0 replaces its PostgreSQL-derived SQL parser with a PEG-based parser while preserving the downstream binder and AST pipeline. The change removes Bison conflict pain, avoids exponential backtracking on malformed input, and enables runtime grammar extensions that can add syntax without reimplementing SQL parsing.
TrueForge is a self-hostable agent harness focused on reducing task cost through context compaction, delayed tool-schema loading, subagents, and sandbox-as-a-tool execution. The release matters for data teams experimenting with governed agents because it separates orchestration from model choice and pairs naturally with gateways for identity, access control, observability, and budgets.
Two projects push DataFusion beyond standard analytics: one combines RDF graphs with time-series and temporal logic, while Cloudflare's AutoSQL uses it for distributed serverless SQL over Iceberg and R2. Both rely heavily on custom planning, pruning, joins, and execution optimizations.