TLDR WEB DEV 2025-07-17

CRAWLING A BILLION WEB PAGES IN JUST OVER 24 HOURS, IN 2025

This dev successfully crawled a billion web pages in just over 24 hours on a budget of a few hundred dollars. The crawler was designed with politeness and fault-tolerance in mind, focusing on HTML content only and adhering to robots.txt rules. The system used a cluster of optimized nodes, each with its own Redis instance, fetcher processes, and parser processes, overcoming bottlenecks like parsing speed by using modern libraries and truncating page content.

DEBUGGING THE ONE-IN-A-MILLION FAILURE: MIGRATING PINTEREST'S SEARCH INFRASTRUCTURE TO KUBERNETES

Pinterest's engineering team encountered a performance issue while migrating their internal search system, Manas, to Kubernetes, where a small percentage of search requests experienced massive latency spikes. Through a process of elimination and profiling, they discovered that the monitoring tool cAdvisor was the culprit, specifically its default collection of the container_referenced_bytes metric. This metric invasively scanned and cleared page table entries, causing memory contention with Manas' memory-intensive search indices. Disabling the problematic metric in cAdvisor resolved the latency spikes.

ARTISANAL HANDCRAFTED GIT REPOSITORIES

You can manually create Git repositories by bypassing standard Git commands and directly manipulating the underlying Git objects. This post explains Git's internal workings, focusing on content addressable storage, commit objects, and tree objects, and walks through how to manually create file blobs, tree blobs, and commits.

I WAS WRONG ABOUT ROBOTS.TXT

This dev initially blocked all web crawlers using robots.txt to protect his data, but this inadvertently broke link previews on his LinkedIn posts. LinkedIn's bot couldn't access the necessary Open Graph meta tags to generate previews because it was blocked by the robots.txt file. To fix this, he updated his robots.txt to specifically allow the LinkedInBot to crawl his website.

I'M SWITCHING TO PYTHON AND ACTUALLY LIKING IT

This dev shifted to Python for AI development. His Python toolbox includes uv for package management, ruff for linting and formatting, and FastAPI for building APIs. For version control and infrastructure management, he uses GitHub Actions, Dependabot, Gitleaks, pre-commit hooks, Make, and Docker.

Skeptical? This case study shows how Salesloft saves $750k/year

in QA engineering + executes >300 tests in parallel on every PR in minutes.

Schedule a demo to learn more

CLAUDE CODE ROUTER

Claude Code Router is a tool that allows devs to route Claude Code requests to different language models and customize requests while still using the official Claude Code infrastructure. It's configured via a `config.json` file, enabling routing based on factors like background tasks, reasoning needs, or context length to services like OpenRouter, DeepSeek, Ollama, Gemini, and Volcengine.

SHIPPING WEBGPU ON WINDOWS IN FIREFOX 141

Firefox 141 will ship WebGPU on Windows, enabling high-performance web applications for games, visualization, and computation. This implementation, based on the Rust crate WGPU, follows the WebGPU and WGSL standards developed with Mozilla's participation.

EULER HS

EulerHS is a Haskell framework that makes the creation of concurrent backend and console applications easier. It has pre-integrated subsystems like SQL databases, logging, and KV databases.

THE PRAGMATIC ENGINEER 2025 SURVEY: WHAT'S IN YOUR TECH STACK?

The Pragmatic Engineer conducted a survey with nearly 3,000 tech workers to analyze the current tech landscape. The survey covered demographics, AI tools, programming languages, IDEs/terminals, version control, CI/CD, and cloud providers. It found that AI tools are rapidly gaining popularity, with GitHub Copilot leading and Cursor showing fast growth. TypeScript and Python are the most-used languages, while JIRA is the most-disliked tool.

LLM DAYDREAMING

A "day-dreaming loop" (DDL) for LLMs might address their lack of novelty and insight. The DDL involves a background process that continuously samples concepts, explores connections, and filters for valuable ideas. This process, while computationally expensive, could kickstart more innovation and generate proprietary training data.

HOW AND WHERE WILL AGENTS SHIP SOFTWARE?

InstantDB provides agents with full-stack backends, including databases, sync engines, and auth tools, to make the creation of applications easier. Agents and humans benefit from built-in abstractions, cost-efficient hosting, and exposed data.

GASLIGHT-DRIVEN DEVELOPMENT

"Gaslight-driven development" is a new trend where AI language models subtly influence API design by consistently suggesting certain features or methods, leading developers to implement them even if they weren't initially planned.

AM I BECOMING IRRELEVANT?

After transitioning from a hands-on role to a leadership position, this person feels more and more irrelevant, grappling with the lack of visible impact and recognition despite enabling their team's success.

SCALABILITY IS NOT PERFORMANCE

Scalability is the ability to adjust system throughput based on demand, and it is distinct from performance, which refers to the throughput itself.

HOW AI VIBE CODING IS DESTROYING JUNIOR DEVELOPERS' CAREERS

Vibe coding makes developers slower, despite their perception of increased productivity, so devs should use AI as a tool to improve their existing skills and focus on mastering the fundamentals of programming.