TLDR WEB DEV 2025-01-06

BUILDING A POLITE & FAST WEB CRAWLER

A Mozilla engineer noticed that poorly-behaved bots, primarily from OpenAI and Amazon, caused 70% of the load on Diapsora's servers (an open-source social network). To address this, he built a “polite and fast” web crawler using Python and gevent, prioritizing respectful rate limiting by defaulting to one request per domain every five seconds and dynamically adjusting based on server responses. The crawler efficiently manages the queue by fetching robots.txt files in parallel and uses a sophisticated refetching strategy using HTTP headers to minimize redundant requests.

YOU DON'T NEED NEXT.JS

ComfyDeploy migrated its dashboard from Next.js to a plain React setup, which improved build times (from 3 minutes to 18 seconds) and hot reload speed (under 200ms). This decision was driven by escalating Vercel costs, testing difficulties with Next.js's server actions, and painfully slow local development. The migration incorporated TanStack Router and Rspack, forcing architectural improvements and cleaner code.

BACK TO BASICS: WHY WE CHOSE LONG POLLING OVER WEBSOCKETS

Inferable chose HTTP long polling over WebSockets for its real-time system due to scalability challenges with its Node.js/TypeScript backend and PostgreSQL database. Long polling, unlike short polling or WebSockets, keeps the connection open until data is available or a timeout is reached, offering a balance between immediacy and resource efficiency. This approach avoided complexities associated with WebSockets, such as adapting its observability stack, authentication mechanisms, and infrastructure configurations.

HOW AI-ASSISTED CODING WILL CHANGE SOFTWARE ENGINEERING: HARD TRUTHS

There are two developer types: "bootstrappers," who use AI for rapid prototyping, and "iterators," who use AI in daily workflows. While AI accelerates development a lot, it introduces challenges like the "70% problem," where achieving the final 30% of software quality requires real human expertise. As a result, AI makes the lives of experienced software engineers easier, but doesn't replace them.

REFLECTIONS

Sam Altman reflects on OpenAI's journey, the unexpected success of ChatGPT's launch, and the subsequent rapid growth. It was challenging and messy to build a company around groundbreaking technology, as shown by his own public firing and subsequent reinstatement. He's learned a lot and will be more confident moving forward.

free

LOGLAYER

LogLayer is a JavaScript logging library that provides a consistent logging experience across different logging providers. It offers a flexible API for adding tags, metadata, and errors to log messages, supporting multiple loggers simultaneously (e.g., DataDog and New Relic).

TAILWIND CSS MOTION

A Tailwind CSS plugin that lets you animate any element with inline styles. It also comes with a Chrome extension to let you play with animating your website directly.

CSS FLEXBOX PLAYGROUND

Experiment with different flex properties to understand how they affect layout. CSS code is generated in real time and controls are available for finetuned adjustments.

YOU WOULDN'T DOWNLOAD AN AI

This developer extracted an AI model from Microsoft's Seeing AI app. The app's model was found encrypted within the APK file as a TensorFlow Lite model. It was then verified using Netron, which led to the successful extraction of the AI model's weights and biases.

CAN LLMS WRITE BETTER CODE IF YOU KEEP ASKING THEM TO “WRITE BETTER CODE”?

An LLM was told to "write better code," which led to performance improvements in a Python program through multiple iterations. However, the improvements eventually plateaued, and the code became over-engineered. A second experiment used more precise prompt engineering, explicitly outlining optimization goals, which eventually led to a 100x speedup.

THANK YOU (2024 EDITION)

The CSS-Tricks website returned to consistent publishing in 2024, leading to 26 million unique pageviews, many significant site updates and content additions, and new goals.

WORLD'S FIRST MIDI SHELLCODE

This author gained remote code execution on a Yamaha PSR-E433 synthesizer via its MIDI interface by reverse engineering its firmware.

PRINTF DEBUGGING IS OK

Devs should take a pragmatic approach to debugging, and all techniques are valid (including just using printf) for efficiently finding and fixing bugs.

THE RECOIL REPOSITORY HAS BEEN ARCHIVED ON JAN 1, 2025. YOU HAD A GOOD RUN (REDDIT THREAD)

After both maintainers of Recoil, a React state management library created by Meta, were laid off, the repository was officially archived as no Meta engineers were willing to take up the work of maintaining it.