WEBLLM

Summary from the TLDR Dev newsletter

WebLLM runs open-source language models entirely inside the browser using WebGPU, with no server required and an OpenAI-compatible API for streaming, JSON mode, and other chat-completion features. It supports model families including Llama, Phi, Gemma, Mistral, and Qwen, plus npm or CDN installation, multiple browser cache backends, and Web Worker or Service Worker engines that keep inference from blocking the UI.