TLDR Dev · ARTICLES & TUTORIALS

FINDING AND FIXING A 50,000 GOROUTINE LEAK THAT NEARLY KILLED PRODUCTION

skoredin.pro8 min read

Summary from the TLDR Dev newsletter

A 50,000 goroutine leak in a WebSocket notification service gradually degraded production over 6 weeks, growing from 1,200 goroutines and 2.1GB RAM to 50,847 goroutines consuming 47GB RAM with 32-second latency. The leak was caused by three bugs: never calling context cancel functions when WebSocket connections closed, not stopping ticker objects in heartbeat goroutines, and not properly closing message channels. The fix involved adding proper WebSocket close handlers, connection monitoring, and making sure all resources (contexts, tickers, and channels) were properly cleaned up when connections died.