GROQ IS SERVING THE FASTEST RESPONSES I'VE EVER SEEN

Summary from the TLDR newsletter

Groq can serve up to 500 tokens per second. It is able to do this because it uses custom hardware that utilizes Linear Processor Units (LPUs) instead of GPUs. LPUs are designed to deliver deterministic performance for AI computations. They offer a more streamlined approach that eliminates the need for complex scheduling hardware, allowing every clock cycle to be utilized effectively. The system ensures consistent latency and throughput. LPUs can be linked together without the traditional bottlenecks found in GPU clusters, making them extremely scalable.