OPENAI'S JALAPEÑO OPTIMIZES INFERENCE THROUGHPUT AND TOKEN LATENCY

Summary from the TLDR AI newsletter

OpenAI's Jalapeño inference chip was designed around its own model workloads, with early benchmarks showing higher peak throughput per kilowatt and lower token latency than the commercial systems tested on