NVIDIA LAUNCHES A FASTER INFERENCE ENGINE FOR LLMS

Summary from the TLDR AI newsletter

Training costs dominated inference when the current AI race began. Inference costs can quickly balloon as more people deploy language models to production. TensorRT has long been the go-to for speed. A version specifically for language models on H100s is now available.