NEMOTRON-H

Summary from the TLDR AI newsletter

Nvidia has released a new set of hybrid Mamba style models trained on 20T tokens. They achieve strong performance, especially on long context tasks where they have extreme sequence scaling benefits. This represents the first successful fp8 large scale training of a hybrid attention model.