ENABLING TRILLION-PARAMETER MODELS ON AWS EFA

Summary from the TLDR AI newsletter

Perplexity published code enabling trillion-parameter mixture-of-experts models to run efficiently across multiple AWS servers for the first time, opening up less advanced data centers to AI inference. The challenge is that large models, even many open weights models, are too large to fit on single servers, but AWS' networking hardware was too slow for the constant inter-server communication these models require.