CAN WE CHANGE AN AI MODEL'S ARCHITECTURE WITHOUT RETRAINING IT FROM SCRATCH?

Summary from the TLDR IT newsletter

Amazon Web Services researchers have introduced priming, a method that reuses pretrained Transformer weights to reduce the pretraining cost of hybrid state space models by over 99%. The technique addresses the Transformer memory tax by replacing certain attention layers with state space layers like Gated KalmaNet, which researchers found can match source Transformer reasoning benchmarks while decoding up to twice as quickly. To foster further research, the team released a zoo of primed hybrid models, training code, and a vLLM serving plugin under the Apache 2.0 License.