TRAINING MRNA LANGUAGE MODELS ACROSS 25 SPECIES FOR $165

Summary from the TLDR AI newsletter

OpenMed built an end-to-end protein AI pipeline that covers structure prediction, sequence design, and codon optimization. The team compared multiple transformer architectures for codon-level language modeling and found that CodonRoBERTa-large-v2 was the clear winner, with a perplexity of 4.10 and a Spearman CAI correlation of 0.40, significantly outperforming ModernBERT. They then scaled to 25 species, trained four production models in 55 GPU-hours, and built a species-conditioned system that no other open-source project offers. This post contains the complete results, architectural decisions, and runnable code.