PACKING FOR LLM TRAINING

Summary from the TLDR AI newsletter

Packing in training models is a way to improve training efficiently by concatenating examples. If done incorrectly, contamination can occur between examples because the attention doesn’t know where to stop. The community has found that EOS is often sufficient, but there still can be problems. This repository provides an implementation in Hugging Face for common models to properly pack input data.