CRAM LESS TO FIT MORE: TRAINING DATA PRUNING IMPROVES MEMORIZATION OF FACTS

Summary from the TLDR AI newsletter

Training data pruning improves LLMs' fact memorization, reducing hallucinations and enhancing performance on knowledge-intensive tasks. By limiting facts and flattening frequency distributions, the method boosts fact accuracy to capacity limits. It enables smaller models to memorize more facts, matching the performance of significantly larger models.