DISCOFORMER: ONE TRANSFORMER FOR DENSITY AND SCORE, ACROSS DISTRIBUTIONS

Summary from the TLDR AI newsletter

DiScoFormer, a transformer model, estimates both density and score from data in a single pass without retraining, surpassing classical kernel density estimation (KDE) in accuracy, especially in high dimensions. Utilizing cross-attention, it adapts to new data distributions on the spot, improving accuracy for generative modeling and Bayesian inference. Trained with Gaussian Mixture Models, DiScoFormer reduces score error by 6.5x and density error by more than 37x over KDE in 100 dimensions.