LARGE LANGUAGE AND VISION EMBEDDING MODELS WITH HARDNESS-WEIGHTED CONTRASTIVE LEARNING

Summary from the TLDR AI newsletter

Building embedding models for vision and language tasks using a contrastive loss can often lead these models to struggle on hard negative pairs. This work proposes a regularization strategy and observed a substantial improvement in difficult retrieval problems. The method also scales to zero shot video retrieval.