LENS (LANGUAGE MODELS ENHANCED TO SEE)

Summary from the TLDR AI newsletter

Can we get models to take images as output without retraining them? Amazingly, if you use small, image-specific modules, you can provide context to a fully frozen language model that allows it to answer questions about images. Paper, code, and demo is included.