LLAMA 3-V: MATCHING GPT4-V WITH A 100X SMALLER MODEL AND 500 DOLLARS

Summary from the TLDR Web Dev newsletter

Llama3-V is an open-source multimodal model created by Stanford ML researchers that's able to keep up with GPT4-Vision. The researchers accomplished this by combining the language capabilities of Llama 3 with a vision encoder from SigLIP, using a projection block for alignment, and prepending image tokens to textual tokens.