OLLM

Summary from the TLDR AI newsletter

oLLM is a lightweight Python library for large-context LLM inference that enables users to run models like Llama-3.1-8B-Instruct on 100k context using around a consumer GPU with 8GB VRAM. It doesn't use any quantization, only fp16 precision.