LEFTOVERLOCALS: LISTENING TO LLM RESPONSES THROUGH LEAKED GPU LOCAL MEMORY

Summary from the TLDR InfoSec newsletter

Trail of Bits is disclosing a vulnerability that it discovered that allows for memory leakage from an optimized GPU region in GPUs from Apple, Qualcomm, AMD, and Imagination. The researchers showed that they could leak 5.5MB per GPU invocation, which translates to 181MB on a 7B llama.cpp model. They demonstrated that this is enough information to reconstruct the LLM response with high accuracy.