HBF FOR HIGH-THROUGHPUT LLM SERVING (UC BERKELEY, FURIOSAAI)

Summary from the TLDR Hardware newsletter

UC Berkeley and FuriosaAI researchers explored using high-bandwidth flash (HBF) to enhance memory capacity for LLM serving. They achieved up to 87% faster completion times and 55.8% energy savings compared to HBM-only systems. Buffered cache-aware scheduling extended HBF write lifetime from 4.77 to 14.82 years, highlighting the impact of data placement strategies on performance and efficiency.