DO YOU KNOW WHAT A CACHED PROMPT OR CACHED TOKEN _REALLY_ IS? NGROK GUIDE

Summary from the TLDR AI newsletter

Cached prompts are 10x cheaper and 85% faster. What actually gets cached isn't the response, but the K and V matrices from the attention