MONITOR PROMPT CACHING TO OPTIMIZE YOUR TOKEN USAGE

Summary from the TLDR DevOps newsletter

Prompt caching can significantly reduce LLM cost and latency as agent systems accumulate large, repetitive system prompts, tool definitions, guardrails, and context. Datadog recommends keeping cached prefixes stable, placing breakpoints around static context and conversation history, and monitoring cache hit rates, writes, token usage, and traces to identify cache-busting changes such as tool reordering, compaction, and dynamic metadata.