Context Caching Mechanics: Slashing LLM Bills by 75% on Repeated Prompt Prefixes
When an LLM processes an input prompt of length $N$, every transformer layer must compute Key and Value tensors ($K, V$) for each token across all attention heads. For massive prompts (such as entire 100,000-token codebases, legal repositories, or long system prompt constitutions), recomputing these KV tensors on every single turn burns massive compute and drives up Time-To-First-Token (TTFT).
Context Caching eliminates this redundant compute by persisting the calculated KV tensors in GPU High Bandwidth Memory (HBM) or ultra-fast NVMe tiers. Subsequent API calls that share an identical token prefix directly load the precomputed KV tensors, slashing both latency and token pricing.
Implementing Anthropic Prompt Caching with Python
Anthropic Claude supports explicit cache breakpoints via the cache_control parameter. Up to 4 distinct cache breakpoints can be placed anywhere within tools, system prompts, or multi-turn conversational histories:
import anthropic
client = anthropic.Anthropic()
# Load a massive 80,000-token codebase or legal document
with open("massive_codebase_context.md", "r") as f:
massive_docs = f.read()
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
system=[
{
"type": "text",
"text": "You are a principal security engineer auditing the attached repository for CVE vulnerabilities."
},
{
"type": "text",
"text": massive_docs,
# Explicit cache boundary: instructs Claude to cache everything up to this token
"cache_control": {"type": "ephemeral"}
}
],
messages=[
{"role": "user", "content": "Analyze authentication flow in src/auth/session.ts"}
]
)
# Inspect cache performance metrics
usage = response.usage
print(f"Tokens read from cache: {getattr(usage, 'cache_read_input_tokens', 0)}")
print(f"Tokens written to cache: {getattr(usage, 'cache_creation_input_tokens', 0)}")
print(f"Uncached tokens: {usage.input_tokens}")