AI Cost Optimization Guides & Best Practices
Practical techniques with explicit assumptions for measuring and reducing LLM API spend.
A repeatable workflow
Optimize cost without hiding quality regressions
Choose one production-like workload and change one lever at a time. The objective is not the lowest token bill in isolation; it is a lower cost for an outcome that still meets your quality and reliability threshold.
- 1
Capture a baseline
Record requests, input, cached input, output, retries, latency, and successful outcomes for a fixed sample.
- 2
Change one lever
Test caching, routing, retrieval, output limits, or agent steps independently so the effect remains attributable.
- 3
Verify the outcome
Run the same evaluation set and reject savings that create unacceptable errors, truncation, or latency.
- 4
Price and document
Use reviewed rates, calculate the monthly effect, and record assumptions, exclusions, and the decision date.
Prompt caching essentials
Reuse eligible stable prompts and reference context at the provider's cached-input rate, then measure the actual hit rate and cost change.
- Place stable content (system prompt, tools, references) at the start.
- Keep variable user input at the end of the prompt.
- Enable caching on the static prefix and measure the hit rate.
Measure before deciding
Eligible repeated prefix tokens, cache hit rate, billed hit and miss tokens, and monthly cache write, storage, and read charges.
Model routing by difficulty
Send simple tasks to a small, cheap model and reserve flagship models for genuine reasoning to slash blended cost.
- Classify each request by complexity before inference.
- Route classification and extraction to a small model.
- Escalate only hard tasks to the flagship model.
Measure before deciding
Task success by route, escalation and fallback rate, latency, and total tokens per successful result.
Output token control
Output costs several times more than input. Cap and shape responses to keep the most expensive tokens in check.
- Set explicit max_tokens on every call.
- Request concise or structured output where a machine consumes it.
- Prefer long context with short answers over verbose generation.
Measure before deciding
Output-token p50 and p95, truncation or retry rate, and acceptance rate for the intended task.
RAG context efficiency
Retrieval quietly inflates input tokens. Tune chunking and reranking to send fewer, sharper passages.
- Test several chunk sizes against a fixed evaluation set.
- Rerank candidates and keep only passages that meet the measured relevance threshold.
- Strip boilerplate and deduplicate before sending.
Measure before deciding
Retrieved context tokens, answer quality on a fixed evaluation set, and embedding, reranking, and storage cost.
AI agent budgeting
Agents re-send growing context on every loop. Cap steps and compress history to keep runs affordable.
- Set a hard step limit per agent run.
- Summarize conversation history instead of carrying full transcripts.
- Use a cheap model for orchestration and tool selection.
Measure before deciding
Steps per run, retry or loop rate, completion rate, and context growth between the first and final step.
Infrastructure cost hygiene
Beyond tokens, retries, embeddings, and idle capacity add up. Make cost observable across the stack.
- Reduce retries with robust error handling and validation.
- Version embeddings and re-embed incrementally.
- Right-size or autoscale self-hosted GPU capacity.
Measure before deciding
Token spend plus retries, evaluations, embeddings, storage, network traffic, and idle compute.
Keep an optimization decision log
For each experiment, save the workload version, model, provider price date, token measurements, quality threshold, result, and owner. Recheck the decision after a model, prompt, retrieval pipeline, or provider price changes.