Anthropic has rolled out a new prompt caching feature in the API for its Claude family of generative AI models, aimed at significantly decreasing costs and latency for developers. Available now in public beta for Claude 3.5 Sonnet and Claude 3 Haiku models, prompt caching allows frequently used prompts to be saved and reused, which cuts down on the need to continually reprocess these prompts. Future support for Claude 3 Opus is also planned.
This new capability promises remarkable efficiency improvements, offering cost reductions of up to 90% and latency reductions of up to 85% for long prompts. Cached contexts can be reused in future API calls, which is especially useful for applications such as coding assistants, long-form document processing, and detailed instruction sets. While there is a 25% increase in cost for writing to the cache, using cached content costs 10% less than generating new prompts.
Developers have shared early reports of notable speed and cost improvements with the caching feature. The system works particularly well for long instructions and embedding documents within prompts. A notable feature of the cache is its 5-minute lifetime, which is refreshed upon each use, ensuring that frequently accessed prompts remain available.
However, the new capability isn’t without potential concerns. There are some security implications related to the sharing of cached prompts, which developers will need to address. Nevertheless, the dramatic efficiency gains make prompt caching a compelling addition for those using Claude 3.5 Sonnet and Claude 3 Haiku in their applications.

Collections

Anthropic has introduced prompt caching in the Claude AI models, significantly reducing costs and latency for developers. Available in public beta for Claude 3.5 Sonnet and Claude 3 Haiku models, prompt caching allows reuse of frequently used prompts, cutting down reprocessing needs. Early reports show 90% cost and 85% latency reductions for long prompts. The cache has a 5-minute lifetime and operates effectively for long instructions and embedded documents, though developers must consider security implications.

Anthropic Introduces Prompt Caching to Reduce Latency and Costs for Claude Models