Explore how Salesforce saved over $500K annually in token-based costs by replacing live LLM dependencies with a controllable simulation layer and more.

The Salesforce Engineering Blog offers a deep dive into Salesforce technologies, providing technical insights, best practices, and real-world examples. Developers can explore topics such as Salesforce development, architecture, and integrations, gaining  knowledge to build innovative solutions on the Salesforce platform. With contributions from Salesforce engineers and industry experts, the blog serves as a resource for the Salesforce community.

Salesforce Engineering

Salesforce built an internal mock LLM service to replace live API calls during performance benchmarking and testing. The service simulates LLM responses with configurable latency and error conditions, enabling teams to validate performance, scalability, and failover behavior without external dependencies. This approach saved over $500K annually in token costs, eliminated latency variability that required repeated test runs, and enabled controlled failure injection for testing circuit breakers and failover from OpenAI to Azure OpenAI. The service handles over 50,000 requests per minute and has been widely adopted across Salesforce's AI platform teams.

How a Mock LLM Service Cut $500K in AI Benchmarking Costs