If you’re building LLM apps at scale, your gateway shouldn’t be the bottleneck. That’s why we built Bifrost — a high-performance, fully self-hosted LLM gateway optimized for speed, scale, and flexibility, built from scratch in Go.
Bifrost is designed to behave like a core infrastructure service. It adds minimal overhead even at extremely high load (e.g. ~11µs at 5K RPS) and gives you fine-grained control across providers, monitoring, and transport.
<h2>What we focused on</h2>
<ul>
<li>Unified OpenAI-style API for 1,000+ models across OpenAI, Anthropic, AWS Bedrock, Google Vertex, Azure, and more</li>
<li>Adaptive load balancing that distributes requests based on latency, error history, TPM limits, and usage</li>
<li>Cluster-mode resilience with peer-to-peer sync so node failures don’t disrupt routing or data</li>
<li>Automatic provider failover and semantic caching to reduce latency and cost</li>
<li>Observability with metrics, logs, and distributed tracing</li>
<li>Extensible plugin system for analytics, monitoring, and custom logic</li>
<li>Flexible configuration via Web UI or file-based setups</li>
<li>Governance features like virtual keys, hierarchical budgets, SSO, alerts, and exports</li>
</ul>
Bifrost is fully self-hosted, lightweight, and built for scale. The goal is to make it easy for developers to integrate multiple LLMs with minimal friction while keeping performance high.
If you’re running into performance ceilings with tools like LiteLLM — or just want something reliable for production — give it a shot.
<ul>
<li>Repo: <a href="https://github.com/maximhq/bifrost" target="_blank" rel="noopener nofollow">https://github.com/maximhq/bifrost</a></li>
<li>Website: <a href="https://getbifrost.ai/" target="_blank" rel="noopener nofollow">https://getbifrost.ai/</a></li>
</ul>
Would love feedback, issues, or contributions from anyone who tries it out.

Self-hosted refers to the practice of running your own applications and services on your own servers or devices, giving you full control over your data and how it is managed.

selfhosted

Bifrost is an open-source, self-hosted LLM gateway built in Go that provides a unified OpenAI-style API for 1,000+ models across multiple providers (OpenAI, Anthropic, AWS Bedrock, Google Vertex, Azure). It features adaptive load balancing, automatic failover, semantic caching, cluster-mode resilience with peer-to-peer sync, and adds minimal overhead (~11µs at 5K RPS). The gateway includes observability tools, an extensible plugin system, governance features like virtual keys and hierarchical budgets, and can be configured via Web UI or files.

Bifrost: An open-source, high-performance, multi-provider LLM gateway for your projects