We deployed Bifrost to production. At 500 RPS, everything worked great. At 2,000 RPS, occasional 502s appeared. At 5,000 RPS, the gateway returned 502s for 30% of requests.
The problem wasn’t obvious. CPU was fine. Memory looked stable. Network had capacity. But requests were failing.
Here’s what we learned debugging production gateway failures.
<a href="https://github.com/maxim-ai/bifrost" target="_blank" rel="noopener nofollow">GitHub</a> | <a href="https://docs.getbifrost.ai/benchmarking" target="_blank" rel="noopener nofollow">Benchmarks</a>
<h2>Problem 1: Connection Pool Exhaustion</h2>
Symptom: 502s increase under sustained load. CPU idle. Memory stable.
What was happening: Each request to OpenAI/Anthropic needs an HTTP connection. Go’s default HTTP client reuses connections, but we weren’t tuning the pool size.
Under load, connection reuse dropped from 90% to 40%. New connections took 50-100ms to establish. Latency spiked. Timeouts triggered. 502s.
The fix: Adaptive connection pooling.
<pre><code class="language-go">// Before: Default http.Transport
client := &amp;http.Client{}

// After: Tuned connection pool
transport := &amp;http.Transport{
 MaxIdleConns: 100,
 MaxIdleConnsPerHost: 20,
 IdleConnTimeout: 90 * time.Second,
 DisableKeepAlives: false,
}
client := &amp;http.Client{Transport: transport}
</code></pre>
Result: Connection reuse jumped to 95%. 502s dropped by 60%.
But we still had 502s at peak load.
<h2>Problem 2: Goroutine Leaks</h2>
Symptom: Memory slowly climbing. Goroutine count increasing over hours. Eventually: 502s everywhere.
What was happening: Error handling paths weren’t properly cleaning up goroutines. When a provider timeout occurred, the goroutine handling that request didn’t terminate. After hours of sustained traffic, thousands of zombie goroutines accumulated.
How we found it:
<pre><code class="language-bash"># Monitor goroutine count
curl http://localhost:8080/debug/pprof/goroutine?debug=1

# After 6 hours at 2K RPS
# Expected: ~200 goroutines
# Actual: 4,783 goroutines
</code></pre>
The fix: Proper context cancellation.
<pre><code class="language-go">// Before: Missing cleanup
func processRequest(req *Request) {
 go func() {
 resp, err := client.Do(req)
 // If this errors, goroutine never terminates
 }()
}

// After: Context-based lifecycle
func processRequest(ctx context.Context, req *Request) {
 ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
 defer cancel() // Always cleanup
 
 go func() {
 select {
 case result := &lt;-process(ctx, req):
 return
 case &lt;-ctx.Done():
 return // Properly terminate on timeout
 }
 }()
}
</code></pre>
Result: Goroutine count stayed stable at ~200 even after days of sustained load.
But 502s still appeared during traffic spikes.
<h2>Problem 3: Worker Pool Backpressure</h2>
Symptom: Brief traffic spikes (2K → 5K RPS) caused 502s. Sustained 5K RPS worked fine after warmup.
What was happening: Our worker pool had fixed sizing. When traffic spiked suddenly, the job queue filled up. New requests had nowhere to go. Queue full → reject → 502.
The fix: Buffered channels with backpressure policy.
<pre><code class="language-go">// Before: Fixed queue size
jobQueue := make(chan *Job, 1000) // Hard limit

// After: Adaptive buffering
type WorkerPool struct {
 jobQueue chan *Job
 config PoolConfig
}

func (wp *WorkerPool) Submit(job *Job) error {
 select {
 case wp.jobQueue &lt;- job:
 return nil // Success
 case &lt;-time.After(100 * time.Millisecond):
 // Queue full, wait briefly
 select {
 case wp.jobQueue &lt;- job:
 return nil
 default:
 return errors.New(&quot;queue full&quot;) // Graceful rejection
 }
 }
}
</code></pre>
Buffer sizing formula we landed on:
<pre><code>optimal_buffer = (target_rps * avg_latency_seconds) * 3
</code></pre>
For 5K RPS with 2s avg latency: <code>5000 * 2 * 3 = 30,000</code> buffer size.
Result: Handled burst traffic without 502s.
Still had one more problem.
<h2>Problem 4: Memory Allocations Under Load</h2>
Symptom: Latency spikes every 10-15 seconds. Brief 502s during spikes. CPU shows GC activity.
What was happening: Every request allocated new buffers for JSON marshaling and HTTP bodies. At 5K RPS, that’s 5,000 allocations per second. Go’s garbage collector paused to clean up. During GC pauses: 502s.
How we measured it:
<pre><code class="language-bash"># Check allocation rate
go tool pprof http://localhost:8080/debug/pprof/allocs

# 372MB allocated per second at 5K RPS
</code></pre>
The fix: Object pooling.
<pre><code class="language-go">// Before: Allocate on every request
func marshalRequest(req *Request) []byte {
 buf := new(bytes.Buffer)
 json.NewEncoder(buf).Encode(req)
 return buf.Bytes()
}

// After: Reuse buffers via sync.Pool
var bufferPool = sync.Pool{
 New: func() interface{} {
 return new(bytes.Buffer)
 },
}

func marshalRequest(req *Request) []byte {
 buf := bufferPool.Get().(*bytes.Buffer)
 defer func() {
 buf.Reset()
 bufferPool.Put(buf)
 }()
 
 json.NewEncoder(buf).Encode(req)
 return buf.Bytes()
}
</code></pre>
We pooled:
<ul>
<li>JSON marshal buffers</li>
<li>HTTP response bodies</li>
<li>Channel objects</li>
<li>Message structures</li>
</ul>
Result:
<ul>
<li>Allocations dropped from 372MB/sec to 140MB/sec</li>
<li>Memory hit ratio: 85% (85% of requests reused pooled objects)</li>
<li>GC pauses reduced by 70%</li>
<li>502s during load: eliminated</li>
</ul>
<h2>Problem 5: Database Logging Kills Performance</h2>
Symptom: Latency increased from 11µs to 200µs. Under sustained load: timeouts and 502s.
What was happening: We were logging every request synchronously to a database for audit trails. Each write took ~150µs. That added directly to request latency.
The fix: Async batch logging.
<pre><code class="language-go">// Before: Synchronous writes
func logRequest(req *Request) {
 db.Insert(req) // Blocks request
}

// After: Async batching
type AsyncLogger struct {
 batch chan *LogEntry
 db *Database
}

func (al *AsyncLogger) Start() {
 go func() {
 buffer := make([]*LogEntry, 0, 1000)
 ticker := time.NewTicker(100 * time.Millisecond)
 
 for {
 select {
 case entry := &lt;-al.batch:
 buffer = append(buffer, entry)
 if len(buffer) &gt;= 1000 {
 al.db.BatchInsert(buffer)
 buffer = buffer[:0]
 }
 case &lt;-ticker.C:
 if len(buffer) &gt; 0 {
 al.db.BatchInsert(buffer)
 buffer = buffer[:0]
 }
 }
 }
 }()
}
</code></pre>
Result: Latency dropped from 200µs back to 11µs. Zero request blocking.
<h2>Final Configuration</h2>
After all optimizations, our production config:
<pre><code class="language-go">config := ProductionConfig{
 // Connection pooling
 MaxIdleConns: 100,
 MaxIdleConnsPerHost: 20,
 
 // Worker pools
 WorkerCount: 200,
 JobQueueSize: 30000,
 
 // Timeouts
 RequestTimeout: 30 * time.Second,
 IdleConnTimeout: 90 * time.Second,
 
 // Memory pools
 BufferPoolSize: 10000,
 ReuseThreshold: 0.85,
 
 // Logging
 AsyncLogging: true,
 BatchSize: 1000,
 FlushInterval: 100 * time.Millisecond,
}
</code></pre>
<h2>Results</h2>
Before optimizations (5K RPS):
<ul>
<li>502 rate: 30%</li>
<li>p99 latency: 8.2s</li>
<li>Memory: 2.1GB</li>
<li>Goroutines: 4,783 (leaking)</li>
</ul>
After optimizations (5K RPS):
<ul>
<li>502 rate: 0%</li>
<li>p99 latency: 1.68s</li>
<li>Memory: 120MB</li>
<li>Goroutines: ~200 (stable)</li>
</ul>
<h2>Key Lessons</h2>
<ol>
<li>
Connection pools need tuning. Don’t rely on defaults. Monitor reuse rates.
</li>
<li>
Context cancellation is not optional. Every goroutine needs a termination path.
</li>
<li>
Backpressure &gt; hard rejection. Brief queuing prevents cascading failures.
</li>
<li>
Object pooling matters at scale. GC pauses kill latency. Reuse allocations.
</li>
<li>
Never block request paths. Move slow operations (logging, analytics) async.
</li>
</ol>
<h2>Monitor These Metrics</h2>
<pre><code class="language-bash"># Goroutine count (should be stable)
curl localhost:8080/debug/pprof/goroutine?debug=1

# Allocation rate (watch for spikes)
go tool pprof localhost:8080/debug/pprof/allocs

# Connection pool stats
curl localhost:8080/metrics | grep conn_pool
</code></pre>
If you’re building high-throughput services, these patterns apply beyond gateways. Connection pooling, goroutine lifecycle, backpressure, and memory pooling are fundamental to Go performance at scale.
<hr>

Squad for devs building apps with GenAI and LLMs! Connect, share, exchange ideas, and get feedback. 

Why Your LLM Gateway Returns 502s Under Load (And How We Fixed It)