Move telemetry flushing off the Lambda response path using the Extensions API and goroutine chaining to eliminate flush-driven gateway timeouts while preserving full observability.

InfoQ is a leading online platform for software developers, architects, and technical leaders, providing news, articles, presentations, and interviews on a wide range of topics, including agile practices, DevOps, microservices, and emerging technologies. With a focus on quality content and expert insights, InfoQ helps professionals stay informed about the latest trends, best practices, and industry developments. Developers can learn from real-world experiences, gain  knowledge, and connect with peers in the global software community through InfoQ's diverse and engaging content.

InfoQ

At Lead Bank, synchronous OpenTelemetry telemetry flushing in AWS Lambda caused intermittent exporter stalls to become user-facing 504 gateway timeouts. The solution uses the AWS Lambda Extensions API to defer telemetry flushing off the response path. By registering an internal extension and using goroutine chaining in Go, the handler returns immediately while the extension withholds its NextEvent() call until the flush completes, preventing Lambda from freezing mid-flush. The naive approach of calling NextEvent() too early caused race conditions under warm reuse. The revised design uses single-shot goroutines that spawn the next only after flush completes. Results: p99 latency dropped from occasional 10s timeouts to 200ms. The pattern also applies to other post-response work like async job triggering, with caveats around slow handlers, short function timeouts, and background workloads where synchronous flushing remains simpler.

Using AWS Lambda Extensions to Run Post-Response Telemetry Flush

Initial Approach and Failure Under Warm Reuse