Engineers at Gearset shared how they tackled asynchronous observability gaps using OpenTelemetry distributed tracing at QCon London 2026. Key insights include: wrapping queue clients to propagate trace context across async boundaries, shifting from queue-size metrics to latency-based SLOs for more meaningful alerting, using OpenTelemetry trace state to embed root span timestamps for accurate async duration tracking, and adopting wide events with rich metadata to enable discovery-based debugging. The OpenTelemetry Collector was used to enrich traces with Kubernetes metadata and scrub sensitive data. Cultural adoption proved as challenging as the technical work, with the team recommending proving value through real incident resolution rather than top-down mandates.
Sort: