Is your Kafka lag dashboard lying? Learn how log compaction and retention can hide massive consumer delays and how to detect inaccurate time lag metrics.

SoftwareMill is a software consultancy specializing in custom software development and distributed systems. Readers can learn about software architecture, cloud computing, and functional programming. With case studies, blog posts, and technical analysis, SoftwareMill provides  insights and expertise for building scalable and resilient software solutions.

SoftwareMill

Kafka time lag metrics can silently underreport consumer delays due to two features: log compaction and retention deletion. When a consumer's committed offset points to a deleted message, Kafka returns the next available message with a newer timestamp, making lag appear smaller than it really is. In extreme cases, reported lag of 30 seconds can mask actual lag of 30 minutes. The klag-exporter tool detects both conditions by comparing requested vs. returned offsets and checking if committed offsets fall below the low watermark, exposing detection flags on metrics. Mitigations include increasing min.compaction.lag.ms, scaling slow consumers, and treating flagged time lag values as lower bounds rather than exact measurements.

Compaction & Retention: Edge Cases That Make Your Kafka Lag Metrics Inaccurate

Retention Deletion: When Your Offsets Fall Off the Cliff

Detection: How klag-exporter Catches These Lies