Building a CloudWatch metrics pipeline: parsing OpenTelemetry data

AppSignal engineers describe the technical challenges of building a parser to ingest AWS CloudWatch metrics delivered via Kinesis Firehose in OpenTelemetry protobuf format. Key problems solved include: handling base64-encoded, optionally gzip-compressed records with LEB128-prefixed protobuf messages; correctly separating aggregate vs. dimensional data points (which are not sums of each other); normalizing duration units to milliseconds and mapping CloudWatch percentage values to AppSignal's Percentage type; and transforming AWS namespace conventions (e.g., AWS/Lambda/Duration) to snake_case for wildcard dashboard matching. The normalized data powers automated AWS dashboards in AppSignal, currently in preview, with setup via AWS Console or CloudFormation.