Inside the Pipe: What the Architecture Diagram Doesn’t Tell You

A practitioner shares hard-won lessons from building a governed cloud data pipeline that moves reference data from on-premises MongoDB through Kafka into a three-layer architecture (Landing, Bronze, Silver) with Athena as the query surface. Key insights cover why three explicit layers beat one, how Kafka Connect's Dead Letter Queue and Schema Registry enforce trustworthiness, the nuances of CDC including handling absent events, why audit columns (VALID_FROM, VALID_TO, DELETE_FLAG, JOB_RUN_ID) are operationally essential rather than compliance decoration, and what distinguishes a reusable data platform from a one-off pipeline.