Learn how the AWAP protocol uses row and batch contracts to prevent schema scars and state corruption. A practical guide to resilient data ingestion using dlt.

dltHub

The AWAP (Audit-Write-Audit-Publish) protocol is presented as a middle ground between brittle strict schema enforcement and permissive auto-evolution that silently corrupts data. It introduces a two-gate validation architecture: a row-level gate (Audit 1) that catches syntactic violations like missing primary keys before they trigger schema mutations, and a batch-level gate (Audit 2) that detects semantic anomalies like abnormal NULL rates or suspicious aggregate patterns before promoting data to production. A practical example using dlt with a street survey dataset demonstrates how to filter invalid rows in-flight, flag suspicious records, compute per-agent suspicious rates in staging, and block untrusted agents from reaching production tables. This prevents both schema scars (permanent DDL mutations from malformed rows) and state corruption (silent overwriting of historical data via MERGE operations).

Row vs. Batch Contracts: Using AWAP to Prevent Schema Scars and State Corruption

The Two-Gate Architecture: Row vs. Batch Link icon

Practical Example: The Street Survey System Link icon

Seeing the bigger picture in staging Link icon

Audit 2: The Batch‑Level Gate Link icon

Conclusion: Engineering for Resilience Link icon