A comparison of data platform architecture versus backend architecture, exploring when and why to offload work from the backend into a separate data platform. Key differences include latency tolerance, query patterns (OLTP vs OLAP), integration database rules, and testing challenges. The author argues that pushing non-latency-critical work (recommendations, reporting, ML model training, search indexing) into batch-oriented data pipelines reduces backend complexity by roughly an order of magnitude. Practical guidance covers why integration databases are acceptable in the data layer but harmful in backend systems, why complex mega-queries are fine in data pipelines, and why achieving meaningful test coverage for data pipelines is genuinely difficult.
Sort: