A modern tech stack typically involves at least a frontend and backend but relatively quickly also grows to include a data platform. This typically grows out of the need for ad-hoc analysis and reporting but possibly evolves into a whole oil refinery of cronjobs, dashboards, bulk data copying, and much more.

Erik Bernhardsson

A comparison of data platform architecture versus backend architecture, exploring when and why to offload work from the backend into a separate data platform. Key differences include latency tolerance, query patterns (OLTP vs OLAP), integration database rules, and testing challenges. The author argues that pushing non-latency-critical work (recommendations, reporting, ML model training, search indexing) into batch-oriented data pipelines reduces backend complexity by roughly an order of magnitude. Practical guidance covers why integration databases are acceptable in the data layer but harmful in backend systems, why complex mega-queries are fine in data pipelines, and why achieving meaningful test coverage for data pipelines is genuinely difficult.

Data architecture vs backend architecture