Speed is all the rage in data engineering nowadays. Stakeholders want their data sooner with higher quality and efficiency! All this demand for speed can cause data engineers to make stupid decisions by not asking the proper follow up questions to decipher the true requirements from the false ones!

Substack is a platform for independent writers and journalists to publish and monetize their content. Through newsletters, articles, and podcasts, Substack offers insights into a wide range of topics such as politics, technology, culture, and personal development. Readers can subscribe to their favorite writers and receive regular updates, analysis, and commentary on the issues that matter to them. Additionally, Substack provides tools and resources for writers to grow their audience, engage with their readers, and monetize their content effectively.

Substack

This post discusses the tradeoffs between batch, micro-batch, and streaming when building a data pipeline. It explores the downsides of streaming pipelines, the advantages of using a batch pipeline, and suggests technologies for batch processing.

How to choose between batch, micro-batch, and streaming when building a data pipeline

The latency / complexity / quality tradeoff

Have clarity when talking with stakeholders

Picking the right technology depending on your use case