If you want to understand the nuances of setting up Apache Spark and Airflow? Or if you want to practice data engineering pipeline from your local machine?
Here I am with a setup that is from scratch…

DevGenius serves as a platform for sharing insights and experiences in software development, offering articles and interviews with industry experts to gain  insights into various aspects of software engineering. By exploring DevGenius's curated content, developers can learn from real-world experiences, best practices, and lessons learned from seasoned professionals across different domains of software development. Whether you're looking for career advice, technical insights, or inspiration for your next project, DevGenius provides a  knowledge and expertise to help you succeed in your software development journey.

Dev Genius

A guide on setting up Apache Spark and Airflow on a local machine using Docker. The setup includes components like Airflow UI, Jupyter Notebook, Spark UI, Spark History Server, and Spark Shell. The post covers architecture details, directory structures, and configuration steps, including cloning the GitHub repository, generating SSH keys, building Docker images, and running the setup.

Apache Spark & Airflow in Docker: Step by Step guide