Community Picks·2yThe State of Data Engineering 2024
The 2024 State of Data Engineering report discusses the influence of GenAI on software infrastructure, the expansion of product offerings due to the economic downturn, and the impact of open table formats and their catalogs in the data lake industry. It also highlights the importance of data version control and observability in AI/ML systems.
Python in Plain English·2yCreating an ETL Data Pipeline Using Bash with Apache Airflow
Learn how to create an ETL data pipeline using bash with Apache Airflow. Extract data from various file formats, transform it, and load it into a new file. Includes steps for starting Apache Airflow, downloading the dataset, creating a DAG, and executing the pipeline.
Medium·2yAll you need to know about the Google File System
The Google File System (GFS) is a distributed large-scale file system designed by Google. It is built to handle component failures, store large files, and optimize performance for file appending. The system features a single master, multiple chunkservers, and multiple clients. GFS uses chunk replication, lease management, and checksumming to ensure high availability and data integrity.
Substack·2yWhy it's hard for data engineers to get promoted after senior engineer
Barriers and obstacles for data engineers to get promoted after senior engineer, lack of visibility compared to other data roles, being viewed as 'less technical' at some companies, smaller teams for data engineering compared to software engineering, and the top out of the data engineering individual contributor track at data architect.
