Best of Data Engineering — July 2025

1. 1
   Article

   Tinybird·38w

   Why LLMs struggle with analytics

   LLMs face significant challenges when working with analytical data, struggling with tabular data interpretation, SQL generation accuracy, and complex database schemas. The key to successful agentic analytics lies in providing comprehensive context through detailed documentation, semantic models, and sample data rather than expecting perfect SQL generation. Building query validation loops with error feedback, using LLM-as-a-judge evaluators, and focusing on business understanding over technical perfection enables more reliable analytical insights.

   90

   5
2. 2
   Article

   Data Engineer Things·40w

   The Ultimate Roadmap to Become a Data Engineer in 2025 (With Free Resources)

   A comprehensive guide for becoming a data engineer in 2025, covering essential skills like SQL, Python, and data modeling, along with big data technologies like Apache Spark and cloud platforms. The roadmap emphasizes free learning resources and practical experience, highlighting that mastering core principles enables quick adaptation to new tools in the rapidly evolving data engineering landscape.

   46

   2
3. 3
   Article

   MotherDuck·38w

   Summer Data Engineering Roadmap

   A comprehensive 3-week structured learning roadmap for aspiring data engineers covering foundational skills (SQL, Git, Linux), core engineering concepts (Python, cloud platforms, data modeling), and advanced topics (streaming, data quality, DevOps). The guide provides curated resources and a progressive learning path from beginner to intermediate level, emphasizing practical skills needed for full-stack data engineering roles.

   38
4. 4
   Article

   Data Engineer Things·40w

   How I Built a Reddit Data Pipeline

   A comprehensive guide to building an end-to-end data pipeline that extracts Reddit data, transforms it using AWS Glue, and stores it in S3 for querying with Athena and Redshift Spectrum. The tutorial covers environment setup with Docker and Airflow, infrastructure provisioning using Terraform, and implementing ETL workflows with proper orchestration. Key components include Reddit API integration, AWS services configuration (S3, Glue, Athena, Redshift), and DAG development for automated data processing.

   23
5. 5
   Article

   Javarevisited·40w

   Top 8 Udemy Courses to Learn Apache Airflow in 2025

   A curated list of 8 Udemy courses for learning Apache Airflow in 2025, ranging from beginner to advanced levels. The courses cover workflow orchestration, DAG creation, cloud deployment, and production-level implementations. Recommendations include Marc Lamberti's hands-on introduction for beginners and advanced courses covering AWS, Docker, and Kubernetes integration for experienced users.

   16
6. 6
   Article

   ByteByteGo·38w

   How Nubank Uses AI Models to Analyze Transaction Data for 100M Users

   Nubank processes transaction data from 100 million users using transformer-based foundation models instead of traditional manual feature engineering. Their system converts raw transactions into tokenized sequences, trains models using self-supervised learning on trillions of transactions, and combines sequential embeddings with tabular data through joint fusion architecture. The centralized AI platform allows teams across the company to access pretrained models for various financial tasks like credit scoring, fraud detection, and personalization.

   15
7. 7
   Article

   Towards Dev·38w

   Industry-Standard Architecture for Data Engineering Projects

   A comprehensive guide to building scalable data engineering architecture using Azure Data Factory and Databricks. The approach involves extracting CSV files from SharePoint, processing them through bronze and silver data layers, and implementing control tables for pipeline management. Key components include parameterized ADF pipelines, progress tracking metadata tables, and automated error handling to support multiple data interfaces efficiently.

   14