A comprehensive guide to implementing machine learning lineage tracking using DVC (Data Version Control) for an ML system deployed on AWS Lambda. Covers the complete pipeline from ETL and data drift detection through preprocessing, model tuning, and fairness evaluation. Demonstrates integration with tools like Evidently AI for drift monitoring, Prefect for workflow orchestration, and AWS S3 for remote storage. Includes practical code examples for tracking data, experiments, models, and predictions while ensuring reproducibility and compliance.
Table of contents
What is Machine Learning Lineage?What We’ll BuildWorkflow in ActionStep 1: Initiating a DVC ProjectStep 2: The ML LineageStep 3: Deploying the DVC ProjectStep 4: Configuring Scheduled Run with PrefectStep 5: Deploying the ApplicationConclusionSort: