Luigi is a Python library developed at Spotify for building complex pipelines of batch jobs with dependency resolution and workflow visualization. It powers thousands of daily tasks at Spotify, including recommendations, A/B test analysis, and dashboards. Conceptually similar to GNU Make, it also has built-in Hadoop support for large-scale data processing.
Sort: