Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.
You can build pretty much any task you want, but Luigi also comes with a toolbox of several common task templates that you use. It includes support for running Python mapreduce jobs in Hadoop, as well as Hive, and Pig, jobs. It also comes with file system abstractions for HDFS, and local files that ensures all file system operations are atomic. This is important because it means your data pipeline will not crash in a state containing partial data.
This is free and open source software.
Website: github.com/spotify/luigi
Support:
Developer: Spotify
License: Apache License 2.0
Luigi is written in Python. Learn Python with our recommended free books and free tutorials.
| Explore Linux | |
|---|---|
 | Best Free and Open Source Software - A huge collection of recommended free and open source software covering every major category. |
 | In-depth Linux Software Reviews - Detailed assessments of Linux software covering features, usability and overall quality. |
 | The Big List of Active Linux Distros - An extensive collection of actively developed Linux distributions. |
 | Open Source Alternatives - Alternatives to software and services from Google, Microsoft, Apple, Adobe, IBM, Autodesk, Oracle and others. |
 | Essential Linux System Tools - Indispensable utilities for monitoring, maintaining and managing Linux systems. |
 | Linux Productivity Tools - Useful utilities for improving productivity and streamlining everyday tasks. |
 | Alternatives to Popular CLI Tools - Modern alternatives to familiar command-line utilities supplied with Linux systems. |
 | Awesome Free Linux Game Tools - Tools, utilities and companion software that improve gaming on Linux. |
 | Machine Learning on Linux - Machine learning and deep learning software, self-hosted applications and practical tools. |
 | Saving Money with Linux - Practical ways to reduce electricity consumption and lower computing costs with Linux. |
 | Linux Candy - Fun, entertaining and sometimes delightfully quirky software from the lighter side of Linux. |
Please read our Comment Policy before commenting.