Last Updated on July 11, 2021
High Performance Analytics Toolkit (HPAT) is an open source big data analytics and machine learning framework that offers Python’s ease of use combined with fast operation. It’s a compiler-based framework for big data. It accelerates data analytics and machine learning on clusters.
HPAT scales analytics/ML codes in Python to bare-metal cluster/cloud performance automatically. HPAT is built on top of Numba and LLVM compilers. code, and provides resiliency.
It compiles a subset of Python (Pandas/NumPy) to efficient parallel binaries with MPI, requiring only minimal code changes. It also provides scripting abstractions in the Julia language for analytics tasks, automatically parallelizes them, generates efficient MPI/C++.
HPAT is orders of magnitude faster than alternatives like Apache Spark. For example, HPAT is 14x to 400x faster than Spark on the Cori supercomputer at LBL/NERSC, and scales better to larger number of nodes.
HPAT depends on MPICH (a high performance and widely portable implementation of the Message Passing Interface (MPI) standard), SciPy, pandas, and numba.
Features include:
- Automatically parallelizes a subset of Python based on the MapReduce parallel pattern. MapReduce provides high-level parallelism abstractions suitable for data-parallel analytics programs, which can also be provided on top of scripting languages.
- Flexible in distributed data structures, which enables the use of existing libraries such as HDF5, ScaLAPACK,and Intel Data Analytics Acceleration Library.
- Offers resiliency using automatic checkpointing and facilitates optimization and fusion of array operations. It has restart capabilities, targeted at iterative machine learning applications like logistic regression and k-means.
- Novel design system.
- Domain-specific partitioning inference and parallelization.
- Parallel I/O code generation.
- Implemented as a Julia package using Julia’s high-level matrix and vector operations. It supports the high-level syntax of the Julia language.
- Good coverage of NumPy operators.
- Good coverage of Pandas operators.
- Supports I/O for the HDF5 and Parquet formats.
- Provides basic ASCII string support.
- Supports basic integer dictionaries.
Website: intellabs.github.io/hpat
Support: GitHub
Developer: Ehsan Totoni (Intel)
License: BSD 2-Clause “Simplified” License
HPAT is written in Python. Learn Python with our recommended free books and free tutorials.
Return to Essential Python Tools Home Page
| Explore Linux | |
|---|---|
 | Best Free and Open Source Software - A huge collection of recommended free and open source software covering every major category. |
 | In-depth Linux Software Reviews - Detailed assessments of Linux software covering features, usability and overall quality. |
 | The Big List of Active Linux Distros - An extensive collection of actively developed Linux distributions. |
 | Open Source Alternatives - Alternatives to software and services from Google, Microsoft, Apple, Adobe, IBM, Autodesk, Oracle and others. |
 | Essential Linux System Tools - Indispensable utilities for monitoring, maintaining and managing Linux systems. |
 | Linux Productivity Tools - Useful utilities for improving productivity and streamlining everyday tasks. |
 | Alternatives to Popular CLI Tools - Modern alternatives to familiar command-line utilities supplied with Linux systems. |
 | Awesome Free Linux Game Tools - Tools, utilities and companion software that improve gaming on Linux. |
 | Machine Learning on Linux - Machine learning and deep learning software, self-hosted applications and practical tools. |
 | Saving Money with Linux - Practical ways to reduce electricity consumption and lower computing costs with Linux. |
 | Linux Candy - Fun, entertaining and sometimes delightfully quirky software from the lighter side of Linux. |