Apache Beam is a unified programming model for defining and executing batch and streaming data processing pipelines.
It provides SDKs for Java, Python, and Go, with pipelines able to run on distributed processing backends such as Apache Flink, Apache Spark, Google Cloud Dataflow, and Hazelcast Jet.
This is free and open source software.
Key Features
- Build both batch and streaming data processing pipelines using a common programming model.
- Offers SDKs for Java, Python, and Go.
- Separates pipeline construction from the underlying execution engine.
- Runs pipelines locally with DirectRunner and PrismRunner.
- Supports distributed execution with runners for Apache Flink, Apache Spark, Google Cloud Dataflow, Hazelcast Jet, and other backends.
- Represents data using bounded or unbounded PCollections.
- Uses PTransforms to express computations that transform input collections into output collections.
- Supports portable pipelines that can move between compatible execution environments.
- Designed for highly parallel data processing workloads.
Website: https://github.com/apache/beam
Support:
Developer: Apache Software Foundation
License: Apache-2.0
Related Software
| Data Analysis Tools | |
|---|---|
| Hadoop | Distributed processing of large data sets across clusters of computers |
| Storm | Distributed and fault-tolerant realtime computation |
| Drill | Distributed system for interactive analysis of large-scale datasets |
| Flink | Framework and distributed processing engine |
| Spark | Unified analytics engine for large-scale data processing |
| Pentaho | Enterprise reporting, analysis, dashboard, data mining, workflow and more |
| HPCC Systems | Designed for the enterprise to resolve Big Data challenges |
| Rapid Miner | Knowledge discovery in databases, machine learning, and data mining |
Read our verdict in the software roundup.
 Explore our comprehensive directory of recommended free and open source software. Our carefully curated collection spans every major software category.This directory is part of our ongoing series of informative articles for Linux enthusiasts. It features hundreds of detailed reviews, along with open source alternatives to proprietary solutions from major corporations such as Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk. You’ll also find interesting projects to try, hardware coverage, free programming books and tutorials, and much more. Discovered a useful open source Linux program that we haven’t covered yet? Let us know by completing this form. |
Please read our Comment Policy before commenting.