Apache Beam is a unified programming model for defining and executing batch and streaming data processing pipelines.
It provides SDKs for Java, Python, and Go, with pipelines able to run on distributed processing backends such as Apache Flink, Apache Spark, Google Cloud Dataflow, and Hazelcast Jet.
This is free and open source software.
Key Features
- Build both batch and streaming data processing pipelines using a common programming model.
- Offers SDKs for Java, Python, and Go.
- Separates pipeline construction from the underlying execution engine.
- Runs pipelines locally with DirectRunner and PrismRunner.
- Supports distributed execution with runners for Apache Flink, Apache Spark, Google Cloud Dataflow, Hazelcast Jet, and other backends.
- Represents data using bounded or unbounded PCollections.
- Uses PTransforms to express computations that transform input collections into output collections.
- Supports portable pipelines that can move between compatible execution environments.
- Designed for highly parallel data processing workloads.
Website: https://github.com/apache/beam
Support:
Developer: Apache Software Foundation
License: Apache License 2.0
Apache Beam is written in Java. Learn Java with our recommended free books and free tutorials.
Related Software
| Data Analysis Tools | |
|---|---|
| Hadoop | Distributed processing of large data sets across clusters of computers |
| Storm | Distributed and fault-tolerant realtime computation |
| Drill | Distributed system for interactive analysis of large-scale datasets |
| Flink | Framework and distributed processing engine |
| Spark | Unified analytics engine for large-scale data processing |
| Pentaho | Enterprise reporting, analysis, dashboard, data mining, workflow and more |
| HPCC Systems | Designed for the enterprise to resolve Big Data challenges |
| Daft | High-performance distributed data processing engine |
| Apache Beam | Unified programming model for batch and streaming data processing |
Read our verdict in the software roundup.
 Explore our carefully curated directory of recommended free and open source software, covering every major software category.The directory forms part of our extensive collection of articles for Linux enthusiasts. It includes hundreds of detailed reviews, together with free and open source alternatives to proprietary software from companies such as Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk. LinuxLinks also covers interesting projects worth exploring, Linux-compatible hardware, free programming books and tutorials, and much more. Know a useful free and open source Linux application that we haven’t covered? Tell us about it using our submission form. |
Please read our Comment Policy before commenting.