Apache Beam is a unified programming model for defining and executing batch and streaming data processing pipelines.
It provides SDKs for Java, Python, and Go, with pipelines able to run on distributed processing backends such as Apache Flink, Apache Spark, Google Cloud Dataflow, and Hazelcast Jet.
This is free and open source software.
Key Features
- Build both batch and streaming data processing pipelines using a common programming model.
- Offers SDKs for Java, Python, and Go.
- Separates pipeline construction from the underlying execution engine.
- Runs pipelines locally with DirectRunner and PrismRunner.
- Supports distributed execution with runners for Apache Flink, Apache Spark, Google Cloud Dataflow, Hazelcast Jet, and other backends.
- Represents data using bounded or unbounded PCollections.
- Uses PTransforms to express computations that transform input collections into output collections.
- Supports portable pipelines that can move between compatible execution environments.
- Designed for highly parallel data processing workloads.
Website: https://github.com/apache/beam
Support:
Developer: Apache Software Foundation
License: Apache License 2.0
Apache Beam is written in Java. Learn Java with our recommended free books and free tutorials.
Related Software
| Data Analysis Tools | |
|---|---|
| Hadoop | Distributed processing of large data sets across clusters of computers |
| Storm | Distributed and fault-tolerant realtime computation |
| Drill | Distributed system for interactive analysis of large-scale datasets |
| Flink | Framework and distributed processing engine |
| Spark | Unified analytics engine for large-scale data processing |
| Pentaho | Enterprise reporting, analysis, dashboard, data mining, workflow and more |
| HPCC Systems | Designed for the enterprise to resolve Big Data challenges |
| Daft | High-performance distributed data processing engine |
| Apache Beam | Unified programming model for batch and streaming data processing |
Read our verdict in the software roundup.
LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |


Please read our Comment Policy before commenting.