Big Data

Apache Storm – distributed stream processing

Apache Storm is a distributed processing system for analysing streams of data as events arrive. Unlike conventional batch-processing frameworks, which operate on datasets with defined boundaries, Storm is designed to run continuously. Applications are expressed as topologies that connect data sources with processing operations, allowing incoming information to pass through a sequence of transformations.

Storm can be used for real-time analytics, event processing, continuous calculations and data transformation pipelines. Its cluster architecture distributes processing tasks among worker processes and manages their execution when machines or processes fail. Applications can use different message-processing guarantees depending on their requirements, including stronger consistency through the Trident API. This distinction matters for workloads where recovering from a failure must not cause incorrect results.

This is free and open source software.

Key Features

  • Processes unbounded streams of data through continuously running topologies.
  • Uses spouts to introduce data into a topology and bolts to process it.
  • Supports configurable parallelism across worker processes and cluster nodes.
  • Provides stream grouping methods for controlling how tuples are distributed between tasks.
  • Uses a tuple-based data model for transferring information between processing components.
  • Supports message acknowledgements and replay after processing failures.
  • Offers best-effort and at-least-once message-processing guarantees.
  • Provides exactly-once processing semantics through the Trident API.
  • Trident supports stateful operations, aggregation and transactional processing.
  • Includes time-based and count-based windowing for stream calculations.
  • Supports persistent state and checkpointing for suitable processing components.
  • Integrates with Apache Kafka and other external messaging systems.
  • Supports distributed remote procedure calls.
  • Provides a cluster architecture using Nimbus and Supervisor services.
  • Supports highly available Nimbus configurations.
  • Includes a web interface for monitoring topologies, workers and processing statistics.
  • Provides APIs and command-line utilities for submitting and managing topologies.
  • Supports authentication, authorisation and secure communication for cluster deployments.

Website: storm.apache.org
Support:
Developer: Apache Software Foundation
License: Apache License 2.0

Storm is written in Java. Learn Java with our recommended free books and free tutorials.


Related Software

Data Analysis Tools
HadoopDistributed processing of large data sets across clusters of computers
StormDistributed and fault-tolerant realtime computation
DrillDistributed system for interactive analysis of large-scale datasets
FlinkFramework and distributed processing engine
SparkUnified analytics engine for large-scale data processing
PentahoEnterprise reporting, analysis, dashboard, data mining, workflow and more
HPCC SystemsDesigned for the enterprise to resolve Big Data challenges
DaftHigh-performance distributed data processing engine
Apache BeamUnified programming model for batch and streaming data processing

Read our verdict in the software roundup.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
0 Comments
Oldest
Newest Most Voted