Apache Flume is a distributed data collection system for gathering, aggregating and transferring large volumes of log events. It is designed for applications that need to move data reliably between services and storage systems, particularly in environments involving Hadoop.
The software uses an architecture based on sources, channels and sinks. Sources receive events, channels provide intermediate storage, and sinks transfer the events to their destinations.
This arrangement allows complex collection pipelines to be assembled from reusable components. Flume supports buffering, transactional processing and configurable routing, helping applications handle interruptions while transferring data between distributed systems.
This is free and open source software.
Key Features
- Collect event data from applications, files and supported network sources.
- Build data pipelines using sources, channels and sinks.
- Transfer events between multiple Flume agents.
- Use memory channels for temporary event buffering.
- Use durable file channels to retain events across supported failures and restarts.
- Provide transactional handling between sources, channels and sinks.
- Configure fan-in and fan-out data flows.
- Support failover and load balancing between destinations.
- Route events according to configured rules and event metadata.
- Apply interceptors to inspect or transform events during collection.
- Extract fields using regular expressions.
- Transfer collected data to Hadoop HDFS and other supported destinations.
- Support source and sink extensions for additional event-processing requirements.
- Configure secure communication and authentication for supported components.
Website: flume.apache.org
Support:
Developer: The Apache Software Foundation
License: Apache License 2.0
Apache Flume is written in Java. Learn Java with our recommended free books and free tutorials.
Related Software
| Log Analyzers | |
|---|---|
| Kibana | Browser based interface for logstash and ElasticSearch |
| logstash | Log processing, search, and analytics |
| OpenObserve | Cloud-native observability platform |
| GoAccess | Real-time web log analyzer and interactive viewer |
| Fluentd | Data collector for unified logging layer |
| Loki | Horizontally-scalable, highly-available, multi-tenant log aggregation system |
| Graphite | Enterprise scalable realtime graphing |
| SigNoz | Monitor your applications and troubleshoot problems |
| Apache Flume | Delivers data from applications to Apache Hadoop's HDFS |
| OpenTSDB | Scalable, distributed Time Series Database |
| VictoriaLogs | High-performance log database designed to ingest, store, and query log data |
| syslog-ng | Collects, filters, and routes logs across systems |
| CLP | Compresses logs for efficient search and analysis |
| Matomo SLA | Imports web server logs into Matomo for analysis |
Read our verdict in the software roundup.
LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |


Please read our Comment Policy before commenting.