Apache Flume is an open source, scalable, distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
The system is centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic applications.
The main goal of Apache Flume is to deliver data from applications to Apache Hadoop’s HDFS. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic applications.
Key Features
- Complex flows:
- build multi-hop flows where events travel through multiple agents before reaching the final destination.
- fan-in and fan-out flows.
- contextual routing.
- backup routes (fail-over) for failed hops.
- Channel-based transactions to guarantee reliable message delivery.
- Supports a durable file channel which is backed by the local file system. Events are staged in the channel, which manages recovery from failure.
- High performance persistent channel – the File Channel.
- ElasticSearch Sink.
- Create a SpoolDirectory Source and Client.
- Regex Extractor Interceptor.
- Load Balancing RPC client.
- Hive Sink based on the new Hive Streaming support.
- End to End authentication in Flume.
- Simple regex search-and-replace interceptor.
Website: flume.apache.org
Support: User Guide
Developer: The Apache Software Foundation
License: Apache License 2.0
Apache Flume is written in Java. Learn Java with our recommended free books and free tutorials.
Related Software
| Log Analyzers | |
|---|---|
| Kibana | Browser based interface for logstash and ElasticSearch |
| logstash | Log processing, search, and analytics |
| OpenObserve | Cloud-native observability platform |
| GoAccess | Real-time web log analyzer and interactive viewer |
| Fluentd | Data collector for unified logging layer |
| Loki | Horizontally-scalable, highly-available, multi-tenant log aggregation system |
| Graylog2 | Log management solution implementation storing logs in ElasticSearch |
| Graphite | Enterprise scalable realtime graphing |
| SigNoz | Monitor your applications and troubleshoot problems |
| Apache Flume | Delivers data from applications to Apache Hadoop's HDFS |
| OpenTSDB | Scalable, distributed Time Series Database |
| VictoriaLogs | High-performance log database designed to ingest, store, and query log data |
| Scribe | Server for aggregating log data that is streamed in real time from clients |
| LogoRRR | Cross-platform log analysis tool |
| Chukwa | Hadoop sub-project devoted to large-scale log collection and analysis |
Read our verdict in the software roundup.
 Explore our carefully curated directory of recommended free and open source software, covering every major software category.The directory forms part of our extensive collection of articles for Linux enthusiasts. It includes hundreds of detailed reviews, together with free and open source alternatives to proprietary software from companies such as Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk. LinuxLinks also covers interesting projects worth exploring, Linux-compatible hardware, free programming books and tutorials, and much more. Know a useful free and open source Linux application that we haven’t covered? Tell us about it using our submission form. |
Please read our Comment Policy before commenting.