Apache Flume – log data aggregation and more

Apache Flume is an open source, scalable, distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

The system is centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic applications.

The main goal of Apache Flume is to deliver data from applications to Apache Hadoop’s HDFS. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic applications.

Features include:

  • Complex flows:
    • build multi-hop flows where events travel through multiple agents before reaching the final destination.
    • fan-in and fan-out flows.
    • contextual routing.
    • backup routes (fail-over) for failed hops.
  • Channel-based transactions to guarantee reliable message delivery.
  • Supports a durable file channel which is backed by the local file system. Events are staged in the channel, which manages recovery from failure.
  • High performance persistent channel – the File Channel.
  • ElasticSearch Sink.
  • Create a SpoolDirectory Source and Client.
  • Regex Extractor Interceptor.
  • Load Balancing RPC client.
  • Hive Sink based on the new Hive Streaming support.
  • End to End authentication in Flume.
  • Simple regex search-and-replace interceptor.

Website: flume.apache.org
Support: User Guide
Developer: The Apache Software Foundation
License: Apache License 2.0

Apache Flume is written in Java. Learn Java with our recommended free books and free tutorials.

Return to Log Analyzers Home Page

Read our complete collection of recommended free and open source software. The collection covers all categories of software.

The software collection forms part of our series of informative articles for Linux enthusiasts. There's tons of in-depth reviews, alternatives to Google, fun things to try, hardware, free programming books and tutorials, and much more.
Share this article

Share your Thoughts

This site uses Akismet to reduce spam. Learn how your comment data is processed.