Log Analyzers

Apache Flume – distributed log data collection system

Apache Flume is a distributed data collection system for gathering, aggregating and transferring large volumes of log events. It is designed for applications that need to move data reliably between services and storage systems, particularly in environments involving Hadoop.

The software uses an architecture based on sources, channels and sinks. Sources receive events, channels provide intermediate storage, and sinks transfer the events to their destinations.

This arrangement allows complex collection pipelines to be assembled from reusable components. Flume supports buffering, transactional processing and configurable routing, helping applications handle interruptions while transferring data between distributed systems.

This is free and open source software.

Key Features

  • Collect event data from applications, files and supported network sources.
  • Build data pipelines using sources, channels and sinks.
  • Transfer events between multiple Flume agents.
  • Use memory channels for temporary event buffering.
  • Use durable file channels to retain events across supported failures and restarts.
  • Provide transactional handling between sources, channels and sinks.
  • Configure fan-in and fan-out data flows.
  • Support failover and load balancing between destinations.
  • Route events according to configured rules and event metadata.
  • Apply interceptors to inspect or transform events during collection.
  • Extract fields using regular expressions.
  • Transfer collected data to Hadoop HDFS and other supported destinations.
  • Support source and sink extensions for additional event-processing requirements.
  • Configure secure communication and authentication for supported components.

Website: flume.apache.org
Support:
Developer: The Apache Software Foundation
License: Apache License 2.0

Apache Flume is written in Java. Learn Java with our recommended free books and free tutorials.


Related Software

Log Analyzers
KibanaBrowser based interface for logstash and ElasticSearch
logstashLog processing, search, and analytics
OpenObserveCloud-native observability platform
GoAccessReal-time web log analyzer and interactive viewer
FluentdData collector for unified logging layer
LokiHorizontally-scalable, highly-available, multi-tenant log aggregation system
GraphiteEnterprise scalable realtime graphing
SigNozMonitor your applications and troubleshoot problems
Apache FlumeDelivers data from applications to Apache Hadoop's HDFS
OpenTSDBScalable, distributed Time Series Database
VictoriaLogsHigh-performance log database designed to ingest, store, and query log data
syslog-ngCollects, filters, and routes logs across systems
CLPCompresses logs for efficient search and analysis
Matomo SLAImports web server logs into Matomo for analysis

Read our verdict in the software roundup.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
0 Comments
Oldest
Newest Most Voted