Log Analyzers

Scribe – distributed log aggregation server

Scribe is a distributed log aggregation system originally developed at Facebook for collecting event data from large numbers of servers. It was designed for environments where applications generate continuous streams of log messages that need to be consolidated for storage and analysis.

The system uses a hierarchy of collection servers. Individual machines can send events to local Scribe servers, which forward the information to central servers or other aggregation layers.

This architecture allows the collection infrastructure to handle temporary network failures and unavailable destinations. Scribe can buffer messages locally and forward them when connectivity is restored, reducing the risk of losing information during interruptions.

This is free and open source software.

Key Features

  • Collect log events streamed from multiple clients.
  • Aggregate events from large numbers of distributed machines.
  • Forward messages through configurable hierarchies of collection servers.
  • Use local buffering when destination servers are temporarily unavailable.
  • Forward buffered messages after a failed destination becomes available again.
  • Write collected events to local or network-mounted filesystems.
  • Configure network stores for forwarding events to other Scribe servers.
  • Use buffer stores with primary and secondary destinations.
  • Distribute messages among configured stores using bucket-based routing.
  • Forward individual events to multiple destinations through multi stores.
  • Discard selected events through null stores.
  • Write event records using Thrift-based file storage.
  • Support applications communicating through Apache Thrift interfaces.
  • Configure separate storage behaviour for different categories of log messages.

Website: github.com/facebookarchive/scribe
Support:
Developer: Facebook
License: Apache License 2.0

Scribe is written in C++. Learn C++ with our recommended free books and free tutorials.


Related Software

Log Analyzers
KibanaBrowser based interface for logstash and ElasticSearch
logstashLog processing, search, and analytics
OpenObserveCloud-native observability platform
GoAccessReal-time web log analyzer and interactive viewer
FluentdData collector for unified logging layer
LokiHorizontally-scalable, highly-available, multi-tenant log aggregation system
GraphiteEnterprise scalable realtime graphing
SigNozMonitor your applications and troubleshoot problems
Apache FlumeDelivers data from applications to Apache Hadoop's HDFS
OpenTSDBScalable, distributed Time Series Database
VictoriaLogsHigh-performance log database designed to ingest, store, and query log data
syslog-ngCollects, filters, and routes logs across systems
CLPCompresses logs for efficient search and analysis
Matomo SLAImports web server logs into Matomo for analysis

Read our verdict in the software roundup.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
1 Comment
Oldest
Newest Most Voted
Jack
Jack
4 years ago

Scribe was developed at Facebook and released as open source but is currently unmaintained.