Scribe is an open source server for aggregating log data that is streamed in real time from clients. It is designed to scale to a very large number of nodes and be robust to network and node failures.
There is a scribe server running on every node in the system, configured to aggregate messages and send them to a central scribe server (or servers) in larger groups. If the central scribe server isn’t available the local scribe server writes the messages to a file on local disk and sends them when the central server recovers. The central scribe server(s) can write the messages to the files that are their final destination, typically on an nfs filer or a distributed filesystem, or send them to another layer of scribe servers.
Scribe is implemented as a thrift service using the non-blocking C++ server.
Scribe was developed at Facebook and released as open source.
Key Features
- Designed to be robust to failure of the network or any specific machine.
- Types of store available:
- file – writes to a file, either local or nfs.
- network – sends messages to another scribe serve.
- buffer – contains a primary and a secondary store. Messages are sent to the primary store if possible, and otherwise the secondary. When the primary store becomes available the messages are read from the secondary store and sent to the primary. Ordering of the messages is preserved. The secondary store has the restriction that it must be readable, which at the moment means it has to be a file store.
- bucket – contains a large number of other stores, and decides which messages to send to which stores based on a hash.
- null – discards all messages.
- thriftfile – similar to a file store but writes messages into a Thrift TFileTransport file.
- multi – a store that forwards messages to multiple stores.
Website: github.com/facebookarchive/scribe
Support:
Developer: Facebook
License: Apache License 2.0
Scribe is written in C++. Learn C++ with our recommended free books and free tutorials.
Related Software
| Log Analyzers | |
|---|---|
| Kibana | Browser based interface for logstash and ElasticSearch |
| logstash | Log processing, search, and analytics |
| OpenObserve | Cloud-native observability platform |
| GoAccess | Real-time web log analyzer and interactive viewer |
| Fluentd | Data collector for unified logging layer |
| Loki | Horizontally-scalable, highly-available, multi-tenant log aggregation system |
| Graylog2 | Log management solution implementation storing logs in ElasticSearch |
| Graphite | Enterprise scalable realtime graphing |
| SigNoz | Monitor your applications and troubleshoot problems |
| Apache Flume | Delivers data from applications to Apache Hadoop's HDFS |
| OpenTSDB | Scalable, distributed Time Series Database |
| VictoriaLogs | High-performance log database designed to ingest, store, and query log data |
| Scribe | Server for aggregating log data that is streamed in real time from clients |
| LogoRRR | Cross-platform log analysis tool |
| Chukwa | Hadoop sub-project devoted to large-scale log collection and analysis |
Read our verdict in the software roundup.
 Explore our comprehensive directory of recommended free and open source software. Our carefully curated collection spans every major software category.This directory is part of our ongoing series of informative articles for Linux enthusiasts. It features hundreds of detailed reviews, along with open source alternatives to proprietary solutions from major corporations such as Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk. You’ll also find interesting projects to try, hardware coverage, free programming books and tutorials, and much more. Discovered a useful open source Linux program that we haven’t covered yet? Let us know by completing this form. |
Scribe was developed at Facebook and released as open source but is currently unmaintained.