Scribe
Scribe is an open source server for aggregating log data that
is streamed in real time from clients. It is designed to scale to a
very large
number of nodes and be robust to network and node failures.
There is a scribe server running on every node in the system,
configured to aggregate messages and send them to a central scribe
server (or servers) in larger groups. If the central scribe server
isnt available the local scribe server writes the messages to a file
on local disk and sends them when the central server recovers. The
central scribe server(s) can write the messages to the files that are
their final destination, typically on an nfs filer or a distributed
filesystem, or send them to another layer of scribe servers.
Scribe is implemented as a thrift service using the
non-blocking C++ server.
Scribe was developed at Facebook and released as open source.
Features include:
- Designed to be robust to failure of the network or any
specific machine
- Types of store available:
- file writes to a file,
either local or nfs
- network sends messages to
another scribe serve
- buffer contains a primary
and a secondary store. Messages are sent to the primary store if
possible, and otherwise the secondary. When the primary store becomes
available the messages are read from the secondary store and sent to
the primary. Ordering of the messages is preserved. The secondary store
has the restriction that it must be readable, which at the moment means
it has to be a file store
- bucket contains a large
number of other stores, and decides which messages to send to which
stores based on a hash
- null discards all messages
- thriftfile similar to a
file store but writes messages into a Thrift TFileTransport file
- multi a store that
forwards messages to multiple stores
Return
to Log Analyzers Home Page
Last Updated Sunday, December 23 2012 @ 02:04 AM EST
| 
