Most components of a web application produce operational log files. Some logs are written by each application in a unique format. Other components generate out-of-the-box logs. Monitoring system logs is an essential activity for anyone charged with taking decisions. System administrators need to monitor logs to look out for unusual activity, to troubleshoot applications and websites that are under their control. By scanning logs, extracting and correlating data, system administrators can investigate and resolve problems, carry out capacity planning, help to detect vulnerabilities, ensure the smooth running of services and balancing capacity, and establish who has used services and when.
Information captured in log files is an important strategic resource to carry out analytics and searches. Making sense of logs helps organisations make better customer-focused decisions.
The purpose of this article is to identify the best open source software for collecting, parsing, storing, and making sense of logs. The applications featured in this article are ideal for organisations of any size.
To provide an insight into the quality of software that is available, we have compiled a list of 11 high quality open source log analyzers that offer an excellent way to manage and interrogate logs. Software featured in this article provided finely grained, real-time monitoring, generating scalable realtime graphing.
Here’s our verdict captured in a legendary LinuxLinks-style ratings chart. Only free and open source software is eligible for inclusion.
Click the links in the table below to learn more about each log analyzer.
| Log Analyzers | |
|---|---|
| Kibana | Browser based interface for logstash and ElasticSearch |
| logstash | Log processing, search, and analytics |
| Fluentd | Data collector for unified logging layer |
| Loki | Horizontally-scalable, highly-available, multi-tenant log aggregation system |
| Graylog2 | Log management solution implementation storing logs in ElasticSearch |
| Graphite | Enterprise scalable realtime graphing |
| SigNoz | Monitor your applications and troubleshoot problems |
| Apache Flume | Delivers data from applications to Apache Hadoop's HDFS |
| OpenTSDB | Scalable, distributed Time Series Database |
| Scribe | Server for aggregating log data that is streamed in real time from clients |
| Chukwa | Hadoop sub-project devoted to large-scale log collection and analysis |
This article has been revamped in line with our recent announcement.
 Read our complete collection of recommended free and open source software. Our curated compilation covers all categories of software. The software collection forms part of our series of informative articles for Linux enthusiasts. There are hundreds of in-depth reviews, open source alternatives to proprietary software from large corporations like Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk. There are also fun things to try, hardware, free programming books and tutorials, and much more. |