Most components of a web application produce operational log files. Some logs are written by each application in a unique format. Other components generate out-of-the-box logs. Monitoring system logs is an essential activity for anyone charged with taking decisions. System administrators need to monitor logs to look out for unusual activity, to troubleshoot applications and websites that are under their control. By scanning logs, extracting and correlating data, system administrators can investigate and resolve problems, carry out capacity planning, help to detect vulnerabilities, ensure the smooth running of services and balancing capacity, and establish who has used services and when.
Information captured in log files is an important strategic resource to carry out analytics and searches. Making sense of logs helps organisations make better customer-focused decisions.
The purpose of this article is to identify the best open source software for collecting, parsing, storing, and making sense of logs. The applications featured in this article are ideal for organisations of any size.
Google Analytics is worthy of a mention in passing. It is the most widely used website statistics server, in part because it is an excellent web log analysis tool and generates first-rate graphs and well-defined reports. However, whilst the basic service is available free of charge, the source code is not available. Understandably, many people are adverse to allowing a large corporation like Google having direct access to their site metrics. The software featured in this article not only means you do not need to disclose your site metrics to third parties, but also offers the ability to analyze all types of logs, not those generated from web servers.
To provide an insight into the quality of software that is available, we have compiled a list of 8 high quality open source log analyzers that offer an excellent way to manage and interrogate logs. Software featured in this article provided finely grained, real-time monitoring, generating scalable realtime graphing.
Now, let’s explore the 8 log analyzers at hand. For each title we have compiled its own portal page, a full description with an in-depth analysis of its features, together with links to relevant resources and reviews.
| Log Analyzers | |
|---|---|
| Graylog2 | Log management solution implementation storing logs in ElasticSearch |
| logstash | Log processing, search, and analytics |
| Apache Flume | Delivers data from applications to Apache Hadoop's HDFS |
| OpenTSDB | Scalable, distributed Time Series Database |
| Graphite | Enterprise scalable realtime graphing |
| Kibana | Browser based interface for logstash and ElasticSearch |
| Scribe | Server for aggregating log data that is streamed in real time from clients |
| Chukwa | Hadoop sub-project devoted to large-scale log collection and analysis |
 Read our complete collection of recommended free and open source software. Our curated compilation covers all categories of software. The software collection forms part of our series of informative articles for Linux enthusiasts. There are hundreds of in-depth reviews, open source alternatives to proprietary software from large corporations like Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk. There are also fun things to try, hardware, free programming books and tutorials, and much more. |