Cybersecurity Log Collection with Graylog and Wazuh: Building a Robust & Open-Source Architecture

INTRODUCTION

In this age of constant cyber threats, the meticulous management of system & application logs have emerged as an imperative.

Not only do these logs allow us to identify and respond to current security incidents, but they also provide a vital window into the past, enabling forensic investigations that can span back several months

This paper presents an architecture for collecting and retaining logs, outlines the challenges in log processing, and provides comprehensive & open-source solutions, such as Graylog & Wazuh (both running on opensearch database engine), to address these issues.

Please note that the bibliographic references are provided on the last page.

KEYWORDS

Logs, Monitoring, Active Response, Graylog, NXLog, Wazuh, OpenSearch

THE SIGNIFICANCE OF LOGS

Logs serve as the silent sentinels that monitor the activities of systems, applications, and users by carrying out the following functions:

- Detection of Security Incidents

Logs are the primary means by which security incidents are detected. They record events such as login attempts, system access, data transfers, and application activity. By carefully monitoring and analyzing them, security teams can identify unusual patterns, anomalies, and potential security breaches. In doing so, logs become the first line of defense in identifying and mitigating cyber threats.

- Security Event Correlation

Logs are not just standalone records; they can be correlated across various systems and applications to provide a holistic view of the security landscape. By combining logs from different sources, security teams can uncover hidden patterns, identify sophisticated attacks, and develop a more accurate understanding of the overall security posture.

- Compliance and Auditing

Logs are essential for demonstrating compliance with various industry regulations and standards. Organizations in sectors like healthcare, finance, and government are often required to retain logs for extended periods to prove they are meeting data security and privacy requirements. Log data serves as evidence that an organization is following the necessary security protocols.

- Forensic Analysis

Beyond their real-time utility, logs play a crucial role in forensic investigations. When a security incident occurs, forensic analysts rely on historical log data to reconstruct events, determine the scope of a breach, and understand the tactics, techniques, and procedures employed by attackers. This historical data can reach back several months, allowing for a comprehensive examination of the incident.

To sum up, logs are the lifeblood of cybersecurity, providing essential visibility, accountability, and historical context, ensuring that organizations can safeguard their digital assets and respond effectively to security challenges.

CHALLENGES IN LOG MANAGEMENT

The collection and retention of logs pose several significant issues:

- Volume and Variety of Logs

One of the foremost challenges in log management is dealing with the sheer volume and diversity of logs generated by various systems, applications, and devices. As business expand and technology becomes more integrated into everyday operations, the quantity of log data can be overwhelming. Diverse log formats and sources further complicate this issue.

To manage this, organizations require robust tools and strategies for log aggregation and normalization.

- Real-Time Analysis & Correlation

Timely detection and response to cyber threats necessitate real-time log analysis. However, efficiently analyzing and correlating logs to identify patterns, anomalies, and potential threats is a complex task. Balancing the need for immediate alerting and response with the practical limitations of real-time monitoring is an ongoing challenge in cybersecurity.

- Log Retention and Scalability

Log retention policies must strike a balance between maintaining historical data for compliance, forensics, and threat analysis and managing storage costs. As organizations generate vast quantities of log data, the cost and logistical complexity of retaining this data for extended periods become a significant issue.

- Log Integrity, Security and Privacy

Log data often contains sensitive information, including user credentials and personal data. Ensuring that logs are adequately secured to prevent unauthorized access and protect privacy is an important concern. Regulatory framework impose specific requirements on the handling and storage of logs to safeguard user privacy.

IMPLEMENTING AN ARCHITECTURE WITH GRAYLOG & WAZUH

In this section, we will delve into the practical implementation of a log collection and retention architecture using Graylog and Wazuh.

We won’t provide here a detailed guide on how to install Graylog and Wazuh on Opensearch engine (natively allowing resiliency and scalability). Instead, we’ll focus on the key steps and best practices for integrating these platforms to optimize log management.

Why combine Graylog & Wazuh ?

Combining Graylog and Wazuh offers a synergistic solution for enhanced log management and cybersecurity.
Graylog excels in log collection, analysis, and visualization, providing a user-friendly interface for monitoring digital activities.

On the other hand, Wazuh specializes in advanced threat detection and response.

By integrating these two platforms, you create a robust architecture that not only centralizes log data but also leverages the power of Wazuh’s real-time threat detection capabilities. This combination offers a comprehensive solution for identifying and mitigating security incidents, making it an ideal choice for organizations looking to bolster their cyber defenses

Real World Implementation

We have implemented Graylog and Wazuh within a Lab composed of several physical servers (Linux and Windows OS), application servers (Web, Mail, Docker,..), workstations (Linux and Windows OS) and two FireWalls (An IPfire instance serving as the external FireWall and a PFSense instance serving as the internal FireWall). This configuration allows us to demonstrate how to manage an heterogeneous IT Infrastructure.

- Architectural principles
  
  As shown in the figure below, we have opted to centralize Graylog within our collection system. This configuration allows Graylog to efficiently manage the collection of logs from various endpoints and to send alerts to Wazuh. While this is not the only possible setup, it empowers us to leverage Graylog’s full capabilities for log management while enabling Wazuh to focus on SIEM functions, specifically alert processing and active responses.

- Graylog configuration
  
  Graylog natively provides many capabilities for processing logs from devices using Syslog, and it can also ingest logs from specific devices like Palo Alto. For Windows EndPoints, additional agent is needed – as NxLOG for instance (see below).
  Graylog dashboards bring efficient monitoring of all incoming logs, and the ‘inputs’ and ‘streams’ functionalities allow you to define specific ports, protocols, and indices (equivalent to database tables in the OpenSearch terminology) for the ingestion of logs from each device or group of devices. For each indice, log rotation can be specified (on time period or on file size criteria).

When it comes to Windows endpoints, NxLOG agents offer a convenient solution to ensure compatibility with Event Monitoring, with a quick installation and a very simple configuration file :

Exemple of

NXLOG configuration file for WINDOWS EndPoint

<Extension gelf>
Module xm_gelf
</Extension>

<Input internal>
Module im_internal
</Input>

<Input eventlog>
Module im_msvistalog
</Input>

<Output graylog_udp>
Module om_udp
Host graylog.dudes.lan
Port 12200
OutputType GELF
</Output>

<Route 1>
Path eventlog, internal => graylog_udp
</Route>

- Triggering WAZUH

In order to relieve the Wazuh server, not all logs ingested by Graylog are forwarded to it, and the implementation of an alert mechanism (activated on a specific stream) notifies Wazuh only in the case of suspicious events (e.g. authentication errors).

Here is an example of event definition and wazuh notification (through a dedicated API configured on Wazuh Server) :

- WAZUH Configuration

Graylog notification can be processed by Wazuh through an alerting rule (with alert level 11 in the example below ) and associated with an active response (which will necessarily take place on a device with Wazuh agent installed).

Hence, Wazuh will process filtered alerts coming from Graylog for every EndPoints from which logs are collected (including workstations or less sensitive apps for instance), while directly operating major EndPoints (FireWalls, Domain Controllers, …) through Wazuh Agent (allowing active responses). See below, Wazuh Dashboard with 7 EndPoints running Wazuh or OSSEC Agent.

SECURITY & PRIVACY CONSIDERATIONS

Ensuring the security and privacy of logs is a fundamental aspect of your log management architecture.

Implementing strict access control measures is essential to protect log data from unauthorized access. Define who can access the log data, make modifications, or delete logs. Utilize role-based access control (RBAC) to assign specific permissions to users or groups, ensuring that only authorized personnel can interact with log data.

Encrypting log data both in transit and at rest enhances its security. Use secure communication protocols for log transmission and implement encryption for log storage to protect against data interception and tampering. In cases where log data contains sensitive or personally identifiable information (PII), consider implementing techniques like data anonymization and redaction. This helps protect privacy while still retaining valuable log data.

And, when sending alerts and log data from Graylog to Wazuh, prioritize secure transmission channels. Ensure that the data is encrypted and protected during transit to maintain its confidentiality and integrity.

CONCLUSION

The integration of Graylog and Wazuh offers a powerful solution for log management and cybersecurity. By centralizing log collection, optimizing security, and streamlining threat detection, this architecture empowers organizations to navigate the complex landscape of modern cyber threats. With scalability, data retention, and a strong focus on log security, this approach ensures the long-term health and integrity of digital infrastructures.