Click on each book below to review & buy on Amazon.

As an Amazon Associate, I earn from qualifying purchases.

RHCSA - Operate Running Systems: Locate & Interpret System Log Files & Journals

Reviewing log files and journals is important in understanding how your system and services are running. The most common tools for logging are rsyslogd and systemd-journald.

systemd-journald

systemd-journald is a component of the systemd system and service manager and is responsible for handling and managing system and service log data. It serves as the central logging daemon, collecting log messages from various sources and providing a centralized location for accessing and managing logs.

Here are some key features and functionalities of systemd-journald:

Centralized Logging: Instead of relying on traditional flat text files, systemd-journald stores log data in a binary format called the "journal." This binary format allows for more efficient storage and retrieval of log data and provides additional metadata along with log messages.
Journal Storage: The log data is stored in a ring-buffer-like structure, rotating and managing log files automatically. This allows for efficient disk space utilization and avoids log files from becoming too large.
Metadata and Tags: The journal includes additional metadata about each log entry, such as the timestamp, log source, log level, process ID (PID), and other relevant information. This makes it easier to search and filter logs based on specific criteria.
Preservation across Reboots: The journal can preserve log data across system reboots, ensuring that important log messages from previous sessions are still accessible during troubleshooting or analysis. (Journal preservation is disable by default, but in the next lesson we look at how we can enable this feature.)
Integration with journalctl: journalctl is a command-line utility that allows users to query and read logs from the systemd journal. It provides powerful filtering and searching capabilities, making it easy to retrieve specific log information.
Forwarding and Remote Access: systemd-journald can forward log data to other logging services or remote servers, facilitating centralized log management in distributed environments.

Locate & Interpret systemd-journald Logs

When log data preservation is enabled in systemd-journald, the logs are stored in the /var/log/journal/ directory. Each host has its own subdirectory within /var/log/journal/, identified by a unique machine ID.

If log data preservation is disabled or not configured, the logs are stored in the /run/log/journal/ directory. The /run directory is a temporary filesystem (tmpfs) that holds volatile data and does not persist across reboots.

These log files are in binary format so reading them is achieved by using the journalctl command. Below are some useful journalctl options:

Command Option	Description	Example Usage
`-f` or `--follow`	Output appended data as new entries are added.	`journalctl -f` (Follow and display real-time log entries)
`-b`	Show logs from a specific boot.	`journalctl -b -1` (Show logs from the previous boot)
`-u`	Filter logs by a specific unit (service).	`journalctl -u crond` (Show logs for the crond service)
`-p`	Filter logs by priority level.	`journalctl -p warning` (Show logs with 'error' priority)
`-x` or `--catalog`	Include message explanations from the journal's catalog.	`journalctl -x` (Display logs with additional metadata)
`--since`	Show logs since a specific date/time.	`journalctl --since "2023-07-01 00:00:00"`
`--until`	Show logs until a specific date/time.	`journalctl --until "2023-07-10 23:59:59"`
`-o` or `--output`	Specify the output format for log entries.	`journalctl -o json` (Output logs in JSON format)
`-k` or `--dmesg`	Show kernel messages (equivalent to `dmesg`).	`journalctl -k` (Show kernel messages)

When using these commands, you might need superuser privileges (root access). However, if you are a member of the wheel group you will have read access.

Follow the below exercises to get used to using the journalctl command to interpret the journal log file. Ensure to have two terminal sessions open:

Follow the journal in real time to see messages as they are written:

In terminal one, run:

journalctl -f

In terminal two, run:

for i in {1..5}
do
  logger This is message $i
  sleep 3
done

You should see the journal display:

Jul 25 08:51:42 rhcsa-install.home.arpa user[3876]: This is message 1
Jul 25 08:51:45 rhcsa-install.home.arpa user[3878]: This is message 2
Jul 25 08:51:48 rhcsa-install.home.arpa user[3880]: This is message 3
Jul 25 08:51:51 rhcsa-install.home.arpa user[3882]: This is message 4
Jul 25 08:51:54 rhcsa-install.home.arpa user[3884]: This is message 5

Press Ctrl+C to exit from the journalctl command.

Show log entries for a specific service, in this case, crond:

journalctl -u crond

The output should show run-parts, anacron & CROND entries.

Display log entries from the last 10 minutes:

journalctl --since '-10 minutes'

The output will show the last 10 minutes of journal entries.

rsyslogd

rsyslogd is a logging daemon used to handle and manage system and application log data. It collects, processes, and distributes log messages from different sources throughout the system.

Here are the key features and functionalities of rsyslogd:

Log Collection and Routing: rsyslogd is responsible for gathering log messages from various applications, services, and the kernel. It can handle logs generated by different sources simultaneously, ensuring a centralized approach to log management.
Configuration Flexibility: rsyslogd boasts an extensive and flexible configuration system. Administrators can easily customize the behavior of the logging daemon, specify log destinations, and apply filters to route logs selectively.
Multiple Log Destinations: The daemon allows log messages to be directed to various destinations. Commonly, logs are written to plain text files, but rsyslogd can also forward logs to remote syslog servers, databases, or other specialized outputs.
Support for Syslog Protocol: rsyslogd is compatible with the traditional Syslog protocol, which enables communication between different devices and systems over a network. This support allows log messages to be sent to and received from remote hosts.
Log Rotation: rsyslogd can manage log rotation, ensuring log files do not grow indefinitely and that old log data is archived or deleted based on specified criteria.
Filtering and Parsing: The logging daemon allows administrators to apply filters to log messages, enabling them to selectively process, discard, or forward specific types of log entries. Additionally, rsyslogd can parse log messages to extract structured data from them for further analysis.
Reliability and Performance: rsyslogd is designed to be robust and efficient, capable of handling a large volume of log messages without significant performance impact on the system.
Integration with Log Analyzers: rsyslogd can work in tandem with log analysis tools, allowing for more in-depth log examination and generating insights from log data.
Log Forwarding: The ability to forward logs to remote syslog servers or other logging services makes rsyslogd suitable for centralized log management in complex, distributed environments.

rsyslogd Facilities

rsyslogd facilities are predefined categories used to classify log messages based on their sources or types, allowing administrators to selectively process, filter, and route logs for better log management and analysis. Each facility represents a distinct aspect of system activity, such as authentication, kernel events, user-level applications, or internal syslog messages.

Below is a table with a summary of each facility:

Facility	Description
auth	Authentication-related messages, including successful and failed login attempts.
authpriv	Private authentication messages that should only be visible to privileged users.
cron	Messages from the cron daemon, indicating scheduled tasks and their results.
daemon	Messages from system daemons that are not covered by other facilities.
kern	Kernel-generated messages, such as hardware and driver-related events.
local0 to local7	Custom local facilities, typically used for application-specific logs.
mail	Messages related to the mail system, including email delivery status and errors.
news	Usenet news messages.
user	Messages from user-level applications.
lpr	Messages related to the printing system.
syslog	Internal messages generated by the syslog daemon itself.
mark	Used for inserting time markers in the log stream, primarily for debugging purposes.
uucp	Messages related to the Unix-to-Unix Copy Protocol (UUCP).

rsyslogd Priorities

The primary purpose of rsyslogd priorities is to facilitate log message classification and determine their severity levels. These priorities enable administrators to efficiently manage and filter log data based on their importance, ensuring critical events receive the appropriate attention while less critical ones are appropriately handled.

Below is a table with a summary of each priority:

Priority	Description
emerg	System is unusable (highest severity level).
alert	Action needs to be taken immediately.
crit	Critical conditions that require attention.
err	Error conditions.
warning	Potential issues that are not necessarily errors but need attention.
notice	Normal but significant events that may require monitoring.
info	General informative messages.
debug	Debugging messages (lowest severity level).

rsyslogd Log Destinations

rsyslogd uses a combination of priorities and facilities to determine which log messages should be written to specific log files. Priorities define the severity level of a log message (e.g., debug, info, error), while facilities categorize messages based on their source or system component (e.g., kernel, user, mail). By configuring rules in its configuration file (/etc/rsyslog.conf), rsyslogd routes log messages to the appropriate log files based on the specified priorities and facilities.

The default log files managed by rsyslogd are configured as follows:

Rules	Log File
*.info;mail.none;authpriv.none;cron.none	/var/log/messages
authpriv.*	/var/log/secure
mail.*	-/var/log/maillog
cron.*	/var/log/cron
uucp,news.crit	/var/log/spooler
local7.*	/var/log/boot.log

The hyphen preceding a log file, like -/var/log/maillog in the above, indicates that the log file does not undergo synchronization after each write. While this approach may result in potential data loss after a system crash, it can lead to improved performance.

Important Log Files

The below log files are important in monitoring and troubleshooting various aspects of the system, helping to identify and resolve issues, track events, and maintain system security and performance.

Log File	Purpose
/var/log/audit/audit.log	Records audit events on systems with auditd enabled.
/var/log/boot.log	Contains boot-up messages and kernel initialization info.
/var/log/cron	Logs scheduled tasks and cron job execution information.
/var/log/maillog	Records email-related activities and mail server info.
/var/log/messages	General system and application log messages.
/var/log/secure	Stores security-related messages, authentication events.
/var/log/spooler	Keeps printing and queuing-related messages.

Make sure to familiarize yourself with the contents of those log files.

Viewing important log files:

sudo less /var/log/audit/audit.log

boot.log has some binary data so using cat and piping to less keeps formatting.

sudo cat /var/log/boot.log | less

sudo less /var/log/cron

sudo less /var/log/maillog

sudo less /var/log/messages

sudo less /var/log/secure

sudo less /var/log/cron

sudo less /var/log/spooler

Support DTV Linux

Click on each book below to review & buy on Amazon. As an Amazon Associate, I earn from qualifying purchases.

NordVPN ®: Elevate your online privacy and security. Grab our Special Offer to safeguard your data on public Wi-Fi and secure your devices. I may earn a commission on purchases made through this link.