Not so simple Syslog
Now that we have our simple log aggregator running as per the Simple Rsyslog post. We need to examine some form of the log analysis system.
As spoken prior, “There are many systems to do this, and they all require some level of caffeine, nicotine, and alcohol-driven insanity. While all the various systems have their advantages, they have the disadvantage of various complexity levels and CPU load levels.”
Most modern log analysis systems desire the logs to be ingested in JSON format, so perhaps utilizing a modern log receiver to perform this function natively is in order. Enter Syslog-ng, or Syslog the next generation.
Let us assume the following:
- A log host with IP address 10.249.4.112
- A log source with IP address 10.249.4.158
We will begin the configuration with the log source.
--- Set a host name
sudo hostnamectl set-hostname log-source-4-158
--- Add the log host to /etc/hosts
echo "10.249.4.112 loghost" >> /etc/hosts
---- And TEST
ping loghost
--- Check our NTP and Time zone; These should match on source and host
# timedatectl
Local time: Mon 2021-03-08 20:44:54 UTC
Universal time: Mon 2021-03-08 20:44:54 UTC
RTC time: Mon 2021-03-08 20:44:55
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
NTP service: active
RTC in local TZ: noWe now have the basic setup on the log source; we will now configure rsyslog to send logs via our log host’s Syslog function. We send all messages from the auth and authpriv facilities to the remote server on port 514 in this specific case.
--- nano /etc/rsyslog.d/50-default.conf --- add the line as shown auth,authpriv.* /var/log/auth.log auth,authpriv.* @@loghost:514
And we will restart the rsyslog service and confirm proper operations.
--- Syntax check our config rsyslogd -N1 --- Restart the service and reread the config # service rsyslog restart --- Check the status service rsyslog status
As our log source is complete, it is time to turn our eyes to the central log host. We begin by setting up some of the same basics as shown above.
--- Set a host name
# sudo hostnamectl set-hostname log-host-4-112
--- Add the log host to /etc/hosts
# echo "10.249.4.112 loghost" >> /etc/hosts
---- And TEST
# ping loghost
--- Check our NTP and Time zone; These should match on source and host
# timedatectl
Local time: Mon 2021-03-08 20:44:54 UTC
Universal time: Mon 2021-03-08 20:44:54 UTC
RTC time: Mon 2021-03-08 20:44:55
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
--- Stop and disable rsyslog
# systemctl stop rsyslog.service
# systemctl disable rsyslog.serviceAnd we begin the installation of Syslog-ng.
--- Update our packages # apt-get update --- Install syslog-ng # apt-install syslog-ng
And begin to configure Syslog-ng; we start by enabling a network source, this is done in the configuration source stanza with the phrase as below:
--- nano /etc/syslog-ng.conf
source s_network {
tcp(ip(0.0.0.0) port(514));
udp(ip(0.0.0.0) port(514));
};This will open port 514 TCP and UDP on all present network interfaces; one can restrict to a specific port by supplying that IP address to replace 0.0.0.0. Likewise, one can move the listening port to a non-standard port by replacing the 514 above with the port desired.
We will now specify a destination phrase in the destination stanza, specifying a destination file, and use the Syslog-ng template to format the log messages into JSON.
--- nano /etc/syslog-ng.conf
destination d_json {
file("/var/log/auth.json" template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
};Lastly, we shall create a log flow with facility filtration in the logs stanza of our configuration.
--- nano /etc/syslog-ng.conf
log {
source(s_network); filter(f_auth); destination(d_json);
};Once our configuration is complete we will need to check it, and restart the service to read the new configuration.
--- Syntax check syslog-ng config
# syslog-ng -s -f /etc/syslog-ng/syslog-ng.conf
-- Restart syslog-ng and confirm status
# service syslog-ng restart
# service syslog-ng status
● syslog-ng.service - System Logger Daemon
Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-03-08 19:52:11 UTC; 1h 8min ago
Docs: man:syslog-ng(8)
Main PID: 1383 (syslog-ng)
Tasks: 1 (limit: 1160)
Memory: 2.2M
CGroup: /system.slice/syslog-ng.service
└─1383 /usr/sbin/syslog-ng -FAs we have configured our log source and our log host it is time to test.
I will ssh to my log-source with both good and bad credentials so we may observe the various messages produced.
# tail -f /var/log/auth.json
-- Proper Credentials
{"TAGS":".source.s_network","SOURCEIP":"10.249.4.158","SOURCE":"s_network","PROGRAM":"sshd","PRIORITY":"info","PID":"1608","MESSAGE":"Accepted publickey for ubuntu from 98.[redacted.redacted].153 port 9219 ssh2: RSA SHA256:[redacted-redacted-redacted]","LEGACY_MSGHDR":"sshd[1608]: ","HOST_FROM":"10.249.4.158","HOST":"10.249.4.158","FACILITY":"auth","DATE":"Mar 8 21:07:24"}
{"TAGS":".source.s_network","SOURCEIP":"10.249.4.158","SOURCE":"s_network","PROGRAM":"sshd","PRIORITY":"info","PID":"1608","MESSAGE":"pam_unix(sshd:session): session opened for user ubuntu by (uid=0)","LEGACY_MSGHDR":"sshd[1608]: ","HOST_FROM":"10.249.4.158","HOST":"10.249.4.158","FACILITY":"authpriv","DATE":"Mar 8 21:07:24"}
{"TAGS":".source.s_network","SOURCEIP":"10.249.4.158","SOURCE":"s_network","PROGRAM":"systemd-logind","PRIORITY":"info","PID":"445","MESSAGE":"New session 9 of user ubuntu.","LEGACY_MSGHDR":"systemd-logind[445]: ","HOST_FROM":"10.249.4.158","HOST":"10.249.4.158","FACILITY":"auth","DATE":"Mar 8 21:07:24"}
-- Improper username
{"TAGS":".source.s_network","SOURCEIP":"10.249.4.158","SOURCE":"s_network","PROGRAM":"sshd","PRIORITY":"info","PID":"1718","MESSAGE":"Invalid user xyz from 98.[redacted.redacted].153 port 3062","LEGACY_MSGHDR":"sshd[1718]: ","HOST_FROM":"10.249.4.158","HOST":"10.249.4.158","FACILITY":"auth","DATE":"Mar 8 21:13:00"}
{"TAGS":".source.s_network","SOURCEIP":"10.249.4.158","SOURCE":"s_network","PROGRAM":"sshd","PRIORITY":"info","PID":"1718","MESSAGE":"Connection reset by invalid user xyz 98.[redacted.redacted].153 port 3062 [preauth]","LEGACY_MSGHDR":"sshd[1718]: ","HOST_FROM":"10.249.4.158","HOST":"10.249.4.158","FACILITY":"auth","DATE":"Mar 8 21:13:01"}
-- Improper SSH Key
{"TAGS":".source.s_network","SOURCEIP":"10.249.4.158","SOURCE":"s_network","PROGRAM":"sshd","PRIORITY":"info","PID":"1724","MESSAGE":"AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:[redacted-redacted-redacted] failed, status 22","LEGACY_MSGHDR":"sshd[1724]: ","HOST_FROM":"10.249.4.158","HOST":"10.249.4.158","FACILITY":"auth","DATE":"Mar 8 21:14:29"}
{"TAGS":".source.s_network","SOURCEIP":"10.249.4.158","SOURCE":"s_network","PROGRAM":"sshd","PRIORITY":"info","PID":"1724","MESSAGE":"Connection reset by authenticating user ubuntu 98.[redacted.redacted].153 port 3188 [preauth]","LEGACY_MSGHDR":"sshd[1724]: ","HOST_FROM":"10.249.4.158","HOST":"10.249.4.158","FACILITY":"auth","DATE":"Mar 8 21:14:29"}
One can use a JSON lint system to review and confirm that the JSON created is proper JSON and can be used as an input to any number of analytics systems.
