Simple rsyslog
As a UNIX / Linux systems administrator, I manage several systems. The key to proper management is the management and review of systems logs. There are many systems to do this, and they all require some level of caffeine, nicotine, and alcohol-driven insanity. While all the various systems have their advantages, they have the disadvantage of various complexity levels and CPU load levels.
I’ll focus on what has become the “standard” logging system; rsyslog.
Rsyslog is a powerful, enterprise-level, secure, and high-performance log processing system that accepts data from different sources (systems/applications) and outputs it into multiple formats.
It is designed in a client/server model; therefore, it can be configured as a client and/or as a central logging server for other servers, network devices, and remote applications.
Prior to configuring we must make sure several items are in place:
- All systems utilize NTP sources.
- Use the timedatectl command.
- All systems utilize the SAME NTP source.
- Use the timedatectl command.
- All systems have the same time zone set.
- Use the timedatectl command.
- All systems have rsyslog installed.
- Use the command service rsyslog status.
Sample output of timedatectl:
Sample output of service rsyslog status:
If the command service rsyslog status returns Unit rsyslog.service can not be found, rsyslog is not installed and should be installed as below:
$ sudo apt update $ apt install rsyslog rsyslog-relp $ sudo systemctl start rsyslog $ sudo systemctl enable rsyslog $ sudo systemctl status rsyslog
Building the log aggregator:
As of this writing, rsyslog is installed by default on Debian and Ubuntu, but ryslog-relp (lossless log shipping) is not.
We will build a minimal configuration on the log host to receive logs and write them to one or more files. (We will not use the standard port 514) This configuration will listen on port 5514 bound to the address 10.249.4.42.
nano /etc/rsyslog.d/00-log-host.conf
------------------------------------
module(load="imrelp")
ruleset(name="receive_from_5514") {
action(type="omfile" file="/var/log/remote/auth.log")
}
# Note the address bound to and the port listening on.
input(type="imrelp" address="10.249.4.42" port="5514" ruleset="receive_from_5514")To preserve logs for historical or forensic reasons we will setup a daily log rotation schedule.
nano /etc/logrotate.d/rsyslog_host
----------------------------------
/var/log/remote/*.log {
rotate 365
daily
compress
missingok
notifempty
dateext
dateformat .%Y-%m-%d
dateyesterday
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}One can test the logrotation via the following:
# logrotate --debug /etc/logrotate.d/rsyslog_host WARNING: logrotate in debug mode does nothing except printing debug messages! Consider using verbose mode (-v) instead if this is not what you want. reading config file /etc/logrotate.d/rsyslog_host Reading state from file: /var/lib/logrotate/status Allocating hash table for state file, size 64 entries Handling 1 logs rotating pattern: /var/log/remote/*.log after 1 days (365 rotations) empty log files are not rotated, old logs are removed considering log /var/log/remote/*.log log /var/log/remote/*.log does not exist -- skipping Creating new state
COnfiguring the Client:
Once the log host is setup we will proceed to setup our client to forward logs to the host
nano /etc/rsyslog.d/90-log-client
-----------------------------------
# Poll each file every 2 seconds
module(load="imfile" PollingInterval="2")
# Create a ruleset to send logs to the right port for our environment
# Note the port and address of the LOG HOST
module(load="omrelp")
ruleset(name="send_to_loghost") {
action(type="omrelp" target="10.249.4.42" port="5514")
}
# Send all files on this server to the same remote, tagged appropriately
input(
type="imfile"
File="/var/log/auth.log"
Tag="auth:"
Facility="local7"
Ruleset="send_to_loghost"
)We will also need to configure the “local7” facility to send logs to our log host
nano /etc/rsyslog.conf ----------------------------- module(load="omrelp") # enables sending RELP messages (over TCP) module(load="immark" interval="180") # enables --MARK-- message capability
nano /etc/rsyslog.d/50-default.conf ----------------------------------- # Send auth messages of `auth,authrpiv` facility both to a local file and to a RELP server auth,authpriv.* :omrelp:10.249.4.42:5514
One can syntax test the various configurations as follows:
# rsyslogd -N1 rsyslogd: version 8.2001.0, config validation run (level 1), master config /etc/rsyslog.conf rsyslogd: End of config validation run. Bye.
As we are using non-standard ports the UFW service must be configured to allow traffic on port 5514. We will do the following on both host and client.
ufw allow 5514
One can apply all forms of filtration at the UFW level, but those are a separate post and discussion for now we will just open the port.
The last steps are to restart rsyslog on both the log host and client.
On the server:
# service rsyslog restart
# service rsyslog status
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2021-03-07 19:35:31 UTC; 4s ago
TriggeredBy: ● syslog.socket
Docs: man:rsyslogd(8)
https://www.rsyslog.com/doc/
Main PID: 7724 (rsyslogd)
Tasks: 5 (limit: 1160)
Memory: 1.8M
CGroup: /system.slice/rsyslog.service
└─7724 /usr/sbin/rsyslogd -n -iNONE
On the client:
# service rsyslog restart
# service rsyslog status
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2021-03-07 19:37:46 UTC; 4s ago
TriggeredBy: ● syslog.socket
Docs: man:rsyslogd(8)
https://www.rsyslog.com/doc/
Main PID: 2536 (rsyslogd)
Tasks: 5 (limit: 1160)
Memory: 1.8M
CGroup: /system.slice/rsyslog.service
└─2536 /usr/sbin/rsyslogd -n -iNONETesting:
In this scenario one can simply ssh to the log-client and provide either a bad password, user or ssh key. This will generate error messages in the client /var/log/auth.log similar to:
Mar 7 20:23:37 ip-10-249-4-149 sshd[3350]: Connection reset by invalid user ahahaha 127.0.0.1 port 2759 [preauth] Mar 7 20:23:38 ip-10-249-4-149 sshd[3352]: Invalid user ahahaha from 127.0.0.1 port 2761 Mar 7 20:23:38 ip-10-249-4-149 sshd[3352]: Connection reset by invalid user ahahaha 98.15.251.153 port 2761 [preauth] Mar 7 20:23:38 ip-10-249-4-149 sshd[3354]: Invalid user ahahaha from 127.0.0.1 port 2762
These should be sent to the log host /var/log/remote/auth.log in a format similar to :
Mar 7 20:23:38 ip-10-249-4-149 sshd[3354]: Invalid user ahahaha from 127.0.0.1 port 2762 Mar 7 20:23:38 ip-10-249-4-149 sshd[3354]: Connection reset by invalid user ahahaha 98.15.251.153 port 2762 [preauth] Mar 7 20:23:39 ip-10-249-4-149 sshd[3356]: Invalid user ahahaha from 127.0.0.1 port 2763 Mar 7 20:23:39 ip-10-249-4-149 sshd[3356]: Connection reset by invalid user ahahaha 127.0.0.1 port 2763 [preauth]
Troubleshooting:
Things that will waste your day:
- The client and host can not communicate
- Ping the host from the client
- Ping the client from the host
- The host-based firewall is blocking you
- Use the command ufw status
- You have not configured the host properly
- Confirm the bind-address and port
- You have not configured the client properly
- Confirm the target address and port
- You do not have permissions to read/write the files
- tail -f /var/log/syslog for errors
Congratulations:
If you have logs forwarded from the client to the host, you have a basic log centralization system in place. From here you can perform additional processing of these raw logs. Perhaps send them to a analytics service, or to a long term storage facility, or even set alarms and alerts for too many failed logins, or web server errors.
You have the basics, now you can make it dance to your tune.
