How to ignore (discard) certain syslog messages in rsyslogd using filters

Written by Claudio Kuenzler - 2 comments

Published on January 7th 2022 - Listed in Linux ELK

---

A central syslog server, using rsyslogd, was being flooded with syslog messages from surrounding servers. Docker logs in particular were filling up the central syslog server's /var/log/syslog in a short amount of time:

root@syslog:~# grep "docker" /var/log/syslog
Jan  7 13:16:18 docker11 systemd[1]: run-docker-runtime\x2drunc-moby-b5f31be74b826201bd84ce712581bfceffaf192b024d68375d94f0947af4ff07-runc.0OXJAR.mount: Succeeded.
Jan  7 13:16:18 docker01 systemd[1]: run-docker-runtime\x2drunc-moby-3ee4af24c1c3cd3f79b579fc81ac0d5af34e78647185aa4e8a2c02d3903b0ec0-runc.KjU36D.mount: Succeeded.
Jan  7 13:16:18 docker23 systemd[1]: run-docker-runtime\x2drunc-moby-b20db8928161fc671b942fc82f98ea3a86ac6c9d09c15a880ecdb416d16700c5-runc.N89XuM.mount: Succeeded.
Jan  7 13:16:18 docker11 systemd[1]: run-docker-runtime\x2drunc-moby-ae1234beb6d40cdc6bdb0e8da07461c6958c130959e1cc5f4cebc037cdce5bab-runc.wTfEVk.mount: Succeeded.
Jan  7 13:16:19 docker24 systemd[1]: run-docker-runtime\x2drunc-moby-6aa9b35b3053ce45cf77ba6707a1e183374bce314c69fb999dd2091f1a6fc083-runc.Z6TlqD.mount: Succeeded.
[...]

A ton of mount:Succeeded log events were logged, multiple times per second.

This is fine on the Docker host's local syslog file, but on the central syslog server (which forwards the logs to an ELK stack) this is just too much (non-helpful) information.

Luckily rsyslogd is capable to discard syslog messages using filters. 

Discarding logs which contain a certain content

In the following example the syslog message field (referred to as ":msg") can be checked if the content contains a certain string:

root@syslog:~# cat /etc/rsyslog.d/10-filter-docker-syslog.conf
# Filter out messages like these:
# Jan  7 13:08:44 docker01 systemd[1]: run-docker-runtime\x2drunc-moby-4cb10df07f04c27fc12255faf5d8d58acdc4ca5fc99b7d59088048022b1d2f38-runc.MprpAa.mount: Succeeded.
# Jan  7 13:14:31 docker23 systemd[1]: var-lib-docker-overlay2-9fcf3bf476a8337799f1f3a58c7f74ce88856f300a62ae63c5549fb6d0e89714-merged.mount: Succeeded.
:msg, contains, "run-docker-runtime"    stop
:msg, contains, "var-lib-docker-overlay2"    stop

The "stop" at the end is the action which rsyslog should take. In this case (stop) the message is simply ignored/discarded. 

Note: In previous rsyslog versions, the hyphen character (~) was used instead of stop

Discarding logs of a specific program/process name

The same works also with other syslog fields. For example if all syslog messages from the systemd process should be discarded:

root@syslog:~# cat /etc/rsyslog.d/10-filter-systemd.conf
# Filter out all systemd messages:
:programname, isequal, "systemd"    stop

Discarding logs from a specific syslog server

Syslog messages from a specific syslog server can also be ignored:

root@syslog:~# cat /etc/rsyslog.d/10-filter-remote-syslogserver.conf
# Filter out messages from a spammy syslog server:
:fromhost-ip, isequal, "10.10.2.133"    stop

More filters using properties and conditions

In general there are many possibilities to create rsyslog filters. Basically the syntax is the following:

:field, condition, "search string" action

A full list of field names can be found in the rsyslog properties documentation.

A full list of possible conditions can be found in the compare operations documentation.

Always make sure to reload or restart rsyslogd after a config change.

The config order is important!

Last but not least it's important to note that these "filter configs" should be named correctly. Rsyslog (by default) reads all *.conf files from the /etc/rsyslog.d/ directory in an alphabetical order. The filters should happen before the file "50-default.conf" is loaded. This is the config responsible for writing the syslog messages into files.

Therefore if you start all your filters with a number prior to 50, the filters should work. In the following example all filter configs start with a 10:

root@syslog:~# ll /etc/rsyslog.d/
total 32
--w----r-T 1 root root  706 Aug 14  2017 01-json.conf
-rw-r--r-- 1 root root  456 Jan  7 13:15 10-filter-docker-syslog.conf
-rw-r--r-- 1 root root   93 Jan  7 14:42 10-filter-remote-syslogserver.conf
-rw-r--r-- 1 root root  314 Sep  8  2015 20-ufw.conf
-rwxr-xr-x 1 root root  255 Jun 29  2017 21-cloudinit.conf
-rw-r--r-- 1 root root 1124 Jan 30  2018 50-default.conf
--w----r-T 1 root root  108 Jan  7 13:30 99-remote.conf
-rw-r--r-- 1 root root  242 Apr 13  2016 postfix.conf

Add a comment

Show form to leave a comment

Comments (newest first)

D McKeon from Oregon, USA wrote on Jun 25th, 2022:

Thanks for an informative post. FYI,
the hyphen '-' or dash character is %2d,
the '~' at %7e can be called 'tilde'

dec oct hex 76543210 char
45 55 2d 00101101 -
126 176 7e 01111110 ~

---

noWinToday from wrote on May 24th, 2022:

nice, but i can't get it to work :(
i had it used some years ago, but now i've inherited a server and cannot get it working again.
rsyslog.conf contains this:
:msg, contains, "Connection discarded" stop
some templates like this:
template(name="stonegate" type="string" string="/var/log/rsyslog/stonegate/SG_%$YEAR%-%$MONTH%-%$DAY%.log")
a ruleset with some actions like this:
ruleset(name="remote-syslog-514") {
if ( $fromhost-ip == '172.18.2.60' or $fromhost-ip == '172.18.173.20' ) then
{
action(type="omfile" DynaFile="stonegate" dynaFileCacheSize="50" asyncWriting="on" flushInterval="2" ioBufferSize="1024")
stop
}
else if ( $fromhost-ip ...
module(load="imudp")
input(type="imudp" port="514" ruleset="remote-syslog-514")
module(load="imptcp")
input(type="imptcp" port="514" ruleset="remote-syslog-514")

---