I am trying to send information to loggly via rsyslog with data from mongodb 4.4.2. However I cannot get the data in a way that I can manipulate it and scrub out certain information. When I follow the guide on the Loggly site it works for non mongodb information. If I leave %$!msg% as %msg% I get the mongodb data but I am not able to manipulate it.
config file for reading mongo logs
module(load="imfile")
#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-bundle.crt
# Input for FILE1
input(type="imfile" tag="mongo_lou_qa" ruleset="filelog" file="/var/log/mongodb/mongod.log") #wildcard is allowed at file level only
$template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [token@41058 tag=\"tag1\" tag=\"tag2\" ] %$!msg%"
set $!msg = $msg;
if re_match($!msg,'([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])')
then
{
set $!ext = re_extract($!msg,'([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])',0,1,"");
set $!msg= replace($!msg, $!ext, "xxxxxxxxx");
}
ruleset(name="filelog"){
action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="LogglyFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
}
Mongodb sample log
{"t":{"$date":"2021-01-01T00:00:00.000-00:00"},"s":"I", "c":"ACCESS", "id":20000, "ctx":"conn79","msg":"Successful authentication","attr":{"mechanism":"ABC","principalName":"__system","authenticationDatabase":"local","client":"0.0.0.0:00000"}}
rulesets are used to partition the inputs so that only some rules apply to them. It seems that variables like
$!msg
and$.msg
are local to each ruleset, so setting them outside a ruleset will not affect the values in an action that is inside a ruleset. The documentation is not very clear on just what else needs to be inside the ruleset, for example the template probably does not need to be.