/var/log/messages keeps on repopulating

1.5k views Asked by At

I need to clear all the rsyslog logs so, I stop the rsyslog and remove file.log as well as /var/log/messages. But as soon as I restart the service, all the logs start popping up again (400k lines). I looked up some more information and it seems that the StateFile is not being saved somewhere. I tried to create /var/spool/rsyslog dir with 777 permissions but that did not solve the issue as well and the file.log keeps on getting repopulated over and over again and it doesnot stop until the log files have all the nonsense logs back in it.

I have a logfile configured as following in ../rsyslog.d local0.* /home/user/file.log

My OS is Pidora and I am running pidora rsyslogd: [origin software="rsyslogd" swVersion="7.4.2" x-pid="1150" x-info="http://www.rsyslog.com"]

How do I clear all the rsyslog cleanly and stop rsyslog from repopulating it???? Been stuck with this for couple of days now.

Misc. Information: I am also noticing whole bunch of the following errors in /var/log/messages Nov 28 12:26:51 pidora rsyslogd-2177: imjournal: begin to drop messages due to rate-limiting Nov 28 12:36:07 pidora rsyslogd-2177: imjournal: 334681 messages lost due to rate-limiting

1

There are 1 answers

6
BobTuckerman On

The reason the logs reappear once you restart rsyslog is because there is a process that is continually logging those messages, so many that rsyslog is rate limiting them. Based on the time between those two log prints you provided, it appears to be printing ~570 logs per second. I need more log output to help you figure out what is going on.

You have two options

  1. (recommended) Figure out why the process in question is spewing those messages and fix it so that the messages will stop. Could it be related to this bug?

  2. (not recommended) Add a filter to your rsyslog config just after the modules section. This is NOT a solution you want to use long term because it removes ALL messages from the process in question, not just those that you want to remove. This could mean that you could miss seeing important error messages. Something like this would do it:

    if $programname == '<processname>' then ~

It's important to resolve the problem and not just filter it, because on a Raspberry Pi you will be running on flash based memory. It's possible that you will wear out the SD card faster if a process is continually logging a ton of messages and you run it for an extended period of time.