MongoDB, log is unable to logrotate

Question

The MongoDB version is v2.6.3 on my server, and mongod is running:

ubuntu@koala:/var/log/mongodb$ ps -ef | grep mongo
root      7434     1 17 Jun16 ?        06:57:26 mongod -f /etc/mongodb-onepiece.conf --fork

I am using logrotate to daily rotate the log file of MongoDB. A strange problem just occurred with logrotate. I check the log file:

ubuntu@koala:/var/log/mongodb$ ls -lth | grep mongodb 
-rw-r--r-- 1 ubuntu ubuntu 1.9G Jun 18 10:23 mongodb-onepiece.log.1
-rw-r--r-- 1 ubuntu ubuntu    0 Jun 17 07:35 mongodb-onepiece.log
-rw-r--r-- 1 ubuntu ubuntu 838M Jun 15 07:35 mongodb-onepiece.log.3.gz
-rw-r--r-- 1 ubuntu ubuntu   22 Jun 14 20:52 mongodb-onepiece.log.2.gz
-rw-r--r-- 1 ubuntu ubuntu 1.1G Jun  4 17:10 mongodb-onepiece.log.4.gz
-rw-r--r-- 1 ubuntu ubuntu  53M May 29 19:14 mongodb-onepiece.log.5.gz

The most up-to-date log file is .log.1 instead of .log. When I use tail -fn to check the log.1 file, I can see that the log is still appending to it, and it's growing:

ubuntu@koala:/var/log/mongodb$ tail -fn 2 mongodb-onepiece.log.1
2015-06-18T10:36:50.163+0800 [initandlisten] connection accepted from 192.168.1.52:50278 #2507 (49 connections now open)
2015-06-18T10:36:50.163+0800 [conn2503] command koala.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 numYields:0  reslen:178 0ms

This means that MongoDB is logging to the file that is't not supposed. As can be seen from the mongod config file, MongoDB should log to the logpath:

ubuntu@koala:/var/log/mongodb$ vim /etc/mongodb-onepiece.conf
dbpath=/var/lib/mongodb-onepiece
logpath=/var/log/mongodb/mongodb-onepiece.log
logappend=true
bind_ip = 192.168.1.*
port = 47017
fork=true
journal=true
master = true

From the above, I assume that the problem was not with the logrotate config, but with MongoDB writing to the wrong file. Everyday when logrotate starts, it only checks .log file and finds out it's empty, then it will stop rotating the log.

If I restart the mongod daemon, the logpath will be correct for a moment (writing to the right log file). For that day, the .log file is not empty, then it will be successfully rotated to .log.1 file. But the same problem will happen again after log rotating ,i.e., MongoDB will be logging to .log.1 file afterwards. The cycle comes here.

The logrotate config file is given here:

ubuntu@koala:/var/log/mongodb$ vim /etc/logrotate.d/mongodb
/var/log/mongodb/*.log {
     daily
     rotate 52
     missingok
     copytruncate
     notifempty
     compress
     delaycompress
}

The same logrotate config just works fine with other MongoDB logs on the other server with MongoDB v2.6.5 and I suppose postrotate is not the trick here (I have also tried postrotate but without luck).

How to solve this problem?

Answer 1

I'm not a mongo expert, but:

1. You should be following the official documentation https://docs.mongodb.org/v2.6/tutorial/rotate-log-files/

2. If you are going to use a logrotate config file, as you indicated, then you need a postrotate lint to your config (failure to do so is why mongodb continues to log to the log.1 file)

   postrotate
       kill -SIGUSR1 `cat /var/run/mongodb.pid` >/dev/null 2>&1 || true