MySQL restarting every 30 minutes on Ubuntu 11.04

Question

I'm having an issue where MySQL 5.1.54 is restarting every 30 minutes on Ubuntu 11.04. When this occurs, the following appears in the MySQL log:

111030 12:01:52 [Note] /usr/sbin/mysqld: Normal shutdown

111030 12:01:52 [Note] Event Scheduler: Purging the queue. 0 events
111030 12:01:52  InnoDB: Starting shutdown...
111030 12:01:54  InnoDB: Shutdown completed; log sequence number 0 875122
111030 12:01:54 [Note] /usr/sbin/mysqld: Shutdown complete

111030 12:01:55 [Note] Plugin 'FEDERATED' is disabled.
111030 12:01:55  InnoDB: Initializing buffer pool, size = 256.0M
111030 12:01:55  InnoDB: Completed initialization of buffer pool
111030 12:01:55  InnoDB: Started; log sequence number 0 875122
111030 12:01:55 [Note] Event Scheduler: Loaded 0 events
111030 12:01:55 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.54-1ubuntu4-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)

This occurs like clockwork every 30 minutes, so it's obviously some service restarting it.

I have checked the crontab of every user on the system (including system users), and none of them have a crontab setup, as you can see in the output below:

# awk -F: '{print $1}' /etc/passwd | xargs -n 1 -i crontab -u {} -l  
no crontab for root
no crontab for daemon
no crontab for bin
no crontab for sys
no crontab for sync
no crontab for games
no crontab for man
no crontab for lp
no crontab for mail
no crontab for news
no crontab for uucp
no crontab for proxy
no crontab for www-data
no crontab for backup
no crontab for list
no crontab for irc
no crontab for gnats
no crontab for nobody
no crontab for libuuid
no crontab for syslog
no crontab for sshd
no crontab for landscape
no crontab for ubuntu
no crontab for statd
no crontab for myproxy
no crontab for condor
no crontab for messagebus
no crontab for avahi
no crontab for joe
no crontab for smmta
no crontab for smmsp
no crontab for postfix
no crontab for deploy
no crontab for mysql
no crontab for redis

My dmesg contains the following each time it is restarted. I'm not an apparmor expert, but I believe this is a normal message obtained each time the MySQL service starts:

[1165328.780405] type=1400 audit(1319976114.984:74): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=31985 comm="apparmor_parser"

Also, here are the contents of the MySQL upstart configuration in /etc/init/mysql.conf:

# MySQL Service

description     "MySQL Server"
author          "Mario Limonciello <[email protected]>"

start on (net-device-up
          and local-filesystems
      and runlevel [2345])
stop on runlevel [016]

respawn

env HOME=/etc/mysql
umask 007

# The default of 5 seconds is too low for mysql which needs to flush buffers
kill timeout 300

pre-start script
    #Sanity checks
    [ -r $HOME/my.cnf ]
    [ -d /var/run/mysqld ] || install -m 755 -o mysql -g root -d /var/run/mysqld
    /lib/init/apparmor-profile-load usr.sbin.mysqld
    LC_ALL=C BLOCKSIZE= df --portability /var/lib/mysql/. | tail -n 1 | awk '{ exit ($4<4096) }'
end script

exec /usr/sbin/mysqld

post-start script
   for i in `seq 1 30` ; do
        /usr/bin/mysqladmin --defaults-file="${HOME}"/debian.cnf ping && {
            exec "${HOME}"/debian-start
            # should not reach this line
            exit 2
        }
        sleep 1
    done
    exit 1
end script

Any idea what might be causing this? It doesn't cause any problems, other than Monit alerts stating that "PID changed Service mysqld" (I have Monit monitoring mysqld -- but it reports no errors with the mysqld process, other than the fact that every 30 minutes, it has its PID changed since MySQL is restarted).

Thanks in advance.

Answer 1

Seems converting the /etc/init.d/mysql start up script to sysV style rather than as an upstart script seemed to correct the problem for me.

Answer 2

You should try to run it without AppArmor: just run /usr/bin/mysqld_safe or /usr/bin/mysqld without using upstart and wait 30 minutes. If mysql does not auto-restart, then disable AppArmor in the /etc/init/mysql.conf file, or configure it differently.

If the problem is still there, read mysql's log. If the log is not enabled by default, you can use the option --log-error=/tmp/mysql.log --log-warnings when launching mysqld.

Answer 3

can you check (and probably post) the definition of your mysql job in upstart? (/etc/init/mysql.conf). OK = try removing the "respawn". It does not work as documented in the upstart documentation. Generally it's used to respawn the process if it's killed by other process, but it seems it does not function as expected. You can see why the apparmour is always loading - because of the pre-start stanza in the script. As upstart is very new, and is still evolving, better is to use the SysV way.

Answer 4

I had this same problem upgrading to Ubuntu 14.04. I found this question because it mentions the AppArmor log message, so thanks! I may not have realised MySQL was restarting otherwise.

Upon investigating /var/log/daemon.log, I found the output of /etc/mysql/debian-start appearing repeatedly. The relevant part was this:

May 18 06:48:18 tom /etc/mysql/debian-start[15525]: Upgrading MySQL tables if necessary.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysql' as: /usr/bin/mysql
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Error: Server version (5.5.35-1ubuntu1) does not match with the version of
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: the server (5.5.37) with which this program was built/distributed. You can
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: use --skip-version-check to skip this check.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: FATAL ERROR: Upgrade failed

I read through the /etc/mysql/debian-start script and tried running the upgrade command as in the script, hoping to debug it (MySQL server needs to be running at the time):

/usr/bin/mysql_upgrade --defaults-extra-file=/etc/mysql/debian.cnf

I then found that this worked without complaint and everything simply worked from that point onward. I don't know why it was failing in the first place, but that seemed to fix it. MySQL hasn't restarted itself since.

Answer 5

are you using chef or puppet, which might be doing something that triggers the reboot?