MariaDB/MySQL all of a sudden crashed and won't restart

Question

For the past 2 weeks my MariaDB has been crashing periodically. Restarting would fix it for a few days, then it would happen again. I planned on looking into the cause of it this weekend when I had more time, but today it crashed and it will not start up again. It is running on an nginx server with a single Drupal 7 installation.

This is the output of systemctl status mariadb.service:

mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled)
   Active: failed (Result: exit-code) since Tue 2015-06-09 19:54:28 EDT; 3min 58s ago
  Process: 4319 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 4317 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 4289 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 4317 (code=exited, status=0/SUCCESS)

Jun 09 19:54:26 kp mysqld_safe[4317]: 150609 19:54:26 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Jun 09 19:54:26 kp mysqld_safe[4317]: 150609 19:54:26 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Jun 09 19:54:27 kp mysqld_safe[4317]: 150609 19:54:27 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Jun 09 19:54:28 kp systemd[1]: mariadb.service: control process exited, code=exited status=1
Jun 09 19:54:28 kp systemd[1]: Failed to start MariaDB database server.
Jun 09 19:54:28 kp systemd[1]: Unit mariadb.service entered failed state.

And this is the output of mariadb.log:

150609 20:27:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150609 20:27:56 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
150609 20:27:56 InnoDB: The InnoDB memory heap is disabled
150609 20:27:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150609 20:27:56 InnoDB: Compressed tables use zlib 1.2.7
150609 20:27:56 InnoDB: Using Linux native AIO
150609 20:27:56 InnoDB: Initializing buffer pool, size = 128.0M
150609 20:27:56 InnoDB: Completed initialization of buffer pool
150609 20:27:56 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 30047078369
150609 20:27:56  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 30047078662
150609 20:27:56  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
150609 20:27:56  InnoDB: Waiting for the background threads to start
150609 20:27:56 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 5.5.41-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 466712 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
/usr/libexec/mysqld(my_print_stacktrace+0x3d)[0x7fdab189479d]
/usr/libexec/mysqld(handle_fatal_signal+0x515)[0x7fdab14aa475]
/lib64/libpthread.so.0(+0xf130)[0x7fdab0bda130]
/usr/libexec/mysqld(+0x64e387)[0x7fdab1659387]
/usr/libexec/mysqld(+0x74ac94)[0x7fdab1755c94]
/usr/libexec/mysqld(+0x73f7d5)[0x7fdab174a7d5]
/usr/libexec/mysqld(+0x64bc99)[0x7fdab1656c99]
/usr/libexec/mysqld(+0x63fc26)[0x7fdab164ac26]
/lib64/libpthread.so.0(+0x7df5)[0x7fdab0bd2df5]
/lib64/libc.so.6(clone+0x6d)[0x7fdaaf4501ad]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150609 20:27:57 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended

I have not changed anything on the website in weeks, so I'm not sure why it has been crashing, and why it all of a sudden won't even restart now. I have searched all over and am out of ideas, none of the above logs have been any help. The only thing I can think of is that the site crawls RSS feeds regularly and adds records from them to the database - it is a quite large database (over 500,000 records when I last checked a couple months ago, now probably closer to 1,000,000) and maybe it got too large and crashed? Anyone who can help point me in the right direction would be greatly appreciated.

Answer 1

Your are not recoverying from a crash properly. Edit your my.cnf and force innodb recovery.

Add innodb_force_recovery = 1;

you may further increase this number from 1 to 6, check the link provided to see what the differences are.

If you continue to see: InnoDB: Waiting for the background threads to start in your error log file.

Also add innodb_purge_threads=0.

Here is a useful link.

If that does not work you can try the TwinDB/percona_recovery_toolkit.