While trying to repair a corrupted table which it stuck at optimizing stage for 12 hours, my smartness went ahead and rebooted the server
After reboot mysql won't start
Running mariadb 10.1.20
Storage Engine AREA
RAM 32 gb
OS: Centos 6.7
My.cnf
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]
[mysqld]
port = 3306
default-storage-engine = aria
aria-pagecache-buffer-size = 1024M
skip_name_resolve
skip-networking
bulk_insert_buffer_size = 8M
connect_timeout = 10
interactive_timeout = 50
query_cache_type = 0
query_cache_limit = 10M
query_cache_min_res_unit = 2k
query_cache_size = 0
table_open_cache = 2048
thread_stack = 256K
transaction_isolation = REPEATABLE-READ
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
table_cache = 4096
join_buffer_size = 10M
key_buffer_size = 512M
max_allowed_packet = 16M
bulk_insert_buffer_size = 64M
myisam_recover_options = BACKUP, FORCE
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 5G
thread_cache_size = 1024
max_heap_table_size = 64M
tmp_table_size = 64M
thread_handling = pool-of-threads
wait-timeout = 30
interactive-timeout = 30
concurrent_insert = 2
net_buffer_length = 2K
wait_timeout = 300
host_cache_size = 2000
max_connections = 1000
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
So i tried myisamchk
myisamchk --silent --force --fast --update-state /var/lib/mysql//.MYI
myisamchk: MyISAM file /var/lib/mysql/fmartinto2/core_members.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/fmartinto2/core_sessions.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/content_gener.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/content.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/content_playlist.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/content_views.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/emails_queue.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/episodes.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/links.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/link_votes.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/messages.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/playlists.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/playlist_views.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/users.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/watched_already.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/martin_new/watchlist.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
Then used it again with more memory
myisamchk --silent --force --fast --update-state \
--key_buffer_size=512M --sort_buffer_size=512M \
--read_buffer_size=4M --write_buffer_size=4M \
/var/lib/mysql/*/*.MYI
Returned empty After i removed --silent and re run it again it return
MyISAM file: /var/lib/mysql/martin_new/users.MYI is already checked
Mysql Error log:
161221 20:30:55 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/sbin/mysqld: Can't read dir of '/etc/my.cnf.' (Errcode: 2 "No such file or directory")
Fatal error in defaults handling. Program aborted
161221 20:30:55 mysqld_safe mysqld from pid file /var/lib/mysql/martin.cr.pid ended
161221 20:35:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/sbin/mysqld: Can't read dir of '/etc/my.cnf.' (Errcode: 2 "No such file or directory")
Fatal error in defaults handling. Program aborted
161221 20:35:00 mysqld_safe mysqld from pid file /var/lib/mysql/martin.cr.pid ended
161221 20:50:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2016-12-21 20:50:32 140464028563488 [Note] /usr/sbin/mysqld (mysqld 10.1.20-MariaDB) starting as process 3489 ...
2016-12-21 20:50:32 140464028563488 [Note] mysqld: Aria engine: starting recovery
recovered pages: 0% 10% 20% 30% 40%161221 20:50:33 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.1.20-MariaDB
key_buffer_size=536870912
read_buffer_size=2097152
max_used_connections=0
max_threads=1001
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10795087 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7fc0552f85cb]
/usr/sbin/mysqld(handle_fatal_signal+0x4d5)[0x7fc054e51375]
/lib64/libpthread.so.0(+0x375780f7e0)[0x7fc0544537e0]
/usr/sbin/mysqld(+0x99f356)[0x7fc055222356]
/usr/sbin/mysqld(+0x9a09ad)[0x7fc0552239ad]
/usr/sbin/mysqld(+0x9a444c)[0x7fc05522744c]
/usr/sbin/mysqld(+0x9a4f0e)[0x7fc055227f0e]
/usr/sbin/mysqld(+0x97dc4d)[0x7fc055200c4d]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x68)[0x7fc054e538d8]
/usr/sbin/mysqld(+0x46ded5)[0x7fc054cf0ed5]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x8da)[0x7fc054cf1e3a]
/usr/sbin/mysqld(+0x3ce045)[0x7fc054c51045]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x1d9b)[0x7fc054c5395b]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc052865d1d]
/usr/sbin/mysqld(+0x3c4999)[0x7fc054c47999]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
161221 20:50:33 mysqld_safe mysqld from pid file /var/lib/mysql/martin.cr.pid ended
161221 20:56:50 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2016-12-21 20:56:51 140139510470688 [Note] /usr/sbin/mysqld (mysqld 10.1.20-MariaDB) starting as process 2830 ...
2016-12-21 20:56:51 140139510470688 [Note] mysqld: Aria engine: starting recovery
recovered pages: 0% 10% 20% 30% 40%161221 20:56:51 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.1.20-MariaDB
key_buffer_size=536870912
read_buffer_size=2097152
max_used_connections=0
max_threads=1001
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10795087 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7f74c66665cb]
/usr/sbin/mysqld(handle_fatal_signal+0x4d5)[0x7f74c61bf375]
/lib64/libpthread.so.0(+0x375780f7e0)[0x7f74c57c17e0]
/usr/sbin/mysqld(+0x99f356)[0x7f74c6590356]
/usr/sbin/mysqld(+0x9a09ad)[0x7f74c65919ad]
/usr/sbin/mysqld(+0x9a444c)[0x7f74c659544c]
/usr/sbin/mysqld(+0x9a4f0e)[0x7f74c6595f0e]
/usr/sbin/mysqld(+0x97dc4d)[0x7f74c656ec4d]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x68)[0x7f74c61c18d8]
/usr/sbin/mysqld(+0x46ded5)[0x7f74c605eed5]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x8da)[0x7f74c605fe3a]
/usr/sbin/mysqld(+0x3ce045)[0x7f74c5fbf045]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x1d9b)[0x7f74c5fc195b]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f74c3bd3d1d]
/usr/sbin/mysqld(+0x3c4999)[0x7f74c5fb5999]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
161221 20:56:51 mysqld_safe mysqld from pid file /var/lib/mysql/martin.cr.pid ended
161221 21:00:05 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2016-12-21 21:00:05 139723942508576 [Note] /usr/sbin/mysqld (mysqld 10.1.20-MariaDB) starting as process 3404 ...
2016-12-21 21:00:05 139723942508576 [Note] mysqld: Aria engine: starting recovery
recovered pages: 0% 10% 20% 30% 40%161221 21:00:06 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.1.20-MariaDB
key_buffer_size=536870912
read_buffer_size=2097152
max_used_connections=0
max_threads=1001
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10795087 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7f14049e85cb]
/usr/sbin/mysqld(handle_fatal_signal+0x4d5)[0x7f1404541375]
/lib64/libpthread.so.0(+0x375780f7e0)[0x7f1403b437e0]
/usr/sbin/mysqld(+0x99f356)[0x7f1404912356]
/usr/sbin/mysqld(+0x9a09ad)[0x7f14049139ad]
/usr/sbin/mysqld(+0x9a444c)[0x7f140491744c]
mysys/stacktrace.c:268(my_print_stacktrace)[0x7f1404917f0e]
maria/ha_maria.cc:3594(ha_maria_init)[0x7f14048f0c4d]
sql/handler.cc:513(ha_initialize_handlerton(st_plugin_int*))[0x7f14045438d8]
sql/sql_plugin.cc:1408(plugin_initialize)[0x7f14043e0ed5]
sql/sql_plugin.cc:1678(plugin_init(int*, char**, int))[0x7f14043e1e3a]
sql/mysqld.cc:5196(init_server_components)[0x7f1404341045]
sql/mysqld.cc:5787(mysqld_main(int, char**))[0x7f140434395b]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f1401f55d1d]
/usr/sbin/mysqld(+0x3c4999)[0x7f1404337999]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
161221 21:00:06 mysqld_safe mysqld from pid file /var/lib/mysql/martin.cr.pid ended
Use the
myisamchecktool to 'repair' the MyISAM tables. I don't know if there is a corresponding took forariatables, but that seems to be where it blew up.Suggest filing a bug report with mariadb.com