From last 4 days we are facing frequent database crashes with mysql infobright engine, there is no recent changes on production environment and no updates. Currently we are using the version 5.1.40.
Find the below dump, can any one help to figure out the issue.
170520 21:12:08 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=1677721600
read_buffer_size=1048576
max_used_connections=75
max_threads=1000
threads_connected=54
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3696548 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0xc2a4bd000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fc0d0bede58 thread_stack 0x80000
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0xaef849]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x412e13]
/lib64/libpthread.so.0(+0xf7e0) [0x7fc0d48c77e0]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0xb10635]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0xb1f123]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x9a9693]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x76ae0c]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x76b594]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x767ab3]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x7694ea]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x72902b]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x422325]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x427573]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x42b38c]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x42c227]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x42cb05]
/usr/local/infobright-4.7.1-x86_64/bin/mysqld() [0x41f06d]
/lib64/libpthread.so.0(+0x7aa1) [0x7fc0d48bfaa1]
/lib64/libc.so.6(clone+0x6d) [0x7fc0d460caad]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xc2da5e410 = SELECT DATE_FORMAT(DATETIME,'%Y%m%d') AS YEAR_MONTH_DAY_SK ,HOUR(DATETIME) AS HOUR_SK, IFNULL(DESTINATION,'0') AS DESTINATION, IFNULL(DATETIME,'1970-01-01 00:00:00') AS DATETIME, IFNULL(CLIENTID,'0') AS CLIENTID, IFNULL(GROUPID,'0') AS GROUPID, IFNULL(TEAMID,'0') AS TEAMID, IFNULL(SERVICEID,'0') AS SERVICEID, IFNULL(CHANNELID,'0') AS CHANNELID, IFNULL(STATUSID,'0') AS STATUSID, CASE REASONCODE WHEN '' THEN NULL WHEN NULL THEN NULL ELSE REASONCODE END AS REASONCODE, CASE REASONDESC WHEN '' THEN NULL WHEN NULL THEN NULL ELSE REASONDESC END AS REASONDESC, IFNULL(ACTIONTYPE1ID,'0') AS ACTIONTYPE1ID, CASE ACTIONTYPE1DESC WHEN '' THEN NULL WHEN NULL THEN NULL ELSE ACTIONTYPE1DESC END AS ACTIONTYPE1DESC, IFNULL(ACTIONTYPE2ID,'0') AS ACTIONTYPE2ID, CASE ACTIONTYPE2DESC WHEN '' THEN NULL WHEN NULL THEN NULL ELSE ACTIONTYPE2DESC END AS ACTIONTYPE2DESC, IFNULL(ATTACHMENT,'0') AS ATTACHMENT, CASE MIMETYPE WHEN '' THEN NULL WHEN NULL THEN NULL ELSE MIMETYPE END AS MIMETYPE, CASE VOICEFLOWNAME WH
thd->thread_id=35918
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170520 21:12:08 mysqld_safe Number of processes running now: 0
170520 21:12:08 mysqld_safe mysqld restarted
tcmalloc: large alloc 1365172224 bytes == 0x4518000 @
Loading configuration for Infobright instance ...
Option: AllowMySQLQueryPath, value: 1.
Option: AutoConfigure, value: 0.
Option: CacheFolder, value: /usr/local/infobright-4.7.1-x86_64/cache.
Option: ControlMessages, value: 0.
Option: IBEngineRevision, value: IEE_4.7.1_r30553_31737.
Option: InternalMessages, value: 0.
Option: InternalMessagesFlushPeriod, value: 60.
Option: KNFolder, value: BH_RSI_Repository.
Option: KNLevel, value: 99.
Option: LicenseCheckInterval, value: 0.
Option: LicenseExpireWarningDays, value: 0.
Option: LicenseFile, value: <unknown>.
Option: LicenseServerIPAddr, value: .
Option: LicenseServerType, value: .
Option: LicenseServerWarningNumber, value: .
Option: LoaderMainHeapSize, value: 800.
Option: PushDown, value: 1.
Option: ServerMainHeapSize, value: 48000.
Option: UseMySQLImportExportDefaults, value: 0.
Option: bherrLogLevel, value: 1.
Infobright instance configuration loaded.
tcmalloc: large alloc 40265318400 bytes == 0x687c8000 @
tcmalloc: large alloc 10066329600 bytes == 0x9cff48000 @
170520 21:12:09 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
170520 21:12:09 [ERROR] Do you already have another mysqld server running on port: 5029 ?
170520 21:12:09 [ERROR] Aborting
170520 21:12:09 [Note] /usr/local/infobright-4.7.1-x86_64/bin/mysqld: Shutdown complete
170520 21:12:09 mysqld_safe mysqld from pid file /data/infobright/data/SH-UMP-CINFBRT2.pid ended