I'm running ejabberd inside a docker container. Everything was working fine for a long time but today I discovered that after an hour or so ejabberd is crashing. Here is the important part for the crash.log file.
/var/log/ejabberd/crash.log
--------------------------------------------------------------------------------
2016-12-27 16:00:25 =CRASH REPORT====
crasher:
initial call: disk_log:init/2
pid: <0.169.0>
registered_name: []
exception exit: {{{failed,{error,{file_error,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG",enoent}}},[{disk_log,reopen,2}]},[{disk_log,do_exit,4,[{file,"disk_log.erl"},{line,1188}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]}
ancestors: [disk_log_sup,kernel_safe_sup,kernel_sup,<0.10.0>]
messages: []
links: [<0.135.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 2586
stack_size: 27
reductions: 40032
neighbours:
2016-12-27 16:00:25 =SUPERVISOR REPORT====
Supervisor: {local,disk_log_sup}
Context: child_terminated
Reason: {{failed,{error,{file_error,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG",enoent}}},[{disk_log,reopen,2}]}
Offender: [{pid,<0.169.0>},{name,disk_log},{mfargs,{disk_log,istart_link,undefined}},{restart_type,temporary},{shutdown,1000},{child_type,worker}]
2016-12-27 16:00:25 =ERROR REPORT====
** gen_event handler mnesia_event crashed.
** Was installed in mnesia_event
** Last event was: {mnesia_system_event,{mnesia_fatal,"~p~n",[{error,{"Cannot rename disk_log file",latest_log,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG",{log_header,trans_log,"4.3","4.12.3",ejabberd@localhost,{1482,854424,897044}},{file_error,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG",enoent}}}],<<131,108,0,0,0,24,104,2,100,0,9,99,114,97,115,104,105,110,102,111,104,2,107,0,4,126,112,126,110,108,0,0,0,1,104,2,
......
Based on the crash error log seems that is something with the logs rotation.
Cannot rename disk_log file",latest_log,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG
The strange thing is whenever I restart the container everything is working fine I obviously can see the /opt/ejabberd/database/ejabberd@localhost/
directory with all the files there and also the LATEST.LOG
file.
After an hour or so, it crashes and then everything inside /opt/ejabberd/database/ejabberd@localhost/
is gone even the ejabberd@localhost
dir is also deleted.
If I go inside the container I cannot create the directory again or any other files in /opt/ejabberd/database/
directory.
$mkdir -p /opt/ejabberd/database/ejabberd@localhost
mkdir: cannot create directory ‘/opt/ejabberd/database/ejabberd@localhost’: No such file or directory
Does not seem to be a permissions issue
$ls -lh /opt/ejabberd/
drwxr-xr-x 0 ejabberd ejabberd 0 Dec 27 15:09 backup
drwxr-xr-x 2 ejabberd ejabberd 4.0K Dec 27 15:09 conf
drwxr-xr-x 0 ejabberd ejabberd 0 Dec 27 16:00 database
drwxr-xr-x 5 ejabberd ejabberd 4.0K Dec 27 15:09 extauth
-rw-r--r-- 1 ejabberd ejabberd 0 Dec 27 15:09 first-start-done
-rwxr-xr-x 1 ejabberd ejabberd 747 Mar 2 2016 init.sh
drwxr-xr-x 15 ejabberd ejabberd 4.0K Jul 15 08:59 scripts
drwxr-xr-x 0 ejabberd ejabberd 0 Dec 27 16:00 ssl
drwxr-xr-x 0 ejabberd ejabberd 0 Dec 27 15:09 upload
Have to say that I did not have this before, does anybody experienced something similar inside a docker container with ejabberd ? Any thoughts about what would be causing this strange error ?
Thanks in advance