Running ejabberd inside docker container is causing PREVIOUS.LOG enoent crash

444 views Asked by At

I'm running ejabberd inside a docker container. Everything was working fine for a long time but today I discovered that after an hour or so ejabberd is crashing. Here is the important part for the crash.log file.

/var/log/ejabberd/crash.log
--------------------------------------------------------------------------------
2016-12-27 16:00:25 =CRASH REPORT====
  crasher:
    initial call: disk_log:init/2
    pid: <0.169.0>
    registered_name: []
    exception exit: {{{failed,{error,{file_error,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG",enoent}}},[{disk_log,reopen,2}]},[{disk_log,do_exit,4,[{file,"disk_log.erl"},{line,1188}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]}
    ancestors: [disk_log_sup,kernel_safe_sup,kernel_sup,<0.10.0>]
    messages: []
    links: [<0.135.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 40032
  neighbours:
2016-12-27 16:00:25 =SUPERVISOR REPORT====
     Supervisor: {local,disk_log_sup}
     Context:    child_terminated
     Reason:     {{failed,{error,{file_error,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG",enoent}}},[{disk_log,reopen,2}]}
     Offender:   [{pid,<0.169.0>},{name,disk_log},{mfargs,{disk_log,istart_link,undefined}},{restart_type,temporary},{shutdown,1000},{child_type,worker}]

2016-12-27 16:00:25 =ERROR REPORT====
** gen_event handler mnesia_event crashed.
** Was installed in mnesia_event
** Last event was: {mnesia_system_event,{mnesia_fatal,"~p~n",[{error,{"Cannot rename disk_log file",latest_log,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG",{log_header,trans_log,"4.3","4.12.3",ejabberd@localhost,{1482,854424,897044}},{file_error,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG",enoent}}}],<<131,108,0,0,0,24,104,2,100,0,9,99,114,97,115,104,105,110,102,111,104,2,107,0,4,126,112,126,110,108,0,0,0,1,104,2,
......

Based on the crash error log seems that is something with the logs rotation.

Cannot rename disk_log file",latest_log,"/opt/ejabberd/database/ejabberd@localhost/PREVIOUS.LOG

The strange thing is whenever I restart the container everything is working fine I obviously can see the /opt/ejabberd/database/ejabberd@localhost/ directory with all the files there and also the LATEST.LOG file. After an hour or so, it crashes and then everything inside /opt/ejabberd/database/ejabberd@localhost/ is gone even the ejabberd@localhost dir is also deleted.

If I go inside the container I cannot create the directory again or any other files in /opt/ejabberd/database/ directory.

$mkdir -p /opt/ejabberd/database/ejabberd@localhost

mkdir: cannot create directory ‘/opt/ejabberd/database/ejabberd@localhost’: No such file or directory

Does not seem to be a permissions issue

$ls -lh /opt/ejabberd/

drwxr-xr-x  0 ejabberd ejabberd    0 Dec 27 15:09 backup
drwxr-xr-x  2 ejabberd ejabberd 4.0K Dec 27 15:09 conf
drwxr-xr-x  0 ejabberd ejabberd    0 Dec 27 16:00 database
drwxr-xr-x  5 ejabberd ejabberd 4.0K Dec 27 15:09 extauth
-rw-r--r--  1 ejabberd ejabberd    0 Dec 27 15:09 first-start-done
-rwxr-xr-x  1 ejabberd ejabberd  747 Mar  2  2016 init.sh
drwxr-xr-x 15 ejabberd ejabberd 4.0K Jul 15 08:59 scripts
drwxr-xr-x  0 ejabberd ejabberd    0 Dec 27 16:00 ssl
drwxr-xr-x  0 ejabberd ejabberd    0 Dec 27 15:09 upload

Have to say that I did not have this before, does anybody experienced something similar inside a docker container with ejabberd ? Any thoughts about what would be causing this strange error ?

Thanks in advance

0

There are 0 answers