Apache 2 on VM *sometimes* not finding mods-enabled directory

Question

I'm running Apache 2 on an Ubuntu 10.04 server VM (ESXi 4.1) and getting something very weird:

Sometimes the mods-enabled directory is found.
Sometimes it's not. It seems to not be about 75% not found.
Nothing is changed in between.

Here's some excerpts from the error.log combined with command lines to demonstrate what's going on:

So I start out by stopping Apache to make it easier to find stuff in the log:

CMDLN: root@bunny:/etc/apache2# apache stop  
APWARN: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName  
ERRLOG: [Tue Feb 15 11:45:40 2011] [notice] caught SIGTERM, shutting down  

Then I start it:

CMDLIN: root@bunny:/etc/apache2# apache start
APWARN: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
ERRLOG: [Tue Feb 15 11:46:09 2011] [notice] Apache/2.2.14 (Ubuntu) proxy_html/3.0.1 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations

Then I restart it:

CMDLIN: root@bunny:/etc/apache2# apache restart
APWARN: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
ERRLOG: [Tue Feb 15 11:46:15 2011] [notice] SIGHUP received.  Attempting to restart

Then I change directories back one to test the theory it's using my shell path

CMDLIN: root@bunny:/etc/apache2# cd ..
CMDLIN: root@bunny:/etc# apache restart
APWARN: apache2: Syntax error on line 33 of /etc/apache2/apache2.conf: Include directory 'mods-enabled' not found
ERRLOG: apache2: Syntax error on line 33 of /etc/apache2/apache2.conf: Include directory 'mods-enabled' not found

Looks like it so far... kinda..

CMDLIN: root@bunny:/etc# apache restart
APWARN: apache2: Syntax error on line 33 of /etc/apache2/apache2.conf: Include directory 'mods-enabled' not found

Still looks like it, but no log error this time. Change back to the apache directory and try it again

CMDLIN: root@bunny:/etc# cd apache2/
CMDLIN: root@bunny:/etc/apache2# apache restart
APWARN: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
APWARN: httpd not running, trying to start
ERRLOG: [Tue Feb 15 11:46:30 2011] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
ERRLOG: [Tue Feb 15 11:46:30 2011] [notice] Apache/2.2.14 (Ubuntu) proxy_html/3.0.1 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations

Hmm, looks like that was it, right? As to why... no idea...

Another OK restart except for that DNS thing...

CMDLIN: root@bunny:/etc/apache2# apache restart
APWARN: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
ERRLOG: [Tue Feb 15 11:46:36 2011] [notice] SIGHUP received.  Attempting to restart

So far so good... wait... Oops, this shows up in the log while I haven't changed directories, while Apache is trying to start, but instead shuts down.

ERRLOG: apache2: Syntax error on line 33 of /etc/apache2/apache2.conf: Include directory 'mods-enabled' not found

I go to restart it and...

CMDLIN: root@bunny:/etc/apache2# apache restart
APWARN: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
APWARN: httpd not running, trying to start
ERRLOG: [Tue Feb 15 11:46:45 2011] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
ERRLOG: [Tue Feb 15 11:46:45 2011] [notice] Apache/2.2.14 (Ubuntu) proxy_html/3.0.1 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations

And it's back up. And I am bewildered.

I have no idea why this is happening, and it doesn't make sense it should be doing this. My instinct was telling me it was somehow not getting the right path and expecting mods-enabled for some reason to be relative to where I was, in my shell... and while that in and of itself would have been hairy, it got worse when Apache shut itself down right in the middle of running for this same "error"

Here's my httpd.conf:

<Location ~ "/ws/?.*$">
    SetHandler perl-script
    PerlResponseHandler Webservices::Qmedtrix
</Location>

<Directory /var/www>
    Options +ExecCGI +Includes
</Directory>

<Directory /usr/share>
    Options ExecCGI FollowSymlinks Indexes
</Directory>

ProxyPass /group http://localhost:8080/group
ProxyPassReverse /group http://localhost:8080/group

Here's my apache2.conf:

LockFile ${APACHE_LOCK_DIR}/accept.lock
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15

StartServers          2
MinSpareThreads      25
MaxSpareThreads      75
ThreadLimit          64
ThreadsPerChild      25
MaxClients          150
MaxRequestsPerChild   0

User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

AccessFileName .htaccess

<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy all
</Files>

DefaultType text/plain
HostnameLookups On
ErrorLog ${APACHE_LOG_DIR}/error.log

LogLevel warn

Include mods-enabled/*.load
Include mods-enabled/*.conf

Include httpd.conf
Include ports.conf

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Include conf.d/

Include sites-enabled/

And here's my default Virtual Domain:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DirectoryIndex index.shtml index.html

    DocumentRoot /var/www
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews +Includes +ExecCGI
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog /var/log/apache2/access.log combined

</VirtualHost>

and my envars file:

#unset HOME
# I have tried the above line enabled and not, no difference.

if [ "${APACHE_CONFDIR##/etc/apache2-}" != "${APACHE_CONFDIR}" ] ; then
        SUFFIX="-${APACHE_CONFDIR##/etc/apache2-}"
else
        SUFFIX=
fi

export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data
export APACHE_PID_FILE=/var/run/apache2$SUFFIX.pid
export APACHE_RUN_DIR=/var/run/apache2$SUFFIX
export APACHE_LOCK_DIR=/var/lock/apache2$SUFFIX
export APACHE_LOG_DIR=/var/log/apache2$SUFFIX
export LANG=C
export LANG

I'm stumped.

Any ideas what's going on here?

Answer 1

apachectl is a bash script. You might want to file a bug report for that behavior to Ubuntu BTS.

Answer 2

You should remove the lines in your /etc/envvars file:

#unset HOME
# I have tried the above line enabled and not, no difference.

if [ "${APACHE_CONFDIR##/etc/apache2-}" != "${APACHE_CONFDIR}" ] ; then
        SUFFIX="-${APACHE_CONFDIR##/etc/apache2-}"
else
        SUFFIX=
fi

Remove all of that.

Then also edit your envvars to remove your $suffix so that your end file looks something like this:

export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data
export APACHE_PID_FILE=/var/run/apache2.pid
export APACHE_RUN_DIR=/var/run/apache2
export APACHE_LOCK_DIR=/var/lock/apache2
export APACHE_LOG_DIR=/var/log/apache2
export LANG=C
export LANG

I had exactly the same problem as you did. And fixed it after playing with the envvars.

Answer 3

I had the exact same problem, also on a KVM virtual client. The solution for me was to uncomment the following line:

#ServerRoot "/etc/apache2"

Hope this can help the next guy who starts wondering why he has to run "apache2ctl start" in the "/etc/apache2" directory in order to succeed ;)