how to improve the performance of Apache with mod_wsgi?

Question

Use Apache/2.4.12(Unix) and mod_wsgi-4.4.11 and blow configuration of apache/conf/extra:

//httpd-mpm.conf
<IfModule mpm_worker_module>
    StartServers             3
    MinSpareThreads          75
    MaxSpareThreads          250 
    ThreadsPerChild          25
    MaxRequestWorkers        400
    MaxConnectionsPerChild   0
</IfModule>


//httpd-vhosts.conf
WSGIRestrictEmbedded On
<VirtualHost *:443>
    ServerName form.xxx.com

    WSGIScriptAlias / /usr/local/apache/services/form/form.wsgi 
    WSGIDaemonProcess paymentform user=test processes=10 threads=5 display-name=%{GROUP} maximum-requests=100

    WSGIApplicationGroup %{RESOURCE}
    WSGIProcessGroup form
    DocumentRoot /usr/local/apache/services/form

    SSLEngine On
    //any certification files  
   <Directory /usr/local/apache/services/form>
      Require all granted
   </Directory>
</VirtualHost>

In this configuration, I use Apache jmeter for testing.

GET : form.xxx.com //only return "index" string
Number of Threads(users):100  
Ramp-up Period : 0
Loop count : 10 

But result is..

samples: 1000
Average: 3069   
Min    : 13
Max    : 22426  
Std.Dev: 6671.693614549157
Error %: 10.0% 
Throughput : 24.1/sec
KB/sec : 10.06/sec
AvgBytes : 428.5

During testing, raise connection refused or connection timeout and stop receving requests in 400~500 requests. Server cpu or memory is not full.

How to improve performance? fix mpm worker configuration? or fix WSGI configuration in httpd-vhosts?

I modify httpd-mpm.conf below, but no difference.

<IfModule mpm_worker_module>
    StartServers             10
    ServerLimit              32
    MinSpareThreads          75
    MaxSpareThreads          250 
    ThreadsPerChild          25
    MaxRequestWorkers        800
    MaxConnectionsPerChild   0
</IfModule>

Answer 1

You have a number of things which are wrong in your configuration. One may be a cut and paste error. Another is a potential security issue. And one will badly affect performance.

The first is that you have:

WSGIProcessGroup form

If that is really want you have, then the web request wouldn't even be getting to the WSGI application and should return a 500 error response. If it isn't giving an error, then your request is being delegated to a mod_wsgi daemon process group not even mentioned in the above configuration. This would all come about as the value to WSGIProcessGroup doesn't match the name of the defined daemon process group specified by the WSGIDaemonProcess directive.

What you would have to have is:

WSGIProcessGroup paymentform

I suspect you have simply mucked up the configuration when you pasted it in to the question.

A related issue with delegation is that you have:

WSGIApplicationGroup %{RESOURCE}

This is what the default is anyway. There would usually never be a need to set it explicitly. What one would normally use if only delegating one WSGI application to a daemon process group is:

WSGIApplicationGroup %{GLOBAL}

This particular value forces the use of the main Python interpreter context of each process which avoids problems with some third party extension modules that will not work properly in sub interpreter contexts.

The second issue is a potential security issue. You have:

DocumentRoot /usr/local/apache/services/form

When using WSGIScriptAlias directive, there is no need to set DocumentRoot to be a parent directory of where your WSGI script file or source code for your application is.

The danger in doing this is that if WSGIScriptAlias was accidentally disabled, or changed to a sub URL, all your source code then becomes downloadable.

In short, let DocumentRoot default to the empty default directory for the whole server, or create an empty directory just for the VirtualHost and set it to that.

The final thing and which would drastically affect your performance is the use of maximum-requests option to WSGIDaemonProcess. You should never use maximum-requests in a production system unless you understand the implications and have a specific temporary need.

Setting this value and to a low value, means that the daemon processes will be killed off and restarted every 100 requests. Under a high volume of requests as with a benchmark, you would be constantly restarting your application processes.

The result of this would be increased CPU load and much slower response times, with potential for backlogging to the extent of very long response times due to overloading the server due to everything restarting all the time.

So, absolute first thing you should do is remove maximum-requests and you should see some immediate improvement.

You also have issues with process restarts in your Apache MPM settings. It is not as major as this only affects the Apache worker processes which are proxying requests, but it will also cause extra CPU usage, plus a potential need for a higher number of worker processes being required.

I have talked about the issue of Apache process churn due to MPM settings before in:

• http://lanyrd.com/2013/pycon/scdyzk/

One final problem with your benchmarking is that your test, if all it is returning is the 'index' string from some simple hello world type program, is that it bears no relationship to your real world application.

Real applications are not usually so simple and time within the WSGI application is going to be much more due to template rendering, database access etc etc. This means the performance profile of a real application is going to be completely different and changes how you should configure the server.

In other words, testing with a hello world program is going to give you the completely wrong idea of what you need to do to configure the server appropriately. You really need to understand what the real performance profile of your application is under normal traffic loads and work from there. That is, hammering the server to the point of breaking is also wrong and not realistic.

I have been blogging on my blog site recently about how typical hello world tests people use are wrong, and give some examples of specific tests which show out how the performance of different WSGI servers and configurations can be markedly different. The point of that is to show that you can't base things off one simple test and you do need to understand what your WSGI application is doing.

In all of this, to really truly understand what is going on and how to tune the server properly, you need to use a performance monitoring solution which is built into the WSGI server and so can give insights into the different aspects of how it works and therefore what knobs to adjust. The blog posts are covering this also.

Answer 2

I encountered a similar problem as ash84 described, I used jmeter to test the performance and found the error % becomes non-zero when the jmeter thread number is set beyond some value (50 in my case).

After I watched Graham Dumpleton's talk, I realized it happens mainly because there are not enough spare MPM threads prepared for serving the upcoming burst jmeter requests. In this case, some jmeter requests are not served in the beginning, even though the number of MPM threads catched up later.

In short, setting MinSpareThreads to a larger value fixed my problem, I raised jmeter threads from 50 to 100 and get 0% error.

    MinSpareThreads          120
    MaxSpareThreads          150 
    MaxRequestWorkers        200

The number of WSGIDaemonProcess processes times the number of WSGIDaemonProcess threads doesn't have to be greater than the number of jmeter threads. But you may need to set them to higher values to make sure WSGIDaemonProcess could handle the requests quickly enough.