Why isn't IIS cleaning up the old worker processes (w3wp.exe) on pool recycle leading to website out of memory exception?

Question

I have an asp.net-mvc site and recently I am getting an out of memory exceptions on my web server. I only have 1 application pool and we recent set IIS to recycle after it hits a certain limit. I went in the other day and saw 4 w3wp.exe processes running (each with ~1.8GB memory being used)

I assume that during the recycle process, it's not killing the old worker process and eventually I get out of memory exceptions on my website because the box only has 8GB memory. I can add memory to the box but I am concerned why these old processes are not being cleaned up.

Are there any recommendations to figure out why this recycle process is not killing the old w3wp.exe processes and leaving them running? Any suggestions around understand both root cause or even workarounds to avoid this risk going forward?

Answer 1

I had similar issue when I was running things like FFMpeg.exe or some PDF conversion with WPF graphics, IIS process won't shutdown and would issue memory not found errors. The problem is not with IIS, but some deadlocks in the process which blocks even after crashing.

Workaround is, divide your website in to two separate websites, one should only do transaction processing with database which usually does not have crashes. The logic like video/photo conversion, PDF conversion, or any other logic that may cause crash should be moved to other web service. And use HTTP call from your website internally to process them over web services.

Now, in this case, there is still no way to get around web service process crashing, so I decided to recycle application pool worker every 100 requests (I chose this number after watching few requests, on an average it would go beyond 1GB only after hitting 200 requests) and I turned application pool into Web Garden by making 4 process per pool.

Advantage of this setup is, you can move your web service process to other machines easily in future, you can increase/decrease number of processes per pool. And your main website, which is simply doing transaction process becomes highly responsive as it is not affected by process recycles of web services.

Answer 2

Application pool recycle usually shuts down old worker processes, so I don't think you hit a bug of IIS.

Note that processes can live longer if they become orphaned, and based on configuration IIS must leave them there for you to troubleshoot,

https://www.iis.net/configreference/system.applicationhost/applicationpools/add/failure

If you are not quite familiar with debugging such processes, I suggest you open a support case via http://support.microsoft.com and let Microsoft support guys help you out.

Answer 3

Right-click the "old" w3wp process in Task Manager, select "Create dump", then explore that dump in Visual Studio (just launch an empty Visual Studio and drag and drop the DMP file onto it).

Then you understand exactly what's going on, without guessing and fortune-telling.

You can click "debug managed code" and see what the process is doing. Is there any async tasks waiting or blocked? Any background tasks that may have cancelled the graceful shutdown via HostedService? What about "parallel stacks" view? How many threads are launched? Which modules are loaded?

P.S. And yes, it is normal for IIS to keep the old process around for 2-5 minutes after recycling.

Answer 4

The primary suspect you should go after are classes that implement IDisposable and any use of them that doesn't call Dispose(). I had a similar issue, which I have already posted as a reply to this question.

The most likely cause is that you are not calling .Dispose() on database connections. People tend to get confused about this because .NET's built-in connection pool makes it sound like you should not dispose connections. However, calling .Dispose() at the end of the request (or when you are finished using the connection) is exactly what you should do to prevent resource leaks. Connection pooling expects this to happen.

I don't think that 4 w3wp.exe processes is much of a concern - it is normal for IIS to spawn multiple processes. But it is clear that your application has resource leaks that need to be addressed. Start by looking at IDisposable as I mentioned above. If you still have issues, take a look at items you have in the cache and try to determine if there is a more efficient caching strategy that you could use. If all else fails, profile the application to see if you can locate the source of the resource leak. Most likely your application should be implementing IDisposable somewhere it is not.

Answer 5

As It is clear from you post that seting IIS to recycle after it hits a certain limit is creating a new application pool.

There can be multiple reasons for older not getting terminated.

One can be memory leak, not disposing off un-managed resources, you can look for this.

To find out another reason, Enable "Process Orphaning" in IIS. To find out the culprit process you can use "Process Explorer".

You also have an option to set an Executable, this will be executed when a Process is orphaned/ abandoned.

Immediate solution also may be to set action to KillW3p after a certain CPU limit.

<applicationPools>
   <add name="DefaultAppPool">
     <cpu limit="80000" action="KillW3wp" resetInterval="00:02:00" />
   </add>
</applicationPools>