Publishing an ASP.NET MVC2 site with Web Deploy

Question

I currently use Web Deploy, http://learn.iis.net/page.aspx/346/web-deploy/ to publish my MVC2 app. It used to work well, but now it is got to the point where I can't continue using it:

When the MVC app was small and had only a few users it was easy to publish. Just right click the project in Visual Studio and choose "Publish". And because there were only a few users it was easy to find a time when no one was using the site to do a quick update.

Then the app got bigger and had a few more users. The "Publish" action started taking longer and longer and occasionally timing out. Even when I recycled the app pool before deploy it still took a long time.

Also it became harder to find a time when no one was using the site so the update could be done without affecting anyone.

Then the "Publish" action started timing out every single time, and I had to switch to manual deployment as per this earlier unanswered question: Visual Studio 2010 - web deploy times out - what to do?

Now the manual deploy is taking longer and longer, from 5 to 20 minutes. And the number of users has grown significantly, so the deployment always affects someone (slow response times, timeouts, site unavailable, etc)

So what can I do? Is there a better alternative to using web deploy?

Edit:

Today's deployment took 18 minutes to publish just 49 changed files. The situation is just ridiculous and is one of the biggest weaknesses of our site right now. So I'm starting a decent sized bounty in the hopes of solving this.

Some more questions that may lead to a solution:

• Why would it take so long when only a few files have been changed?
• Why does the web deploy zip always include the entire codebase and not just changed files?
• Why don't I just manually copy the changed files myself and skip the whole web deploy? But it is hard to manually work out what files have changed. I use SVN - does it have a way to output only files that have changed between two branches?
• What other questions should I be asking but haven't thought of yet?

In reply to answers:

Re: http://www.troyhunt.com/2010/11/you-deploying-it-wrong-teamcity_24.html This is exactly how I was doing the deploy, and would be an ideal method. Web deploy does correctly identify which files have changed, however it times out and no publish occurs. There are around 2500 files in the solution, perhaps it is taking too long to identify which ones are changed? Or it could be that publish has a short timeout value and that just uploading the 15mb zip file uses all that time up.

I do have full control over the server, and it does support web deploy. There are actually 2 servers: the primary live server, and a redundant server that we keep ready in case the first falls over. So any solution has to be easy to deploy to more than one server (web deploy was ideal until it stopped working).

The suggestion of creating a new folder for each release and then just changing IIS to point to that new folder sounds like it will result in lower downtime/slowtime when during the publish. But it is a very manual process and I would prefer something more automated.

Edit #2

I have managed to narrow it down, and found exactly where it is slow - but not why. This is from the deploy log:

[9/02/2011 12:11:56 a.m.] Performing synchronization pass #1.
[9/02/2011 12:11:56 a.m.] Parameter entry 'IIS Web Application Name/1' is applicable to 'iisApp/C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp' because of its scope.
[9/02/2011 12:11:56 a.m.] Parameter entry 'IIS Web Application Name/2' is applicable to 'setAcl/C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp' because of its scope.
[9/02/2011 12:11:56 a.m.] Parameter entry 'IIS Web Application Name/2' is applicable to 'setAcl/C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp' because of its scope.
[9/02/2011 12:11:56 a.m.] Parameter entry 'Add write permission to App_Data Folder/1' is applicable to 'setAcl/C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp\App_Data' because of its scope.
[9/02/2011 12:11:56 a.m.] Source createApp (C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp) does not match destination (Default Web Site/virtual-dir/) differing in attributes (isDest['False','True']). Update pending.
[9/02/2011 12:11:56 a.m.] Update operation on createApp (C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp) skipped because of rule CreateApplicationRule.
[9/02/2011 12:11:56 a.m.] Source filePath (C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp\App_Data\Create.sql) does not match destination (Default Web Site/virtual-dir/App_Data\Create.sql) differing in attributes (size['259691','259697'],lastWriteTime['02/08/2011 10:45:20','02/06/2011 03:48:16']). Update pending.

[400 lines of file updates skipped, time expired 2 seconds ....]

[9/02/2011 12:11:58 a.m.] Delete operation on filePath (Default Web Site/v2/zzz_app_offline.htm) skipped because of rule DoNotDeleteRule.
[9/02/2011 12:11:58 a.m.] Source setAcl (C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp) does not match destination (Default Web Site/virtual-dir/) differing in attributes (isDest['False','True'],setAclUser,setAclAccess). Update pending.
[9/02/2011 12:11:58 a.m.] Updating setAcl (Default Web Site/virtual-dir/).
[9/02/2011 12:13:47 a.m.] Source setAcl (C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp) does not match destination (Default Web Site/virtual-dir/) differing in attributes (isDest['False','True'],setAclUser,setAclAccess). Update pending.
[9/02/2011 12:13:47 a.m.] Updating setAcl (Default Web Site/virtual-dir/).
[9/02/2011 12:17:11 a.m.] Source setAcl (C:\src\Site.2010\Site.UI\obj\Release\Package\PackageTmp\App_Data) does not match destination (Default Web Site/virtual-dir//App_Data) differing in attributes (isDest['False','True'],setAclUser,setAclAccess). Update pending.
[9/02/2011 12:17:11 a.m.] Updating setAcl (Default Web Site/virtual-dir//App_Data).
[9/02/2011 12:17:11 a.m.] The dependency check 'DependencyCheckInUse' found no issues.
[9/02/2011 12:17:11 a.m.] The synchronization completed in 1 pass(es).

The cause of the slowness is the "Updating setAcl" component. I am examining the ACLs of the development box and server box to see what is different. However it seems like an extremely bad idea to copy the ACL from a dev box to a server box! I already had the ACL set up just fine on the server.

Answer 1

I'd start by trying to isolate where the timeout is happening. You've mentioned a 15MB zip with 2,500 files which doesn't strike me as particularly large. Have you tried creating a deployment package in Visual Studio then running it directly on the server? This will take network latency out of the picture which is a pretty fundamental variable when it comes to timeouts.

As for why a zip with the entire application needs to be uploaded, you need to remember the actual identification of what has changed and subsequent deployment into IIS all happens on the server. It's not Visual Studio or msdeploy on your local machine calling the shots on this.

As for why you don't just manually copy the changed files over, it's summarised in my blog post you've referenced but in short, it's laborious and error prone. It means you need to consciously work through the thought process of "which of my 2,500 files just changed" rather than simply saying "make my target site match my development version". You haven't mentioned if you're publishing the web.config or not but obviously config transforms is another important reason why the simple CTRL-C then CTRL-V approach is cumbersome.

Trying to just take a change directly from SVN is also risky. Your first problem is you need to have complete confidence in the integrity and accuracy of the revision you're updating from if you're to get the appropriate changes published. You're then left with trying to sync these to the target and you're back at the same issues raised in the previous paragraph. The other big problem is versioning object code is always nasty; you'll be in a perpetual state of conflict with anyone else on the project and VCS is simply not intended to function this way.

My advice would be to focus on solving the root cause of the problem - Web Deploy is timing out - rather than simply trying to work around the symptoms. Manually publishing changes only or messing around with IIS bindings is only going to create more trouble for you in the long run and a lot more work in the immediate term. See how you go sharing the results of creating a package, copying it to the server then executing it locally and we'll take it from there. Once you have it working as designed, you should be seeing deployments no more than a few minutes and site outage measured in seconds.

BTW - You might also like to add what sort of latency you have between your PC and the server and how long it would normally take to transfer a 15MB file over HTTP.

Answer 2

Some other options you could consider:

1) Deploy to a fresh directory each time and then switch between directories using IIS.

2) Use a separate Subversion repository for your binaries. svn-load-dirs.pl can load the binaries in to subversion and can correctly mark files that have been added, removed or changed. You can run svn load-dirs on the end of your automated build process. The build process is the only 'user' checking into this repository and the production server is the only place checking it out so there are no versioning conflicts.

To deploy you simply svn update the working directory from the binary subversion repository. It efficiently copies down just the changed files in an atomic fashion (i.e. if the download fails it doesn't replace any files). If you find a problem right after deployment you can quickly go back to the previous version. If you want to update a single aspx file you can do a targeted svn update on that single file.

If you have TortoiseSVN installed on the server you can also see immediately if anyone has changed any files on the server without going through the correct build->checkin->deploy path.

The only caveat is that subversion doesn't handle binary diffs so your repository will grow rapidly, but you can simply wipe it and start again occasionally if that ever becomes a problem.

Another plus is that you have a complete record of every version you've ever deployed.

I've used this technique for several projects and wish all deployment systems worked more like it: i.e. atomic update, easy to roll back, targeted single file update, complete version history, ...

Answer 3

If you have control over the server a really good option is to manually upload the zip file. Unzip it and then use IIS manager to point to the new codebase. This way the downtime should be minimal. And if something goes wrong, you have your last version intact and can just point IIS to that folder again.

On a shared host this won't work as well though. Maybe there is some way to adapt the same strategy by uploading the code and then rename folders to make it point to the new files.

Anyway it seems like web deploy should support sending only changed content. But I think you host need to support Web deploy in that case: http://www.troyhunt.com/2010/11/you-deploying-it-wrong-teamcity_24.html

If you don't have that I guess you could use a script to detect modifications in SVN. Here are some information about how to find changed files: http://blog.lysender.com/2010/11/svn-list-modified-files-between-revisions/ You would have to remember that code files are compiled into dll files so I would imagine something like this:

1. Publish the website(Or use the files from a staging server whatever makes most sense in your case)
2. Have the script build a list of files to modify(translating code files to their dll)
3. Use an ftp that can be controlled through the command line to push the changes.

Answer 4

On the build/dev box, use command-line MSBuild to build the project's SLN (or wdproj). Make sure to precompile everything. Use a separate output path that you clean before the build. Get the results zipped up, and transfer that to the web server out-of-band (via UNC path or FTP server, or whatever). On the server, unzip and do a xcopy deploy.

To minimize transfer time, use rsync (there are versions for Windows), or use 7-zip with maximum settings to zip binaries.

Server downtime is minimized since it would be only copying from local disk to local disk. Even though it's fast, the IIS app pool will recycle, to make up for that you'll need two machines behind a load balancer, so that you can update one while the other serves requests. (Maybe overkill, investigate using IIS's web-gardens)

To automate the process, use Powershell, or maybe even simpler, with batch files and PSExec for running remote commands.

Answer 5

The fact that you are deploying from VS and not from a build/continuous integration server is the problem here.

Answer 6

@JK from the info you provided it feels like a timeout issue. I agree with @TroyHunt 2500 files in 15 megs should delploy quickly. In particular the the output showing a delay while applying ACLs (whether or not they need to be changed). If it were me, I would start out doing some non-web deploy health checks. A few thoughts come to mind

• are the webservers in a domain or workgroup?
• what does dcdiag show ?
• what does netdiag show ?
• are the dev boxes and the prod boxes in the same domain?

Is there a chance you are trying to apply users or groups that don't exist any more, are disabled , or come from domains other than the web server domain? It's possible your organization has a domain hierarchy with child domains or domain trusts, that are valid, but communication is blocked in the production data center.

I think i would also manually look at the ACLs and see if there are entries that can't be resolved (they show up as SIDS before resolution).

HTH, -eric

Answer 7

Here's how I'm doing the publish:

1. I have a hook on teamcity and everytime I do a commit with svn it uses a rakefile to copy files to a dropbox folder where I also have a git repository.
2. Do a commit/push for the files that have been changed (for this git repository). I also have dropbox installed on the server
3. Pull changes from dropbox (with git) to a staging application to check things again.
4. Once everything is ok for the staging application, i switch it with the production one from iis
5. When I want to publish again, once everything is synchronized with dropbox, i do a pull again, on the previous production which now becomes the staging one, and I switch again the applications.

Result->Users get 0 seconds downtime.

If you want to cut some corners. You can do a git pull directly on the production site. Result->1 2 seconds downtime for users

If you really really want to cut some corners. You can install the dropbox folder directly into your production folder and dropbox will sync everything. Result 5 6 or more seconds downtime for users.

Answer 8

Why not use Dropbox?! Seriously....

WARNING: THIS HASN'T BEEN TESTED BY ME, JUST AN HYPOTHETICAL ANSWER

Solution 1: In a non-professional way, I would install Dropbox on all servers including the staging server. And just web deploy from Visual studio to the staging.

Dropbox syncs files very quickly especially when you enable "Network downloading" where files get downloaded from local servers instead from dropbox!

Solution 2: Create deployment package from Visual Studio and save it to a local folder that syncs with dropbox. Then create a scheduled task that auto-runs deploy.cmd and clears the content of the deployment folder once done to avoid re-deploying over and over.

Problems with using Dropbox

• ACL won't be synced :(
• Remote desktop session should be always active as dropbox runs in the tray (not a service)
• Folders should not contain any "data" files that gets changed or else conflicts will occur

Answer 9

I just had a similar issue where webdeploy began to take a very long time, especially when it got to "Updating setAcl". I was reluctant to disable updating the ACL as was suggested by some of the previous answers, especially when it worked fine deploying the same project to a different server.

So I looked at what was different between the two target machines and the solution was pretty obvious in hindsight. On the production machine we had a lot of temporary local files that were being created and stored for a month (by design). I reduced the number of files, and the publish time went from 30 minutes to about 15 seconds.