Why is my work item migration taking so long?

Question

I have an Azure Devops 2019 project with 2200 workitems, I expected this to take less then a few hours. However, it took the migration tool over 3 days to complete. Is this a normal migration duration or could there be something wrong with either our setup or maybe with the tool?

I have found there are some gaps in the logging where it doesnt seem to do anything

10:33:35 WotrkItemQuery: keyToFiund: XXX
10:33:55 [Test Case] [Complete: 1606/2215] work item has 2 revisions and revision migration is set to True
10:44:16: [Test Case] [Complete 1606/2215] Found 2 revisions to migrate on Work Item:XXX

Answer 1

The length of migration for each work item depends on:

1. Number of revisions
2. Number of Attachments
3. Number of Links

It's not uncommon for 2k work items to turn into 20k saves, or even 200k if you have rollup addons for your TFS. I have seen it take 3 seconds per work item, and also 30 seconds.

In addition, when Azure DevOps detects that you are making a lot of calls it may actively introduce Rate Limits to the process to protect the services of other users. This is normal, and there is no way around it, nor should there be. Your migration should never take president over production users. You can see your usage and any rate limits applied by going to /_settings/usage on your Org.

We have added a few features since the tool was launched in 2016 to help speed up the process:

• We now do the work item, attachment, and links in a single pass; this reduces the load on the Azure DevOps services and reduces the frequency and severity of rate limits.
• We have also added the ability to resume a migration, which also enables the ability to bring across new changes from the source system.
• We have added FilterWorkItemsThatAlreadyExistInTarget to minimise the number of items that are processed, but not that this prevents the changes being brought across.

Options for Speeding it up:

1. Only take the Tip - You can turn of revisions so that it only does 1 state of a work item, the latest. You can set ReplayRevisions to false to only write the latest, and you can use CollapseRevisions set to true to attach a JSON with the history.
2. Run closer to Target - All of the long calls are against the target with only a single query against the Source. I usually try to run the migratory as close to the target as possible, so running on an AzureVM in the same data centre as the target is a good call. However, you may be limited if your source is not accessible.
3. Scoped Query - If I am doing a large migration I will often scope the query for a first run only to those work items that are currently in scope for the teams that I need to get up and running soonest. Maybe only non-closed work items that have been edited in the last 60 days. Then I would increase the scope on subsequent runs.