.ToList on IQueryable causes NullReferenceException when executed in background

Question

My code works fine when executed synchronously, but fails when triggered via QueueBackgroundWorkItem.

UPDATE: the issue seems to be with the Sitefinity dynamic module manager I am using requiring HTTP context within the thread created by QueueBackgroundWorkItem, so this is a Sitefinity specific question. I've found an article that seems to point to a solution: http://www.sitefinity.com/developer-network/forums/sitefinity-sdk/errors-with-managers-when-multi-threading

Relevant portions of my code

ApiController action that starts background job:

// GET: api/Sync/MFP
[HttpGet]
public IHttpActionResult MFP()
{
    HostingEnvironment.QueueBackgroundWorkItem(ct => startMFPSync());
    return Ok("Sync Started!");
}

private void startMFPSync()
{
    var sourcesConfig = Config.Get<SyncSourcesSettingsConfig>();

    MFPApi api = new MFPApi(new MFPConfig
    {
        Url = sourcesConfig.MFPUrl,
        Key = sourcesConfig.MFPKey,
        Password = sourcesConfig.MFPPassword,
    });

    DataSync dataSync = new DataSync(api);
    dataSync.RunSync();
}

DataSync class:

public class DataSync
{
    IDataSource dataSource;

    public DataSync(IDataSource source)
    {
        dataSource = source;
    }

    public void RunSync()
    {
        dataSource.GetResponse();

        List<SyncContent> dataToSync = dataSource.GetDataForSync();
        SFDynamicModuleSync destinationSync = new SFDynamicModuleSync(dataToSync);

        // function call where exception occurs
        destinationSync.CacheModuleData();

        // other sync operations
    }
}

CacheModuleData() function in class below is where exception occurs:

public class SFDynamicModuleSync : IDataDestinationSync
{
    /// <summary>
    /// List of SyncContent objects to sync
    /// </summary>
    private List<SyncContent> dataToSync;

    /// <summary>
    /// Used to store results of CacheModuleData
    /// </summary>
    private List<List<DynamicContent>> modulesItems;

    /// <summary>
    /// Sitefinity dynamic module manager
    /// </summary>
    private DynamicModuleManager dynamicModuleManager;

    /// <summary>
    /// Initializes a new instance of the <see cref="SFDynamicModuleSync"/> class
    /// </summary>
    /// <param name="dataToSync">List of SyncContent objects to sync</param>
    public SFDynamicModuleSync(List<SyncContent> dataToSync)
    {
        this.dataToSync = dataToSync;
        this.modulesItems = new List<List<DynamicContent>>();
        this.dynamicModuleManager = DynamicModuleManager.GetManager();
    }

    /// <summary>
    /// Retrieves all data from dynamic modules and places in modulesItems
    /// </summary>
    public void CacheModuleData()
    {
        foreach (string contentType in this.dataToSync.Select(e => e.ContentTypeName))
        {
            Type type = TypeResolutionService.ResolveType(contentType);

            IQueryable<DynamicContent> moduleItems = this.dynamicModuleManager.GetDataItems(type)
                .Where(i => i.Status == ContentLifecycleStatus.Master);

            if(moduleItems != null)
            {
                // The .ToList() here causes a NullReferenceException when code is triggered by background job
                List<DynamicContent> moduleItemsList = moduleItems.ToList();
                this.modulesItems.Add(moduleItemsList);
            }
        }
    }

    // other sync methods - not included here for abbrevity 
}

Stack trace:

System.NullReferenceException was unhandled by user code
  HResult=-2147467261
  Message=Object reference not set to an instance of an object.
  Source=Unity_ILEmit_DynamicClasses
  StackTrace:
       at DynamicModule.ns.Wrapped_OpenAccessDynamicModuleProvider_81d3fcbe95dd4a47b8c1cb1cc5a692ab.ApplyFilters(IDataItem item)
       at Telerik.Sitefinity.Security.FieldsPermissionsApplierEnumerator`1.Demand(T forItem)
       at Telerik.Sitefinity.Security.PermissionApplierEnumeratorBase`1.MoveNext()
       at Telerik.Sitefinity.Data.Linq.DataItemEnumerator`1.MoveNext()
       at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
       at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
       at TeamSI.Sitefinity.DataSync.DataDestinations.SFDynamicModuleSync.CacheModuleData() in C:\Projects\SIEQ\TeamSI.Sitefinity.DataSync\DataDestinations\SFDynamicModuleSync.cs:line 75
       at TeamSI.Sitefinity.DataSync.DataSync.RunSync() in C:\Projects\SIEQ\TeamSI.Sitefinity.DataSync\DataSync.cs:line 28
       at SitefinityWebApp.Mvc.Controllers.SyncController.startMFPSync() in C:\Projects\SIEQ\Main_Site\Mvc\Controllers\SyncController.cs:line 72
       at SitefinityWebApp.Mvc.Controllers.SyncController.<MFP>b__1_0(CancellationToken ct) in C:\Projects\SIEQ\Main_Site\Mvc\Controllers\SyncController.cs:line 52
       at System.Web.Hosting.HostingEnvironment.<>c__DisplayClass91_0.<QueueBackgroundWorkItem>b__0(CancellationToken ct)
       at System.Web.Hosting.BackgroundWorkScheduler.<RunWorkItemImpl>d__7.MoveNext()
  InnerException: 

Answer 1

Turns out this was an issue with the Sitefinity CMS I was querying data from requiring HTTP context, which is of course lost within threads. The NullReferenceException was occuring within the dynamicModuleManager.GetDataItems() method.

With some help from friends and awesome SO contributors including @Igor, @Hogan, @PeterBons and @daramasala who helped me understand the problem I was able to solve it by upping privileges to SF and simulating HttpContext within the thread.

http://www.sitefinity.com/developer-network/forums/sitefinity-sdk/errors-with-managers-when-multi-threading

Only SF documentation outside of the forums I could find on using this method is here: http://docs.sitefinity.com/bug-tracker-create-the-savebug-action

Updated RunSync() method in my DataSync class:

public void RunSync()
{
    SystemManager.RunWithElevatedPrivilegeDelegate worker = new SystemManager.RunWithElevatedPrivilegeDelegate(args => {

        dataSource.GetResponse();
        List<SyncContent> dataToSync = dataSource.GetDataForSync();
        var destinationSync = new SFDynamicModuleSync(dataToSync);

        destinationSync.CacheModuleData();

        // complete sync operations for each content type
        for (int i = 0; i < dataToSync.Count; i++)
        {
            destinationSync.DeleteOldItems(i);
            destinationSync.AddItems(i);
            destinationSync.UpdateItems(i);
        }
    });

    SystemManager.RunWithElevatedPrivilege(worker);
}