How can I optimize slow (not-so) complex queries in Entity Framework Core 2.1

Question

I have a LINQ query that makes string search within a few tables. The query however is painfully slow on big tables. At my first attempt, I was getting a timeout. I was able to improve the performance a little. This is the first version of the code:

 public ListResponse<UserDTO> GetUsers(FilterParameters filter)
 {
     var query = from user in _dbContext.Users
                    .Include(w => w.UserRoles).ThenInclude(u => u.Role)
                 join accountHolder in _dbContext.AccountHolders
                    .Include(c => c.OperationCountry)
                    .Include(x => x.Accounts)
                         .ThenInclude(x => x.Currency)
                  on user.Id equals accountHolder.ObjectId into aHolder
                  
                  from a in aHolder.DefaultIfEmpty()
                  select new UserDTO
                    {
                        Id = user.Id,
                        FirstName = user.FirstName,
                        LastName = user.LastName,
                        Username = user.UserName,
                        Email = user.Email,
                        Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
                        LastActivity = user.LastActivity,
                        CreatedAt = user.CreatedAt,
                        EmailConfirmed = user.EmailConfirmed,
                        AccountBalance = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().Balance,
                        AccountReference = a.Accounts.Where(p => p.CurrencyId == a.OperationCountry.LocalCurrencyId).Single().AccountRef
                    };

        // Apply search term
        if (!IsNullOrEmpty(filter.SearchTerm))
            query = query.Where(w =>
                w.FirstName.Contains(filter.SearchTerm) 
                w.LastName.Contains(filter.SearchTerm) ||
                w.Email.Contains(filter.SearchTerm) ||
                w.AccountReference.Contains(filter.SearchTerm));

        if (filter.ColumnFilters != null)
        {
            if (filter.ColumnFilters.ContainsKey("EmailConfirmed"))
            {
                var valueStr = filter.ColumnFilters["EmailConfirmed"];
                if (bool.TryParse(valueStr, out var value))
                    query = query.Where(x => x.EmailConfirmed == value);
            }
        }

        // Get total item count before pagination
        var totalItemCount = query.Count();
        // Apply pagination
        query = query.ApplySortAndPagination(filter);
        var userDtoList = query.ToList();

        return new ListResponse<UserDTO>()
        {
            List = userDtoList,
            TotalCount = totalItemCount
        };
 }

I suspected non-database code in the query (such as Single, and Mapping) was causing a slow query so I made an effort to get rid of them. I am still curious how to get a single Account without calling Single() inside the query. Here's the modified version.

public ListResponse<UserDTO> GetUsers(FilterParameters filter) 
{
    var query = from user in _dbContext.Users
                      .Include(w => w.UserRoles)
                         .ThenInclude(u => u.Role)
                      .Include(w => w.AccountHolder)
                         .ThenInclude(c => c.OperationCountry)
                      .Include(w => w.AccountHolder)
                         .ThenInclude(c => c.Accounts)
                         .ThenInclude(x => x.Currency)
                       select user;

     if (!IsNullOrEmpty(filter.SearchTerm)) 
     {
        query = query.Where(w =>
            w.FirstName.StartsWith(filter.SearchTerm) || 
            w.LastName.StartsWith(filter.SearchTerm) || 
            w.UserName.StartsWith(filter.SearchTerm) || 
            w.AccountHolder.Accounts.Any(x => x.AccountRef.StartsWith(filter.SearchTerm))); 
     }

     // total before pagination
     var totalItemCount = query.Count();

     // Nothing fancy, just OrderBy(filter.OrderBy).Skip(filter.Page).Take(filter.Length)
     query = query.ApplySortAndPagination(filter);  
     
     userList = query.ToList()  //To deal with "Single" calls below, this returns at most filter.Length records
     var userDtoResult = (from user in query
                          select new UserDTO
                          {
                              Id = user.Id,
                              FirstName = user.FirstName,
                              LastName = user.LastName,
                              Username = user.UserName,
                              Email = user.Email,
                              Roles = Mapper.Map<IList<RoleDTO>>(user.UserRoles.Select(i => i.Role)),
                              LastActivity = user.LastActivity,
                              CreatedAt = user.CreatedAt,
                              EmailConfirmed = user.EmailConfirmed,
                              AccountBalance = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).Balance
                              AccountReference = user.AccountHolder.Accounts.Single(p => p.CurrencyId == user.AccountHolder.OperationCountry.LocalCurrencyId).AccountRef
                              }).ToList();

            return new ListResponse<UserDTO>()
            {
                List = userDtoResult,
                TotalCount = totalItemCount
            };
}

The SQL query generated by this query runs slow too, whereas if I write a join query in SQL, it completes in a few hundred milliseconds. I am suspecting I am suffering from N+1 Query problem, but not sure since EF seems to generate a single query when I trace in the SQL Server Profiler.

This is the query generated by the Entity framework and runs in about 8 seconds when I run on the SSMS:

exec sp_executesql N'SELECT TOP(@__p_4) [w].[Id], [w].[AccessFailedCount], [w].[ConcurrencyStamp], [w].[CreatedAt], [w].[CreatedBy], [w].[DeletedAt], [w].[DeletedBy], [w].[DetailId], [w].[Email], [w].[EmailConfirmed], [w].[EmailConfirmedAt], [w].[FacebookId], [w].[FirstName], [w].[GoogleId], [w].[IsActive], [w].[IsDeleted], [w].[LastActivity], [w].[LastName], [w].[LockoutEnabled], [w].[LockoutEnd], [w].[NormalizedEmail], [w].[NormalizedUserName], [w].[Password], [w].[PasswordHash], [w].[PhoneNumber], [w].[PhoneNumberConfirmed], [w].[RoleId], [w].[SecurityStamp], [w].[TwoFactorEnabled], [w].[UpdatedAt], [w].[UpdatedBy], [w].[UserName], [w].[WorkflowId], [t].[Id], [t].[AccountHolderLevel], [t].[AccountHolderType], [t].[CreatedAt], [t].[CreatedBy], [t].[DeletedAt], [t].[DeletedBy], [t].[IsDeleted], [t].[ObjectId], [t].[OperationCountryId], [t].[UpdatedAt], [t].[UpdatedBy], [t0].[Id], [t0].[ContinentId], [t0].[CountryCode], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[DeletedAt], [t0].[DeletedBy], [t0].[ISOCode2], [t0].[IsActive], [t0].[IsDeleted], [t0].[IsOperational], [t0].[LocalCurrencyId], [t0].[Name], [t0].[PhoneCode], [t0].[PostCodeProvider], [t0].[Regex], [t0].[SmsProvider], [t0].[UpdatedAt], [t0].[UpdatedBy]
FROM [Users] AS [w]
LEFT JOIN (
    SELECT [a].[Id], [a].[AccountHolderLevel], [a].[AccountHolderType], [a].[CreatedAt], [a].[CreatedBy], [a].[DeletedAt], [a].[DeletedBy], [a].[IsDeleted], [a].[ObjectId], [a].[OperationCountryId], [a].[UpdatedAt], [a].[UpdatedBy]
    FROM [AccountHolders] AS [a]
    WHERE [a].[IsDeleted] = 0
) AS [t] ON [w].[Id] = [t].[ObjectId]
LEFT JOIN (
    SELECT [c].[Id], [c].[ContinentId], [c].[CountryCode], [c].[CreatedAt], [c].[CreatedBy], [c].[DeletedAt], [c].[DeletedBy], [c].[ISOCode2], [c].[IsActive], [c].[IsDeleted], [c].[IsOperational], [c].[LocalCurrencyId], [c].[Name], [c].[PhoneCode], [c].[PostCodeProvider], [c].[Regex], [c].[SmsProvider], [c].[UpdatedAt], [c].[UpdatedBy]
    FROM [Countries] AS [c]
    WHERE [c].[IsDeleted] = 0
) AS [t0] ON [t].[OperationCountryId] = [t0].[Id]
WHERE ([w].[IsDeleted] = 0) AND ((((([w].[FirstName] LIKE @__filter_SearchTerm_0 + N''%'' AND (LEFT([w].[FirstName], LEN(@__filter_SearchTerm_0)) = @__filter_SearchTerm_0)) OR (@__filter_SearchTerm_0 = N'''')) OR (([w].[LastName] LIKE @__filter_SearchTerm_1 + N''%'' AND (LEFT([w].[LastName], LEN(@__filter_SearchTerm_1)) = @__filter_SearchTerm_1)) OR (@__filter_SearchTerm_1 = N''''))) OR (([w].[UserName] LIKE @__filter_SearchTerm_2 + N''%'' AND (LEFT([w].[UserName], LEN(@__filter_SearchTerm_2)) = @__filter_SearchTerm_2)) OR (@__filter_SearchTerm_2 = N''''))) OR EXISTS (
    SELECT 1
    FROM [Accounts] AS [x]
    WHERE (([x].[IsDeleted] = 0) AND (([x].[AccountRef] LIKE @__filter_SearchTerm_3 + N''%'' AND (LEFT([x].[AccountRef], LEN(@__filter_SearchTerm_3)) = @__filter_SearchTerm_3)) OR (@__filter_SearchTerm_3 = N''''))) AND ([t].[Id] = [x].[AccountHolderId])))
ORDER BY [w].[LastActivity] DESC, [w].[Id], [t].[Id]',N'@__p_4 int,@__filter_SearchTerm_0 nvarchar(100),@__filter_SearchTerm_1 nvarchar(100),@__filter_SearchTerm_2 nvarchar(256),@__filter_SearchTerm_3 nvarchar(450)',@__p_4=10,@__filter_SearchTerm_0=N'james',@__filter_SearchTerm_1=N'james',@__filter_SearchTerm_2=N'james',@__filter_SearchTerm_3=N'james'

Finally this is my SQL query that returns whatever is necessary in less than 100 ms:

declare @searchTerm varchar(100) = '%james%'
select top 10 
    u.Id,
    u.UserName,
    u.FirstName,
    u.LastName,
    u.LastActivity,
    u.CreatedAt,
    a.Balance,
    a.AccountRef,
    ah.AccountHolderLevel,
    u.Email,
    r.Name
from Users u
join AccountHolders ah on ah.ObjectId = u.Id
join Accounts a on ah.Id = a.AccountHolderId
join UserRoles ur on ur.UserId = u.Id
join Roles r on r.Id = ur.RoleId
where FirstName like @searchTerm or LastName like @searchTerm  or u.UserName like @searchTerm or FirstName + ' ' + LastName like @searchTerm or a.AccountRef like @searchTerm
and a.CurrencyId = ah.OperationCountryId

The columns I am searching are all indexed by the way, so that's not a problem. I know that the new EF-Core has many performance improvements. Unfortunately, I cannot update due to sheer number of breaking changes.

I am not sure splitting query into 2 (one for users and one for account) would work well, because there will be joins all over again. If I cannot find a solution using I plan converting my query to a view, but I want to do it as a last resort, since our convention is to use EF as much as possible. And I refuse to believe that EF does not have a solution. This is not actually a complex query at all and I am sure a fairly common use case.

So, what is the best way to optimize this query using EF-Core?

Answer 1

So, what is the best way to optimize this query using EF-Core?

Many things have changed in EF Core query pipeline since 2.1 (3.0, 3.1, 5.0 and now working on 6.0), but some general rules can be used, with the goal of getting rid of the client side query evaluation (which starting with 3.0 is not supported at all, so it's good to start preparing for the switch - support for 2.1 ends August this year).

The first would be to remove all these Include / ThenInclude. If the query is projecting the result in DTO without involving entity instances, then all these are redundant/not needed and removing them will ensure the query gets fully translated to SQL.

var query = _dbContext.Users.AsQueryable();
// Apply filters...

The next is the Roles collection. You must remove Mapper.Map call, otherwise it can't be translated. In general either use AutoMapper mappings and ProjectTo to fully handle the projection, or not use it at all (never put Map method calls inside query expression tree). According to your SQL, it should be something like this

Roles = user.UserRoles.Select(ur => ur.Role)
    .Select(r => new RoleDTO { Name = r.Name })
    .ToList(),

Actually EF Core will execute this as separate query (a behavior broken by "single query mode" in 3.x, and brought back optionally with 6.0 "split query mode"), so it is is important to have ToList() call at the end, otherwise you'll get N + 1 queries rather than 2.

Finally, the Single() call. It can be avoided by flattening the sub collection using correlated SelectMany, or its query syntax equivalent

from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId

The let statement is not mandatory, I've added it just for readability. Now you can use the range variables user, ah and a in the final select similar to table aliases in SQL.

Also since your SQL query doesn't really enforce single account match, there is no such enforcement in the LINQ query as well. If it was needed, then the equivalent of the Single can be achieved with SelectMany + Where + `Take(1), e.g.

from user in query
let ah = user.AccountHolder
from a in ah.Accounts
    .Where(a => a.CurrencyId == ah.OperationCountryId)
    .Take(1)

(a mixture of query and method syntax, but LINQ allows that)

So the final query would be something like this

from user in query
let ah = user.AccountHolder
from a in ah.Accounts
where a.CurrencyId == ah.OperationCountryId
select new //UserDTO
{
    Id = user.Id,
    FirstName = user.FirstName,
    LastName = user.LastName,
    Username = user.UserName,
    Email = user.Email,
    Roles = user.UserRoles.Select(ur => ur.Role)
       .Select(r => new RoleDTO { Name = r.Name })
       .ToList(),
    LastActivity = user.LastActivity,
    CreatedAt = user.CreatedAt,
    EmailConfirmed = user.EmailConfirmed,
    AccountBalance = a.Balance,
    AccountReference = a.AccountRef
}

and should translate to very similar to the handcrafted SQL. And hopefully execute faster similar to it.