How to optimize a big query?

Question

I have a table with 10000 users and there is a relationship with work_hours table.

This is my User Model and how I get sum of work hours.

protected $appends = ['sum_work_hours'];

public function work_hours()
{
    return $this->hasMany(WorkHour::class);
}

public function getSumWorkHoursAttribute()
{
    return $this->work_hours->sum('hours_total');
}

And I made a query to show all users and their sum of work hours like this:

public function getHoursDatatable()
{
    $users = User::with('work_hours')->get();

    return Datatables::of($users)->make();
}

And this query takes about 7-8 seconds to show the table which is very slow.

I know that this is because of summing up the total work hours from work_hours table for each user and I want to learn how to optimize this query if this is even possible considering the amount of data?

Update:

    $users = User::with('work_hours')->select(['users.name', 'users.email']);

    return Datatables::of($users)
        ->add_column('sum_work_hours', function(User $user) {
            return $user->work_hours->sum('hours_total');
        })
        ->make();

This gives me an error:

DataTables warning: table id=users-table - Ajax error. For more information about this error, please see http://datatables.net/tn/7

Update 2

This works:

public function getHoursDatatable()
{
    $users = User::with('work_hours');

    return Datatables::of($users)
        ->addColumn('sum_work_hours', function(User $user) {
            return $user->work_hours->sum('hours_total');
        })
        ->make();
}

But also takes about 7-8 seconds to load...

Answer 1

At some point your data will just get too big to query everything at once.

Basically you have 2 options:

1. Since you already have 10K users (and that number might still grow?) I would suggest switching to async (serverSide: true option). You are using datatables, so obviously you do not need all 10K users in 1 page, nor do you need all of them for a single report.

Datatables do allow serverside processing and it does allow pagination, while keeping the options to filter and sort like you're probably doing right now.

The yajra package will keep doing what it is doing right now, the thing you will have to change is that you do not assign all users to the view and build it like that, you'll need an emtpy table and let the frontend populate it. Just create an api endpoint where you can point your datatable to, in order to fetch the paginated list from it.

So instead of using @foreach ($users as $user) in your view, do not fetch it in the controller action that displays the table, let the api endpoint handle it.

Example:

<table id="example" class="display" cellspacing="0" width="100%">
    <thead>
        <tr>
            <th>First name</th>
            <th>Last name</th>
            <th>Total Hours</th>
        </tr>
    </thead>
    <tfoot>
        <tr>
            <th>First name</th>
            <th>Last name</th>
            <th>Total Hours</th>
        </tr>
    </tfoot>
</table>

And you'll need to do this (here or in some lib you are using)

$(document).ready(function() {
    $('#example').DataTable( {
        "processing": true,
        "serverSide": true,
        "ajax": "../server_side/scripts/server_processing.php"
    } );
} );

You will also need:

//code.jquery.com/jquery-1.12.4.js https://cdn.datatables.net/1.10.16/js/jquery.dataTables.min.js

If you did not already include that in your project, that's not clear from you info.

For more info see: Datatables Server-side processing

2. Store the total sum of hours somewhere else. You could simply update the sum of hours per user when changes to the Workhour model are done. That way you will not need to do the sum every time you request a single or a list (or all) users.

My suggestion would be to do at least #1 and #2 will be a good optimization after that.