Script/Queue keeps running out of memory

Question

My application has a huge amount of data it needs to process whenever a user requests it. The script was originally organised in a foreach loop but that caused PHP to timeout everytime. I moved to using Redis queues, but then I was having memory issues.

mmap() failed: [12] Cannot allocate memory

and

PHP Fatal error:  Out of memory (allocated 79691776) (tried to allocate 134217728 bytes) 

Now I've set the queue to only have one process. It works better but after a while I start getting the memory errors again. And this is just me testing it. Once users start using it, it will just fall over.

I allocate the script 1024MB because it just runs out of memory if I don't on single use. I'm wondering if there's something I can do after each time the script is run to free up memory. Like unsetting variables? I can't see how this would help though, seeing as the script ends and is run again from scratch by the queue worker.

I'm using a vagrant machine (Homestead) with 2GB RAM

UPDATE:

The backtest begins when we execute the dispatcher and runs through 10 leagues and 10 years.

Dispatcher class:

class Dispatcher
{
    use Attributes;
    use DataRetriever;
    public function runBacktest($token)
    {
        $authUserId = Auth::user()->id;
        $system = System::where('token', $token)->first();
        $this->getSystem( $authUserId, $system->id);
        foreach ($this->leagues as $league) {
            foreach ($this->years as $year) {
                BacktestJob::dispatch($authUserId, $system->id, $token, $league, $year);
            }
        }
    }
}

The dispatcher executes the job:

class BacktestJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    private $token;
    private $league;
    private $year;
    private $authUserId;
    private $systemId;

    public $timeout = 10000;


    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct($authUserId, $systemId, $token, $league, $year)
    {
        $this->token = $token;
        $this->league = $league;
        $this->year = $year;
        $this->authUserId = $authUserId;
        $this->systemId = $systemId;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $backtest = new Backtest;
        $backtest->init($this->authUserId, $this->systemId, $this->token, $this->league, $this->year);
    }
}

Below is a stripped down version of the main script, because it does quite a lot:

public function init($authUserId, $systemId, $token, $league, $year)
    {
//      ini_set('memory_limit', -1);
        ini_set('max_execution_time', 300); //300 seconds = 5 minutes
        ini_set('memory_limit', '1024M'); // or you could use 1G
        $this->authUserId = $authUserId;
        $this->systemId = $systemId;
        $this->token = $token;
include(storage_path("app/matches/{$league->key}/{$year}.php"));
        $this->leagueResults[$league->key][$year] = collect($this->leagueResults[$league->key][$year]);
//Loops through the data - saves to new array
fwrite($file, serialize($backtest));

Originally the data was pulled from a 50MB json file. I replaced the json file with a hardcoded PHP array (File size 100MB). I know the newer file is bigger, but I assumed not going though json_decode would speed things up.

I also removed a db insert at the end of the script, but I would prefer it to stay in as it would make my life easier.

Answer 1

Ok, so I fixed the issue by breaking up the data file. Because I have full access to the original data file, and I don't need all the data in it for every request, I broke it up into about 50 smaller files.

I was surprised by the increased performance. From 30+ seconds plus timeouts, it went down to less than 2 seconds. The majority of requests completed in less than a second.