Your application database records will increase by every day. As a developer, we faced performance and server memory issues when working with large table records. In this blog, we going to process the large table records and explain the importance of the Eloquent chunk method.

We need a demo application to work with large records.

Laravel Installation

As usual, we going to install Basic Laravel Admin Panel locally. This Basic admin comes with users with roles and permissions.

The Basic Laravel Admin Panel is based on Laravel Sail. What is Sail? Sail is a built-in solution for running your Laravel project using Docker.

Refer to the https://github.com/balajidharma/basic-laravel-admin-panel#installation step and complete the installation.

Demo data

For demo records, we going to create dummy users on the user’s table using the Laravel seeder. To generate a seeder, execute the make:seeder Artisan command.

./vendor/bin/sail php artisan make:seeder UserSeeder

INFO Seeder [database/seeders/UserSeeder.php] created successfully.

Open the generated seeder file located on database/seeders/UserSeeder.php and update with the below code.

<?php
namespace Database\Seeders;
use Illuminate\Database\Seeder;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Hash;
use Illuminate\Support\Str;
class UserSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        for ($i=0; $i < 1000; $i++) { 
            DB::table('users')->insert([
                'name' => Str::random(10),
                'email' => Str::random(10).'@gmail.com',
                'password' => Hash::make('password'),
            ]);
        }
    }
}

Now run the seeder using the below Artisan command. It will take extra time to complete the seeding.

./vendor/bin/sail php artisan db:seed --class=UserSeeder

After the Artisan command, verify the created users on the user list page http://localhost/admin/user

Processing large records

Now we going to process the large user records. Assume we need to send black Friday offers notifications emails to all the users. Usually, we generate new Artisan command and send the email by using the scheduler job.

Memory issue

We will fetch all the users and send emails inside each loop.

$users = User::all();
$users->each(function ($user, $key) {
    echo $user->name;
});

If you have millions of records or if your result collection has a lot of relation data means, your server will throw the Allowed memory size of bytes exhausted error.

To overcome this issue we will process the limited data by saving the limit in the database or cache.

Example: First time we fetch the 100 records and save the 100 on the database table.
Next time fetch 100 to 200 records and save the 200 in the database. So this method involved additional fetch and update. Also, we need to stop the job once processed all the records.

Laravel provides the inbuild solution of the Eloquent chunk method to process the large records

Laravel Eloquent chunk method

The Laravel ELoquent check method retrieves a small chunk of results at a time and feeds each chunk into a Closure for processing.

User::chunk(100, function ($users) {
    foreach ($users as $user) {
        echo $user->name;
    }
});

Understand the chunk method

I will create one function in the user controller and explain the check method detailed.

Open the routes/admin.php and add the below route

Route::get('send_emails', 'UserController@sendEmails');

Now open the app/Http/Controllers/Admin/UserController.php and add the sendEmails method.

Without chunk:
After adding the below code open the http://localhost/admin/send_emails page

public function sendEmails()
{
    $users = User::all();
    $users->each(function ($user, $key) {
        echo $user->name;
    });
}

Open the Laravel Debugbar queries panel. The select * from users will fetch all the 1000+ records.

With chunk method:
Replace the same function with the below code and check the page in the browser.

public function sendEmails()
{
    User::chunk(100, function ($users) {
        foreach ($users as $user) {
            echo $user->name;
        }
    });
}

The chunk method adds limits and processes all the records. So if using chunk, it processes 100 records collection at the time. So no more memory issues.

What is chunkById?

This chunkById the method will automatically paginate the results based on the record’s primary key. To understand it, again update the sendEmails the method with the below code

public function sendEmails()
{
    User::chunkById(100, function ($users) {
        foreach ($users as $user) {
            echo $user->name;
        }
    });
}

Now user id is added on where condition along with the limit of 100.

// chunkById
select * from `users` where `id` > 100 order by `id` asc limit 100
select * from `users` where `id` > 200 order by `id` asc limit 100
select * from `users` where `id` > 300 order by `id` asc limit 100
// chunk
select * from `users` order by `users`.`id` asc limit 100 offset 0
select * from `users` order by `users`.`id` asc limit 100 offset 100
select * from `users` order by `users`.`id` asc limit 100 offset 200

This chunkById is recommended when updating or deleting records inside the closure (in the loop).

Conclusion

The Eloquent chunk method is a very useful method when you work with large records. Also, read about the collection check method.