Codeus logo

Laravel Batch of Chains and the finally() callback

Laravel Batch of Chains and the finally() callback
Napisao/la Matija Bojanić | 22 min čitanja

How we found the issue

We were working on an ETL pipeline for a client who uses Laravel as the core of their stack. At the center of it was a single Job that would download files for a user, parse them, deduplicate, enrich, and store them, file by file, all within that one job. As time passed, it got harder to maintain, and the process got slower.

We came in and proposed moving the ETL pipeline into an event-driven system, a perfect fit for Laravel's async features. We would dispatch a batch of chains, each chain handling the steps for one file (download, process, save), with each step dispatching events onto which we could hook additional actions. After everything finished, post-processing would kick off. Laravel batches offer callbacks for exactly this: then(), catch(), and finally().

Now, some things can naturally break in a process like this. Parsers might encounter bad data. Files might fail to download. So we decided to hook the post-processing into the finally() block instead of then(), and enabled allowFailures() on the batch. That way it would fire regardless of individual job failures, or so we thought.

We tested on a smaller dataset with sandboxed environments. We simulated errors by intentionally throwing exceptions at the end of our chains. Everything looked good.

We deployed to UAT for review and stress testing. Suddenly, there were processes that simply weren't finishing. No matter what we tried, the finally block wasn't being executed.

We triple-checked the code. And then we realized...

Laravel Batches of Chains never execute finally() when a job fails mid-chain

I went back to the docs to make sure we weren't missing something. The documentation describes finally clearly:

To dispatch a batch of jobs, you should use the batch method of the Bus facade. Of course, batching is primarily useful when combined with completion callbacks. So, you may use the thencatch, and finally methods to define completion callbacks for the batch. Each of these callbacks will receive an Illuminate\Bus\Batch instance when they are invoked.

use App\Jobs\ImportCsv;
use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;
use Throwable;

$batch = Bus::batch([
    new ImportCsv(1, 100),
    new ImportCsv(101, 200),
    new ImportCsv(201, 300),
    new ImportCsv(301, 400),
    new ImportCsv(401, 500),
])->before(function (Batch $batch) {
    // The batch has been created but no jobs have been added...
})->progress(function (Batch $batch) {
    // A single job has completed successfully...
})->then(function (Batch $batch) {
    // All jobs completed successfully...
})->catch(function (Batch $batch, Throwable $e) {
    // Batch job failure detected...
})->finally(function (Batch $batch) {
    // The batch has finished executing...
})->dispatch();

return $batch->id;

And for chained batches:

You may define a set of chained jobs within a batch by placing the chained jobs within an array. For example, we may execute two job chains in parallel and execute a callback when both job chains have finished processing

use App\Jobs\ReleasePodcast;
use App\Jobs\SendPodcastReleaseNotification;
use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;

Bus::batch([
    [
        new ReleasePodcast(1),
        new SendPodcastReleaseNotification(1),
    ],
    [
        new ReleasePodcast(2),
        new SendPodcastReleaseNotification(2),
    ],
])->then(function (Batch $batch) {
    // All jobs completed successfully...
})->dispatch();

Nothing in there says "unless a job fails mid-chain." The docs describe then, catch, and finally exactly as you'd expect from any try/catch/finally construct -- finally always runs.

To reproduce this cleanly, I spun up a fresh Laravel 12 installation. I created a TestJob, which just logs a success message. Then I wrote a simple Artisan command to dispatch a batch of three chains, each chain consisting of two TestJobs.

  <?php

  namespace App\Jobs;

  use Illuminate\Bus\Batchable;
  use Illuminate\Contracts\Queue\ShouldQueue;
  use Illuminate\Foundation\Queue\Queueable;
  use Illuminate\Support\Facades\Log;

  class TestJob implements ShouldQueue
  {
      use Batchable;
      use Queueable;

      public function __construct(
          public readonly string $name
      ) {}

      public function handle(): void
      {
          Log::info("success: {$this->name}");
      }
  }
  <?php

  namespace App\Console\Commands;

  use App\Jobs\TestJob;
  use Illuminate\Bus\Batch;
  use Illuminate\Console\Command;
  use Illuminate\Support\Facades\Bus;
  use Illuminate\Support\Facades\Log;
  use Throwable;

  class DispatchBatchChains extends Command
  {
      protected $signature = 'app:dispatch-batch-chains';

      protected $description = 'Dispatch 3 chained job pairs as a batch (one chain intentionally fails)';

      public function handle(): void
      {
          $batch = Bus::batch([
              [new TestJob('chain-1 job-1'), new TestJob('chain-1 job-2')],
              [new TestJob('chain-2 job-1'), new TestJob('chain-2 job-2')],
              [new TestJob('chain-3 job-1'), new TestJob('chain-3 job-2')],
          ])
              ->then(function () {
                  Log::info('batch completed');
              })
              ->finally(function (Batch $batch) {
                  Log::info('batch finally');
              })
              ->catch(function (Batch $batch, ?Throwable $e) {
                  Log::info('batch failed');
              })
              ->allowFailures()
              ->dispatch();

          $this->comment("Batch dispatched: {$batch->id}");
      }
  }

All six jobs run, then() fires, finally() fires. Everything works as expected.

  [2026-03-13 19:47:03] local.INFO: success: chain-1 job-1
  [2026-03-13 19:47:03] local.INFO: success: chain-2 job-1
  [2026-03-13 19:47:03] local.INFO: success: chain-3 job-1
  [2026-03-13 19:47:03] local.INFO: success: chain-1 job-2
  [2026-03-13 19:47:03] local.INFO: success: chain-2 job-2
  [2026-03-13 19:47:03] local.INFO: success: chain-3 job-2
  [2026-03-13 19:47:03] local.INFO: batch completed
  [2026-03-13 19:47:03] local.INFO: batch finally

Now we add some failures into the mix. I created a FailingJob that will fail intentionally, and updated the dispatch to place it at the end of chain 2.

<?php

namespace App\Jobs;

use Illuminate\Bus\Batchable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Log;

class FailingJob implements ShouldQueue
{
  use Batchable;
  use Queueable;

  public int $tries = 1;

  public function __construct(
      public readonly string $name
  ) {}

  public function handle(): void
  {
      Log::info("failing intentionally: {$this->name}");

      $this->fail();
  }
}
  $batch = Bus::batch([
      [new TestJob('chain-1 job-1'), new TestJob('chain-1 job-2')],
      [new TestJob('chain-2 job-1'), new FailingJob('chain-2 job-2')],
      [new TestJob('chain-3 job-1'), new TestJob('chain-3 job-2')],
  ])
      ->then(function () {
          Log::info('batch completed');
      })
      ->finally(function (Batch $batch) {
          Log::info('batch finally');
      })
      ->catch(function (Batch $batch, ?Throwable $e) {
          Log::info('batch failed');
      })
      ->allowFailures()
      ->dispatch();

Since we now have a job that fails, catch() should trigger. But what about finally()?

  [2026-03-13 19:50:08] local.INFO: success: chain-1 job-1
  [2026-03-13 19:50:08] local.INFO: success: chain-2 job-1
  [2026-03-13 19:50:08] local.INFO: success: chain-3 job-1
  [2026-03-13 19:50:08] local.INFO: success: chain-1 job-2
  [2026-03-13 19:50:08] local.INFO: failing intentionally: chain-2 job-2
  [2026-03-13 19:50:08] local.INFO: batch failed
  [2026-03-13 19:50:08] local.INFO: success: chain-3 job-2
  [2026-03-13 19:50:08] local.INFO: batch finally

finally() fires as well. So far so good. But what if the failure is in the middle of a chain? After all, chains guarantee that jobs in the chain won't process unless the previous job succeeded.

$batch = Bus::batch([
  [new TestJob('chain-1 job-1'), new TestJob('chain-1 job-2')],
  [new FailingJob('chain-2 job-1'), new TestJob('chain-2 job-2')],
  [new TestJob('chain-3 job-1'), new TestJob('chain-3 job-2')],
])
  [2026-03-13 19:51:23] local.INFO: success: chain-1 job-1
  [2026-03-13 19:51:23] local.INFO: failing intentionally: chain-2 job-1
  [2026-03-13 19:51:23] local.INFO: batch failed
  [2026-03-13 19:51:24] local.INFO: success: chain-3 job-1
  [2026-03-13 19:51:24] local.INFO: success: chain-1 job-2
  [2026-03-13 19:51:24] local.INFO: success: chain-3 job-2

catch() fires. The other two chains finish. And then nothing. No finally. The batch just stops.

So why does this happen? Laravel determines whether to fire finally() using a single condition, found in Illuminate/Bus/UpdatedBatchJobCounts.php:

public function allJobsHaveRanExactlyOnce()
{
    return ($this->pendingJobs - $this->failedJobs) === 0;
}

When a job fails mid-chain, Laravel abandons the rest of that chain. Those jobs are never dispatched, never run, and never decrement pendingJobs. The condition never becomes true, and finally() never fires. The batch sits there, waiting for jobs that will never come.

Turns out I'm not the first to notice this. There are two GitHub issues going back to 2021 and 2022, both reporting the exact same behavior. Both were closed without a fix.

Taylor Otwell acknowledged it:

"I can see the reasoning behind the current behavior to some extent - the 'finally' callback executes when all of the jobs in the batch have run exactly once (success or failure - doesn't matter)... in the case of your chain, the jobs after the failing job have not executed at all, so finally is never firing."

So the behavior is understood. It's just not fixed. The community's reaction was about what you'd expect:

"I'm confused why this would just be closed. It's clearly not the proper behavior for a finally block. The docs describe then, catch, and finally as they are understood in general, which is to say that finally is always called."

The Laravel team's response was essentially: open a PR.

Which, to be fair, is the only reasonable answer when you're maintaining a framework used by hundreds of thousands of developers. But it doesn't make it less frustrating when you're the one debugging a stalled batch in UAT at the end of a client sprint.

Our fix

Fortunately, we had already been tracking progress ourselves. We had created a log table that tracks each step of the pipeline separately: how many documents were expected, how many succeeded, and how many failed.

Schema::create('pipeline_logs', function (Blueprint $table) {
  $table->id();
  $table->timestamps();

  $table->timestamp('started_at')->nullable();
  $table->timestamp('finished_at')->nullable();
  $table->string('step')->nullable();
  $table->string('status')->nullable();
  $table->integer('total_documents_count')->nullable();
  $table->integer('successful_documents_count')->nullable();
  $table->integer('failed_documents_count')->nullable();

  $table->unsignedBigInteger('process_id');
  $table->foreign('process_id')->references('id')->on('processes')->onDelete('cascade');

  $table->index(['process_id', 'step']);
});

Each job dispatches an event on completion or failure. A listener catches those events and atomically increments the corresponding counter:

public function handleDocumentStored(DocumentProcessedEvent $event): void
{
  $result = DB::select(
      "UPDATE pipeline_logs
      SET successful_documents_count = successful_documents_count + 1, updated_at = NOW()
      WHERE process_id = ? AND step = ?
      RETURNING total_documents_count, successful_documents_count, failed_documents_count",
      [$event->processId, PipelineLog::STEP_STORED]
  );

  if (!empty($result)) {
      $this->checkAndCompleteStoreStep($result[0], $event->processId);
  }
}

We use a raw UPDATE ... RETURNING query rather than Eloquent's increment() followed by a refresh(). The reason is simple: with dozens of jobs updating the same row concurrently, there's a race window between the increment and the read. The atomic query closes that window entirely.

Once the numbers add up, successful + failed >= total, we know the step is done, regardless of how many jobs failed. At that point we dispatch DocumentStoreStepCompletedEvent, onto which we hook post-processing: generate reports, send email notifications, and so on.

private function checkAndCompleteStoreStep(object $counters, int $processId): void
{
  $total = $counters->total_documents_count;
  $successful = $counters->successful_documents_count;
  $failed = $counters->failed_documents_count;

  if ($total !== null && ($successful + $failed) >= $total) {
      PipelineLog::where('process_id', $processId)
          ->where('step', PipelineLog::STEP_STORE)
          ->update([
              'status' => PipelineLog::STATUS_COMPLETED,
              'finished_at' => now(),
          ]);

      DocumentStoreStepCompletedEvent::dispatch($processId);
  }
}

This sidesteps the finally() issue entirely. We're not relying on Laravel's batch to tell us when things are done, we're tracking it ourselves.

Is it more work than a finally() callback? Yes. But it gives us a reliable trigger for post-processing regardless of failures, as well as full visibility into where processes get stuck, which steps are failing, and how many documents made it through each stage.

We had to implement this quickly, as explaining to a client that the framework has an issue is near impossible. So we pivoted, and found a good solution that is now in production, and just last week processed 250,000 documents in 72 hours (the only thing slowing us down was the infra budget).

The next challenge for us is to dig deeper into the chain-batch interaction in the framework itself and fix it properly. Keep an eye out for our PR in the upcoming weeks.

Povezane usluge