Symfony version(s) affected

6.4.12, (7+)

Description

Hello

We use the scheduler since more than one year with (20+ scheduled task), but we want to accelerate and scale the number of task, but we encounter scalability issue.

Following the schedule documentation, we want to manage multiple schedule in a scalable way, but it seems impossible or very hard to manage.

If I have 2 scheduled job, taskA, taskB running every minute, and taking 10 secondes each.
configured to use the default scheduler, with a cache and a lock (redis) (like the documentation suggest)

If I start two consumers (bin/console messenger:consume scheduler_default)

The job taskA is executed as scheduled, the taskB is waiting for taskA to finished and run on the first consumer while the second consumer will do nothing, just wait for the lock to be acquired.

if I use a dedicated scheduler for each task and start two consumers (bin/console messenger:consume scheduler_workerA scheduler_workerB)

The result is the same the job taskA is executed as scheduled, the taskB is always late while the second consumer will do nothing, just wait for the lock to be acquired (both locks been acquired by consumer one).

If I add a jitter option on each task, then it seems work as expected, TaskA is picked by first consumer and taskB is picked by the second consumer. (was lucky, the 2 scheduler is locked on a consumer and mostly on the first one, unless the running task that take 10 seconds is due instantly, then the lock of the second scheduler is delayed and picked up by the second consumer)

How to reproduce

Create two tasks

#[AsCronTask(expression: '* * * * *')]
class TestTaskA
{
    public function __invoke(): void
    {
        sleep(10);
        $this->logger->info('Task A done');
    }
}

configure a lock and a cache for the default scheduler

Start 2 consumer

bin/console messenger:consume  scheduler_default

TaskA and TaskB should run nearly at the same time

Possible Solution

In my point of view the lock done in the MessageGenerator should be done on a combination of the woker name, message name (signature) and (optionnal) the next run signature. This would allow a better scalability and also a better DX, no need to add jitter everywhere and no need to create a schedule + config for each task.

Additional Context

I would be happy to work on a PR if you think the scheduler should be scalable without having dedicated scheduler and consumer per task.