Symfony version(s) affected

6.4.4

Description

When using the AWS SQS transport with a FIFO queue, the transport automatically adds a MessageDeduplicationId

https://github.com/symfony/amazon-sqs-messenger/blob/7.2/Transport/Connection.php

        if (self::isFifoQueue($this->configuration['queue_name'])) {
            $parameters['MessageGroupId'] = null !== $messageGroupId ? $messageGroupId : __METHOD__;
            $parameters['MessageDeduplicationId'] = null !== $messageDeduplicationId ? $messageDeduplicationId : sha1(json_encode(['body' => $body, 'headers' => $headers]));
            unset($parameters['DelaySeconds']);
        }

This is unexpected: Using a FIFO queue should not also activate deduplication.

In our example, we have messages with the same payload. E.g.

UpdateItemMessage
{
   itemid: 123
}

In out case, this message takes some time to be processed. If the user makes an another change in the meantime, another message is put onto the queue. Since they share the same payload and headers, the message gets the same MessageDeduplicationID and therefore handled as duplicate by SQS.

There is no way on AWS side to disable the deduplication on AWS SQS FIFO queues. As a workaround, users have to set their own unique Deduplication IDs.

How to reproduce

• Create a FIFO queue on AWS SQS
• Create a new Message
• Create MessageHandler that takes a bit longer to process (up to 5min)
• Add n messages to the FIFO queue with the same payload. They wont be delivered to the worker.

Possible Solution

Do not automatically add MessageDeduplicationId and MessageGroupId to the messages unless the Message implements MessageDeduplicationAwareInterface and MessageGroupAwareInterface. Do not override if they return null.

Additional Context

This is not new behavior. The code has been added 5 years ago.

You can see the removed deduplicated messages in the monitoring section of you FIFO queue