Symfony version(s) affected: 4.3-beta1

Description
After handling a message which fails, it will be retried 3 times. If all of those retries fail, then the message should be added to the failure transport (as explained here: https://symfony.com/blog/new-in-symfony-4-3-messenger-failure-transport)
However, for me at least, this is not the case.

I use the following configuration, but after the 3rd retry I can see this in the logs in this order:
messenger.WARNING: An exception occurred while handling message CLASS... at vendor/symfony/messenger/Middleware/HandleMessageMiddleware.php:82 at MY_HANDLER_CLASS.
messenger.INFO: Rejected message CLASS will be sent to the failure transport failed.
messenger.INFO: Sending message CLASS with "Symfony\Component\Messenger\Transport\AmqpExt\AmqpTransport"...
messenger.CRITICAL: Rejecting CLASS (removing from transport). at vendor/symfony/messenger/Middleware/HandleMessageMiddleware.php:82 at MY_HANDLER_CLASS

From what I see, the last messenger.CRITICAL error seems to come from the messenger component trying to call my handler again for some reason but I am not sure why. I use routing keys and also amqp for the failed queue, this is my configuration:

messenger:
    failure_transport: failed
    transports:
        amqp:
            dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
            options:
                exchange:
                    type: topic
                    name: topic
                queues:
                   QUEUE:
                        binding_keys: ['KEY.#']
        failed:
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
                options:
                    exchange:
                        type: topic
                        name: topic
                    queues:
                        QUEUE_FAILED:
                            binding_keys: ['#']

I am really not sure why this does not work. What I also found out is that when the Symfony\Component\Messenger\EventListener\SendFailedMessageToFailureTransportListener is called, it does not remove the DelayStamp from the envelope, so when the message arrives in the Symfony\Component\Messenger\Transport\AmqpExt\Connection class, the publishWithDelay function will be called, which will publish the message again to the delay queue (delay_queue_KEY.TEST_4000 in this case), with the exchange being delay and the routing key delay_queue_KEY.TEST_4000. Moreover, the delay queue is setup in the same way, with the routing key being my initial routing key KEY.TEST and the exchange name being topic, so the message will get published to the QUEUE queue not to the QUEUE_FAILED queue because of the routing key.

So how can this be prevented?Shouldn't the routing key be set to null before publishing to the failed transport, so that way I can put the QUEUE_FAILED to be bound for the null key? Or should I create an AmqpStamp using a middleware in this case, and add a new routing key for the failed transport?But then, why does the failure_transport config option even exist if it does not work properly?

How to reproduce
Use routing keys for messages, have the handler fail 3 times and see the results. The message will not be added to the failure transport as intended.

Possible Solution
Ignore all routing keys, exchanges etc when using the failure_transport config option when the message fails after the set number of retries, and have the failure_transport and no other transports be used for this situation.