Hi @michaelklishin thank you for the answer!

We noticed the issue once we migrated from 3.11.3 to 4.1.3. And it was unexpected behaviour for us, which wasn't mentioned anywhere.

Could you please give some advice/answers to the next questions? It will help us investigate further and find an optimal solution

What I found in 3.11.3 was a configuration - queue_explicit_gc_run_operation_threshold. And I suspect, because of it, segments were cleaned up well in the previous version.
I haven't found it in the docs for the new version. Do we have something similar in the new version to force GC?

4.1.0 made segment cleanup significantly more aggressive for the typical case where messages are consumed.

As you said, acknowledgement of the message will work better. We can replace the message TTL queue config with a consumer which will move messages according to their time in the queue, acknowledge them and move to another queue. What's bothering is that there is delivery limit for messages. Did I get it right that we can still set up such queues with unlimited delivery limits? The problem is that it may consume disk space, but since it will be acknowledged in the end, it should be cleaned up.

There is also documentation on quorum queue tuning for large messages.

Yep, I am currently testing different configurations. Just for clarification, did I get it correctly that a lower segment_max_entries number causes the creation of more segment files. And large number of segment files will also trigger clean-up more frequently.
But in general, it should consume more than wal_max_size_bytes.

Thank you very much for help!

Just a bit more context.
We use queues with TTL for the retry mechanism. Once a message is discarded by an ordinary consumer, it comes to the retry queue with a TTL. After TTL expiration, it tries for more time to be acknowledged in another queue.
And there is a chain of queues with different TTLs.

So there is a trend of memory leaks.

@kjnilsson Hi!

Thank you for the suggestion.
I made a bit bigger test, it is 2k messages and a chain of queues with different TTL

It is better with your suggestion, but still memory consumption is fast.
Is it possible to set such values through Rabbit configuration?

Screenshots from the second test

@vaimer quorum_queue_checkpoint_config is not exposed to rabbitmq.conf and is not really configurable (without rabbitmqctl eval) at all, the default values it uses are constants:

-define(CHECK_MIN_INTERVAL_MS, 1000).
-define(CHECK_MIN_INDEXES, 4096 * 2).
-define(CHECK_MAX_INDEXES, 666_667).

They can potentially be exposed (this is not a promise of delivery) to rabbitmq.conf but there hasn't been any need for that in over seven years of quorum queue existence.

If you want the minimum memory footprint, use a stream or superstream. Minimum memory footprint with this TTL-based workload with large messages of at least 15 MiB is not what quorum queues are optimized for, and that is very unlikely to change. However, streams keep significantly less data in memory and TTL is not an edge case scenario but rather a fundamental feature of streams.

I'm afraid I do not understand what the metrics you are comparing mean specifically. The first two probably mean the memory footprint at the beginning and the end of each test but I'm not sure what "Returning memory" is.

So, give a stream with a comparable data retention policy a try. Or store those messages in a blob store with a TTL and pass around their blob store keys in the messages flowing through the queues (or streams), which is a standard recommendation for messages of tens of MiBs in size or more.

Sorry for the confusion, I missed it during copying from my internal notes.

It is a memory once all messages end up in dead letter queue. You can see this step in the provided graphic. Memory jumped to this value, right after processing is done.

I will check out your recommendation, thank you!

if you keep publishing some more message will it eventually reclaim?