Using RQ + Redis in production with a large number of workers and tasks (and not deleting any finished tasks for better debugging / tracking purposes) requires constantly increasing Redis' memory. This is so because Redis doesn't support tiered storage, meaning that it needs to load into memory its entire dataset.

Compressing Redis' memory and using custom RQ serializers (eg json instead of pickle) reduces the pace at which the memory needs to be increased, but it doesn't solve the general problem of memory usage increasing constantly.

Valkey (a drop-in replacement for Redis) is currently discussing the topic of adding a tiered storage solution, but nothing has been decided.

Postgres is able to replicate the atomicity of Redis, it's fast (arguably as fast as Redis in certain scenarios; definitely "fast enough" to act as a broker for RQ), which makes me wonder if implementing Postgres as a broker for RQ would be a good idea or even if it's something that you're willing to consider, @selwin .