Support for native delayed messages using RabbitMQ

[jmfn]

I understand there are 2 techniques for delaying messages with RabbitMQ, using the dead lettering "trick" or the Delayed Message plugin that (more info).

Has this been explored to date?

[mookid8000]

I don't know of anyone who has tried it with Rebus.

The Rebus defer API also promises more than the RabbitMQ dead-letter-defer can hold, as it is possible for Rebus to defer messages with a per-message delay – e.g.

await bus.Defer(TimeSpan.FromSeconds(30), "last");
await bus.Defer(TimeSpan.FromSeconds(10), "first");

is possible with Rebus.

If course one could emulate the behavior by creating TTL queues with differing TTLs, e.g. rounding to something sensible like

rebus-ttl-10s
rebus-ttl-20s
rebus-ttl-30s
rebus-ttl-60s

allowing for deferring messages 10, 20, 30, and 60 seconds respectively.

[mookid8000]

In a project I am currently involved with, we have created a dedicated timeout manager that consumes messages from the timeouts queue, and then all other endpoints simply

Configure.With(...)
    .(...)
    .Timeouts(t => t.UseExternalTimeoutManager("timeouts"))
    .Start();

I think this approach is pretty smooth too.

[mookid8000]

(PS: I closed the "issue" because it's not really an issue – we can still keep discussing it in here 😄 )

[jmfn]

Thanks for the comments.

After some more reading, I understand that both native RabbitMQ delayed message solutions have limitations. The plugin option could be implemented like your Azure and SQS transports have done, with an x-delay magic header. The dead lettering option I would choose for higher volume use cases, as long as you only need one message TTL.

Some findings worth sharing about the 2 native solutions--

The Delayed Message Plugin has reliable delivery issues when the number of delayed messages grows too large, 100K+ (rabbitmq/rabbitmq-delayed-message-exchange#72). Would work just fine for some use cases, but not for those that may reach high volumes.

Dead lettering messages only expire when they hit the head of the queue per per-message-ttl caveats (http://www.rabbitmq.com/ttl.html#per-message-ttl-caveats). Now I understand your comment about using multiple queue-ttl’s. So this is definitely workable if your message TTL’s are the same, but not so reliable if you’re sticking messages in with variable “expiration” values. Still, doesn’t lend itself well to the nice rebus.Defer(timeout, message) API since the timeout argument isn’t guaranteed.