Thread: Under the hood with delayed messages in RabbitMQ
Delaying message delivery can be tricky, especially on some messaging infrastructure. Let's look at what happens when a message gets sent 10 sec into the future using the RabbitMQ transport.
Feb 13, 2020 · 4:22 PM UTC
What we mean by "sending a message 10-sec into the future" is that the message delivery will be delayed by 10 sec. This can be very helpful in a variety of situations.
1
On the infrastructure side, delayed delivery can be used for delayed retries, so that you can avoid hammering a slightly-delayed web service to death and instead give it some time to recover. particular.net/blog/but-all-…
On the business logic side, delayed retries are useful for modeling a time-triggered action. In fact, most batch jobs are better modeled using delayed message delivery, so you don't have to alter every row of your database every night. particular.net/blog/death-to…
Some messaging infrastructure (Azure Service Bus, Azure Queues, Amazon SQS) support delayed delivery natively, at least to some degree. What makes our scenario interesting is that RabbitMQ support for delays is quite limited.
1
1
Any queue created in RabbitMQ comes with a set of properties. Two of these, x-message-ttl and x-dead-letter-exchange can be used for delayed delivery. The first specifies time-to-live for the queue and the second an exchange to which messages get routed when TTL the elapses.
1
1
E.g. to support 1 min delays I can create delay-for-1-min queue, with x-message-ttl=60000 and x-dead-letter-exchange=exchange. With that in place, any message sent to delay-for-1-min queue gets delayed for 1 min before reaching the destination specified in the routing key.
1
1
The message sits in the delay-for-1-min queue until the TTL elapses (60 sec) at which point the broker moves it to the exchange, which in turn routes it to the final destination based on the routing key specified. Done ... or is it?
1
1
The TTL + dead letter exchange approach is the de-facto standard delay technique for RabbitMQ. But it comes with one constraint—for any delay value we need a dedicated queue. In scenarios when possible delays are known up-front and there is only a handful of these, that's fine.
Unfortunately, for us this assumption doesn't hold. Delay values are not known upfront, the required resolution is at the level of seconds and on top of that a 1-year delay for a message is nothing unusual. To support this we would need ~30M queues!
1
Our RabbitMQ transport builds on-top (literally) of the described technique. With some extensions, it supports message delays between 1 sec and 8.5 years with 1-second resolution using 28 exchanges and 28 queues. How do we do that?
1
1
The idea is based on the observation that any value from 1 sec to 8.5 years can be represented by 28 bits. If we had 28 queues with TTL from 2^0 to 2^27 then all we need is for a message to be routed through the queues that have TTLs summing up to the delay value we need.
1
1
We do this by stacking 28 exchanges and using the message routing key for queue selection. At each level in the stack there is an exchange that either moves the message to the n-th delay queue (with TTL=2^n) or passes it the lower level—based on the routing key of that message.
1
1
Now when I want to send a message with 10-sec delay, first I calculate the binary representation of 10 which is 1010b and put that into my routing key together with a final destination which gives me 1.0.1.0.final-destination.
2
1
Before sending it, I prepend 24 leading zeros (to cater for all 28 levels) and send the message to the top of the stack. And that's it! Based on the routing key information the message will flow through queues with total TTL of 2^4+2^1=10 sec - which is exactly what we wanted.
1
1
Want to know more? Check out the RabbitMQ transport delayed delivery documentation docs.particular.net/transpor…
1
2
Or, check out the RabbitMQ transport delayed delivery source code
github.com/Particular/NServi…
1


