Rabbitmq- Designing a message replay service

Question

I am trying to design a replay mechanism that will enable users to replay messages from the queues. The best design I have come up for an exchange that contains multiple queues and multiple consumers is:

1. Create a recorder service that will:

   • Create a queue and bind all routing keys to it.
   • Consume all messages from the exchange.
   • Save all messages to the DB.

2. Subscriber request for replay.

   • Each subscriber creates a new exchange, queue and binds to it with same bindings as its regular queue.
   • Subscriber sends a rest requests to a web server to start replay with a filter ( startdate, etc). Request contains its replay exchange name.
   • Web server pulls data from the DB and publishes it to the specific exchange
   • refinements can be added like attaching RequestId and echoing it back.

Questions:
1. Does that make sense?
2. Am I inventing the wheel? Is there a rabbit inherent solution? plugin?
3. Does creating multiple exchanges considered a good practice?
In this solutionan exchange for each queue is created in order to publish the same message.

Another solution:
1. Create an additional queue "ReplayQueue" for each queue. set a TTL (let's say a month).
2. Each time a user requests a replay let him replay from its own ReplayQueue without acking.

This solution is a bit problematic because.

• In order to replay last day, consumers will have to fetch all earlier 29 days and filter them out.
• This solution scales up - Queues will get larger (unlike db storage that can scale out).

Answer 1

1. Does that make sense?

Yes

2. Am I inventing the wheel? Is there a rabbit inherent solution? plugin?

You are not reinventing the wheel. There is AFAIK no rabbit solution and no out of the box solution.

Your first solution is in my opinion good. The Another solution is very problematic, because a healthy rabbit is an empty one and rabbit is not a datastore.

You will have a queue (STORE) where all published messages should be routed to. Instead of binding STORE with all the binding keys, you could consider using a topic exchange. At the price that binding key must not contain . # * and a slight overhead when the message is routed. The STORE queue will bind with the binding key #.

You could have a look at firehose.

Why a web request? You can use Rabbit with an RPC call:

• Subscriber sends an amqp request to start replay with a filter ( startdate, etc).
• Request contains the reply-to queue name. The queue may be exclusive to the client and request.
• RPC server pulls data from the DB and publishes it to the reply-to queue

You could also have a look at the direct-reply-to pattern.

3. Does creating multiple exchanges considered a good practice?

Yes, as soon as you need it. For your specific case, in my opinion an exchange per subscriber is not necessary. The request contains already the queue name. You could simply publish to this queue using the default exchange amq.direct with the routing key equal to the queue name. If you want an exchange I would create an unique exchange (eg. "replay") and have each subscriber bind their replay queues to this exchange. The "replay" exchange can be a convention or sent with the request.

Answer 2

The first solution is feasible. Given that rabbit MQ ships with a tracing plugin, storing the message in DB becomes even more easier as it simply boils down to consuming from a queue bound to amq.rabbitmq.trace exchange.

This exchange is specific to a vhost and every vhost has its own amq.rabbitmq.trace exchange. In addition when creating a new trace it is possible to restrict on which queue/exchange to enable tracing , therby giving the flexibility to disable tracing where it is not required.

Refer the following links to configure tracing:
https://www.rabbitmq.com/firehose.html
https://www.rabbitmq.com/blog/2011/09/09/rabbitmq-tracing-a-ui-for-the-firehose/