Best way to ensure an event is eventually published to a message queuing sytem

2.7k views Asked by At

Please, imagine you have a method like the following:

public void PlaceOrder(Order order)
{
     this.SaveOrderToDataBase(order);
     this.bus.Publish(new OrderPlaced(Order));    
}

After the order is saved to the database, an event is published to the message queuing system, so other subsystems on the same or another machine can process it.

But, what happens if this.bus.Publish(new OrderPlaced(Order)) call fails? Or the machine crashes just after saving the order into the database? The event is not published and other subsystems cannot process it. This is unacceptable. If this happens I need to ensure that the event is eventually published.

What are the acceptable strategies can I use? Which is the best one?

NOTE: I don't want to use distributed transactions.

EDIT:

Paul Sasik is very close, and I think I can achieve 100%. This is what I thought:

first create a table Events in the database like the following:

CREATE TABLE Events (EventId int PRIMARY KEY)

You may want to use guids instead of int, or you may use sequences or identities.

Then do the following pseudocode:

open transaction
save order and event via A SINGLE transaction
in case of failure, report error and return
place order in message queue
in case of failure, report error, roll back transaction and return
commit transaction

All events must include EventId. When event subscribers receive an event, they first check EventId existence in database.

This way you get 100% realiability, not only 99.999%

2

There are 2 answers

6
Jesús López On BEST ANSWER

The correct way to ensure the event is eventually published to the message queuing sytem is explained in this video and on this blog post

Basically you need to store the message to be sent into the database in the same transaction you perform the bussines logic operation, then send the message to the bus asynchronously and delete the message from the database in another transaction:

public void PlaceOrder(Order order)
{
     BeginTransaction();
     Try 
     {
         SaveOrderToDataBase(order);
         ev = new OrderPlaced(Order);
         SaveEventToDataBase(ev);
         CommitTransaction();
     }
     Catch 
     {
          RollbackTransaction();
          return;
     }

     PublishEventAsync(ev);    
}

async Task PublishEventAsync(BussinesEvent ev) 
{
    BegintTransaction();
    try 
    {
         await DeleteEventAsync(ev);
         await bus.PublishAsync(ev);
         CommitTransaction();
    }
    catch 
    {
         RollbackTransaction();
    }

}

Because PublishEventAsync may fail you have to retry later, so you need a background process for retrying failed sendings, something like this:

foreach (ev in eventsThatNeedsToBeSent) {
    await PublishEventAsync(ev);
}
3
Paul Sasik On

You can make the this.bus.Publish call part of a database transaction of the this.SaveOrderToDataBase. This means that this.SaveOrderToDataBase executes in transaction scope and if the db call fails you never call the mq and if the mq call fails then you roll back the db transaction leaving both systems in a consistent state. If both calls succeed you commit the db transaction.

Pseudocode:

open transaction
save order via transaction
in case of failure, report error and return
place order in message queue
in case of failure, report error, roll back transaction and return
commit transaction

You didn't mention any specific db technology so here's a link to a wiki article on transactions. Even if you're new to transactions, it's a good place to start. And a bit of good news: They are not hard to implement.