Most ideal way to handle RabbitMQ error queue

1.7k views Asked by At

My application has a .net core Microservice for handling notifications and It has been deployed on Kubernetes. In there NotificationRequestConsumer as follows, (please be noted this is just a code snippet to elaborate my question)

public class NotificationRequestConsumer : IConsumer<INotificationRequest>
{
    public NotificationRequestConsumer()
    {
        
    }
    public Task Consume(ConsumeContext<INotificationRequest> context)
    {
        // notification request logic goes here
        return Task.CompletedTask;
    }
}

how configured the Masstransit in the startup.

public static IServiceCollection AddMassTransitConnection(this IServiceCollection services, IConfiguration configuration)
{
    services.AddMassTransit(x =>
    {
        x.AddBus(context => Bus.Factory.CreateUsingRabbitMq(c =>
        {
            c.Host(configuration["RabbitMQ:HostUrl"]);
            c.ConfigureEndpoints(context);
        }));
        
        x.AddConsumer<NotificationRequestConsumer>(c => c.UseMessageRetry(r => r.Interval(1,500)));
    });

    services.AddMassTransitHostedService();

    return services;
}

As per the above code, I have set interval few milliseconds to retry if any error occurs while alert processing. If there is a problem, I use the fault consumer to store the data of the relevant request in the DB, for future use (to send the relevant notification manually in the future).

public class NotificationRequestFaultConsumer : IConsumer<Fault<INotificationRequest>>
{
    public Task Consume(ConsumeContext<Fault<INotificationRequest>> context)
    {
        //For future use, I store the relevant data here 
        return Task.CompletedTask;
    }
}

Even if I did this, the relevant exception would be added to the RabbitMQ error queue. As far as I know, That is part of how transport works.

My concerns are as follows,

  1. Does the continuous growth of the error queue cause the cluster to crash?
  2. Is it a good approach to log only to the ELK Stack without throwing exceptions and not adding them to the RabbitMQ error queue?
  3. Is it possible to give specific expiration criteria to delete the error queue automatically and is it a good thought?
1

There are 1 answers

0
developer_009 On

You can use dead-letter-queues, which is a rabbitMQ built-in mechanism for handling messages in those cases refering the official documentation:

  1. The message is negatively acknowledged by a consumer using basic.reject or basic.nack with requeue parameter set to false.
  2. The message expires due to per-message TTL; or
  3. The message is dropped because its queue exceeded a length limit