Using Hangfire with Redis, is there an easy way to partition jobs to the various Redis servers using key-hashes?

Question

We're currently using Hangfire with MSSQL and would like to up the throughput of our tasks by moving to Redis.

Sharding (partitioning) by key-hash is very simple, scalable and easy to maintain. We don't have access to Redis clusters currently. So there are two requirements we'd like to fulfill:

• Have a list of Redis servers passed into Hangfire via a config file.
• Have any given key go to either server predictably (similar to Memcached)

That way we could have any number of jobs split between 'n' number of Redis servers.

I realize that there are two parts to Hangfire: storage and jobs. I'm curious how those two components would work in a Redis-sharded environment.

I realize that third party components such as Twemproxy help with these concerns, but since the key-hashing approach is so straight-forward I thought I'd exhaust that avenue prior to implementing Twemproxy.

Thanks!

Answer 1

You're right in discerning the two part of Hangfire: the engine know nothing about the storage and viceversa.

Jobs are picked up from the storage using lock mechanisms:

When Hangfire starts, it spin-up as many thread as configured (Worker number). Each Worker ask then to the storage if there are jobs to be processed, moving them in processing state. THAT operation has to be made using some sort of locking mechanism.

How this lock is implmented is up to the storage itself, I believe that on SQL Server the storage use UPDATE MERGE SQL statements, whereas the official Redis version use Blocking commands (BRPOPLPUSH and alikes) while others like mine, leverages Redis Publish-Subscribe mechanism.

Going to your requirements, I can speak about that Redis implementation

• The list of Redis server is passed to the storage through the Storage Connection string, like "redis0:6380,redis1:6380,allowAdmin=true" (see this page for detail)
• As far as I know the Redis client supports Redis Cluster and it works sharding the Keys and letting the client connect to the correct Redis instance

I'm using replicated Hangfire server with Redis in production but I'm using only Redis master-slave failover, not Redis Cluster