Why am i getting a redis timeout error when multi threading with the stackexchange.redis multiplexer?

40 views Asked by At

I'm facing a timeout problem using StackExchange.Redis library in C#, especially when multi-threading.

I have a very simple algorithm: Set a key, wait for 5s to simulate some database queries or other IO stuff, and get this same key. And do this with as much parallel tasks as you can.

Here is a code snippet (a lot simplier that what I really want to do):

using StackExchange.Redis;

var start = DateTime.Now.Ticks;
int globalCounter = 0;

var connectionMultiplexer = new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect("localhost,asyncTimeout=1000,syncTimeout=1000"));
var _redisDatabase = connectionMultiplexer.Value.GetDatabase();
var _redisServer = connectionMultiplexer.Value.GetServer(connectionMultiplexer.Value.GetEndPoints()[0]);

List<Task<string>> tasks = new List<Task<string>>();

for (int i = 0; i < 20; i++)
{
    tasks.Add(RedisCallAsync());
}

await Task.WhenAll(tasks);

Console.WriteLine("End : " + (DateTime.Now.Ticks - start));

async Task<string> RedisCallAsync()
{
    globalCounter++;
    int counter = globalCounter;
    Console.WriteLine(DateTime.Now.ToString("HH:mm:ss.fff") + " - Starting thread #" + counter);
    await _redisDatabase.StringSetAsync("key" + counter, "value" + counter, TimeSpan.FromMinutes(1));
    System.Threading.Thread.Sleep(5000);
    var result = await _redisDatabase.StringGetAsync("key" + counter);
    Console.WriteLine(DateTime.Now.ToString("HH:mm:ss.fff") + " - Killing thread #" + counter);

    return result;
}

I have 6 cores on my machine, and i'm running 20 thread at once. I'm running this code on WSL under Windows 11.

Here is the result :

10:31:58.085 - Starting thread #1
10:31:58.093 - Starting thread #2
10:31:58.093 - Starting thread #3
10:31:58.093 - Starting thread #4
10:31:58.093 - Starting thread #5
10:31:58.093 - Starting thread #6
10:31:58.093 - Starting thread #7
10:31:58.093 - Starting thread #8
10:31:58.093 - Starting thread #9
10:31:58.093 - Starting thread #10
10:31:58.093 - Starting thread #11
10:31:58.093 - Starting thread #12
10:31:58.093 - Starting thread #13
10:31:58.093 - Starting thread #14
10:31:58.093 - Starting thread #15
10:31:58.093 - Starting thread #16
10:31:58.093 - Starting thread #17
10:31:58.093 - Starting thread #18
10:31:58.093 - Starting thread #19
10:31:58.093 - Starting thread #20
10:32:08.042 - Killing thread #11
10:32:08.097 - Killing thread #14
10:32:08.097 - Killing thread #12
10:32:08.097 - Killing thread #13
10:32:08.097 - Killing thread #17
10:32:08.097 - Killing thread #15
10:32:08.097 - Killing thread #16
10:32:09.038 - Killing thread #18
10:32:10.053 - Killing thread #19
10:32:11.041 - Killing thread #20
Unhandled exception. StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 1940ms elapsed, timeout is 1000ms), command=GET, next: GET key3, inst: 0, qu: 0, qs: 8, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 96, in-pipe: 0, out-pipe: 0, last-in: 2, cur-in: 0, sync-ops: 0, async-ops: 28, serverEndpoint: localhost:6379, conn-sec: 7, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: LSXB-001468(SE.Redis-v2.7.17.27058), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=11,Free=32756,Min=6,Max=32767), POOL: (Threads=11,QueuedItems=7,CompletedItems=44,Timers=1), v: 2.7.17.27058 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__RedisCallAsync|1>d.MoveNext() in /home/cristof/Dev/testredis/console/Program.cs:line 28
--- End of stack trace from previous location ---
   at Program.<Main>$(String[] args) in /home/cristof/Dev/testredis/console/Program.cs:line 17
   at Program.<Main>(String[] args)

As you can see, one of the redis operation is taking more than the configured timeout (1000ms here). How can redis take more than a second to handle such a little query (set + get) ?

I tried a lot of things, without really understand them : longrunning task, task factory, ...

I suspect threading oversubscription : maybe am i asking too much ? (Really, waiting is too much ?). When i try to avoid threading oversubscription with this code update :

for (int i = 0; i < 4; i++)
{
    for (int j = 0; j < Environment.ProcessorCount - 1; j++)
    {
        tasks.Add(RedisCallAsync());
    }
    await Task.WhenAll(tasks);
}

It works, but in my case it's 4 times slower (because I sleep 4 times instead of 1). Moreover, how can I be sure that no other threads are running, which could create oversubscription?

Now 2 clear questions: Why am i getting a timeout witout handling ProcessCount? How can i optimize this code to avoid waiting for every chunk (depending of ProcessCount)?

0

There are 0 answers