I have a very basic question on Redis specifically and single threaded applications in general
I am using Redis in Cluster mode on AWS EC2 instances. Currently I have 6 nodes in the cluster ( 3 master + 3 replica) and I am using r6a.2xlarge instances ( 4 core, 8 vCPU)
As I understand Redis is single threaded - I want to understand the implication for that for scaling
When I run htop on my instances I see mostly only 1 core is getting used and others are idle
Question If instead of 6 r6a.2xlarge nodes ( 3 master + 3 replica), I go to 12 r6a.xlarge nodes ( 6 master + 6 replica) will I be able to achieve double (or nearly double) the scale in terms of compute (assume size of data is not a constraint) at the same cost. Conversely if I go to 6 r6a.4xlarge nodes (3 master + 3 replica), will the scale I am able to achieve remain the same even though I am doubling my cost
If yes, then does it mean it always makes sense to use lower instance type for Redis (or any other software which is single threaded and always rely on horizontal scale) If not, what I am missing here
Redis version I am using in case that is relevant : 6.2.6
You should be able to setup a Redis cluster that runs multiple Redis processes for different shards on the same machine. If you do it properly you should be able to get to a much better CPU utilization. Have a look at the Redis docs on scaling.
Have you given Redis Cloud Pro a try yet? Instead of worrying about optimizing the infrastructure, you simply provide the memory and throughput you require and you are good to go. In addition, you will enjoy the benefits of Redis Enterprise including advanced capabilities and data structures. Have a look here.
Hope this helps!
Disclaimer: I work for Redis.