Using ServiceStack Redis with Twemproxy

Question

I've been using ServiceStack PooledRedisClientManager with success. I'm now adding Twemproxy into the mix and have 4 Redis instances fronted with Twemproxy running on a single Ubuntu server.

This has caused problems with light load tests (100 users) connecting to Redis through ServiceStack. I've tried the original PooledRedisClientManager and BasicRedisClientManager, both are giving the error No connection could be made because the target machine actively refused it

Is there something I need to do to get these two to play nice together? This is the Twemproxy config

alpha:
  listen: 0.0.0.0:12112
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  redis: true
  timeout: 400
  server_retry_timeout: 30000
  server_failure_limit: 3
  server_connections: 1000
  servers:
   - 0.0.0.0:6379:1
   - 0.0.0.0:6380:1
   - 0.0.0.0:6381:1
   - 0.0.0.0:6382:1

I can connect to each one of the Redis server instances individually, it just fails going through Twemproxy.

Answer 1

I haven't used twemproxy before but I would say your list of servers is wrong. I don't think you are using 0.0.0.0 correctly.

Your servers would need to be (for your local testing):

servers:
 - 127.0.0.1:6379:1
 - 127.0.0.1:6380:1
 - 127.0.0.1:6381:1
 - 127.0.0.1:6382:1

You use 0.0.0.0 on the listen command to tell twemproxy to listen on all available network interfaces on the server. This mean twemproxy will try to listen on:

• the loopback address 127.0.0.1 (localhost),
• on your private IP (i.e. 192.168.0.1) and
• on your public IP (i.e. 134.xxx.50.34)

When you are specifying servers, the server config needs to know the actual address it should connect on. 0.0.0.0 doesn't make sense. It needs a real value. So when you come to use different Redis machines you will want to use, the private IPs of each machine like this:

servers:
 - 192.168.0.10:6379:1
 - 192.168.0.13:6379:1
 - 192.168.0.14:6379:1
 - 192.168.0.27:6379:1

Obviously your IP addresses will be different. You can use ifconfig to determine the IP on each machine. Though it may be worth using a hostname if your IPs are not statically assigned.

---

Update:

As you have said you are still having issues, I would make these recommendations:

1. Remove auto_eject_hosts: true. If you were getting some connectivity, then after time you end up with no connectivity, it's because something has caused twemproxy to think there was something wrong with the Redis hosts and reject them.

   So eventually when your ServiceStack client connects to twemproxy, there will be no hosts to pass the request onto and you get the error No connection could be made because the target machine actively refused it.

2. Do you actually have enough RAM to stress test your local machine this way? You are running at least 4 instances of Redis, which require real memory to store the values, twemproxy consumes a large amount of memory to buffer the requests it passes to Redis, this memory pool is never released, see here for more information. Your ServiceStack app will consume memory - more so in Debug mode. You'll probably have Visual Studio or another IDE open, the stress test application, and your operating system. On top of all that there will likely be background processes and other applications you haven't closed.

   A good practice is to try to run tests on isolated hardware as far as possible. If it is not possible, then the system must be monitored to check the benchmark is not impacted by some external activity.

   You should read the Redis article here about benchmarking.

3. As you are using this in a localhost situation use the BasicRedisClientManager not the PooledRedisClientManager.