Timeout in grails application with memcached(ElastiCache) on heavy load and big cache files

Question

I have a grails application that using memcached to store requests results all times.

On request where result stored in cache is small(100kb) works fine with 50K requests, but in one case, where the result is aproximally 800kb the application throw an exception when have more than 1K requests minute.

I tried to fix this on last two days but all answers are inconclusives. I start to think that the problem can be on ElastiCache.

For the implementation, i use the AWS ElastiCache Cluster Client, that is a fork the spymemcached.

I tried to fix changing the client to default spymemcached and xmemcached but the error is the same.

The exception is: java.util.concurrent.ExecutionException: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out.

My service implementation:

import net.spy.memcached.*
import grails.converters.JSON
import org.springframework.beans.factory.InitializingBean

class MemCacheService implements InitializingBean {

    String ELASTIC_CACHE_CLUSTER_IP = "teste.ptsfty.cfg.use1.cache.amazonaws.com";
    def ELASTIC_CACHE_CLUSTER_PORT = 11211
    def MemcachedClient client
    def expireTime = 21600

    def void afterPropertiesSet() {
        client = new MemcachedClient(new InetSocketAddress(ELASTIC_CACHE_CLUSTER_IP, ELASTIC_CACHE_CLUSTER_PORT));
    }

    def getDataOnCache(key) {
        try {
            def result = client.get(key)
            if (result) return JSON.parse(result)
            else return  null
        } catch(Exception e) {
            println e.getMessage()
            return null
        } 
    }

    def saveOnCache(key, value) {
        try {
            def result = client.add(key, expireTime, value);
            return result
        } catch(Exception e) {
            println e.getMessage()
            return null
        }

    }

    def removeOnCache(key) {
        def result = client.delete(key);
        return result
    }


}

Answer 1

if the Elasticache stops working as the number of connections increases, try to increase the memcached_connections_overhead parameter. Your Elasticache instance may be out of memory for handling incoming connections. Each instance uses a small amount of memory to handle each connection, and this parameter usually has a low default value.