How to get the data/value size in Redis keys with prefix?

Question

I stored a lot of stuff in Redis. One group of them are with namespace cache (key starts with cache:). I want to know the size of the data/values with namespace cache. Can I achieve this in Redis? Any suggestions?

Answer 1

You may use scan with memory usage commands. Depending on the size of your database(you may check it with DBSIZE) - you may arrange the count option of the scan command. The following command is going to scan the database with matching to the cache: prefix.

SCAN 0 MATCH cache:* COUNT 2000

Then you may execute MEMORY USAGE on the individual keys. You can achieve it in your favorite programming language with available redis library.

The lua example could be something like this(i don't have enough experience on lua but it looks like working). It is going to return total size of the values in bytes.

local response = redis.call("SCAN", 0, "MATCH", "cache:*", "count", 2000)
local keys = response[2]
local total = 0

for i = 1, #keys do
    total = total + redis.call("MEMORY", "USAGE", keys[i])
end

return total

it may not be the best "performing" solution for large databases. you may need to update your cursor.

Edit: As @for_stack pointed out in the comment, it will not work when the count is less than your total key size when the count is less, it needs to be iterated multiple times.

Answer 2

You can do this with RedisGears (https://oss.redislabs.com/redisgears/) with a single line:

RG.PYEXECUTE "GB().map(lambda x: int(execute('MEMORY', 'USAGE', x['key']))).aggregate(0, lambda a,x: a+x, lambda a,x: a+x).run('cache:*')"

The first map operation get the size of each key and the aggregate operation sums it. the argument to the run function is the keys prefix to run on.

Answer 3

nodejs snippet using ioredis:

const Redis = require("ioredis");
let redisClient = new Redis({
  port: 6379, // Redis port
  host: "127.0.0.1", // Redis host
  username: "default", // needs Redis >= 6
  password: "my-top-secret",
  db: 0, // Defaults to 0
});
async function calculateKeysSize(redisClient, matchPattern) {
  let iterations = 0;
  let totalKeys = 0;
  let totalBytes = 0;
  let nextCursor, currCursorKeys;
  while (nextCursor !== "0") {
    [nextCursor, currCursorKeys] = await redisClient.scan(nextCursor || 0, "match", matchPattern);

    totalKeys += currCursorKeys.length;

    const pipeline = redisClient.pipeline();
    for (const currKey of currCursorKeys) {
      pipeline.memory("usage", currKey);
    }
    const responses = await pipeline.exec();
    const sizes = responses.map((response) => response[1]);
    totalBytes += sizes.reduce((a, b) => a + b, 0);

    if (iterations % 1000 == 0) {
      console.log(`scanned ${totalKeys} so far.. total size: ${totalBytes} Bytes`);
    }
    iterations++;
  }

  return { totalKeys, totalBytes };
}

await calculateKeysSize(redisClient, "cache:*");

gist link for ready-to-use script.