Migrating and merging multiple standalone Redis instances into a new Redis cluster on AWS Elasticache

Question

Right now we have our data split between three standalone Redis instances, no clustering or sharding. We're switching from our current cloud service provider over to AWS and want to store all this data in a new Elasticache Redis cluster so we can scale horizontally.

Unfortunately, the AWS-CLI tool for migrating does not work when the target instance is in clustered mode and I'm unable to manually use the SLAVEOF command in ElastiCache. It also looks like the dump.rdb files are only able to be imported when you first start up your redis-server or when first creating your new EC cluster.

Any pointers on how to achieve this migration and merge process would be appreciated. Is there a way I can merge multiple dump.rdb files into one? Or somehow pipe each server's data to the configuration endpoint / master node one by one?

Answer 1

Not an elegant solution, but here's what I did. I had to upload and consolidate all of our Redis data into a single instance.

Spun up a new Redis instance with enough memory to hold the data of all three of our Redis instances and then installed redis-rdb-tools which allows you to parse and convert dump.rdb files into a different format:

https://github.com/sripathikrishnan/redis-rdb-tools

For each dump.rdb file, I ran the following command which generates files consisting of Redis protocol which can later be piped into the new Redis server:

rdb -c protocol dump1.rdb > dump1.protocol

Piped each generated file into the new redis instance:

cat dump1.protocol | redis-cli --pipe

After that was done, ran a BGSAVE (background save command) using redis-cli to generate a single dump.rdb file.

You can then upload the dump.rdb file to S3 and select it to seed all the data when setting up a new Redis cluster in ElastiCache.

Answer 2

Have you looked to the RIOT project:

https://github.com/redis-developer/riot

You can do a live migration. -- I have to admit that I have used to migrate data from Elasticache to Redis Cloud, but you can use it in any direction.