Efficient way to read/store multiple objects in Riak Python Client?

Question

I'm writing a restore riak database tool, and I need to write hundreds of thousands of keys from one riak server to another. Doing so for about 91,000 keys takes 11 minutes (not efficient at all). The basic outline of my code is as follows:

data_objects = bucket.multiget(keys)
for data_object in data_objects:
    newEntry = bucket.new(data_object.key, data=data_object.data)
    newEntry.store()

The python app spends about 7 minutes in the single multiget() function call which creates the data_objects list of 91,000 Riak Objects, and it takes about four minutes in the 91,000 calls to the store() function.

Is there a more efficient way to do read the objects? A more efficient way to store the objects? There is a multiget function in the riak python library, but not a multistore function.

Answer 1

Don't use multiget, create your own threads, share keys to them equally, get from old bucket, store to new one. gevent.monkey_patch.all() may help too.