My requirement is to update partition keys in a table in DynamoDB. In order to update Partition key, the older record should be deleted and new record with updated partition key should be added. The issue is my table size is around 400 MB and it has around 2 million records.
If I scan whole table then its not efficient and high memory consumption happens. Wanted to do batch scan of tables. My ambiguity is, since new elements keep on getting added (because of partition key updation as mentioned above), batch calls mind not end itself.
I was able to do batch scan using LastEvaluatedKey that dynamodb provided in updating other attributed. Wanted to know if LastEvaluatedKey will work when updating partition key (when new record gets added and older record gets deleted).
I'm not sure how you're doing your batch job, but last evaluated key is a pointer to the last item you read in the scan. Even if that item is deleted, the pointer remains in that position and continues reading forward on subsequent requests.
You mention you continue to write into DynamoDB, of those writes don't have the correct Partition key format you're aiming to achieve, then you will have to continually start your Scan to keep getting those items which need changing.
So if you're looking to update all the items, even when new ones are coming in you should probably configure a new table, anything that you scan gets written to the new table. Any new items written to the source table will be consumed by DynamoDB stream and you place a Lambda function to put it in the new table. That means that you only scan the table once.
Keep in mind 400MB is an extremely small table. If you stopped writes for 1 minute you could make the conversion with no interruptions.