I am having a hard time uploading data to my CloudKit container in a series of
'modify records' operations. I have an 'uploader' function in my app that can populate the CloudKit private database with a lot of user data. I batch the records into multiple CKModifyRecordsOperations, with 300 records in each operation as a maximum, before I upload them. When I do this with a bit of data (less than 50MB even), it can take dozens of minutes to do a simple upload. This includes a robust retry logic that takes the CKErrorRetryAfterKey key from any timed-out operations and replays them after the delay (which after happens frequently).
I checked the CloudKit dashboard, and for the container's telemetry section, the 'server latency' seems very very high (over 100,000 for 95% percentile). It also suggests the 'average request size' is 150KB on average across the last few days as I've been testing this, which doesn't seem like a lot, but the server response time is 10 seconds on each operation on average! This seems super slow.
I've tried throttling the requests so only 20 modify operations are sent at a time, but it doesn't seem to help. I have 'query' indexes for 'recordName' field for each recordType, and 'query, searchable, sortable' on some of the custom fields on the recordTypes (though not all). The CKModifyRecordsOperations' configurations have 'qualityOfService' set to 'userInitiated'. But none of this seems to help. I'm not sure what else I can try to improve the 'upload' times (downloading records seem to be happen as expected).
Is there anything else I can try to improve the time it takes to upload a few thousand records? Or is it out of my control?