What happen if gsutil mv command fails?

Question

Suppose we have 50 objects and we want to move them from bucket_a to bucket_b with gsutil. We run the command and it fails halfway through. Will we have 25 objects in bucket_a and 25 objects in bucket_b? Is there any official documentation about this?

gsutil mv gs://bucket_a/* gs//bucket_b/

Answer 1

gsutil, like gcloud only a convenient tool that wrap API calls to the services. If you have a look to CLoud Storage API, you can see that the "move" operation does not exist.

In fact the move is a combination of 2 operations:

1. Copy the source to the destination
2. Delete the source

Note that is the same scenario even if the source and destination are in the same bucket

Therefore, if your command fail or if you lost the network, it depends on where it fails. If it fail during the copy call, the object won't be deleted and you could have the source object and the destination object in the same time (the deletion has not been called due to the issue).

Note: here, if the copy API call has been sent, and even if the gsutil lost the connection, the API call is still completed on the Cloud side

Answer 2

As mentioned in other answers, the API call(s) that fails will determine the end state. That is, either the initial copy operation of the move could fail, meaning your object never got moved to the new bucket, or the delete op could fail, meaning your object was copied to the new bucket but not removed from the old one.

And yes, gsutil's docs do mention the failure behavior for these commands (cp, mv, etc.). If you're not utilizing a flag to continue upon encountering an error, gsutil will fail on the first non-recoverable error it encounters in the given sequence of operations. In the docs for the top-level options, specifically for -m:

Also, although most commands normally fail upon encountering an error when the -m flag is disabled, all commands continue to try all operations when -m is enabled with multiple threads or processes, and the number of failed operations (if any) are reported as an exception at the end of the command's execution.

And in the docs for the cp command's options (which are referenced by docs for the mv command), under -c:

-c

If an error occurs, continue attempting to copy the remaining files. If any copies are unsuccessful, gsutil's exit status is non-zero, even if this flag is set. This option is implicitly set when running gsutil -m cp...