This post comes from my recent attempt at solving https://github.com/ropensci/targets/issues/1172 (more background: https://books.ropensci.org/targets/cloud-storage.html). In a reproducible pipeline, the targets
R package needs to check the existence of a specific version of each AWS S3 object in its data store. The version of a given object is the version ID recorded in the metadata, which may or may not be the most current version in the bucket. Currently, targets
accomplishes this by sending a HEAD request for each relevant object-version pair.
Is there a more efficient/batched way to do this for each version/object pair? list_object_versions()
returns every version of every object of interest, which is way too many versions to download efficiently, and neither list_objects()
nor list_objects_v2()
return any version IDs at all.