I am trying to modify some existing Puppet code that uses rclone sync
to fetch files from an S3 bucket. The issue I'm running into is this is not exactly idempotent. From my understanding of how the existing works is that it will execute the rclone command every time, but will only update the file if the source file differs from the destination file.
This is the relevant portion of that S3 fetcher module:
# rclone sync the file
exec { "fetch_${file[s3_filename]}_from_${file[s3_bucket_name]}":
user => "bolt",
command => "/usr/bin/rclone sync --ignore-checksum --config /usr/local/etc/rclone/${file[s3_bucket_name]}.conf ${file[s3_bucket_name]}:${file[s3_bucket_name]}/${file[s3_path]}/${file[s3_filename]} /tmp/rclone",
}
# Force file permissions
-> file { "${file[s3_filename]}":
path => "/tmp/rclone/${file[s3_filename]}",
mode => '0700',
owner => "bolt",
group => "bolt",
}
# Creates MD5 file for the checksum validation
-> file { "${file[s3_filename]}.md5":
path => "/tmp/rclone/${file[s3_filename]}.md5",
content => $md5_file_content,
mode => '0700',
owner => "bolt",
group => "bolt",
}
and this is the code I am trying to implement after-the-fact:
exec { "decompress_${file[s3_filename]}":
command => "/usr/bin/tar xzf /tmp/rclone/${file[s3_filename]}",
cwd => "/tmp/rclone",
creates => "/tmp/rclone/test.tar",
}
exec { "load_docker_image_${file[s3_filename]}":
command => "/usr/bin/docker load -i /tmp/rclone/test.tar",
require => [
Exec["check_md5_${file[s3_filename]}"],
Exec["decompress_${file[s3_filename]}"],
]
}
My goal is to decompress the archive that is synced from the S3 bucket and then use docker load
on the decompressed file. The problem I'm running into is that because the rclone sync
exec
runs every time (it just doesnt change the file unless its different) the decompression exec
will always run. I am trying to find a good way to only run the decompression if the compressed archive changes.
I have tried to use a mix of
audit => content,
in the File section that forces the file permissions on the downloaded file,
and
subscribe => File["${file[s3_filename]}"],
refreshonly => true,
in the decompression exec
, but I still cant achieve what I'm looking for. Has anyone run into a similar problem?