Puppet idempotency with rclone sync

45 views Asked by At

I am trying to modify some existing Puppet code that uses rclone sync to fetch files from an S3 bucket. The issue I'm running into is this is not exactly idempotent. From my understanding of how the existing works is that it will execute the rclone command every time, but will only update the file if the source file differs from the destination file. This is the relevant portion of that S3 fetcher module:

# rclone sync the file
exec { "fetch_${file[s3_filename]}_from_${file[s3_bucket_name]}":
  user    => "bolt",
  command => "/usr/bin/rclone sync --ignore-checksum --config /usr/local/etc/rclone/${file[s3_bucket_name]}.conf ${file[s3_bucket_name]}:${file[s3_bucket_name]}/${file[s3_path]}/${file[s3_filename]} /tmp/rclone",
}
# Force file permissions
-> file { "${file[s3_filename]}":
    path           => "/tmp/rclone/${file[s3_filename]}",
    mode           => '0700',
    owner          => "bolt",
    group          => "bolt",
}
# Creates MD5 file for the checksum validation
-> file { "${file[s3_filename]}.md5":
    path           => "/tmp/rclone/${file[s3_filename]}.md5",
    content        => $md5_file_content,
    mode           => '0700',
    owner          => "bolt",
    group          => "bolt",
}

and this is the code I am trying to implement after-the-fact:

exec { "decompress_${file[s3_filename]}":
    command => "/usr/bin/tar xzf /tmp/rclone/${file[s3_filename]}",
    cwd     => "/tmp/rclone",
    creates => "/tmp/rclone/test.tar",
}
exec { "load_docker_image_${file[s3_filename]}":
  command => "/usr/bin/docker load -i /tmp/rclone/test.tar",
  require => [
    Exec["check_md5_${file[s3_filename]}"],
    Exec["decompress_${file[s3_filename]}"],
  ]
}

My goal is to decompress the archive that is synced from the S3 bucket and then use docker load on the decompressed file. The problem I'm running into is that because the rclone sync exec runs every time (it just doesnt change the file unless its different) the decompression exec will always run. I am trying to find a good way to only run the decompression if the compressed archive changes.

I have tried to use a mix of

audit          => content,

in the File section that forces the file permissions on the downloaded file,

and

subscribe => File["${file[s3_filename]}"],
refreshonly => true,

in the decompression exec, but I still cant achieve what I'm looking for. Has anyone run into a similar problem?

0

There are 0 answers