How do I transfer small files quickly over the network with zstd?

Question

As the question states, I want to backup many small files and send them via ssh to a destination. Does rsync speed things up significantly vs tar?

Answer 1

This works quite well, significantly faster than gzip.

Push (Upload)

tar -c --zstd src_dir | ssh user@dest_addr "cd dest_dir && tar -x --zstd"

This does the following

1. Creates a tar file using Zstd and outputs it via STDOUT
2. Connects via ssh, piping STDOUT over the network
3. Reads data from STDIN, and extracts it

Custom zstd flags

This uses maximum compression (default level is 3) and multithreading.

tar -c -I "zstd -19 -T0" src_dir | ssh user@dest_addr "cd dest_dir && tar -x --zstd"

With progress

tar -c --zstd src_dir | pv --timer --rate | ssh user@dst_addr "cd dest_dir && tar -x --zstd"

Pull (Download)

ssh user@dest_addr "tar --zstd -cf - src_dir" | tar -x --zstd --directory dest_dir