Archiving, compressing and copying large number of files over a network

Question

I am trying to copy files exceeding 500000 (around 1 TB) , through ssh however the pipe fails as I have exceeded the time limit for the ssh into the remote computer,

so I moved on to archiving and compressing (using tar and gzip) all the files on the remote computer, however even if i leave the process in the background, since I exceed the time for ssh'ing' into the remote computer the process is cancelled.

finally, I moved on to compressing the files one by one and then tarring ( based on a suggestion that archiving consumes a lot of time for large number of files ) however, I get error that argument list is too long.

Since all these files are spread in 20 such folders, I do not want to enter each and divide into further folders and archiving & compressing it.

Any suggestions would be really helpful.

Answer 1

Definitely tar and gz either the whole thing or the 20 directories individually (I would do the latter to divide and conquer at least a little.) That reduces overall transfer time and provides a good error check on the other end.

Use rsync through ssh to do the transfer. If it gets hosed in the middle, use rsync --append to pick up where you left off.