Linux - sh script - download multiple files from FTP

Question

I need script that can download files from ftp via my sh code

I have use expect with ftp, but if I do for loop inside code, I got

wrong # args: should be "for start test next command" while executing "for v in "a b c""

My code

/usr/bin/expect << EXCEPT_SCRIPT
set timeout 1
spawn ftp -n ${HOST}
send "user ${USER} ${PASSWD}\n" 
expect "ftp>"
send "bin\n"
expect "ftp>"
send "dir\n"

for v in "${files_to_download[@]}"
do
  ftp_file="${v}.bz2"

  #download file
  echo ${ftp_file}  

  #put code to dl here

done 

 expect "ftp>"
 send "bye\n"
EXCEPT_SCRIPT

Answer 1

wget

expect can be tricky to work with so I'd prefer to use GNU Wget as an alternative. The following should work as long as you don’t have any spaces in any of the arguments.

for v in "${files_to_download[@]}"
do
  ftp_file="${v}.bz2"
  wget --user=${USER} --password=${PASSWD} ${HOST}/${ftp_file}
done

Request multiple resources using only one FTP connection

If it’s an issue, you can avoid having to make multiple FTP connections to the server by using FTP globbing (wildcard characters), e.g.

wget --user=${USER} --password=${PASSWD} "${HOST}/*.bz2"

Make sure the URL is quoted so the shell doesn’t expand the wildcard. You can also use the -nc, --no-clobber option to avoid re-downloading files that have already been downloaded.

On the other hand, requesting multiple resources (wget url2 url2 ...) does result in multiple FTP logins.

Note: I checked both of the above operations by monitoring Port 21 on my own FTP server with tcpdump.

ncftp

You may also be interested in ncftp. While I haven’t used it in years, I found it – and its ncftpget and ncftpput utilities – much easier to use for command line scripting than the standard UNIX ftp program.