Uploads via ftplib in python taking signicantly longer than doing it manually

Question

SO,

For some reason the little bit of code below is taking SIGNIFICANTLY longer to upload than if I was to upload via a FTP program or via the website itself.

The files come to 600kb which via ftp programs/directly via the web takes 1-3 seconds max however through ftplib it takes 15-30 seconds... that's over 15 times slower

The files that I'm uploading are single line html/php files (minified), is there a faster way to upload...?

In my actual code, I am uploading 25+ files all coming to a max of 1mb in total file size, is there a batch upload method with the .tmp renaming (prevents partially updated pages from loading) which would work better for me?

import ftplib

print "Uploading FULL.html"
## Login using the ftplib library and set the session as the variable ftp_session
ftp_session = ftplib.FTP('ftp.website.com','[email protected]','Password1','',60)
## Open a file to upload
ftp_file = open('OUTPUT/DATA/FULL.html','r')
## Open a folder in the ftp server
ftp_session.cwd("/DATA/")
## Send/upload the file to the ftp server
ftp_session.storlines('STOR FULL.html.tmp', ftp_file)
## Close the ftp_file
ftp_file.close()
## Rename temp file
ftp_session.rename("FULL.html.tmp", "FULL.html")

print "Uploading index.php"
## Open a file to upload
ftp_file = open('OUTPUT/index.php','r')
## Send/upload the file to the ftp server
ftp_session.storlines('STOR index.php.tmp', ftp_file)
## Close the ftp_file
ftp_file.close()
## Rename temp file
ftp_session.rename("index.php.tmp", "index.php")

print "Uploading secondary.php"
## Open a file to upload
ftp_file = open('OUTPUT/secondary.php','r')
## Send/upload the file to the ftp server
ftp_session.storlines('STOR secondary.php.tmp', ftp_file)
## Close the ftp_file
ftp_file.close()
## Rename temp file
ftp_session.rename("secondary.php.tmp", "secondary.php")

## Quit out of the FTP session
ftp_session.quit()

Example upload output....

Running...
Uploaded MAIN.html (284KB) in 9.677 seconds at 29.40 kbps
Uploaded GROUP1.html (12KB) in 0.660 seconds at 19.12 kbps
Uploaded GROUP10.html (9KB) in 1.137 seconds at 8.07 kbps
Uploaded GROUP11.html (14KB) in 1.014 seconds at 14.56 kbps
Uploaded GROUP12.html (17KB) in 0.913 seconds at 18.87 kbps
Uploaded GROUP13.html (12KB) in 0.675 seconds at 18.66 kbps
Uploaded GROUP14.html (13KB) in 0.738 seconds at 18.54 kbps
Uploaded GROUP15.html (16KB) in 0.971 seconds at 16.53 kbps
Uploaded GROUP16.html (14KB) in 1.755 seconds at 8.44 kbps
Uploaded GROUP17.html (13KB) in 1.504 seconds at 9.05 kbps
Uploaded GROUP18.html (14KB) in 0.725 seconds at 20.47 kbps
Uploaded GROUP19.html (14KB) in 1.691 seconds at 8.76 kbps
Uploaded GROUP2.html (10KB) in 0.541 seconds at 18.90 kbps
Uploaded GROUP20.html (13KB) in 0.799 seconds at 16.69 kbps
Uploaded GROUP21.html (14KB) in 0.713 seconds at 20.42 kbps
Uploaded GROUP22.html (14KB) in 0.776 seconds at 18.61 kbps
Uploaded GROUP23.html (14KB) in 0.592 seconds at 24.99 kbps
Uploaded GROUP24.html (13KB) in 0.734 seconds at 18.86 kbps
Uploaded GROUP25.html (20KB) in 0.845 seconds at 24.49 kbps
Uploaded GROUP26.html (18KB) in 0.637 seconds at 28.91 kbps
Uploaded GROUP27.html (11KB) in 0.781 seconds at 14.54 kbps
Uploaded GROUP28.html (16KB) in 0.798 seconds at 20.25 kbps
Uploaded GROUP29.html (12KB) in 0.542 seconds at 23.46 kbps
Uploaded GROUP3.html (16KB) in 0.585 seconds at 27.47 kbps
Uploaded GROUP4.html (14KB) in 0.622 seconds at 23.97 kbps
Uploaded GROUP5.html (14KB) in 0.716 seconds at 20.54 kbps
Uploaded GROUP6.html (18KB) in 1.109 seconds at 16.48 kbps
Uploaded GROUP7.html (11KB) in 0.654 seconds at 17.52 kbps
Uploaded GROUP8.html (9KB) in 0.560 seconds at 16.25 kbps
Uploaded GROUP9.html (12KB) in 0.582 seconds at 21.42 kbps
The whole routine took 72.468 seconds

Answer 1

There are several reasons that your transfer is about as slow as possible and the application is sending much faster:

1. You are sending one line at a time, the ftp application will send one block of 8192 bytes at a time,
2. You are sending in text mode, the application is probably sending in binary compressed mode,

Unless there is a problem with line endings use open(filename, "rb") then tp_session.storbinary('STOR index.php.tmp', ftp_file) you can also try sending the command 'MODE COMPRESSED' and checking the reply and you could play with the block size to find the optimum.

The other thing I would set up would be something like:

import glob, os, ftplib
from time import time

sourcedir = '/where/your/files/are/locally'
destdir = '/where/to/put/them'
##    filelist = ['file2.ext', 'file2.ext', 'etc']
filelist = glob.glob(os.path.join(sourcedir, '*.php'))

ftp_session = ftplib.FTP('ftp.website.com','[email protected]','password123','',60)
ftp_session.cwd(destdir)

for fname in filelist:
    filename = fname.replace(sourcedir+'\\', "")
    ftp_file = open(fname,'rb')
    result = ftp_session.storbinary('STOR %s.tmp' % filename, ftp_file)
    ftp_file.close()
    if 199 < int(result[0:3]) < 300: # Positive Completion
        ftp_session.rename('%s.tmp' % filename, filename)
    else:
        print "Upload of %s resulted in code %s" % (filename, result)
        break

You could potentially zip or tar.gz your files and send that but you would need a secure socket connection and appropriate permissions to unpack separately.

Answer 2

define blocksize along with storbinary of ftp connection,so you will get 1.5-3.0x more faster connection than FTP Filezilla :)

from ftplib import FTP

USER = "Your_user_id"
PASS = "Your_password"
PORT = 21
SERVER = 'ftp.billionuploads.com' #use FTP server name here

ftp = FTP()
ftp.connect(SERVER, PORT)
ftp.login(USER, PASS)

try:
    file = open(r'C:\Python27\1.jpg','rb')
    ftp.storbinary('STOR ' + '1.jpg', file,102400) #here we store file in 100kb blocksize
    ftp.quit()
    file.close()
    print "File transfered"
except:
    print "Error in File transfering"