Upload a huge file to Google Drive using PyDrive

Question

Currently, I am using PyDrive to upload my backup (.tar file) to google drive.

Is there a special thing to do with this library to upload a huge file to Google Drive (around 5gb). In the Google Drive API documentation, it says that we must use the Resumable upload ? https://developers.google.com/drive/web/manage-uploads

My problem is that when I try to send a huge file, the script executes without any errors, really quickly and the file does not appear in GoogleDrive. However, if I do this with a small file around 100mb, everything works perfectly fine...

My code is the following:

def upload(self, backupFile, backupFileName):

    json_data=open(os.path.join(__location__, 'client_secrets.json'))

    data = json.load(json_data)

    """Email of the Service Account"""
    SERVICE_ACCOUNT_EMAIL = data['client_email']

    """Path to the Service Account's Private Key file"""
    SERVICE_ACCOUNT_PKCS12_FILE_PATH = os.path.join(__location__, 'key.p12')

    f = file(SERVICE_ACCOUNT_PKCS12_FILE_PATH, 'rb')
    key = f.read()
    f.close()

    credentials = SignedJwtAssertionCredentials(SERVICE_ACCOUNT_EMAIL, key,
        scope='https://www.googleapis.com/auth/drive', sub='email')
    http = httplib2.Http()
    credentials.authorize(http)

    gauth = GoogleAuth()
    gauth.credentials = credentials

    drive = GoogleDrive(gauth)

    file1 = drive.CreateFile({'title': backupFileName, "parents" : [{"id":"0B7FoN03AUUdZVlNETEtWLS1VTzQ"}]} )  # Create GoogleDriveFile instance with title 'Hello.txt'

    file1.SetContentFile(backupFile);
    file1.Upload()

When I try to send a large file, no errors are returned whatsoever. The python script simply ends without anything being shown...

Answer 1

There are 100MB limits on some types of files

https://support.google.com/drive/answer/37603?hl=en

Size limits

Documents: 1,024,000 characters, regardless of the number of pages or font size. If using the new version of Drive (with the red "New" button on the left side), uploaded document files that are converted to the Google documents format can’t be larger than 50 MB. If using the classic version of Drive, you can't convert document files that are larger than 10 MB.

Spreadsheets: 400,000 cells, with a maximum of 256 columns per sheet. Uploaded spreadsheet files that are converted to the Google spreadsheets format can’t be larger than 100 MB, and need to be under 400,000 cells and 256 columns per sheet.

More information about spreadsheet size limits All spreadsheet limits mentioned above have been removed in the new version of Google Sheets. The new version of Google Sheets should support 2 million cells of data, though please note that extremely large spreadsheets may have slower performance. Learn more about switching to the new version of Google Sheets. Presentations: Presentations created in Google Slides can be up to 100 MB. Uploaded presentation files that are converted to Google Slides can also be up to 50 MB.

Drawings: We’ve never seen anyone make a drawing that was too big (but that’s not a dare).

Other files: Files that you upload but don’t convert to a Google Docs, Sheets, or Slides format can be up to 5 TB each.

Across Google Drive, Gmail, and Google+ Photos, every user is given 15 GB of free storage space, and can purchase additional storage as well.

Answer 2

You can upload any size by passing a file like object to .content of GoogleDriveFile

file1.content=fileobj

from my experience it reads in 8192 bytes at at ime first it seeks 0,2, then seeks 0,0, then calls tell() before every read(8192) and it also calls read(8192) at seek(0,2)+1

so you have to put read(),tell() and seek() methods in the object this is probably from the http.py portion of the api