python requests ChunkedEncodingError when receiving and writing some data from a server

679 views Asked by At

I am working on a python script (as a smaller part of a project) that would accept some URLs and receive the data sequentially using the requests module and write it to a file.

The thing is, I guess, that the server I am receiving from, does not handle the connection effectively. Consequently, the connection seems to get broken in the middle of the process.

For example, I was receiving a video file (~150MB) from the said server (hosted on Heroku) among some other smaller files, but while writing the video, the process ended with about just 40MB of data written (No runtime errors so far though).

Here is the code snippet:

with open(f"./files/{filename}", 'wb') as file:
    size = 0
    res = requests.get(url, headers=headers, stream=True)
    for chunk in res.iter_content(chunk_size=1024):
        size+=len(chunk)
        print(f"{round(size/(1024*1024),3)}MB",end="\r")
        if chunk:
            file.write(chunk)

I tried different values for chunk_size like 10240, 8192 etc. But all seemed to have the same problem and wrote unequal smaller sized files but never complete.

This was until I read the docs and tried to set chunk_size=None. This resulted in the following error after about 100MB of data was written:

requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(109051904 bytes read, 53405476 more expected)', IncompleteRead(109051904 bytes read, 53405476 more expected))

I know it's obvious from the exception that the connection was broken before all the bytes could be received but I had a hard time all day trying to figure out the fix.

Also, I tried try-except for the above exception but that seemed to have corrupted the video as some parts of it were missing.

So, my question is, how can I make the script reconnect, or wait for the server (or my computer) to re-establish the connection to receive those missing bytes?

0

There are 0 answers