I am trying to download a large amount of images. I have 1,300,000+ urls to images in my database. The images are hosted on many other external servers.
The image links do serve up images, but they do so by redirecting the request to a temporary url which is then deleted. Looking at an image in chrome in the networks tab will display the text "Failed to load data: No content available because this request was redirected"
Downloading them one by one with file_put_contents() in php works, but it's very, very slow, around 2 seconds for each image, which comes out to roughly 30 days to get all the images.
What I need to do is to bulk download them, what is the best approach of doing that?
Things to note:
Creating an xmlHttp request to the image link in javascript gives a CORS error.
Looping out the links in tags and the right clicking the page and selecting save as... -> xxx.html does not save the images, due to the image redirect issue that can be seen in the chrome network tab.
You can generate .har files in chrome that does store the images as base64, but I don't know how to access this data with javascript. I suspect this data is the same one used when you copy/paste a page to Microsoft Word.