I'm trying to download data from an external API. There's going to be a lot of downloads, so I want to use pipelines for easier parallelization. The way the API is set up, I can make a request to start a download job, and pass a postback url in that request. When the download job finishes, their API sends a POST to the given url. I want to do the following:
class DownloadPipeline(pipeline.Pipeline):
async = True
public_callbacks = True
def run(self, filename):
postback = self.get_callback_url()
# make API request with postback as a param
def callback(self):
# Read data from the POST
However, all the docs I've read online only have examples of GET requests on the callback url, where data is passed through a query string on the URL. Is there a way to read POST data instead?
Looks like both the POST and GET both call over to run_callback() ... so you should be able to do either