how to send if-none-match header with scrapy or requests in python?

Question

I am scraping a site with Scrapy but some of it's API's are not returning JSON data without the 'if-none-match' header.

I have greater than 100 API's list so I want to generate automatic headers for getting a valid JSON file. anybody knows how to handle this or there is any other method to get rid of it.

Thanks in advance.

Answer 1

You can use the DEFAULT_REQUEST_HEADERS setting if you want to define headers for all requests:

# settings.py
DEFAULT_REQUEST_HEADERS={
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en',
    'If-None-Match': '*',
}

or the headers parameter for individuals requests:

req = scrapy.Request(url, callback=self.parse, headers={'If-None-Match': '*'})