How to get data from SEC Edgar python and a json

Question

on the following page below there is as Data source a json link: https://www.sec.gov/edgar/browse/?CIK=1067983&owner=exclude Data source: CIK0001067983.json -> https://data.sec.gov/submissions/CIK0001067983.json

This is my code (it works fine!):

headers = {
"Host": "www.sec.gov",
"User-Agent": "jo boulement [email protected]",
"Accept-Encoding": "gzip, deflate" 
}

sec_url = "https://data.sec.gov/submissions/CIK0001067983.json"
resp = requests.get(sec_url, headers=headers)
with open("e:\\sec_api_of_1448574_7.html", "w", encoding="utf-8") as my_file:
my_file.write(resp.text)

but as result I get a file looks like this: enter image description here

Error 404: Page Not Found Oops! Page Not Found.

What is here going wrong? The json-link: https://data.sec.gov/submissions/CIK0001067983.json is fine, because download by hand from the page works fine. Hope somebody could give me a hint! Thx!

Answer 1

The HTML that gets returned includes this <script> tag:

<script src="/files/js/js_DkdESgtfPfV7guog-Lhz7nda0K-ISZe0-gHU4CF6Wo0.js"></script>

My guess is that the script referenced by the tag is what causes the JSON data to be returned. A browser will run that script as part of rendering the HTML. The Requests package doesn't do this. It just returns the raw HTML. You might need to use something like Puppeteer or Selenium to get the JSON via that URL.

Answer 2

A web server checks the headers that you send in your request and might decide to return an error page if you don't include certain headers. In this case, it looks like they return an error if you don't include a valid user agent.

This works for me:

import requests

headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36'}

url = "https://data.sec.gov/submissions/CIK0001067983.json"

payload={}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Answer 3

thx, for your help ... I have the solution ...

the documentation of the sec.gov says the following:

https://www.sec.gov/os/webmaster-faq#user-agent

but the header "HOST" lead to the "404 page not found" ...

but this header works fine:

headers = {
"User-Agent": "jo boulement [email protected]",
"Accept-Encoding": "gzip, deflate" 
}

crazy! because the documentation says something else :(