Append to a JSON file using requests.get

Question

I'm trying to pull information from NIST's NVD, and I'm having trouble appending to an existing JSON as I grab data. I see how the code overwrites existing data, but I am unsure of how to tell the for loop to append instead of overwrite.

#Block of code:

jsonPathMedium = (filepath)
jsonPathMediumCompressed = (filepath)

base_url = "https://services.nvd.nist.gov/rest/json/cves/2.0?cvssV3Severity=MEDIUM&resultsPerPage=2000&"
headers = {"Accept": "application/json", "Authorization": "Bearer 123456  "}
ids = ['startIndex=0', 'startIndex=2000']
jsonAppend = []
for id in ids:
    responseMedium = requests.get(base_url + str(id), headers=headers)
    jsonAppend.append(responseMedium)
    print('Grabbed data, making next request.')
print('Finishing pulling data.')
#converts data into json
jsonPrintMedium = responseMedium.json()

jsonObjectMedium = json.dumps(jsonPrintMedium, indent=4)

with open(jsonPathMedium, "w") as jsonFileMedium:
    jsonFileMedium.write(str(jsonObjectMedium))
    jsonFileMedium.close
    print('Wrote to medium severity JSON file.')
    mediumIn= open(jsonPathMedium, 'rb')
    mediumOut = gzip.open(jsonPathMediumCompressed, 'wb')
    mediumOut.writelines(mediumIn)
    mediumOut.close
    mediumIn.close
    print('Compressed medium severity JSON file.')

Answer 1

Let's think about this in words. If I understand correctly, you want to do something like this:

for each id in a list of ids
    get the JSON from an HTTP request for that specific id
    append this JSON to a list of all of the results
write the list of results to a file

You already have some of the code for this, so I will borrow from it. There are a few details you are not quite getting right and I'll comment on those later. Here's what I suggest the code should be:

base_url = "https://services.nvd.nist.gov/rest/json/cves/2.0?cvssV3Severity=MEDIUM&resultsPerPage=2000&"
headers = {"Accept": "application/json", "Authorization": "Bearer 123456  "}
ids = ['startIndex=0', 'startIndex=2000']
jsonAppend = []
for id in ids:
    responseMedium = requests.get(base_url + str(id), headers=headers)
    jsonAppend.append(responseMedium.json()) # <----- parse the JSON from the request here
    print('Grabbed data, making next request.')

json.dump(jsonPathMedium, jsonAppend) # <---- write all of the list to a single file, no need to make this more complicated

If you want to write the JSON to a compressed file, then I recommend you just do that directly. Don't write it to an uncompressed file first. I will leave the implementation as an exercise for the reader.