Python http.client giving empty string when I use response.read().decode()

Question

I am using http.client to hit a POST api and get the json response. My code is working correctly when I print response.read(). However, for some reason, this response is limited to only 20 results and the total count of results is over 20,000. I want to get the complete response in a variable using response.read().decode(), I am hoping that the variable will contain the complete json string. The issue is that I am getting an empty string when I used decode(). How do I get this done? How do I get the complete results?

import http.client

host = 'jooble.org'
key = 'API_KEY'

connection = http.client.HTTPConnection(host)
#request headers
headers = {
    "Content-type": "application/json"}
#json query
body = '{ "keywords": "sales", "location": "MA"}'
connection.request('POST','/api/' + key, body, headers)
response = connection.getresponse()
print(response.status, response.reason)
print(response.read())
print(response.read().decode())

Answer 1

Don't call response.read() twice. response is a stream, so each call to read() continues from where the previous one ended. Since the first call is reading the entire response, the second one doesn't read anything.

If you want to print the encoded and decoded response, assign response.read() to a variable, then decode that.

data = response.read()
print(data)
print(data.decode())

But this can be done more simply using the requests module.

import requests

host = 'jooble.org'
key = 'API_KEY'

body = { "keywords": "sales", "location": "MA"}

response = requests.post(f'https://{host}/api/{key}', json=body)

print(response.content)

Note that in this version body is a dictionary, not a string. The json parameter automatically converts it to JSON.