In this code snippet, I parse json in a loop which makes the process slower. Instead of json, I tried using the rapidjson library for .loads and .dumps functions. However, the overall results are the same and no time is saved. Why could that be?
def get_data(taskid, article_list, **kwargs):
logging.info(f"Total number of ids: {len(article_list)}")
chunks = [article_list[x:x + 50] for x in range(0, len(article_list), 50)]
logging.info("Starting content API requests")
content = []
for i in chunks:
try:
response = requests.post(
'https://api-serverxxx',
headers=headers, json=i)
content_chunk = response.text
content.append(rapidjson.loads(response.text))
if ('{"message":"Forbidden"}' in content_chunk):
logging.info(f"Error during lean data request: {content_chunk}")
exit()
except Exception as e:
logging.info(f"Exception: {e}")
logging.info("Finished API requests")
logging.info("Starting writing file to S3")
try:
filename = datetime.now().strftime("%Y%m%d_%H-%M-%S").replace("-", "_").replace(":", "_")
json_data = rapidjson.dumps(content)
...
Is there any way I can use the rapidjson library to further speed up my code?