Linked Questions

Popular Questions

How to speed up parsing json and writing in the database?

Asked by At

I need to parse json file size of 200MB, at the end I would like to write data from the file in sqlite3 database. I have a working python code, but it takes around 9 minutes to complete the task.

@transaction.atomic
def create_database():
    with open('file.json') as f:
         data = json.load(f)
         cve_items = data['CVE_Items']


        for i in range(len(cve_items)):
            database_object = Data()
            for vendor_data in cve_items[i]['cve']['affects']['vendor']['vendor_data']:
            database_object.vendor_name = vendor_data['vendor_name']


                for description_data in cve_items[i]['cve']['description']['description_data']:
                    database_object.description = description_data['value']


                for product_data in vendor_data['product']['product_data']:
                    database_object.product_name = product_data['product_name']
                    database_object.save()


                    for version_data in product_data['version']['version_data']:
                        if version_data['version_value'] != '-':
                            database_object.versions_set.create(version=version_data['version_value'])

Is it possible to speed up the process?

Related Questions