"Slowing" Importing with Multiple Large Databases

20 views Asked by At

In Brightway2, I'm writing many prospective ecoinvent databases using premise which is fine but the project size becomes very large (20+ GB).

However, afterwards I am importing and writing large excel files also to match to various databases but the process becomes incredibly slow taking up to 20 mins and increasing (usually 1-5 min).

I would like to understand what are the main reasons for this slowing, and if and how this process can be optimized w/o needing to increase computational power?

Below, is the standard import code for the excel databases that take a long time.

    i = bw.ExcelImporter(fp)
    i.apply_strategies()
    i.match_database(f"ecoinvent_cutoff_3.9_tiam_SSP2-RCP{rcp_scenario}_{year}", 
                     fields=('name', 'unit', 'location', 'reference product'))
    i.match_database("biosphere3", fields=('name', 'unit', 'categories'))
    i.match_database(fields=('name', 'unit', 'location'))
    i.statistics()
    i.write_database()

I am going to explore to the superstructure approach for the prospective databases (https://doi.org/10.1007/s11367-021-01974-2) to see if this can speed things up, but I would incredibly value any input! :)

0

There are 0 answers