In Brightway2, I'm writing many prospective ecoinvent databases using premise which is fine but the project size becomes very large (20+ GB).
However, afterwards I am importing and writing large excel files also to match to various databases but the process becomes incredibly slow taking up to 20 mins and increasing (usually 1-5 min).
I would like to understand what are the main reasons for this slowing, and if and how this process can be optimized w/o needing to increase computational power?
Below, is the standard import code for the excel databases that take a long time.
i = bw.ExcelImporter(fp)
i.apply_strategies()
i.match_database(f"ecoinvent_cutoff_3.9_tiam_SSP2-RCP{rcp_scenario}_{year}",
fields=('name', 'unit', 'location', 'reference product'))
i.match_database("biosphere3", fields=('name', 'unit', 'categories'))
i.match_database(fields=('name', 'unit', 'location'))
i.statistics()
i.write_database()
I am going to explore to the superstructure approach for the prospective databases (https://doi.org/10.1007/s11367-021-01974-2) to see if this can speed things up, but I would incredibly value any input! :)