I have a database that I connect to but queries take 30s to get back a response. I'm making about 50 queries each time I want to generate a report. This will take 25mins to get my report generated and is not ideal. The most I can currently do is use threads to execute my query in batches of 10 with 10 database connections. I'm fairly new to python so I'm having a hard time finding resources for this specific task. I'm using snowflake-connector to create connections to my database. Since the most connections I can use is 10. I know I have to create a class to handle these connections but I can't figure out the logic in Python to know when a connector is ready to be used to then attach it to a thread to execute a query on the connector. I'm not sure how to handle the main thread as it's waiting for connections to finish executing their query. Do I use sleep or wait on the main thread. Should I use a while loop on the main thread to check if the connectors are ready? Should I store these connectors as instances of a class?
If anyone has examples they can share or any resources to help me figure this out would be greatly appreciated. I'm fairly new to threading in general and coming from a Java background, my approach in Python is just using a class but I'm not sure if this approach is "pythonic"