Python - Running multiple mysql inserts without waiting for completion using a queue

1.4k views Asked by At

Problem in its most basic form. I have a multidimensional list I am generating from a csv. I have it so it operates as so:

data = [['hello', 'world'],['hello','universe']]
for i in data:
    try:
        cursor.execute("""INSERT INTO MYTABLE (word1,word2) VALUES ('%s','%s')""" % (i[0],i[1]))
        cursor.execute("""INSERT random command here""" % ())
        conn.commit()
    except Exception:
        conn.rollback()

This works. However, it has to wait for the response before it attempts to commit the next one. It takes a fair amount of time to complete going one at a time, so I was hoping to use queue/threading in order to send multiple of these queries (10 or so) at a time while it goes through a list of several hundred of these.

I have read several tutorials on queues and multithreading, but can't wrap my head around how to address specific items out of my list (or really even how queuing and multithreading work). I tried below (and a couple of other variations) but am unable to comprehend how to work with my values being called this way:

def stuff(q):
    while True:
        try:
            cursor.execute("""INSERT INTO MYTABLE (word1,word2) VALUES ('%s','%s')""" % (q.get(x[0]),q.get(x[1])))
            cursor.execute("""Random command here""" % ())
            conn.commit()
        exception Exception:
            conn.rollback()
            cursor.execute(""""insert statement here""" % ())            
        q.task_done()

q = Queue(maxsize=0)
num_threads = 2
array = [['Hello','World'],['This','Is']]

for i in range(num_threads):
    worker = Thread(target=stuff, args=(q,))
    worker.setDaemon(True)
    worker.start()

for x in array:
    q.put(x)

I am unsure of how to work with the other items in the array inside of the "stuff" function, and as a result, have just been guessing. I have a mediocre grasp of queue/threading, but all of the tutorials I have found googling are single dimension arrays. Any insight would be appreciated, as I have been having a really hard time trying to get my head around this. Thanks.

EDIT: Updated example to be more specific. I have a list of unique ticket numbers, it first queries which ones aren't in the database. The ones that aren't in, it puts into a mulidimensional list. Then I try to make the insert into the live table, but if one of the key identifiers don't match, it throws an exception, and then I want them to roll back and insert into a different table

1

There are 1 answers

0
xcs491 On

I ended up packing all my variables into q.put(), and splitting them inside the function. I am guessing there is much better ways to go, but this is the best I could figure out. I also rolled in Blender's suggestion like so. Hopefully this will help someone else:

def stuff(q):
    while True:
        try:
            one, two = q.get().split(',')
            cursor.execute("""INSERT INTO MYTABLE (word1,word2) VALUES (%s,%s)""", (one,two)))
            cursor.execute("""Random command here""", ())
            conn.commit()
        exception Exception:
            conn.rollback()
            cursor.execute(""""insert statement here""", ())            
        q.task_done()

q = Queue(maxsize=0)
num_threads = 2
array = [['Hello','World'],['This','Is']]

for i in range(num_threads):
    worker = Thread(target=stuff, args=(q,))
    worker.setDaemon(True)
    worker.start()

for x in array:
    q.put(x[0]+','+x[1])