Python newbie questions about multiprocessing

30 views Asked by At

Python newbie question

I've been using Python for about a week and I'm focusing on speed so I've come up with a test project to learn about multithreading and multiprocessing.

I have a table of 5000 empolyee records that need to be 'analyzed' I have a table of 50 tests (in user-defined python code) I have a blank table for storing the test results by employee ID I am running in Windows on an 4-core i7 processor

a) for each employee record b) for each test c) run the test d) log the test results next test next employee

QUESTIONS:

  • Should (a) be multithreaded because it's reading an employee record from a database (I/O bound)?
  • Should (b) be multithreaded because it's reading a test record from a database (I/O bound)?
  • Should (c) be multiprocessed because it's reading/writing in memory (CPU bound)?
  • Should (d) be multithreaded because it's writing to a database (I/O bound)?

My current code is something like this:

import datetime
from multiprocessing import Process
import sqlite3

connection=sqlite3.connect('EmpTestPython.db', timeout=90)
cursor=connection.cursor()

def ForEachEmp(empId, msg):
    print(f'    ForEachEmp -> {empId}')    
    
    testProcs=[]
    for testId in range(1, 11): # actual testId numbers will come from DB
        print(f'        ForEachTest -> {empId} {testId}')
        testProc=Process(target=ForEachTest, args=(empId, testId, ''))
        testProc.start()    
    for p in testProcs:
        p.join()      

def ForEachTest(empId, testId, msg):
    print(f'            Launching testId {testId} for emp {empId}')
    testCode = GetCode(testId)
    testDate = datetime.datetime.now
    result = Eval(testCode)
    cursor.execute(
        '''insert into TestResults
        (TestDate, EmpId, TestId, Result)
        values(?, ?, ?, ?)''',
        (testDate, empId, testId, result))
    connection.commit()

def GetCode(testId):
    # this is example-only code    
    cursor.execute('''SELECT TestCode FROM Tests WHERE ID={testId}''')
    return TestCode
        
if __name__ == '__main__':
    empProcs=[]
    for empId in range(1, 51): # actual empId numbers will come from DB
        print(f'Launching empId {empId}')
        empProc=Process(target=ForEachEmp, args=(empId, ''))
        empProc.start()
        empProcs.append(empProc)
    for p in empProcs:
        p.join()
0

There are 0 answers