Python newbie question
I've been using Python for about a week and I'm focusing on speed so I've come up with a test project to learn about multithreading and multiprocessing.
I have a table of 5000 empolyee records that need to be 'analyzed' I have a table of 50 tests (in user-defined python code) I have a blank table for storing the test results by employee ID I am running in Windows on an 4-core i7 processor
a) for each employee record b) for each test c) run the test d) log the test results next test next employee
QUESTIONS:
- Should (a) be multithreaded because it's reading an employee record from a database (I/O bound)?
- Should (b) be multithreaded because it's reading a test record from a database (I/O bound)?
- Should (c) be multiprocessed because it's reading/writing in memory (CPU bound)?
- Should (d) be multithreaded because it's writing to a database (I/O bound)?
My current code is something like this:
import datetime
from multiprocessing import Process
import sqlite3
connection=sqlite3.connect('EmpTestPython.db', timeout=90)
cursor=connection.cursor()
def ForEachEmp(empId, msg):
print(f' ForEachEmp -> {empId}')
testProcs=[]
for testId in range(1, 11): # actual testId numbers will come from DB
print(f' ForEachTest -> {empId} {testId}')
testProc=Process(target=ForEachTest, args=(empId, testId, ''))
testProc.start()
for p in testProcs:
p.join()
def ForEachTest(empId, testId, msg):
print(f' Launching testId {testId} for emp {empId}')
testCode = GetCode(testId)
testDate = datetime.datetime.now
result = Eval(testCode)
cursor.execute(
'''insert into TestResults
(TestDate, EmpId, TestId, Result)
values(?, ?, ?, ?)''',
(testDate, empId, testId, result))
connection.commit()
def GetCode(testId):
# this is example-only code
cursor.execute('''SELECT TestCode FROM Tests WHERE ID={testId}''')
return TestCode
if __name__ == '__main__':
empProcs=[]
for empId in range(1, 51): # actual empId numbers will come from DB
print(f'Launching empId {empId}')
empProc=Process(target=ForEachEmp, args=(empId, ''))
empProc.start()
empProcs.append(empProc)
for p in empProcs:
p.join()