File cannot be found when i use #Fabric put# in multi-threading to copy a temp file created by tempfile.mkstemp to a remote file

Question

Python Version: 2.6.4

Fabric Version:1.9.0

I have an automation testing framework to execute cases in parallel by using threading.Thread(3 threads in my case).

Each thread worker uses fabric put(we do some wrapper on this function though) to copy a temporary file created by tempfile.mkstemp to a remote file.

The question is that it always gives me an error that file cannot be found, the error happens during 'put' from the exception tips.

here is the code when do 'put':

MyShell.py(Parent class of MyFabShell)

def putFileContents(self, file, contents):

    fd, tmpFile= tempfile.mkstemp()

    os.system('chmod 777 %s' % tmpFile)

    contentsFile = open(tmpFile, 'w')
    contentsFile.write(contents)
    contentsFile.close()

    dstTmpFile = self.genTempFile()

    localshell = getLocalShell()

    self.get(localshell, tmpFile, dstTmpFile) # use local shell to put tmpFile to dstTmpFile


    os.close(fd)

    os.remove(tmpFile)
    #os.remove(tmpFile)

MyFabShell.py:

def get( self, srcShell, srcPath, dstPath ):
    srcShell.put( self, srcPath, dstPath )

def put(self, dstShell, srcpath, dstpath):

    if not self.pathExists( srcPath ):  # line 158
        raise Exception( "Cannot put <%s>, does not exist." % srcPath )

    # do fabric get/put in the following
    # ...

The call of put results in an error:

...
self.shell.putFileContents(configFile, contents)
File "/path/to/MyShell.py", line 401, in putFileContents
self.get(localShell, tmpFile, dstTmpFile)
File "/path/to/MyFabShell.py", line 134, in get
srcShell.put( self, srcPath, dstPath )
File "/path/to/myFabShell.py", line 158, in put
raise Exception( "Cannot put <%s>, does not exist." % srcPath )
Exception: Cannot put </tmp/tmpwt3hoO>, does not exist.

I initially doubt the file could be removed during put, so I commented os.remove. However, I got the same error again. From the exception log, it should not be the problem of 'fabric put' since exception throws before the execution of fabric get/put Is mkstemp NOT safe when multithreading is involved? but the document says that "There are no race conditions in the file’s creation" or does my case fail because of GIL? I suspect this is because when I use only 1 thread, everything will be fine.

Could anyone give me some clue on my error? I have being struggling with the problem for a while:(

Answer 1

My problem is solved when i 'join' the thread explicitly. all my threads are not daemon threads, and each of thread has pretty much of I/O operation(e.g. file write/read). 'join' explicitly will make sure each thread's job is completed. I am still not sure the root cause of my problem...the temp file is actually there but the thread complains "cannot find" when multiple thread are working together, the only guess i can give is: when a thread A does I/O operation, the thread A will release GIL so that thread B(or C, D...) can acquired GIL during A's I/O operation time. the problem might happen during the I/O time because Thread A is not in the Python Interpreter any more...that's the reason "file cannot be found" by A, however, When we join A explicitly, A will always make sure to complete its job by reentering the GI(Global Interpreter).