Windows PermissionError & tempfile module

Question

We use tempfile module to generate temporary files & dirs during our tests. We recently started testing on Windows as well and ran into a bunch of Windows PermissionErrors.

I initially thought this may be because Windows, unlike Unix does not allow simultaneous access to a file. When I looked at the code, this was happening even without simultaneous file access.

The one common thread seemed to be the error happened when we used tempfile module's structures in context (with statement). The error disappears when we use a manual try-except-finally.

I am working on ways to fix this, the aforementioned try-except-finally is on the table as well.

Anyone else has encountered this or has any insight?

This is our test function (nosetests but I don't think that's the problem)

from nibabel.tmpdirs import InTemporaryDirectory
def test_high_level_glm_with_data():
    with InTemporaryDirectory():
        shapes, rk = ((7, 8, 7, 15), (7, 8, 7, 16)), 3
        mask, fmri_data, design_matrices = write_fake_fmri_data(shapes, rk)
        multi_session_model = FirstLevelModel(mask=mask).fit(
            fmri_data, design_matrices=design_matrices)
        z_image = multi_session_model.compute_contrast(
            np.eye(rk)[:2], output_type='z_score')
        variance_image = multi_session_model.compute_contrast(
            np.eye(rk)[:2], output_type='effect_variance')

        assert_array_equal(z_image.get_data() == 0., load(mask).get_data() == 0.)  # no error
        assert_true(
            (variance_image.get_data()[load(mask).get_data() > 0] > .001).all())  # error

Here's the traceback:

Traceback (most recent call last):
  File "C:\Users\kshit\AppData\Local\conda\conda\envs\nistats-py37-latest\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "C:\Users\kshit\OneDrive\workspace\nistats-org\nistats-repo\kchawla-pi\nistats\nistats\tests\test_first_level_model.py", line 104, in test_high_level_glm_with_data
    (variance_image.get_data()[load(mask).get_data() > 0] > .001).all())
  File "C:\Users\kshit\AppData\Local\conda\conda\envs\nistats-py37-latest\lib\site-packages\nibabel\tmpdirs.py", line 76, in __exit__
    return super(InTemporaryDirectory, self).__exit__(exc, value, tb)
  File "C:\Users\kshit\AppData\Local\conda\conda\envs\nistats-py37-latest\lib\site-packages\nibabel\tmpdirs.py", line 48, in __exit__
    self.cleanup()
  File "C:\Users\kshit\AppData\Local\conda\conda\envs\nistats-py37-latest\lib\site-packages\nibabel\tmpdirs.py", line 44, in cleanup
    shutil.rmtree(self.name)
  File "C:\Users\kshit\AppData\Local\conda\conda\envs\nistats-py37-latest\lib\shutil.py", line 507, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\kshit\AppData\Local\conda\conda\envs\nistats-py37-latest\lib\shutil.py", line 391, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\kshit\AppData\Local\conda\conda\envs\nistats-py37-latest\lib\shutil.py", line 389, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\kshit\\AppData\\Local\\Temp\\tmpbmgjqk03\\mask.nii'

If I do not use with InTemporaryDir(): the tests work.

Answer 1

Replacing nibabel.tmpdirs.InTemporaryDirectory with Python's tempfile.TemporaryDirectory solves the problem.

Perhaps Nibabel's version is not closing the file before cleanup? Not sure what's going on here, this solved my question.