Handling concurrent file access in Common Lisp

Question

Multiple users need to access the same directory of files using an interface created in Common Lisp. Many race conditions appear when this happens. For example, when more than one user adds or deletes a file with the same time. Is there a way within lisp to "lock" a specific directory while work is being done? This would be a similar concept to the "synchronized" block in a multithreaded environment, but I have separate Lisp instances. I am using Allegro CL on Windows.

Edit: Ideas for a different solution to this problem would also be appreciated.

Answer 1

OS-level

CLISP provides stream-lock and with-stream-lock which interface to fcntl or LockFileEx. These will lock both open streams and files.

You can use FFI to call those OS function in other CL implementations.

A directory is merely a (special) file, so fcntl should be able to lock it (one has to think carefully about what it means to "write to a directory" though).

Windows world is much more complicated though. I don't think it is possible to lock a directory using a library function.

App-level

You can implement collaborative locking yourself. This would mean that only the applications using your library would respect the locking, so you will be able to fix possible issues outside the app.

E.g. (untested!):

(defun file-lock (f)
  "return the name of the lock file for this file"
  (concatenate 'sting f "-my-lock-suffix")) ; or use pathname functions...
(defun lock-file-once (f)
  "try to lock file once"
  (open (file-lock f) :direction :probe :if-exists nil))
(defun lock-file (f)
  "block until the file is locked"
  (loop :until (lock-file-once f)
    :do (sleep 1)))
(defun unlock-file (f)
  "remove the lock"
  (delete-file (file-lock f)))
(defmacro with-lock-file (f &body body)
  "lock the file, run body, unlock it"
  (let ((fn (gensym "with-lock-file-f")))
    `(let ((,fn ,f))
       (unwind-protect
            (progn (lock-file ,fn)
                   ,@body)
         (unlock-file ,fn)))))

Locking whole directories would require non-trivial finesse to avoid deadlocks: locking a directory means locking all its descendants, so acquiring a lock on a file requires first locking everyting above that file, then locking the file, then unlocking everyhing above. This opens us to a race condition.

The simple solution is to have a master lock which is required for any locking operation:

(defvar *master-lock* (pathname .....))
(defun lock-file-or-directory-once (path)
  "lock file or directory or fail"
  (with-lock-file *master-lock*
    scan everything below and also above(!) path
    return nil if any relevant locks are found,
    i.e., if anything below path is locked
    or any directory above path is locked))
(defun lock-file-or-directory (path)
  "block until success"
  (loop :until (lock-file-or-directory path)
    :do (sleep 1)))

Answer 2

Allegro CL provides a number of operating system specific locking mechanisms. Have a look at Appendix A.10 OSI file locking functions. In particular, the Windows-specific solution looks like it's locking, but that lock-stream and unlock-stream, described on the same page, are the platform agnostic versions that would make for more portable code. These do look file specific, though, since they're based on having the file already open. I don't know whether there are options for locking a whole directory or not.

locking

Function

Package: excl.osi

Arguments: stream mode length

Apply, test or remove a Windows lock on an open stream. See the Windows API documentation for _locking for more information on how to use this library function.

This function is not supported on UNIX, where it signals an error of class osi-not-supported.