We are using R 3.3.1 and the packages ff and Rserve under Win 7 to write to an ff file on some server from separate Rserve processes on different hosts. The Rserve processes are constantly performing tasks and are updating the central ff file at the proper indices after completing the task.
A problem occurs when I am "monitoring the progress", i.e. continuously plotting the contents of the central ff file in an RGui console. I do this by opening the ff file, plotting the contents and leaving it open for further plotting. It seems that the Rserve process on the other hosts (other than the one i am monitoring from) sometimes can't finalize its changes to the ff file: The remote host Rserve process checks whether its changes were made after updating the central ff file and the check never fails (this is done by checking whether the contents at the updated position are not equal to the initial value) - yet the changes for some tasks "never make it to the file" - After all the tasks are done some updates from the remote Rserve process are still missing, even after closing and reopening the ff file.
We suspect this might have something to do with caching, an option which we leave at default when creating or manipulating ff files.
Now all seemed fine in a test run where i omitted monitoring (i.e. the ff file was not open all the time); every task could properly update the ff file at the proper indices. However we fear that there could be a similar problem when Rserve processes on different hosts are trying to update the central ff file at the same time - this is something we rather not restricted.
Please feel free to ask for code excerpts or the likes if something was left unclear.