APPE FTP command transfer completion detection serverside

Question

I have a service which handles incoming files through FTP.

On my Debian server where Proftpd is installed I use an iNotify event to determine when a file transfer is complete and I can start working with it.

The event is triggered once a file is no longer written to.

This worked perfectly until I discovered that the new Nikon D4 camera and WT-5 wireless transmitter uses the APPE FTP command to send files.

From what I've read it seems the APPE FTP command sends files in data chunks which are appended to the existing file on the server after the first chunk creates the file.

However this result in several iNotify events being triggered as the file is witten to several times instead of once until the connection closes.

Since the files are handled before the file is completed they will result in an error once handled. And I always remove the file once handled.

Since this service of mine must remain fast in handling incoming files I really liked this solution with iNotify and don't really want to timing if the file-size has remained unchanged for n seconds or whatever to determine if the file transfer is complete.

My question is: Is there any way to determine if the file transfer is actually completed without having to check the file-size or comparing the last modified date?

I've tried to find a way to do this in proftpd with no avail.

The xferlog states that the same file was completed several times:

Fri May 11 14:15:41 2012 2 host-95-199-17-243.mobileonline.telia.com 131072 /var/app/incoming/file.JPG b _ i r auser ftp 0 * c
Fri May 11 14:15:43 2012 2 host-95-199-17-243.mobileonline.telia.com 262144 /var/app/incoming/file.JPG b _ i r auser ftp 0 * c
Fri May 11 14:15:47 2012 3 host-95-199-17-243.mobileonline.telia.com 385624 /var/app/incoming/file.JPG b _ i r auser ftp 0 * c

The c at the end means that the transfer was complete.

So if I had to check if the file is actually completed for every incoming file it would mean an unnecessary delay for files which are actually completed.

Seems like this shouldn't be an unusual problem, but cannot find anything about it.

Any ideas?

Answer 1

Unfortunately there's no good answer for this sort of use case. The issue is that only the FTP client knows when it is done uploading data to the server, and it is the FTP client which knows when a "file" is done. The protocol operates in terms of streams of bytes. And, as you've noticed with the APPE command, uploading can happen in chunks, in addition to just one single upload. Note that the same behavior can happen when the FTP client uses a REST command, followed by a STOR. (SFTP, for comparison, only allows uploading by chunks, i.e. it uses: OPEN, WRITE, WRITE, WRITE, ..., CLOSE, mapping more closely to the Unix system calls used for writing a file.)

You might even be tempted to use the QUIT command as a trigger, to know that that client has finished uploading all of its chunks to your server. And this might work, assuming that your FTP client opens only one single FTP session at a time.

Depending on your specific use cases/needs, it might be possible to figure out some other solution; feel free to email me about it if you'd like.

Hope this helps!

Answer 2

Got also to deal with detecting end of JPG transfers and use those solutions winch can be integrated in scripts :

Catching error: Corrupt JPEG data: premature end of data segment

A simple way to check if the JPEG data is complete or not is to check the first and last two bytes for FF D8 and FF D9 respectively. Those two bytes identify the start and end of a JPEG file respectively.

Or using ImageMagick http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=8483

Hope this might help with the D5 ;) Mathieu