Reliable folder watching with large file copies.

Chokidar is a pretty bullet proof file watcher solution for node but one issue I have with it ( in fact it has functionality to try to elevate) is with add events being received when copying large files and the file still being in the throws of being copied. This was manifested by my file copy task signalling that the file was still busy (EBUSY) i.e. still being copied in the watch folder when it tried to copy the file itself.

The trick that it uses (when switched on) is to poll the files size with stat and only send the add event after its size hasn’t changed for  a predefined amount of time. I’ve seen other solutions for this that involve polling and checking file modified time with stat; in fact my data importer uses this; but I find it fiddly so decided to work out another solution.

The solution I use again uses polling but what is does is try to fopen the file for read; if it gets an EBUSY then it waits for another poll interval.  When the file has finished being copied and the fopen is a success, immediately close the file and pass it on to be copied.

As of present i have only tested this on Windows and Linux but the fopen interface it relies on is pretty consistent between platforms ( I might have to come back to this little gem if it still proves an issue).

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s