WatchService performance with many directories

Question

I want to use the Java WatchService to listen for changes on a big number of directories (many hundreds of thousands) but I don't know if it is appropriate for such numbers of watched directories.

Does anyone have experience with WatchService with such numbers of directories?

If it helps, the WatchService will be used on CentOS 6.5 with an EXT4 file system.

Thanks, Mickael

Answer 1

This situation is fairly common for IDEs. They often use directory watching for complex directory structures and many 10s of thousands of files.

There is two things to note:

• On Linux you often need to tune the OS to monitor so many files. https://confluence.jetbrains.com/display/IDEADEV/Inotify+Watches+Limit

To prevent this situation it is recommended to increase the watches limit (to, say, 512K). You can do it by adding following line to the /etc/sysctl.conf file:

    fs.inotify.max_user_watches = 524288

This example tunes the system to monitor 512k files.

• If you have an HDD, it won't make it spin any faster and it most likely does 80 - 120 IOPS (IO Per Second) and this is more likely to be a performance bottleneck than you might like.

Like many IO operations in Java, it is wrapper around a facility which is actually implemented by the OS.