In jdk7 watch service API, when will the OVERFLOW event be thrown?

Question

The documentation for the overflow states:

OVERFLOW – Indicates that events might have been lost or discarded.

It does not say under what circumstance should I expect event to be lost or discarded? At first I thought it would be a result of writing a lot of files very fast to the folder. I created a few thousand files with zero size and moved them to a monitored directory. No OVERFLOW.

What am I missing?

Answer 1

"File systems may report events faster than they can be retrieved or processed and an implementation may impose an unspecified limit on the number of events that it may accumulate. Where an implementation knowingly discards events then it arranges for the key's pollEvents method to return an element with an event type of OVERFLOW. This event can be used by the consumer as a trigger to re-examine the state of the object."

From JavaDoc.

See also Steven C's answer. His point about unconsumed events makes the difference, I think.

Answer 2

I created a few thousand files with zero size and moved them to a monitored directory. No OVERFLOW.

Presumably, you are consuming the events in parallel with creating them. If you want to trigger an overflow, try pausing the consumption of events, generating lots of them (as above) and then resuming consumption. There is bound to be a limit on the number of unconsumed events that the operating system can buffer.

Answer 3

Minimal example that generates overflows

Just create the files after watcherService.register and before watcherService.take.

Invoke with:

java Overflow 256

to control the number of events.

Java 7 and Ubuntu 14.04 maxes out at 512 events.

Each time there was an overflow, only a single event was returned by pollEvents(), but this is not very clearly specified on the Javadoc.

I find it weird that this limit is so small, considering that:

• my sysctl fs.inotify.max_user_watches is 524288
• https://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached
• JDK seems to use inotify on Linux: http://hg.openjdk.java.net/jdk7u/jdk7u60/jdk/file/33c1eee28403/src/solaris/native/sun/nio/fs/LinuxWatchService.c#l36

The code:

import java.io.File;
import java.io.IOException;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchService;
import java.util.List;

public class Overflow {

    @SuppressWarnings("unchecked")
    static <T> WatchEvent<T> cast(WatchEvent<?> event) {
        return (WatchEvent<T>)event;
    }

    public static void main(final String[] args) throws InterruptedException, IOException {
        int nfiles;
        if (args.length > 0)
            nfiles = Integer.parseInt(args[0]);
        else
            nfiles = 10_000;
        Path directory = Files.createTempDirectory("watch-service-overflow");
        final WatchService watchService = FileSystems.getDefault().newWatchService();
        directory.register(
                watchService,
                StandardWatchEventKinds.ENTRY_CREATE,
                StandardWatchEventKinds.ENTRY_DELETE);
        final Path p = directory.resolve(Paths.get("Hello World!"));
        for (int i = 0; i < nfiles; i++) {
            Files.createFile(p);
            Files.delete(p);
        }
        List<WatchEvent<?>> events = watchService.take().pollEvents();
        for (final WatchEvent<?> event : events) {
            if (event.kind() == StandardWatchEventKinds.OVERFLOW) {
                System.out.println("Overflow.");
                System.out.println("Number of events: " + events.size());
                return;
            }
        }
        System.out.println("No overflow.");
        Files.delete(directory);
    }
}

Answer 4

From source code JDK17 in OpenJDK

abstract class AbstractWatchKey implements WatchKey {

    /**
     * Maximum size of event list (in the future this may be tunable)
     */
    static final int MAX_EVENT_LIST_SIZE    = 512;

should be "tunable" since 2011........

So you can modify inotify.max_user_watches in Linux as you want it has no effect...

Whe are in July 2022...Nothing has been done since 11 years. This bug has been reported since 2013:

https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8034864

Don't loose time with Java, Oracle or OpenJDK...they just don't care.

Try watchdog 2.1.9 API with Python with thousands of files, just to see, and it's working like a charm....no matters number of events.