Is there any way to execute a callback (on Linux) when a file descriptor is closed

Question

I'm working on a kevent/kqueue emulation library for Linux. I'm a new maintainer on this project, and unfortunately, the previous maintainer is not involved much anymore (so I can't pick their brains about this).

Under FreeBSD and macOS when you close() the file descriptor provided by kqeueue() you free any resources and events associated with it.

It seems like the existing code doesn't provide a similar interface. Before I add a function to the API (or revive an old one) to explicitly free kqueue resources, I was wondering if there was any way to associate triggers with a file descriptor in linux, so that when it's closed we can cleanup anything associated with the FD.

The file descriptor itself could be any type, i.e. one provided by eventfd, or epoll or anything else that creates file descriptors.

Answer 1

When the last write file descriptor from a pipe() call is closed epoll()/poll() waiters will see an [E]POLLHUP event on any read file descriptors still open. Presumably the same is true of any fd that represents a connection rather than state.

Answer 2

The solution to this is fairly simple, if a little annoying to implement. It relies on an fcntl called F_SETSIG, to specify the signal used to communicate FD state changes, and an fcntl called F_SETOWN_EX to specify what thread the signal should be delivered to.

When the application starts it spawns a separate monitoring thread. This thread is used to receive FD generated signals.

In our particular use case, the monitoring thread must be started implicitly the first time a monitored FD is created, and destroyed without an explicit join. This is because we're emulating a FreeBSD API (kqueue), which does not have explicit init and deinit functions.

The monitoring thread:

1. Listens on a the signal we passed to F_SETSIG.
2. Gets its thread ID, and stores it in a global.
3. Informs the application that the monitoring thread has started (and the global is filled) using pthread_cond_broadcast.
4. Calls pthread_detach to ensure it's cleaned up correctly without another thread needing to do an explicit pthread_join.
5. Calls sigwaitinfo to wait on delivery of a signal.

The application thread(s):

1. Uses pthread_once to start the monitoring thread the first time a FD is created, then waits for the monitoring thread to start fully.
2. Uses F_SETSIG to specify the signal sent when the FD is open/closed, and F_SETOWN_EX to direct those signals to the monitoring thread.

When a monitored FD is closed the sigwaitinfo call in the monitoring thread returns. In our case we're using a pipe to represent the kqueue, so we need to map the FD we received the signal for, to the one associated with the resources (kqueues) we need to free. Once this mapping is done, we may (see below for more information) cleanup the resources associated with the FD pair, and call sigwaitinfo again to wait for more signals.

One of the other key pieces to this strategy, is that the resources associated with the FDs are reference counted. This is because the signals are not synchronously delivered, so an FD can be closed, and a new FD can be created with the same number, before the signal indicating the original FD was closed, was delivered and acted on. This would obviously cause big issues with active resources being freed.

To solve this we maintain a mutex synchronised FD to resource mapping array. Each element in this array contains a reference count for a particular FD.

In the case where the signal is not delivered before the FD is reused when creating a new pipe/resource pair the reference count for that particular FD will be > 0. When this occurs we immediately free the resource, and reinitialise it, increasing the reference count. When the signal indicating the FD was closed is delivered, the reference count is decremented (but not to zero), and the resource is not freed.

Alternatively if the signal is delivered before the FD was reused, then the monitoring thread will decrement the reference count to zero, and immediately free the associated resources.

If this description is a bit confusing you can look over our real world implementation using any of the links above.

Note: Our implementation isn't exactly as was described above (notably we don't check the reference count of an FD when creating a new FD/resource mapping). I think this is because we rely on the fact that closing one of the pipe doesn't necessarily result in the other end being closed, so the open end's FD isn't available for reuse immediately. Unfortunately the developer who wrote the code isn't available for querying.