libcurl async integration with kevent on macOS Sierra

Question

I'm integrating curl into an asynchronous I/O event loop based on kqueue.

libcurl has an excellent API for integrating into the applications event loop.

You provide libcurl with two callbacks, one to set a timer (used to limit request/connect times), and the other to register a libcurl's file descriptors for read/write/error events.

The documentation for the callback used to perform FD registration is here: CURLMOPT_SOCKETFUNCTION

The argument that informs the callback of what events libcurl is interested in, has four enum values:

CURL_POLL_IN

Wait for incoming data. For the socket to become readable.

CURL_POLL_OUT

Wait for outgoing data. For the socket to become writable.

CURL_POLL_INOUT

Wait for incoming and outgoing data. For the socket to become readable or writable.

CURL_POLL_REMOVE

The specified socket/file descriptor is no longer used by libcurl.

Although not explicitly documented, libcurl expects, that on subsequent calls to the callback, that the filter state of the event loop, be updated to match what it passed. i.e. If on the first call it passed CURL_POLL_IN (EVFILT_READ) and on a subsequent call it passed CURL_POLL_OUT (EVFILT_WRITE), then the original EVFILT_READ filter would be removed.

I updated the FD registration code to handle this.

int fr_event_fd_insert(fr_event_list_t *el, int fd,
               fr_event_fd_handler_t read,
               fr_event_fd_handler_t write,
               fr_event_fd_handler_t error,
               void *ctx)
{
    int       filter = 0;
    struct kevent evset[2];
    struct kevent *ev_p = evset;
    fr_event_fd_t *ef, find;

    if (!el) {
        fr_strerror_printf("Invalid argument: NULL event list");
        return -1;
    }

    if (!read && !write) {
        fr_strerror_printf("Invalid arguments: NULL read and write callbacks");
        return -1;
    }

    if (fd < 0) {
        fr_strerror_printf("Invalid arguments: Bad FD %i", fd);
        return -1;
    }

    if (el->exit) {
        fr_strerror_printf("Event loop exiting");
        return -1;
    }

    memset(&find, 0, sizeof(find));

    /*
     *  Get the existing fr_event_fd_t if it exists.
     */
    find.fd = fd;
    ef = rbtree_finddata(el->fds, &find);
    if (!ef) {
        ef = talloc_zero(el, fr_event_fd_t);
        if (!ef) {
            fr_strerror_printf("Out of memory");
            return -1;
        }
        talloc_set_destructor(ef, _fr_event_fd_free);
        el->num_fds++;
        ef->fd = fd;
        rbtree_insert(el->fds, ef);
    /*
     *  Existing filters will be overwritten if there's
     *  a new filter which takes their place.  If there
     *  is no new filter however, we need to delete the
     *  existing one.
     */
    } else {
        if (ef->read && !read) filter |= EVFILT_READ;
        if (ef->write && !write) filter |= EVFILT_WRITE;

        if (filter) {
            EV_SET(ev_p++, ef->fd, filter, EV_DELETE, 0, 0, 0);
            filter = 0;
        }

        /*
         *  I/O handler may delete an event, then
         *  re-add it.  To avoid deleting modified
         *  events we unset the do_delete flag.
         */
        ef->do_delete = false;
    }

    ef->ctx = ctx;

    if (read) {
        ef->read = read;
        filter |= EVFILT_READ;
    }

    if (write) {
        ef->write = write;
        filter |= EVFILT_WRITE;
    }
    ef->error = error;

    EV_SET(ev_p++, fd, filter, EV_ADD | EV_ENABLE, 0, 0, ef);
    if (kevent(el->kq, evset, ev_p - evset, NULL, 0, NULL) < 0) {
        fr_strerror_printf("Failed inserting event for FD %i: %s", fd, fr_syserror(errno));
        talloc_free(ef);
        return -1;
    }
    ef->is_registered = true;

    return 0;
 }

Unfortunately, it doesn't work. kevent does not appear to remove the old filters (we continue to receive notifications from them).

What's weirder is if I apply the two operations in two separate calls, it works perfectly.

if (filter) {
    EV_SET(&evset, ef->fd, filter, EV_DELETE, 0, 0, 0);
    kevent(el->kq, evset, ev_p - evset, NULL, 0, NULL);
    filter = 0;
}

Is this a bug in Sierra's kevent implementation, or did I misunderstand how kevent should work?

Answer 1

The problem here, is that you can't 'or' together the EVFILT_READ and EVFILT_WRITE flags.

When enabling or disabling multiple filters you need to call EV_SET() multiple times, on multiple evset structures.

The non-functional code in the example above:

struct kevent evset[2];
struct kevent *ev_p = evset;

if (read) {
    ef->read = read;
    filter |= EVFILT_READ;
}

if (write) {
    ef->write = write;
    filter |= EVFILT_WRITE;
}
ef->error = error;

EV_SET(ev_p++, fd, filter, EV_ADD | EV_ENABLE, 0, 0, ef);
event(el->kq, evset, ev_p - evset, NULL, 0, NULL)

becomes:

int count = 0;
struct ev_set[2];

if (read) {
    ef->read = read;
    EV_SET(ev_set[count++], fd, EVFILT_READ, EV_ADD | EV_ENABLE, 0, 0, ef);
}

if (write) {
    ef->write = write;
    EV_SET(ev_set[count++], fd, EVFILT_WRITE, EV_ADD | EV_ENABLE, 0, 0, ef);
}
ef->error = error;
kevent(el->kq, ev_set, count, NULL, 0, NULL)

After making this change everything worked as expected.