Can I limit the memory usage of bufferevents in libevent?

Question

Does libevent have a way to restrict the memory usage or the outstanding unwritten data with the bufferevents API? Alternatively, what happens when libevent runs out of memory?

My application reads data from a file, parses it, and sends it over the network (pseudocode below, my application is C++):

for line in file: host = parse_host(line) socket[host].send(line)

Previously, I split this up into two threads to ensure that we don't block the parsing of the file unnecessarily if one host is slow:

for line in file: host = parse_host(line) fixed_sized_queues[host].append(line)

while stillParsingFile(): sockets.poll(timeout=1s) for queue in fixed_sized_queues: if queue is not Empty and socket[queue.host] is ReadyForWrite: socket[queue.host].send(q.pop())

The logic for the second thread is very clunky, and I thought it would be better to use libevents:

for line in file: host = parse_host(line) bufferevents[host].add(line)

My understanding is that appending to a bufferevent is non-blocking (i.e. it will do dynamic memory allocation and asynchronously write the data to the socket). How do I prevent the parse thread from outpacing the network and filling up all of memory by allocating space for data that can't be sent yet?

The ideas I have currently are:

• Use evbuffer_add_reference to mitigate the memory usage (still doesn't provide a bound, but at least allocations should be small)
• Use event_set_mem_functions and custom locking to force libevent to block and wait for memory when it hits a cap (this sounds very dangerous, as it only works if libevent only does allocations when adding to a bufferevent and nowhere else)
• In the parse thread, grab a lock before adding to the evbuffer and check evbuffer_get_length(); in the send thread, signal the parse thread in a low water write callback from libevent.