QuestDB: LineTcpConnectionContext [151] buffer overflow

Question

I'm experiencing a problem with QuestDB ingestion via the Influx Line Protocol.

The setting is the following: from time to time I create a JSON value (for one column in a table) that has two lists of about 400 floats. I'm using a Sender to flush each such value to QuestDB, but the client is getting the following error: questdb.ingress.IngressError: Could not flush buffer: Broken pipe (os error 32)

The server log shows the following output:

2023-10-17T13:07:33.505800Z D tcp-line-server epoll [n=1]
2023-10-17T13:07:33.505882Z I tcp-line-server connected [ip=192.168.65.2, fd=151]
2023-10-17T13:07:33.505936Z D tcp-line-server pending [row=0, fd=151]
2023-10-17T13:07:33.897433Z D tcp-line-server epoll [n=1]
2023-10-17T13:07:33.897470Z D tcp-line-server fired [fd=151, op=1, pos=3]
2023-10-17T13:07:33.898413Z E i.q.c.l.t.LineTcpConnectionContext [151] buffer overflow [line.tcp.msg.buffer.size=32768]
2023-10-17T13:07:33.898438Z I tcp-line-server scheduling disconnect [fd=151, reason=0]
2023-10-17T13:07:33.898460Z I tcp-line-server disconnected [ip=192.168.65.2, fd=151, src=queue]

Answer 1

This is likely due to a single message being so long it does not fit into a server protocol buffer. The error message contains a hint: line.tcp.msg.buffer.size=32768 - this is the buffer size in bytes.

You can increase this value in server.conf. Open server.conf and set the following key option: line.tcp.msg.buffer.size=65536 if that does not help try to double it further.