How to receive WebSocket messages larger than 16kB with Spring WebSockets and Undertow

Question

I'm having some trouble receiving large subprotocol messages using Spring Boot, Spring WebSocket and Undertow. The messages are cut off after 16kB. After doing some digging I found the following configuration property which seems to do what I want:

server.undertow.buffer-size=32768

This configuration property seems to be properly picked up when checking the /configprops actuator endpoint. Unfortunately, this doesn't seem to help in receiving messages larger than 16kB.

I also stumbled upon this ominous line from the Undertow documentation (emphasis mine):

For servers the ideal size is generally 16k, as this is usually the maximum amount of data that can be written out via a write() operation (depending on the network setting of the operating system).

This confirms what I've been experiencing that setting the server.undertow.buffer-size has no effect as it's capped by an OS level setting. As I'm using Ubuntu Linux I have been fiddling around with net.core.rmem_* and net.core.wmem_* settings but these don't seem to have any effect either. It's not possible to reproduce this issue on macOS.

Does anyone know how to configure Undertow, Spring Boot, and/or Spring WebSocket to support these messages?

Answer 1

I directly answered the question above myself. The settings I proposed in Spring Boot do work! The problem we had was that the load balancer, in our case haproxy, in front of it was cutting of the messages after 16kB. Tuning haproxy to allow larger messages solved the issue. In the meanwhile, we tweaked the protocol we were using to be more efficient so we don't require these large messages anymore so our haproxy solution wasn't tested in production (YMMV).

Because the developers were all working on macOS and Windows and the issue only occurred on the acceptance and production environments which ran Ubuntu we incorrectly assumed this was the cause.

Lessons learned (these are all really dumb and basic but we made these mistakes anyway):

• Be sure to validate all your assumptions! If we thought Ubuntu was the issue we should've singled out Ubuntu in tests earlier. For example, by using a VM with Ubuntu on which to validate our assumptions in quarantaine.
• Make sure your development environment matches your production environment! In our development environments we weren't running haproxy. As "high level" developers we tend to put aside load balancers and web servers as commodity infrastructure but this example once again shows that these "commodities" can very directly impact the working of your application.