I am working on a backend team which is handling HTTP requests routed through app gateway into a microservice from an embedded device.
The device is making requests every half second roughly, and to the best of my understanding should be keeping the connection alive.
Both the backend team and the embedded team have noticed that on the 100th request going into App Gateway, without any logging from the app gateway, nor the service, a 'Connection: Close' is sent back to the device, which is causing issues. We see this 100 connection then reset pattern looking at the Azure Diagnostics logs for the field noOfConnectionRequests_d.
In test code, the built in HttpClient handles this without an issue, and gets a new connection, but we have found no documentation to clarify the meaning of the fieldnoOfConnectionRequests_d and no documentation around if that limit can be modified or removed. Does anyone know why the connection would be getting closed after successful traffic, and what that field actually correlates to? Does this have to be handled by the embedded device due to a limitation of app gateway?
Here are the diagnostic log docs I am referring to, which do not reference the field.
I have additionally attached a screenshot showing the field from Azure Log Analytics. This number goes up to a max of 100 before a connection (not pooled) is closed. The embedded device does not currently pool connections so the close ends its flow.
final edit: I was able to get an answer on reddit. https://old.reddit.com/r/AZURE/comments/1bt73t0/app_gateway_closes_connections/kxtg328/
This is controlled from Microsoft's end without any transparency in Azure. The embedded device must expect and handle this, and the backend services have little to no control over this.