I am responsible for a bunch of servers and to ease my job I developed a simple monitoring tool that is able to send out SOAP, REST and JDBC requests on a regular basis. The result (i.e. whether the server responds correctly) is then visualized on a simnple "dash-board" providing me the overview which server is up and properly responding and which is not.
With one of our Tomcat-based servers I have the odd effect, that the configured requests work and the server responds properly but every second request yields a 403 error ("forbidden"). This yields the annoying effect that this server's widget toggles between green/OK and red/failed every other minute.
I am scratching my head since a few days how or why the very same request (i.e. same HTTP-headers, same payload, same credentials, same client, same key- and trust-stores, same what-ever) works fine in one request but not the subsequent one. The third request then works again, the forth fails, etc. etc.
Any idea anyone, what could be the reason for this behavior?
In case this is useful: The server that shows this effect is an old Tomcat 7.0.47. The odd fact is that I have a bunch of other servers with exactly the same setup (in fact, their images were originally all cloned from the same template) and these work all fine. It's really a single server only that has shows this toggling between "200" and "403" and I con
Note: there is a FW between my laptop and these servers, but it has no restrictions re. maximum number of requests - at least definitely not on such a low level (1-2 requests/minute from the same source IP).