we attempted to enable the JFR (Java Flight Recorder) feature in our production environment. We used the following startup parameters:
-XX:StartFlightRecording=disk=true,dumponexit=true,name=profile_online,filename=/tmp/profile_`hostname`-`date +%Y_%m_%d_%H_%M_%S`.jfr,maxsize=4096m,maxage=2d,settings=/opt/app/WEB-INF/tars/prod/custom.jfc,path-to-gc-roots=false -XX:FlightRecorderOptions=maxchunksize=32m
We were using JDK version 11.0.14, and the garbage collector in use was G1.
After enabling the above JFR configuration on one of our applications, we encountered bulk Out of Memory (OOM) issues on 7 machines four days later.
We reviewed the monitoring data and found that heap conditions appeared relatively normal. However, we noticed that the mem rss usage, which was around 15GB (peaking at 16GB) before the recent deployment, increased to approximately 15.5GB after the deployment.
To address the issue, we attempted to analyze it using Native Memory Tracking (NMT) and discovered that after four days, the memory overhead for Tracing metrics had reached around 400MB. We suspect that Tracing metrics may include memory overhead from JFR.
We are perplexed about why bulk OOM issues occurred precisely four days after the deployment and whether JFR's memory consumption is related to runtime duration. We are also curious about the primary sources of JFR's memory overhead and whether there is a way to estimate JFR's memory overhead based on startup parameters.