I have several Payara Micro Docker services in a stack, with some services having more than one replica. Their containers are of images that extent Payara Micro's image. It's pretty basic:
FROM payara/micro:5.2022.5-jdk11
COPY target/*.war $DEPLOY_DIR
EXPOSE 8080 8181
CMD ["--deploymentDir", "/opt/payara/deployments", "--contextroot", "a"]
These applications do nothing more than expose a couple of REST endpoints and marshal data to/from their databases (SQL Server and/or MongoDB) - with very little data manipulation/processing/shaping:
JAX-RS Endpoint
-> Local EJB
-> Database
Without any resource restrictions in place, the containers allocate 1GB of memory!
I thought that I could drop the container memory allocation to less than 100MB, but the containers just fail due to memory requirements. I've settle on max 700MB with an initial of 500MB, and the containers seem to initially use a whopping 460MB of memory just idling. Keep in mind these are every small apps that do little more than basic CRUD (some are even just read only!).
a-service:
image: repo/services-a:1.2.3
depends_on:
- mongo
ports:
- "8081:8080"
deploy:
replicas: 2
env_file: .env
networks:
- net
environment:
- MEM_MAX_RAM_PERCENTAGE=80
deploy:
resources:
limits:
cpus: '1.00'
memory: 700M
reservations:
cpus: '0.50'
memory: 500M
Here is a screenshot from Live Charts in Docker Desktop of one of the services starting up:
Here is another screenshot after running a simple stress test:
for run in {1..2000}; do curl -s http://127.0.0.1:8081/a/api/v1/document/details/62fbd94e2b276e0f3d6fcc30 > /dev/null; done
As you can see, there is an insignificant bump in memory usage (~10MB) for the entire 2000 requests.
Even at 500MB/service, this is excessive. Couple that with several services/containers, it's prohibitive.
What can I do to reduce the memory footprint of these Payara Micro containers? Ideally, I'd like to get them down to about 50MB (or even 100MB), but any reduction at this point is welcomed.
Metrics:
base_classloader_loadedClasses_count 22590
base_classloader_loadedClasses_total_total 22617
base_classloader_unloadedClasses_total_total 27
base_cpu_availableProcessors 1
base_cpu_systemLoadAverage 0.09912109375
base_gc_time_total{name="MarkSweepCompact"} 451
base_gc_time_total{name="Copy"} 639
base_gc_total_total{name="MarkSweepCompact"} 6
base_gc_total_total{name="Copy"} 136
base_jvm_uptime_seconds 1665.407
base_memory_committedHeap_bytes 1.73539328E8
base_memory_committedNonHeap_bytes 2.04734464E8
base_memory_maxHeap_bytes 5.67672832E8
base_memory_maxNonHeap_bytes -1
base_memory_usedHeap_bytes 1.38958704E8
base_memory_usedNonHeap_bytes 1.982758E8
base_thread_count 86
base_thread_daemon_count 72
base_thread_max_count 97
vendor_system_cpu_load 0.00928
$ jstat -gc 1
S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT CGC CGCT GCT
5824.0 5824.0 970.5 0.0 46848.0 41339.2 116800.0 93516.7 136320.0 132541.5 17024.0 15556.6 142 0.659 6 0.451 - - 1.111