How to prevent a Java application from executing processes on GNU/Linux?

Question

In other words, are modern GNU/Linux or JVM (ideally Java 11+) able to prevent a Java process from executing other processes?

Issues such as Log4Shell and Spring4Shell seem possible because the JVM allows a Java application to execute other programs.

At the OS level, I imagine it may be possible with some process isolation, jails etc. Are there complete examples about that?

How about the JVM itself? With the Security Manager gone in recent versions, what options exist to harden a Java Virtual Machine?

Answer 1

I'm not sure there's a good way to do that. As discussed in Prevent forking in a child process, you can set RLIMIT_NPROC limit.

However, don't do that for your java app - you will break it. Threads are just a special type of a process on Linux and if you restrict nproc you won't be able to create new threads.

Observe this behavior on a running java application

prlimit
...
NPROC      max number of processes                15107     15107 processes

prlimit --pid $(pgrep java) --nproc=10

Then on the stdout of the java app

[9816123.415s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.

I tried this for a production app running in a docker container.

Note that nproc limit isn't enforced if you the process is running as root.

Morever, it's quite easy to reproduce it with a simple Clojure REPL (or JShell variant of it):

# start a clojure/java process
user=> (require '[clojure.java.shell :as sh])

user=> (dotimes [i 100] (future (sh/sh "sleep" "2")))
nil


# then get change the nproc limit for the process - make sure to pass proper pid
prlimit --pid 2769227 --nproc=99

# then again in the Clojure REPL: 
user=> (dotimes [i 100] (future (sh/sh "sleep" "2")))
[200.656s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[200.656s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[200.657s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[200.657s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[200.657s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Execution error (OutOfMemoryError) at java.lang.Thread/start0 (Thread.java:-2).
unable to create native thread: possibly out of memory or process/resource limits reached

Resources:

• Prevent forking in a child process: Prevent forking in a child process
• Does nproc in limits.conf refers to number of processes or number of threads? https://unix.stackexchange.com/questions/322056/does-nproc-in-limits-conf-refers-to-number-of-processes-or-number-of-threads
• Multiple Java webapps total thread limit on linux Multiple Java webapps total thread limit on linux
• prlimit: https://manpages.ubuntu.com/manpages/xenial/man1/prlimit.1.html