hostPathVolume not mounting on Kubernetes pods(Dynamic Jenkins Agent)

Question

Environment: 
Jenkins - Version 2.235.5 (LTS)
Kubernetes Jenkins Plugin  - 1.27.7

K8s-Master & Worker Server Details:-
OS - CentOS Linux release 7.9.2009 (Core) & Kernal - 3.10.0-1160.6.1.el7.x86_64
Docker Version - 19.03.12
Client Version: v1.19.0
Server Version: v1.19.0

I am trying to mount a hostPathVolume through Jenkins Kubernetes-plugin podTemplate and when the pod gets kicked off via Jenkins master the path of my K8s worker not getting mounted on K8s pods(Jenkins Agent).

I have the below scripted pipeline.

podTemplate(label: 'mypod', containers: [
    containerTemplate(name: 'git', image: 'ubuntu/16.04', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'maven', image: 'ubuntu/16.04', command: 'cat', ttyEnabled: true)
  ],
  volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
  ]
  ) {
    node('mypod') {
        stage('Check running containers') {
            container('docker') {
                // example to show you can run docker commands when you mount the socket
                sh 'hostname'
                sh 'hostname -i'
                sh 'docker ps'
            }
        }
..... further stages....

K8s cluster Jenkins agent pod log

# kubectl logs -f jenkins_job-10-b04wk-dn4gf-llpbp -c jnlp
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: jenkins_job-10-b04wk-dn4gf-llpbp
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Dec 05, 2020 5:41:16 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins-server/jenkins/]
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: jenkins-server
  Agent port:    40003
  Identity:      23:66:a9:70:0b:f2:03:e0:47:44:0d:ee:7e:ce:8c:31
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-server:40003
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 23:66:a9:70:0b:f2:03:e0:47:44:0d:ee:7e:ce:8c:31
Dec 05, 2020 5:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Dec 05, 2020 5:41:21 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call
INFO: Disabled slave engine reconnects.
Dec 05, 2020 5:41:21 AM hudson.util.ProcessTree get
WARNING: Error while determining if vetoers exist
hudson.remoting.RequestAbortedException: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. It is not open for write
        at hudson.remoting.Request.abort(Request.java:340)
        at hudson.remoting.Channel.terminate(Channel.java:1081)
        at hudson.remoting.Channel.close(Channel.java:1482)
        at hudson.remoting.Channel.close(Channel.java:1446)
        at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1313)
        at hudson.remoting.Channel$1.handle(Channel.java:606)
        at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:203)
        at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:189)
        at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:187)
        at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:206)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
        at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
        at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$2200(BIONetworkLayer.java:48)
        at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkins-server/10.20.30.35:40003
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
                at hudson.remoting.Request.call(Request.java:202)
                at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
                at com.sun.proxy.$Proxy5.fetch3(Unknown Source)
                at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:211)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
                at jenkins.agents.AgentComputerUtil.getChannelToMaster(AgentComputerUtil.java:48)
                at hudson.util.ProcessTree.get(ProcessTree.java:432)
                at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
                at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
                at hudson.remoting.UserRequest.perform(UserRequest.java:211)
                at hudson.remoting.UserRequest.perform(UserRequest.java:54)
                at hudson.remoting.Request$2.run(Request.java:369)
                at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                ... 4 more
Caused by: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. It is not open for write
        at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:305)
        at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:304)
        at hudson.remoting.Channel.send(Channel.java:764)
        at hudson.remoting.Channel.close(Channel.java:1479)
        ... 18 more

Dec 05, 2020 5:41:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Dec 05, 2020 5:41:21 AM hudson.util.ProcessTree get
WARNING: Failed to load winp. Reverting to the default
java.lang.NoClassDefFoundError: hudson/util/ProcessTree$Linux$LinuxProcess
        at hudson.util.ProcessTree.get(ProcessTree.java:451)
        at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
        at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
        at hudson.remoting.UserRequest.perform(UserRequest.java:211)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$Linux$LinuxProcess
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:173)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
        ... 12 more

Dec 05, 2020 5:41:21 AM hudson.remoting.Request$2 run
INFO: Failed to send back a reply to the request hudson.remoting.Request$2@1affe1cf: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@157a1c18:JNLP4-connect connection to jenkins-server/10.20.30.35:40003": channel is already closed

openjdk version "1.8.0_272" installed in (Jenkins master, K8s Master and K8s-Worker) systems, Also have restarted all three systems however still no success.

I expect the volume to be mounted in my pod, I have tried with K8s v1.19.4 and Kubernetes Jenkins Plugin - 1.27.0. Not sure what is causing the problem on CentOS-7.9. Whereas the same is working perfectly on CentOS - 8.2.2004 (Core) & 4.18.0-193.14.2.el8_2.x86_64 kernal of K8s master host's native OS.

Answer 1

I have upgraded my docker image from Ubuntu 16.04.6 LTS - 3.10.0-1160.6.1.el7.x86_64 to Ubuntu 16.04.7 to 3.10.0-1160.6.1.el7.x86_64 still didn't work. But it seems Kubernetes on CentOS-7.9 & with Kernal-3.10.0-1160.6.1.el7.x86_64 is having kernel specific issue. So i have upgraded my native host's OS to CentOS-8.2 with - 4.18.0-193.28.1.el8_2.x86_64 kernel, Now my pipeline build working fine.