Channel unregistered before processing completed

Question

I have a very unclear and difficult to understand problem. In the NiFi flow, certain processors use services:

• DistributedMapCacheClientService (v. 1.20.0)
• DistributedMapCacheServer (v. 1.20.0)

and DistributedMapCacheClientService connects to DistributedMapCacheServer.

For some reason, we sometimes get:

java.io.IOException: Request invocation failed
    at org.apache.nifi.distributed.cache.client.CacheClientRequestHandler.invoke(CacheClientRequestHandler.java:103)
    at org.apache.nifi.distributed.cache.client.MyDistributedCacheClient.invoke(DistributedCacheClient.java:69)
    at org.apache.nifi.distributed.cache.client.NettyDistributedMapCacheClient.getAndPutIfAbsent(NettyDistributedMapCacheClient.java:140)
    at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:154)
    at jdk.internal.reflect.GeneratedMethodAccessor471.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
    at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
    at com.sun.proxy.$Proxy175.getAndPutIfAbsent(Unknown Source)
    at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:184)
    at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
    at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1357)
    at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
    at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
    at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Channel unregistered before processing completed: [id: 0x7288e0ed, L:0.0.0.0/0.0.0.0:38868]
    at org.apache.nifi.distributed.cache.client.CacheClientRequestHandler.channelUnregistered(CacheClientRequestHandler.java:70)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:219)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:195)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:188)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1388)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:215)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:195)
    at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:821)
    at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:821)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ... 1 more

This is the configuration of DistributedMapCacheClientService: Host name and port are all well configured.

I have examined all the log files, but I'm unable to find the root cause of this problem.

We have been debugging the project by connecting from the NiFi source to the remote JVM (i.e. NiFi running instance). We've managed to find out, that handShakeHandler, from the source below (CacheClientRequestHandler), occasionally happens to be null.

Why handShakeHandler can be null and in what circumstances?