I am running a spark streaming job deployed in yarn client mode which will frequently dealing with HDFS, Our hadoop cluster version is hadoop-2.6.0-cdh5.7.3 and the patch file in jira HDFS-9276 has been introduced into this version, but I still got some errors as below after a couple of days(mostly 7 days):
> 18-09-2017 10:05:48 CST crm_user_select ERROR - 17/09/18 10:05:48 WARN security.UserGroupInformation: PriviledgedActionException as:bd_recom@FHC (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for bd_recom: HDFS_DELEGATION_TOKEN owner=bd_recom@FHC, renewer=yarn, realUser=, issueDate=1505095524480, maxDate=1505700324480, sequenceNumber=2244503, masterKeyId=504) is expired
18-09-2017 10:05:48 CST crm_user_select ERROR - 17/09/18 10:05:48 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for bd_recom: HDFS_DELEGATION_TOKEN owner=bd_recom@FHC, renewer=yarn, realUser=, issueDate=1505095524480, maxDate=1505700324480, sequenceNumber=2244503, masterKeyId=504) is expired
18-09-2017 10:05:48 CST crm_user_select ERROR - 17/09/18 10:05:48 WARN security.UserGroupInformation: PriviledgedActionException as:bd_recom@FHC (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for bd_recom: HDFS_DELEGATION_TOKEN owner=bd_recom@FHC, renewer=yarn, realUser=, issueDate=1505095524480, maxDate=1505700324480, sequenceNumber=2244503, masterKeyId=504) is expired
18-09-2017 10:05:48 CST crm_user_select ERROR - 17/09/18 10:05:48 WARN hdfs.LeaseRenewer: Failed to renew lease for [DFSClient_NONMAPREDUCE_-2053099090_1] for 30 seconds. Will retry shortly ...
18-09-2017 10:05:48 CST crm_user_select ERROR - org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for bd_recom: HDFS_DELEGATION_TOKEN owner=bd_recom@FHC, renewer=yarn, realUser=, issueDate=1505095524480, maxDate=1505700324480, sequenceNumber=2244503, masterKeyId=504) is expired
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.ipc.Client.call(Client.java:1471)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.ipc.Client.call(Client.java:1408)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
18-09-2017 10:05:48 CST crm_user_select ERROR - at com.sun.proxy.$Proxy14.renewLease(Unknown Source)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:576)
18-09-2017 10:05:48 CST crm_user_select ERROR - at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
18-09-2017 10:05:48 CST crm_user_select ERROR - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
18-09-2017 10:05:48 CST crm_user_select ERROR - at java.lang.reflect.Method.invoke(Method.java:606)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
18-09-2017 10:05:48 CST crm_user_select ERROR - at com.sun.proxy.$Proxy15.renewLease(Unknown Source)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:941)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
18-09-2017 10:05:48 CST crm_user_select ERROR - at java.lang.Thread.run(Thread.java:745)
and following error info:
> 18-09-2017 10:19:35 CST crm_user_select ERROR - 17/09/18 10:19:35 WARN security.UserGroupInformation: PriviledgedActionException as:bd_recom@FHC (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for bd_recom: HDFS_DELEGATION_TOKEN owner=bd_recom@FHC, renewer=yarn, realUser=, issueDate=1505095524480, maxDate=1505700324480, sequenceNumber=2244503, masterKeyId=504) can't be found in cache
18-09-2017 10:19:35 CST crm_user_select ERROR - 17/09/18 10:19:35 WARN hdfs.LeaseRenewer: Failed to renew lease for [DFSClient_NONMAPREDUCE_-2053099090_1] for 857 seconds. Will retry shortly ...
18-09-2017 10:19:35 CST crm_user_select ERROR - org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for bd_recom: HDFS_DELEGATION_TOKEN owner=bd_recom@FHC, renewer=yarn, realUser=, issueDate=1505095524480, maxDate=1505700324480, sequenceNumber=2244503, masterKeyId=504) can't be found in cache
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.ipc.Client.call(Client.java:1471)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.ipc.Client.call(Client.java:1408)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
18-09-2017 10:19:35 CST crm_user_select ERROR - at com.sun.proxy.$Proxy14.renewLease(Unknown Source)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:576)
18-09-2017 10:19:35 CST crm_user_select ERROR - at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
18-09-2017 10:19:35 CST crm_user_select ERROR - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
18-09-2017 10:19:35 CST crm_user_select ERROR - at java.lang.reflect.Method.invoke(Method.java:606)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
18-09-2017 10:19:35 CST crm_user_select ERROR - at com.sun.proxy.$Proxy15.renewLease(Unknown Source)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:941)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
18-09-2017 10:19:35 CST crm_user_select ERROR - at java.lang.Thread.run(Thread.java:745)
Bye the way: 1. NameNode HA is enabled. 2. Kerberos is enabled. 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with NameNode.
I have tried to use the configuration " --conf spark.hadoop.fs.hdfs.impl.disable.cache=true", but it didn't work. So anyone could help me, I would really appreciate!