I was installing a Hive on Ambari. But hive metastore is working well. only hiveserver2 isn't working.
the following error occurred:
*** stderr: /var/lib/ambari-agent/data/errors-362.txt**
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/my_repo/3.0/services/HIVE/package/scripts/hive_server.py", line 161, in <module>
HiveServer().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/my_repo/3.0/services/HIVE/package/scripts/hive_server.py", line 71, in start
hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/stacks/my_repo/3.0/services/HIVE/package/scripts/hive_service.py", line 101, in hive_service
wait_for_znode()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 54, in wrapper
return function(*args, **kwargs)
File "/var/lib/ambari-agent/cache/stacks/my_repo/3.0/services/HIVE/package/scripts/hive_service.py", line 184, in wait_for_znode
raise Exception(format("HiveServer2 is no longer running, check the logs at {hive_log_dir}"))
Exception: HiveServer2 is no longer running, check the logs at /var/log/hive
*** stdout: /var/lib/ambari-agent/data/output-362.txt**
...OMISSION...
2022-04-23 03:22:36,437 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://my_bigdatamn002.my_FQDN.io:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp_iJ6Ce 2>/tmp/tmpMTb03U''] {'quiet': False}
2022-04-23 03:22:36,506 - call returned (0, '')
2022-04-23 03:22:36,506 - get_user_call_output returned (0, u'{\n "beans" : [ {\n "name" : "Hadoop:service=NameNode,name=FSNamesystem",\n "modelerType" : "FSNamesystem",\n "tag.Context" : "dfs",\n "tag.HAState" : "active",\n "tag.TotalSyncTimes" : "21 23 7 ",\n "tag.Hostname" : "my_bigdatamn002.my_FQDN.io",\n "MissingBlocks" : 0,\n "MissingReplOneBlocks" : 0,\n "ExpiredHeartbeats" : 0,\n "TransactionsSinceLastCheckpoint" : 2739,\n "TransactionsSinceLastLogRoll" : 15,\n "LastWrittenTransactionId" : 2755,\n "LastCheckpointTime" : 1650707679000,\n "CapacityTotal" : 283346115674112,\n "CapacityTotalGB" : 263887.0,\n "CapacityUsed" : 11776085343,\n "CapacityUsedGB" : 11.0,\n "CapacityRemaining" : 280452166158831,\n "ProvidedCapacityTotal" : 0,\n "CapacityRemainingGB" : 261191.0,\n "CapacityUsedNonDFS" : 0,\n "TotalLoad" : 24,\n "SnapshottableDirectories" : 0,\n "Snapshots" : 0,\n "NumEncryptionZones" : 0,\n "LockQueueLength" : 0,\n "BlocksTotal" : 220,\n "NumFilesUnderConstruction" : 3,\n "NumActiveClients" : 2,\n "FilesTotal" : 1019,\n "PendingReplicationBlocks" : 0,\n "PendingReconstructionBlocks" : 0,\n "UnderReplicatedBlocks" : 0,\n "LowRedundancyBlocks" : 0,\n "CorruptBlocks" : 0,\n "ScheduledReplicationBlocks" : 0,\n "PendingDeletionBlocks" : 90,\n "LowRedundancyReplicatedBlocks" : 0,\n "CorruptReplicatedBlocks" : 0,\n "MissingReplicatedBlocks" : 0,\n "MissingReplicationOneBlocks" : 0,\n "HighestPriorityLowRedundancyReplicatedBlocks" : 0,\n "HighestPriorityLowRedundancyECBlocks" : 0,\n "BytesInFutureReplicatedBlocks" : 0,\n "PendingDeletionReplicatedBlocks" : 90,\n "TotalReplicatedBlocks" : 220,\n "LowRedundancyECBlockGroups" : 0,\n "CorruptECBlockGroups" : 0,\n "MissingECBlockGroups" : 0,\n "BytesInFutureECBlockGroups" : 0,\n "PendingDeletionECBlocks" : 0,\n "TotalECBlockGroups" : 0,\n "ExcessBlocks" : 0,\n "NumTimedOutPendingReconstructions" : 1,\n "PostponedMisreplicatedBlocks" : 0,\n "PendingDataNodeMessageCount" : 0,\n "MillisSinceLastLoadedEdits" : 0,\n "BlockCapacity" : 33554432,\n "NumLiveDataNodes" : 3,\n "NumDeadDataNodes" : 0,\n "NumDecomLiveDataNodes" : 0,\n "NumDecomDeadDataNodes" : 0,\n "VolumeFailuresTotal" : 0,\n "EstimatedCapacityLostTotal" : 0,\n "NumDecommissioningDataNodes" : 0,\n "StaleDataNodes" : 0,\n "NumStaleStorages" : 0,\n "TotalSyncCount" : 15,\n "NumInMaintenanceLiveDataNodes" : 0,\n "NumInMaintenanceDeadDataNodes" : 0,\n "NumEnteringMaintenanceDataNodes" : 0\n } ]\n}', u'')
2022-04-23 03:22:36,509 - NameNode HA states: active_namenodes = [(u'nn2', 'my_bigdatamn002.my_FQDN.io:50070')], standby_namenodes = [(u'nn1', 'my_bigdatamn001.my_FQDN.io:50070')], unknown_namenodes = []
2022-04-23 03:22:36,511 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://my_bigdatamn002.my_FQDN.io:50070/webhdfs/v1/user/hive/warehouse/sys.db/dag_data?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpum2PTq 2>/tmp/tmptPODjm''] {'logoutput': None, 'quiet': False}
2022-04-23 03:22:36,592 - call returned (0, '')
2022-04-23 03:22:36,592 - get_user_call_output returned (0, u'{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File does not exist: /user/hive/warehouse/sys.db/dag_data"}}404', u'')
2022-04-23 03:22:36,595 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://my_bigdatamn002.my_FQDN.io:50070/webhdfs/v1/user/hive/warehouse/sys.db/dag_data?op=MKDIRS&user.name=hdfs'"'"' 1>/tmp/tmppjxyFM 2>/tmp/tmpvKAiSE''] {'logoutput': None, 'quiet': False}
2022-04-23 03:22:36,686 - call returned (0, '')
2022-04-23 03:22:36,686 - get_user_call_output returned (0, u'{"boolean":true}200', u'')
2022-04-23 03:22:36,689 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://my_bigdatamn002.my_FQDN.io:50070/webhdfs/v1/user/hive/warehouse/sys.db/dag_data?op=SETPERMISSION&user.name=hdfs&permission=1777'"'"' 1>/tmp/tmpQS1XC0 2>/tmp/tmpp3LzCu''] {'logoutput': None, 'quiet': False}
2022-04-23 03:22:36,774 - call returned (0, '')
2022-04-23 03:22:36,775 - get_user_call_output returned (0, u'200', u'')
2022-04-23 03:22:36,777 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://my_bigdatamn002.my_FQDN.io:50070/webhdfs/v1/user/hive/warehouse/sys.db/dag_data?op=SETOWNER&owner=hive&group=&user.name=hdfs'"'"' 1>/tmp/tmpJDnQBA 2>/tmp/tmpgwTWkm''] {'logoutput': None, 'quiet': False}
2022-04-23 03:22:36,874 - call returned (0, '')
2022-04-23 03:22:36,874 - get_user_call_output returned (0, u'200', u'')
2022-04-23 03:22:36,876 - HdfsResource['/user/hive/warehouse/sys.db/app_data'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/3.0.1.0-187/hadoop/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://nameservice1', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': 'missing_principal', 'user': 'hdfs', 'owner': 'hive', 'hadoop_conf_dir': '/usr/hdp/3.0.1.0-187/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/user/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 01777}
2022-04-23 03:22:36,878 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://my_bigdatamn001.my_FQDN.io:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpm6Tuwr 2>/tmp/tmpE7ocu1''] {'quiet': False}
2022-04-23 03:22:36,957 - call returned (0, '')
2022-04-23 03:22:36,958 - get_user_call_output returned (0, u'{\n "beans" : [ {\n "name" : "Hadoop:service=NameNode,name=FSNamesystem",\n "modelerType" : "FSNamesystem",\n "tag.Context" : "dfs",\n "tag.HAState" : "standby",\n "tag.TotalSyncTimes" : "",\n "tag.Hostname" : "my_bigdatamn001.my_FQDN.io",\n "MissingBlocks" : 0,\n "MissingReplOneBlocks" : 0,\n "ExpiredHeartbeats" : 0,\n "TransactionsSinceLastCheckpoint" : 2724,\n "TransactionsSinceLastLogRoll" : 0,\n "LastWrittenTransactionId" : 23,\n "LastCheckpointTime" : 1650707502000,\n "CapacityTotal" : 283346115674112,\n "CapacityTotalGB" : 263887.0,\n "CapacityUsed" : 11776102400,\n "CapacityUsedGB" : 11.0,\n "CapacityRemaining" : 280452166402954,\n "ProvidedCapacityTotal" : 0,\n "CapacityRemainingGB" : 261191.0,\n "CapacityUsedNonDFS" : 0,\n "TotalLoad" : 24,\n "SnapshottableDirectories" : 0,\n "Snapshots" : 0,\n "NumEncryptionZones" : 0,\n "LockQueueLength" : 0,\n "BlocksTotal" : 220,\n "NumFilesUnderConstruction" : 3,\n "NumActiveClients" : 2,\n "FilesTotal" : 1015,\n "PendingReplicationBlocks" : 0,\n "PendingReconstructionBlocks" : 0,\n "UnderReplicatedBlocks" : 0,\n "LowRedundancyBlocks" : 0,\n "CorruptBlocks" : 0,\n "ScheduledReplicationBlocks" : 0,\n "PendingDeletionBlocks" : 0,\n "LowRedundancyReplicatedBlocks" : 0,\n "CorruptReplicatedBlocks" : 0,\n "MissingReplicatedBlocks" : 0,\n "MissingReplicationOneBlocks" : 0,\n "HighestPriorityLowRedundancyReplicatedBlocks" : 0,\n "HighestPriorityLowRedundancyECBlocks" : 0,\n "BytesInFutureReplicatedBlocks" : 0,\n "PendingDeletionReplicatedBlocks" : 0,\n "TotalReplicatedBlocks" : 220,\n "LowRedundancyECBlockGroups" : 0,\n "CorruptECBlockGroups" : 0,\n "MissingECBlockGroups" : 0,\n "BytesInFutureECBlockGroups" : 0,\n "PendingDeletionECBlocks" : 0,\n "TotalECBlockGroups" : 0,\n "ExcessBlocks" : 0,\n "NumTimedOutPendingReconstructions" : 0,\n "PostponedMisreplicatedBlocks" : 0,\n "PendingDataNodeMessageCount" : 0,\n "MillisSinceLastLoadedEdits" : 85711,\n "BlockCapacity" : 33554432,\n "NumLiveDataNodes" : 3,\n "NumDeadDataNodes" : 0,\n "NumDecomLiveDataNodes" : 0,\n "NumDecomDeadDataNodes" : 0,\n "VolumeFailuresTotal" : 0,\n "EstimatedCapacityLostTotal" : 0,\n "NumDecommissioningDataNodes" : 0,\n "StaleDataNodes" : 0,\n "NumStaleStorages" : 0,\n "TotalSyncCount" : 0,\n "NumInMaintenanceLiveDataNodes" : 0,\n "NumInMaintenanceDeadDataNodes" : 0,\n "NumEnteringMaintenanceDataNodes" : 0\n } ]\n}', u'')
2022-04-23 03:22:36,961 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://my_bigdatamn002.my_FQDN.io:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpeyUCm5 2>/tmp/tmpLJp7u3''] {'quiet': False}
2022-04-23 03:22:37,043 - call returned (0, '')
2022-04-23 03:22:37,044 - get_user_call_output returned (0, u'{\n "beans" : [ {\n "name" : "Hadoop:service=NameNode,name=FSNamesystem",\n "modelerType" : "FSNamesystem",\n "tag.Context" : "dfs",\n "tag.HAState" : "active",\n "tag.TotalSyncTimes" : "21 23 7 ",\n "tag.Hostname" : "my_bigdatamn002.my_FQDN.io",\n "MissingBlocks" : 0,\n "MissingReplOneBlocks" : 0,\n "ExpiredHeartbeats" : 0,\n "TransactionsSinceLastCheckpoint" : 2739,\n "TransactionsSinceLastLogRoll" : 15,\n "LastWrittenTransactionId" : 2755,\n "LastCheckpointTime" : 1650707679000,\n "CapacityTotal" : 283346115674112,\n "CapacityTotalGB" : 263887.0,\n "CapacityUsed" : 11776085343,\n "CapacityUsedGB" : 11.0,\n "CapacityRemaining" : 280452166158831,\n "ProvidedCapacityTotal" : 0,\n "CapacityRemainingGB" : 261191.0,\n "CapacityUsedNonDFS" : 0,\n "TotalLoad" : 24,\n "SnapshottableDirectories" : 0,\n "Snapshots" : 0,\n "NumEncryptionZones" : 0,\n "LockQueueLength" : 0,\n "BlocksTotal" : 220,\n "NumFilesUnderConstruction" : 3,\n "NumActiveClients" : 2,\n "FilesTotal" : 1019,\n "PendingReplicationBlocks" : 0,\n "PendingReconstructionBlocks" : 0,\n "UnderReplicatedBlocks" : 0,\n "LowRedundancyBlocks" : 0,\n "CorruptBlocks" : 0,\n "ScheduledReplicationBlocks" : 0,\n "PendingDeletionBlocks" : 90,\n "LowRedundancyReplicatedBlocks" : 0,\n "CorruptReplicatedBlocks" : 0,\n "MissingReplicatedBlocks" : 0,\n "MissingReplicationOneBlocks" : 0,\n "HighestPriorityLowRedundancyReplicatedBlocks" : 0,\n "HighestPriorityLowRedundancyECBlocks" : 0,\n "BytesInFutureReplicatedBlocks" : 0,\n "PendingDeletionReplicatedBlocks" : 90,\n "TotalReplicatedBlocks" : 220,\n "LowRedundancyECBlockGroups" : 0,\n "CorruptECBlockGroups" : 0,\n "MissingECBlockGroups" : 0,\n "BytesInFutureECBlockGroups" : 0,\n "PendingDeletionECBlocks" : 0,\n "TotalECBlockGroups" : 0,\n "ExcessBlocks" : 0,\n "NumTimedOutPendingReconstructions" : 1,\n "PostponedMisreplicatedBlocks" : 0,\n "PendingDataNodeMessageCount" : 0,\n "MillisSinceLastLoadedEdits" : 0,\n "BlockCapacity" : 33554432,\n "NumLiveDataNodes" : 3,\n "NumDeadDataNodes" : 0,\n "NumDecomLiveDataNodes" : 0,\n "NumDecomDeadDataNodes" : 0,\n "VolumeFailuresTotal" : 0,\n "EstimatedCapacityLostTotal" : 0,\n "NumDecommissioningDataNodes" : 0,\n "StaleDataNodes" : 0,\n "NumStaleStorages" : 0,\n "TotalSyncCount" : 15,\n "NumInMaintenanceLiveDataNodes" : 0,\n "NumInMaintenanceDeadDataNodes" : 0,\n "NumEnteringMaintenanceDataNodes" : 0\n } ]\n}', u'')
2022-04-23 03:22:37,046 - NameNode HA states: active_namenodes = [(u'nn2', 'my_bigdatamn002.my_FQDN.io:50070')], standby_namenodes = [(u'nn1', 'my_bigdatamn001.my_FQDN.io:50070')], unknown_namenodes = []
2022-04-23 03:22:37,047 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://my_bigdatamn002.my_FQDN.io:50070/webhdfs/v1/user/hive/warehouse/sys.db/app_data?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpoEc7Xv 2>/tmp/tmptp1Uh0''] {'logoutput': None, 'quiet': False}
2022-04-23 03:22:37,126 - call returned (0, '')
2022-04-23 03:22:37,127 - get_user_call_output returned (0, u'{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File does not exist: /user/hive/warehouse/sys.db/app_data"}}404', u'')
2022-04-23 03:22:37,128 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://my_bigdatamn002.my_FQDN.io:50070/webhdfs/v1/user/hive/warehouse/sys.db/app_data?op=MKDIRS&user.name=hdfs'"'"' 1>/tmp/tmpEjPcYD 2>/tmp/tmpOLNo2G''] {'logoutput': None, 'quiet': False}
2022-04-23 03:22:37,215 - call returned (0, '')
2022-04-23 03:22:37,216 - get_user_call_output returned (0, u'{"boolean":true}200', u'')
2022-04-23 03:22:37,217 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://my_bigdatamn002.my_FQDN.io:50070/webhdfs/v1/user/hive/warehouse/sys.db/app_data?op=SETPERMISSION&user.name=hdfs&permission=1777'"'"' 1>/tmp/tmp4WqYrG 2>/tmp/tmppqYALi''] {'logoutput': None, 'quiet': False}
2022-04-23 03:22:37,305 - call returned (0, '')
2022-04-23 03:22:37,305 - get_user_call_output returned (0, u'200', u'')
2022-04-23 03:22:37,308 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://my_bigdatamn002.my_FQDN.io:50070/webhdfs/v1/user/hive/warehouse/sys.db/app_data?op=SETOWNER&owner=hive&group=&user.name=hdfs'"'"' 1>/tmp/tmpCVYOd7 2>/tmp/tmpnzTLdm''] {'logoutput': None, 'quiet': False}
2022-04-23 03:22:37,394 - call returned (0, '')
2022-04-23 03:22:37,394 - get_user_call_output returned (0, u'200', u'')
2022-04-23 03:22:37,396 - HdfsResource[None] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/3.0.1.0-187/hadoop/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://nameservice1', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': 'missing_principal', 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/3.0.1.0-187/hadoop/conf', 'immutable_paths': [u'/user/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp']}
2022-04-23 03:22:37,401 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2022-04-23 03:22:37,401 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-hive.json
2022-04-23 03:22:37,401 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-hive.json'] {'content': Template('input.config-hive.json.j2'), 'mode': 0644}
2022-04-23 03:22:37,402 - Ranger Hive plugin is not enabled
2022-04-23 03:22:37,402 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'cat /var/run/hive/hive-server.pid 1>/tmp/tmp8LUyGM 2>/tmp/tmpRR982N''] {'quiet': False}
2022-04-23 03:22:37,473 - call returned (1, '')
2022-04-23 03:22:37,473 - Execution of 'cat /var/run/hive/hive-server.pid 1>/tmp/tmp8LUyGM 2>/tmp/tmpRR982N' returned 1. cat: /var/run/hive/hive-server.pid: No such file or directory
2022-04-23 03:22:37,473 - get_user_call_output returned (1, u'', u'cat: /var/run/hive/hive-server.pid: No such file or directory')
2022-04-23 03:22:37,474 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'hive --config /usr/hdp/current/hive-server2/conf/ --service metatool -listFSRoot' 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://nameservice1' | head -1'] {}
2022-04-23 03:22:44,722 - call returned (0, '')
2022-04-23 03:22:44,723 - Execute['/var/lib/ambari-agent/tmp/start_hiveserver2_script /var/log/hive/hive-server2.out /var/log/hive/hive-server2.err /var/run/hive/hive-server.pid /usr/hdp/current/hive-server2/conf/ /etc/tez/conf'] {'environment': {'HIVE_BIN': 'hive', 'JAVA_HOME': u'/usr/java/jdk1.8.0.171', 'HADOOP_HOME': u'/usr/hdp/current/hadoop-client'}, 'not_if': 'ls /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps -p >/dev/null 2>&1', 'user': 'hive', 'path': [u'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hive-server2/bin:/usr/hdp/3.0.1.0-187/hadoop/bin']}
2022-04-23 03:22:44,810 - Execute['/usr/java/jdk1.8.0.171/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-server2/lib/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://my_bigdatamn001.my_FQDN.io/my_bigdata_hive' my_bigdata_hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': 10}
2022-04-23 03:22:45,089 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:22:45,641 - call returned (1, 'Node does not exist: /hiveserver2')
my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:23:27,906 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:23:27,906 - Will retry 25 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:23:37,917 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:23:38,467 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:23:38,468 - Will retry 24 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:23:48,479 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:23:49,026 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:23:49,027 - Will retry 23 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:23:59,035 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:23:59,594 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:23:59,595 - Will retry 22 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:24:09,604 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:24:10,152 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:24:10,154 - Will retry 21 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:24:20,165 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:24:20,716 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:24:20,716 - Will retry 20 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:24:30,719 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:24:31,262 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:24:31,263 - Will retry 19 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:24:41,274 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:24:41,824 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:24:41,825 - Will retry 18 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:24:51,836 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:24:52,402 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:24:52,404 - Will retry 17 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:25:02,415 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:25:02,978 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:25:02,979 - Will retry 16 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:25:12,990 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:25:13,537 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:25:13,537 - Will retry 15 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:25:23,548 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:25:24,102 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:25:24,103 - Will retry 14 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:25:34,110 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:25:34,663 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:25:34,664 - Will retry 13 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:25:44,670 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:25:45,228 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:25:45,230 - Will retry 12 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:25:55,241 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:25:55,801 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:25:55,802 - Will retry 11 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:26:05,813 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:26:06,375 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:26:06,376 - Will retry 10 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:26:16,387 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:26:16,933 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:26:16,933 - Will retry 9 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:26:26,944 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:26:27,494 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:26:27,495 - Will retry 8 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:26:37,497 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:26:38,044 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:26:38,045 - Will retry 7 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:26:48,056 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server my_bigdatamn001.my_FQDN.io:2181,my_bigdatacn001.my_FQDN.io:2181,my_bigdatamn002.my_FQDN.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-04-23 03:26:48,610 - call returned (1, 'Node does not exist: /hiveserver2')
2022-04-23 03:26:48,611 - Will retry 6 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-04-23 03:26:58,622 - Process with pid 350616 is not running. Stale pid file at /var/run/hive/hive-server.pid
Command failed after 1 tries
So, I checked other questions. But this error is still same. I checked zookeeper's znode(/hiveserver2). But there was nothing!
[zk: localhost:2181(CONNECTED) 0] ls /hiveserver2
Node does not exist: /hiveserver2
How can I solve this problem??
Plz somebody save me ....
Restart hive and it will re-write to the Zookeeper node space. (If it can)