I am working in setting Remote PostgreSQL Database for Hive Metastore using
Hive 3.1.3
Hadoop 3.3.4
Spark 3.2.4
PostgresSQL 14.9
I could use Hive Schema Tool to create metastore tables in PostgreSQL using /home/ivan/hive/bin/schematool -dbType postgres -initSchema
with output
Metastore connection URL: jdbc:postgresql://localhost:5432/metastore
Metastore Connection Driver : org.postgresql.Driver
Metastore connection User: hive
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.postgres.sql
Initialization script completed
schemaTool completed
I checked the schema in pgAdmin and everything works as expected so far. However, when I run hive --service metastore
and hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.root.logger=INFO,console
to start Hive Metastore Server and HiveServer2 services respectively, I get
hive --service metastore
2023-11-20 17:15:26: Starting Hive Metastore Server
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/ivan/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/ivan/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
and
hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.root.logger=INFO,console
2023-11-20 17:17:25: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/ivan/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/ivan/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 003418b8-aae6-4061-ac77-6c36451ce5a2
2023-11-20T17:17:26,960 INFO [main] SessionState: Hive Session ID = 003418b8-aae6-4061-ac77-6c36451ce5a2
Hive Session ID = f9e4647e-d61c-4960-8fc4-32198b868c9f
2023-11-20T17:18:27,410 INFO [main] SessionState: Hive Session ID = f9e4647e-d61c-4960-8fc4-32198b868c9f
Hive Session ID = 5de05aeb-df6e-4ce8-8502-48d24e38699a
2023-11-20T17:19:27,449 INFO [main] SessionState: Hive Session ID = 5de05aeb-df6e-4ce8-8502-48d24e38699a
Hive Session ID = 140b29ae-a899-4bce-a873-3bba14ac896a
2023-11-20T17:20:27,489 INFO [main] SessionState: Hive Session ID = 140b29ae-a899-4bce-a873-3bba14ac896a
What proves that hiveserver2 is hanging. I run beeline
(the HiveServer2 CLI), followed by !connect jdbc:hive2://localhost:10000
and I get the following error
beeline> !connect jdbc:hive2://localhost:10000
Connecting to jdbc:hive2://localhost:10000
Enter username for jdbc:hive2://localhost:10000: ivan
Enter password for jdbc:hive2://localhost:10000: ****
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
I run the command netstat -ntpl | egrep "10000|10001|10002"
to check if there is a hiveserver2 port running, but I get
netstat -ntpl | egrep "10000|10001|10002"
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
what proves that hiveserver2 is still hanging. I would really appreciate your help
The most important settings in the hive-site.xml are the following ones
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:postgresql://localhost:5432/metastore</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.postgresql.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>1234</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://192.168.0.5:9000/user/hive/warehouse</value>
</property>
<property>
<name>hive.server2.thrift.http.port</name>
<value>10001</value>
<description>Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'http'.</description>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
<description>Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'binary'.</description>
</property>
</configuration>