docker-compose with hadoop-spark-hive-hdfs services

265 views Asked by At

i have on my windows pc docker-desktop. I've downloaded this image: https://github.com/Marcel-Jan/docker-hadoop-spark

i run it with docker-compose up and start well.

but i dont undstand where i can use on same node all 3 services(hdfs-spark-hive), infact if i type:

docker exec -it hive-server bash 
hive 
create table mytable(id int,name string, cognome string);

after i open a new power shell console and type:

docker exec -it spark-master bash 
cd spark/bin
./pyspark
spark.sql("show tables").show()
>>> spark.sql("show tables").show()
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
+--------+---------+-----------+

i edit spark-master in my docker-compose.yml as:

spark-master:
image: bde2020/spark-master:3.0.0-hadoop3.2
container_name: spark-master
depends_on:
  - namenode
  - datanode
  - hive-server
ports:
  - "8080:8080"
  - "7077:7077"
environment:
  - INIT_DAEMON_STEP=setup_spark
  - CORE_CONF_fs_defaultFS=hdfs://namenode:9000
  - HADOOP_CONF_fs_defaultFS=hdfs://namenode:9000
  - SPARK_CONF_spark_eventLog_enabled=true
  - SPARK_CONF_spark_eventLog_dir=hdfs://namenode:9000/spark-logs
  - SPARK_CONF_spark_history_fs_logDirectory=hdfs://namenode:9000/spark-logs
  - SPARK_CONF_spark_history_store_path=hdfs://namenode:9000/spark-logs
  - SPARK_CONF_spark_sql_warehouse_dir=hdfs://namenode:9000/user/hive/warehouse
  - SPARK_CONF_spark_yarn_historyServer_address=http://historyserver:8188

but always i dont view nothing. Seems that spark dont view hive server. any tips? thanks Regards

0

There are 0 answers