i have on my windows pc docker-desktop. I've downloaded this image: https://github.com/Marcel-Jan/docker-hadoop-spark
i run it with docker-compose
up and start well.
but i dont undstand where i can use on same node all 3 services(hdfs-spark-hive), infact if i type:
docker exec -it hive-server bash
hive
create table mytable(id int,name string, cognome string);
after i open a new power shell console and type:
docker exec -it spark-master bash
cd spark/bin
./pyspark
spark.sql("show tables").show()
>>> spark.sql("show tables").show()
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
+--------+---------+-----------+
i edit spark-master in my docker-compose.yml as:
spark-master:
image: bde2020/spark-master:3.0.0-hadoop3.2
container_name: spark-master
depends_on:
- namenode
- datanode
- hive-server
ports:
- "8080:8080"
- "7077:7077"
environment:
- INIT_DAEMON_STEP=setup_spark
- CORE_CONF_fs_defaultFS=hdfs://namenode:9000
- HADOOP_CONF_fs_defaultFS=hdfs://namenode:9000
- SPARK_CONF_spark_eventLog_enabled=true
- SPARK_CONF_spark_eventLog_dir=hdfs://namenode:9000/spark-logs
- SPARK_CONF_spark_history_fs_logDirectory=hdfs://namenode:9000/spark-logs
- SPARK_CONF_spark_history_store_path=hdfs://namenode:9000/spark-logs
- SPARK_CONF_spark_sql_warehouse_dir=hdfs://namenode:9000/user/hive/warehouse
- SPARK_CONF_spark_yarn_historyServer_address=http://historyserver:8188
but always i dont view nothing. Seems that spark dont view hive server. any tips? thanks Regards