I'm trying to use streamparse on a local cluster with docker containers
Here's my docker-compose.yml
version: '3'
services:
zookeeper:
image: zookeeper
container_name: zookeeper
nimbus:
image: storm:1.0.2
container_name: nimbus
command: storm nimbus
depends_on:
- zookeeper
supervisor:
image: simsimy/docker-storm-supervisor-with-streamparse
container_name: supervisor
command: storm supervisor
depends_on:
- nimbus
- zookeeper
storm-ui:
image: storm:1.0.2
container_name: storm-ui
command: storm ui
depends_on:
- nimbus
- zookeeper
- supervisor
The streamparse sample is out of the box (created with the "sparse quickstart wordcount" command)
I just have changed some parameters in the wordcount quickstart example to match my storm version 1.0.2 (in project.clj file)
Btw, if you have any solution with another storm version, i can change mine
Here is my config.json :
{
"serializer": "json",
"topology_specs": "topologies/",
"virtualenv_specs": "virtualenvs/",
"envs": {
"prod": {
"user": "",
"ssh_password": "",
"nimbus": "localhost",
"workers": ["localhost"],
"log": {
"path": "",
"max_bytes": 1000000,
"backup_count": 10,
"level": "info"
},
"use_virtualenv": true,
"virtualenv_root": "/home/myhome/virtualenvs"
}
}
}
./wordcount/virtualenvs/wordcount.txt contains only "steamparse" (out of the box config file)
When i run Sparse submit, the topology fails with the message :
java.io.IOException: Cannot run program "/home/myhome/virtualenvs/wordcount/bin/streamparse_run" (in directory "/data/supervisor/stormdist/wordcount-10-1529168923/resources")
I have checked the /data/supervisor/stormdist/wordcount-10-1529168923/resources directory, it only contains spouts and bolts...
The topology only works when i set "use_virtualenv": false
Here is the log file of the dependencies collect : https://pastebin.com/Tf48kpBb
Any help would be very appreciated
Thanks