Get access to local folder in Heron

Question

I've written Heron topology which takes data from local folder and processes it. However, heron can't see any other folders except the one described in .pex file. How can I read data outside of .pex declared folders?

Answer 1

Once a heron instance is launched, it should have access to files inside the same container. One example is they can write logs into ./logs/INSTANCE_ID.log, which means it has the write access to those files.

Could you check if the data files are available inside the running container and the instance is using the correct file path to access those files?

Answer 2

Ideally, you would not rely on local data in you topology. It would be better to place any data your bolts need access to in an external system like a database or key:value store. Then you would get your bolts to query (and cache locally in memory) that data. This also has the advantage of scaling better if you need to add more copies of each bolt, as you avoid possible resource contention on the local files.

If you really want the data to be locally accessible to the bolts the answer depends on what scheduler you are using. It could involve adding the files to the heron instance container image that your scheduler uses and could mean rebuilding the containers every time you change the data. Again, I would suggest using an external datastore.