open file inside Ray

Question

I'm using RAY, and created a custom env. However, the custom env needs to open a file, and ray creates workers in a different location. Therefore, I can't access the file.

when printing the worker location I'm getting : city_v1/DQN/DQN_CityFlows_074cc_00000_0_2022-04-20_13-09-56

file exists in:

examples/1x1/file.json

Answer 1

As you mentioned, Ray worker processes may run in different locations, even different machines.

The recommended solution for making files available to workers is to explicitely specify them via runtime environments. https://docs.ray.io/en/latest/ray-core/handling-dependencies.html

In your case you may want to do

runtime_env = {"working_dir": "./"}

to copy the local working directory.

Note that the these directories are sent from the head node to all the workers, which could be a slow process if your directory is large (in which case, you should try to minimize the number of files that need to be uploaded).