Correct way to get the logs from the remote functions in worker node in ray

Question

Similar question has been asked at How can I use the python logging in Ray? but the solution provided is not working for me

I'm not able to get the logs from the worker node in the stdout /logfile. warning and error logs are shown in stdout with different pid but not other logs below log.warning level. An stdout example is shown below. I have tried this both on ray 1.0.0 and 0.9dev versions. Similar issues were raised before here and on stackoverflow but there it is mentioned its rectified. For worker node ray.init, I have set the logging level to DEBUG still facing issues.

ray.init(address=ray_cluster_addr, ignore_reinit_error=True, logging_level=10)

Can someone please help out if I'm missing anything

2020-10-15 13:49:35,907 ERROR worker.py:643 -- Calling ray.init() again after it has already been called.
2020-10-15 13:49:35.908 INFO - emagent.py - start_em_worker[63] Running EM algorithm on sample file 
(pid=92) Ignoring first 2 columns in init_b_params
(pid=92) Ignoring first 2 columns in init_b_params    


  [1]: https://stackoverflow.com/questions/55272066/how-can-i-use-the-python-logging-in-ray

Answer 1

Please refer to this Github discussion thread. https://github.com/ray-project/ray/discussions/11426