Ray Tune conflicting with the `breakpoint()` function

140 views Asked by At

The code section attached below, which is basically copy-pasted from the official documentation of Ray Tune runs as expected until the last line, but calling breakpoint() after tuner.fit() breaks the debugger (I can't see any call stack or local variables in PyCharm's debugger), and prints the following warning:

PYDEV DEBUGGER WARNING:
sys.settrace() should not be used when the debugger is being used.
This may cause the debugger to stop working correctly.
If this is needed, please check: 
http://pydev.blogspot.com/2007/06/why-cant-pydev-debugger-work-with.html
to see how to restore the debug tracing back correctly.
Call Location:
  File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\bdb.py", line 334, in set_trace
    sys.settrace(self.trace_dispatch)

*** Temporarily disabling Ray worker logs ***
--Return--
> c:\users\username\appdata\local\jetbrains\toolbox\apps\pycharm-p\ch-0\223.7571.203\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_breakpointhook.py(13)pydevd_breakpointhook()->None
-> sys.__breakpointhook__(*args, **kwargs)
(Pdb) 
>? 

How could I solve the issue and make the breakpoint() function and the debugger work as expected?

I'm using PyCharm, Python 3.8.10, Windows 10.

main.py

from ray import tune
from ray.tune.search.bayesopt import BayesOptSearch
from ray import air
from ray.air import session

def objective(x, a, b):  # Define an objective function.
    return a * (x**0.5) + b


def trainable(config):  # Pass a "config" dictionary into your trainable.

    for x in range(20):  # "Train" for 20 iterations and compute intermediate scores.
        score = objective(x, config["a"], config["b"])

        session.report({"score": score})  # Send the score to Tune.


# Define the search space
search_space = {"a": tune.uniform(0, 1), "b": tune.uniform(0, 20)}

algo = BayesOptSearch(random_search_steps=4)

tuner = tune.Tuner(
    trainable,
    tune_config=tune.TuneConfig(
        metric="score",
        mode="min",
        search_alg=algo,
    ),
    run_config=air.RunConfig(stop={"training_iteration": 20}),
    param_space=search_space,
)

breakpoint()  # works as expected

r = tuner.fit()

breakpoint()  # raises warning


requirements.txt

pip==21.3.1
setuptools==60.2.0
wheel==0.37.1
ray[tune]==2.2.0
bayesian-optimization==1.4.2

What have I tried?

Inside the PDB session, using traceback.print_stack() I get the following:

  ... pydev/pydevd.py ...
  File "C:\Repos\RayTune\main.py", line 38, in <module>
    breakpoint()  # raises warning
  File "C:\Users\username\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\223.7571.203\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_breakpointhook.py", line 27, in breakpointhook
    pydevd_breakpointhook()
  File "C:\Users\username\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\223.7571.203\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_breakpointhook.py", line 13, in pydevd_breakpointhook
    sys.__breakpointhook__(*args, **kwargs)
File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\bdb.py", line 92, in trace_dispatch
    return self.dispatch_return(frame, arg)
  File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\bdb.py", line 151, in dispatch_return
    self.user_return(frame, arg)
  File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\pdb.py", line 294, in user_return
    self.interaction(frame, None)
  File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\pdb.py", line 357, in interaction
    self._cmdloop()
  File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\pdb.py", line 322, in _cmdloop
    self.cmdloop()
  File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\pdb.py", line 423, in onecmd
    return cmd.Cmd.onecmd(self, line)
  File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\cmd.py", line 216, in onecmd
    return self.default(line)
  File "C:\Users\username\AppData\Local\Programs\Python\Python38\lib\pdb.py", line 381, in default
    exec(code, globals, locals)
  File "<stdin>", line 1, in <module>

locals() evaluate to:

{'args': (), 'kwargs': {}, 'hookname': 'ray.util.rpdb._driver_set_trace', '__return__': None, 'traceback': <module 'traceback' from 'C:\\Users\\username\\AppData\\Local\\Programs\\Python\\Python38\\lib\\traceback.py'>}
0

There are 0 answers