STAN on Databricks - AttributeError: 'ConsoleBuffer' object has no attribute 'closed'

Question

Running STAN (pystan) on Databricks 8.2 ML throws the following Error

To reproduce, just run the simple example from https://pystan.readthedocs.io/en/latest/

Seems like the ConsoleBuffer Class doesn't have an implementation for closed? Have others run into this issue? Any workarounds recommended? I am currently using a single node Cluster and ideally don't want to run this on a local machine.

Stack Trace

AttributeError                            Traceback (most recent call last)
<command-261559943577864> in <module>
      3                 "sigma": [15, 10, 16, 11,  9, 11, 10, 18]}
      4 
----> 5 posterior = stan.build(schools_code, data=schools_data)
      6 fit = posterior.sample(num_chains=4, num_samples=1000)
      7 eta = fit["eta"]  # array with shape (8, 4000)

/databricks/python/lib/python3.8/site-packages/stan/model.py in build(program_code, data, random_seed)
    468 
    469     try:
--> 470         return asyncio.run(go())
    471     except KeyboardInterrupt:
    472         return  # type: ignore

/databricks/python/lib/python3.8/asyncio/runners.py in run(main, debug)
     41         events.set_event_loop(loop)
     42         loop.set_debug(debug)
---> 43         return loop.run_until_complete(main)
     44     finally:
     45         try:

/databricks/python/lib/python3.8/asyncio/base_events.py in run_until_complete(self, future)
    614             raise RuntimeError('Event loop stopped before Future completed.')
    615 
--> 616         return future.result()
    617 
    618     def stop(self):

/databricks/python/lib/python3.8/site-packages/stan/model.py in go()
    438     async def go():
    439         io = ConsoleIO()
--> 440         io.error("<info>Building...</info>")
    441         async with stan.common.HttpstanClient() as client:
    442             # Check to see if model is in cache.

/databricks/python/lib/python3.8/site-packages/clikit/api/io/io.py in error(self, string, flags)
     84         The string is formatted before it is written to the output.
     85         """
---> 86         self._error_output.write(string, flags=flags)
     87 
     88     def error_line(self, string, flags=None):  # type: (str, Optional[int]) -> None

/databricks/python/lib/python3.8/site-packages/clikit/api/io/output.py in write(self, string, flags, new_line)
     59                 formatted += "\n"
     60 
---> 61             self._stream.write(to_str(formatted))
     62 
     63     def write_line(self, string, flags=None):  # type: (str, Optional[int]) -> None

/databricks/python/lib/python3.8/site-packages/clikit/io/output_stream/stream_output_stream.py in write(self, string)
     19         Writes a string to the stream.
     20         """
---> 21         if self.is_closed():
     22             raise io.UnsupportedOperation("Cannot write to a closed input.")
     23 

/databricks/python/lib/python3.8/site-packages/clikit/io/output_stream/stream_output_stream.py in is_closed(self)
    114         Returns whether the stream is closed.
    115         """
--> 116         return self._stream.closed

AttributeError: 'ConsoleBuffer' object has no attribute 'closed'

Answer 1

After trying some old clusters, I realized that pystan 3 is a complete re-write. So one workaround is to go back to pystan==2.19.1.1