Runtime Error on Cloud Run when using Dash App and Papermill executions on background tasks

Question

I am facing an issue in CloudRun: Below I am only showing the scheduling part (background tasks), but I also have some code for a Dash application. I would like to have a Dash application and scheduler that can execute Jupyter Notebooks on the same cloud run application. I am deploying using a dockerFile. I have tried the CloudRun Revision with 4CPU and 4GiB memory.

Everything is working locally (Dash App and Scheduling) but it seems that the papermill execution are not working in CloudRun. The Notebook is simply sending a mail for now. All advices are welcome! Thank you Here is my code:

def run_continuously(interval=1):
    cease_continuous_run = threading.Event()

    class ScheduleThread(threading.Thread):
        @classmethod
        def run(cls):
            while not cease_continuous_run.is_set():
                schedule.run_pending()
                time.sleep(interval)

    continuous_thread = ScheduleThread()
    continuous_thread.start()
    return cease_continuous_run


def runPapermill1():
    print('running 1')
    sec = str(datetime.now().minute)
    pm.execute_notebook(r'testPapermill.ipynb', r"OutputNotebook/output-" + sec + ".ipynb", kernel_name='python3', start_timeout=120)


if __name__ == '__main__':
    schedule.every(50).seconds.do(runPapermill1)
    # Start the background thread
    stop_run_continuously = run_continuously()

    server_port = os.environ.get('PORT', '8080')
    app.run_server(debug=True, port=server_port, host='0.0.0.0')

    # Stop the background thread
    print('Stopping background Task')
    stop_run_continuously.set()

HERE IS AN ERROR IN CloudRun:

2022-03-21T15:18:54.442802ZTraceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/app.py", line 90, in run schedule.run_pending() File "/usr/local/lib/python3.8/site-packages/schedule/init.py", line 780, in run_pending default_scheduler.run_pending() File "/usr/local/lib/python3.8/site-packages/schedule/init.py", line 100, in run_pending self._run_job(job) File "/usr/local/lib/python3.8/site-packages/schedule/init.py", line 172, in _run_job ret = job.run() File "/usr/local/lib/python3.8/site-packages/schedule/init.py", line 661, in run ret = self.job_func() File "/app/app.py", line 118, in runPapermill1 pm.execute_notebook(r'testPapermill.ipynb', r"OutputNotebook/output-" + sec + ".ipynb", kernel_name='python3', File "/usr/local/lib/python3.8/site-packages/papermill/execute.py", line 107, in execute_notebook nb = papermill_engines.execute_notebook_with_engine( File "/usr/local/lib/python3.8/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs) File "/usr/local/lib/python3.8/site-packages/papermill/engines.py", line 359, in execute_notebook cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs) File "/usr/local/lib/python3.8/site-packages/papermill/engines.py", line 418, in execute_managed_notebook return PapermillNotebookClient(nb_man, **final_kwargs).execute() File "/usr/local/lib/python3.8/site-packages/papermill/clientwrap.py", line 43, in execute with self.setup_kernel(**kwargs): File "/usr/local/lib/python3.8/contextlib.py", line 113, in enter return next(self.gen) File "/usr/local/lib/python3.8/site-packages/nbclient/client.py", line 562, in setup_kernel self.start_new_kernel_client() File "/usr/local/lib/python3.8/site-packages/nbclient/util.py", line 84, in wrapped return just_run(coro(*args, **kwargs)) File "/usr/local/lib/python3.8/site-packages/nbclient/util.py", line 62, in just_run return loop.run_until_complete(coro) File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/local/lib/python3.8/site-packages/nbclient/client.py", line 532, in async_start_new_kernel_client await ensure_async(self.kc.wait_for_ready(timeout=self.startup_timeout)) File "/usr/local/lib/python3.8/site-packages/nbclient/util.py", line 96, in ensure_async result = await obj File "/usr/local/lib/python3.8/site-packages/jupyter_client/client.py", line 184, in _async_wait_for_ready raise RuntimeError("Kernel didn't respond in %d seconds" % timeout) RuntimeError: Kernel didn't respond in 120 seconds

SOMETIMES, IT STARTS RUNNING AND JUST STOPS as below Exeample Here

Answer 1

Solution Found:

I just needed to set "CPU is always alloacted" on the CloudRun. This allows me to have the dash app and background tasks (papermill execution) running at the same time. And I set the min instances to 1 as well.