I am running Dask Gateway in a Kubernetes namespace. I am able to connect to the Gateway using the following code, while not running in a Docker container.
from dask.distributed import Client
from dask_gateway import Gateway
gateway = Gateway('http://[redacted traefik ip]')
cluster = gateway.new_cluster()
However, when I run the same code from a Docker container, I get this warning after gateway.new_cluster().
distributed.comm.tcp - WARNING - Closing dangling stream in <TLS local=tls://[local ip]:51060 remote=gateway://[redacted ip]:80/dask-gateway.e71c345decde470e8f9a23c3d5a64956>
What is the cause for this? I have also tried running this with --net=host on the Docker container, that resulted in the same error.
Additional Info: This doesn't appear to be a Docker networking issue... I am able to use the Coiled clusters from within a Docker container, but not the Dask-Gateway clusters...
It appears that the initial outgoing connection from the docker container to the traefik pod succeeds. A dask-scheduler is successfully spun up in the cluster. However, the connection drop (timeout?) that prevents further interactions.