Cassandra on Docker - nodetool connection issue

Question

I was given a R&D task on running Cassandra on Docker. I'm working on XUbuntu 14.04.2 64 bit (installed on VirtualBOX in Windows7 32 bit OS).

I was following the steps given in Docker on Ubuntu and Pokle cassandra tutorials and I got a ConnectException when I tried to check the status of a particular node using the nodetool.

Following is the list of steps that I did.

asela@teamlk:~$ docker run -d --name cass1 poklet/cassandra start
7b2ddfb60adf1c270a425c4440fe263e1fe56f5dc1c3205372bee3b8626e0624

asela@teamlk:~$ docker run -d --name cass2 poklet/cassandra start $(/data/cassandra/scripts/ipof.sh cass1)
4f98a194214e603242279226bb94cee20774058e84b6d0171a8d14d648e19016

asela@teamlk:~$ docker run -d --name cass3 poklet/cassandra start $(/data/cassandra/scripts/ipof.sh cass1)
394f9ce918121c0b79ed8282ba95c8f1badd4e3536d4ad3f24c4c21651e50b3c

asela@teamlk:~$ docker ps
CONTAINER ID        IMAGE                     COMMAND              CREATED              STATUS              PORTS                                                                      NAMES
394f9ce91812        poklet/cassandra:latest   "start 172.17.0.1"   6 seconds ago        Up 3 seconds        22/tcp, 7000-7001/tcp, 7199/tcp, 8012/tcp, 9042/tcp, 9160/tcp, 61621/tcp   cass3               
4f98a194214e        poklet/cassandra:latest   "start 172.17.0.1"   31 seconds ago       Up 29 seconds       22/tcp, 7000-7001/tcp, 7199/tcp, 8012/tcp, 9042/tcp, 9160/tcp, 61621/tcp   cass2               
7b2ddfb60adf        poklet/cassandra:latest   "start"              About a minute ago   Up About a minute   22/tcp, 7000-7001/tcp, 7199/tcp, 8012/tcp, 9042/tcp, 9160/tcp, 61621/tcp   cass1               

asela@teamlk:~$ docker run -i -t poklet/cassandra nodetool -h $(/data/cassandra/scripts/ipof.sh cass1) status
nodetool: Failed to connect to '172.17.0.1:7199' - ConnectException: 'Connection refused'.

ifconfig shows me that my Docker is running on 172.17.42.1

Can someone please tell me if I have missed any configurations and/or if I have done something wrong?

Answer 1

I added the line - "LOCAL_JMX=no" to my docker-compose.yml file, so I have a section

environment:
  - "MAX_HEAP_SIZE=256M"
  - "HEAP_NEWSIZE=128M"
  - "LOCAL_JMX=no"

Answer 2

I am using Fedora 22 but I don't see why it shouldn't work on Ubuntu.

I was having the exact same problem and the following command worked for me:

docker exec cass-1 nodetool status

Notice that cass1 is the name of the docker container for the cassandra node, its used instead of the IP Address.

• I tried this after reading this post which proved to be quite helpful.
• The docker exec command definition - Run a command in a running container.

My output:

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns (effective)  Host ID                               Rack
UN  172.17.0.14  51.6 KB    256     100.0%            3dec79f4-4a6f-46e6-b4d5-14b8f56b6c13  rack1

Answer 3

I am a little late but here are some insights:

If you start a docker container you need to map ports to the outside world or link containers together.

Port Mapping is done by the -p parameter while starting Docker containers. Documentation can be found: https://docs.docker.com/articles/networking/

Containerlinking is done with the --link parameter and it works as if both linked containers are running on the same host. Documentation: https://docs.docker.com/userguide/dockerlinks/

To make your example work (accessing other containers by their IP) you have to expose ports. In case of cassandra this would be

-p 7000:7000 -p 9160:9160 -p 9042:9042 -p 7199:7199

For Reference this is the command I execute on 2 physical hosts to start a cassandra cluster. Because of the port mappings only one instance per host can be started! (to have the container detached from commandline you need to add the -d switch)

docker run --rm --name cassandra-`hostname \
  -v /data/cassandra/projectname:/var/lib/cassandra/data \
  -e CASSANDRA_CLUSTER_NAME=project-database 
  -e CASSANDRA_BROADCAST_ADDRESS=`hostname` 
  -p 7000:7000 -p 9160:9160 -p 9042:9042 -p 7199:7199 \
  -e CASSANDRA_SEEDS=<ip_of_host1>,<ip_of_host2>
  cassandra:2.2.1

Answer 4

In cassandra-env.sh there is a setting "LOCAL_JMX". By default it was "yes", setting it "no" will allow remote connections to port 7199.

Also, you may disable authentication in the same config file by adding: JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"

-- Regarding pokle's docker image, I've made a pull request with the above changes to start.sh(github.com/pokle/cassandra/pull/28) which was merged and now the docker image should work as expected in README.md.

Answer 5

Use docker's exec instead of run.

Most of the Cassandra images out there in the world ship with a reasonable, local-only JMX configuration. However, it's sometimes easy to get confused about where commands are running. docker run will always spin up a new container (conceptually a tiny VM) to execute a command. docker exec will connect to an already running container and execute a command within it. Since we're talking about local-only JMX control via nodetool, we need exec to run the command inside the same container where cassandra itself is operating.

$ docker run -d cassandra:3.0
6ebda615efd2f997f95aece8f7e8ae0faafa72145a3225831fe31513557dc2bb

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                                                              NAMES
6ebda615efd2        cassandra:3.0       "docker-entrypoint.s…"   3 seconds ago       Up 2 seconds        7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp                                                                                        confident_margulis

$ docker run cassandra:3.0 nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.

$ docker exec 6ebda615efd2 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.2  103.06 KB  256          100.0%            72bc1f12-538c-4222-9931-40aab96089b3  rack1

Notice that the exec command requires the identifier of the running container, where run takes the image name.