Wanted: Instructions for adding client-node encryption enabled Cassandra cluster to DataStax OpsCenter 5.1.0

Question

I have a Cassandra cluster with client-node encryption enabled. I am trying to add this cluster to an instance of OpsCenter 5.1.0, but it is not able to connect to the cluster. The log file seems to complain about not being able to verify the SSL certificate:

` 
INFO: Starting factory opscenterd.ThriftService.NoReconnectCassandraClientFactory instance at 0x7f2ce05c8638>
2015-06-10 15:09:46+0000 []  WARN: Unable to verify ssl certificate.
2015-06-10 15:09:46+0000 [] Unhandled Error
        Traceback (most recent call last):
          File "/opt/opscenter-5.1.0/lib/py-debian/2.7/amd64/twisted/python/log.py", line 84, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "/opt/opscenter-5.1.0/lib/py-debian/2.7/amd64/twisted/python/log.py", line 69, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "/opt/opscenter-5.1.0/lib/py-debian/2.7/amd64/twisted/python/context.py", line 59, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/opt/opscenter-5.1.0/lib/py-debian/2.7/amd64/twisted/python/context.py", line 37, in callWithContext
            return func(*args,**kw)
        --- exception caught here ---
          File "/opt/opscenter-5.1.0/lib/py-debian/2.7/amd64/twisted/internet/epollreactor.py", line 217, in _doReadOrWrite
            why = selectable.doRead()
          File "/opt/opscenter-5.1.0/lib/py-debian/2.7/amd64/twisted/internet/tcp.py", line 137, in doRead
            return Connection.doRead(self)
          File "/opt/opscenter-5.1.0/lib/py-debian/2.7/amd64/twisted/internet/tcp.py", line 452, in doRead
            data = self.socket.recv(self.bufferSize)
          File "build/lib/python2.7/site-packages/opscenterd/SslUtils.py", line 12, in ssl_simple_verifyCB


    opscenterd.Utils.SSLVerifyException: SSL certificate invalid

My question is: what are the step-by-step instructions for being able to add a client-node encrypted cluster to opscenter? Which .pem and .keystore files are needed exactly, how do I get hold of them?

The DataStax documentation on that topic is not detailed enough and therefore not really helpful. I assume some people out there must have managed to set this up successfully and I am sure that a detailed explanation / instructions would be appreciated by many.

Answer 1

One thing to note here, although the docs do mention generating a key per node, in practice this isn't very scalable. In most systems it is common to create the one keystore with the required keys and certificate(s) and then use this across all the nodes in your cluster and your client applications as needed. You export the certificate from this keystore and use this for OpsCenter. OpsCenter is (as far as SSL is concerned) a SSL client like any other client.

So you have to export your key from your java keystore, convert it to .pem format and use that for the opscenterd process. The agents are java based so they can use the java keystore. The DS docs are there but its a bit fragmented so its a question of looking in the right places :-)

I'm going to use the OpsCenter latest docs here as a reference. I'm assuming you are only using SSL between OpsCenter and Cassandra and OpsCenter Agents and Cassandra

Prepping the server certificates:

https://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html

Configuring client to node SSL:

https://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLClientToNode_t.html

using cqlsh with SSL (optional):

https://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureCqlshSSL_t.html

To convert the key to a pem format see step 7 here:

https://docs.datastax.com/en/latest-opsc/opsc/online_help/opscAddingCluster_t.html

Examples

Note all these examples assume 1-way SSL. You generated a key in a file called /etc/dse/keystore and the certificate in a file called /etc/dse/truststore

To be honest I've never really had a lot of luck in adding SSL enabled clusters directly in the OpsCenter UI. I've always found creating the cluster.conf file and agent address.yaml files by hand far quicker and easier.

Note the SSL files like truststore, key.pem etc need to be on all the local machines that need them.

Example agent /var/lib/datastax-agent/conf/address.yaml file (note the use_ssl is for the opscenter <> agents SSL which we're not using here)

stomp_interface: 192.168.56.29
use_ssl: 0
# ssl_keystore settings if using ssl
ssl_keystore: /etc/dse/truststore
ssl_keystore_password: datastax

Example opscenter /etc/opscenter/clusters/<cluster_name>.conf file

[jmx]
username = 
password = 
port = 7199

[kerberos_client_principals]

[kerberos]

[agents]
ssl_keystore = /etc/dse/truststore
ssl_keystore_password = datastax

[kerberos_hostnames]

[kerberos_services]

[cassandra]
ssl_ca_certs = /etc/dse/key.pem
ssl_validate = False
seed_hosts = 192.168.56.22

Other tips etc

I always find if Im troubleshooting SSL connections in DSE / Cassandra. I'll strip out all the SSL and get the cluster working nomrmally first, then I'll configure SSL one step at a time, like turning on node to node SSL, then client to node, then OpsCenter and so on. Debugging all the SSL errors is not for the feint hearted!

Links

Other doc links you might find useful:

https://docs.datastax.com/en/opscenter/5.2/opsc/configure/opscConnectionConfig_r.html

https://docs.datastax.com/en/opscenter/5.2/opsc/configure/agentAddressConfiguration.html