two way security (multi-protocol)

Question

I'm implementing an one-to-many multi-protocol server (+ clients) and I'd like to add 2-way security. Here's what I'd like to accomplish:

1. both client and server authenticate to each other in a secure way. there is no human interaction involved on the client side.
2. client's code checksum is validated on the server.
3. client's code may be written in an interpreted language (such as python or javascript), so I'd like to prevent the possibility to compromise the network after someone gains access to the client (this may be an overkill though, because my clients won't be executing anything on the server, just reporting the results of their actions)

How should I design the authentication flow? What techniques should I use/google for, or - on a lower level - what existing solutions could I try? (my prototype is written using node.js)

Answer 1

SSL can do authentication both ways. Out of the box, nothing special needed. One can even get the certificates for free (self-signed or from recognized CAs).

Client certificates can be used to distinguish clients if that's a need, similarly they can be used to prevent copies of clients that log in simultaneously.

What you fundamentally cannot do is prevent a smart malicious user from controlling a client in such a manner as that they reverse engineer how it interacts with the server and instead of running your intended client, run their own that still acts as if it is the real client but isn't.

The solution to the impossibility of trusting the client is to not let it do things that you have to trust it is running your code unaltered. That often means moving from a 2 tier model (heavy client - server) to a 3 tier model where the code that you want to run is kept on hardware you control, and only an (untrusted) user interfacing is pushed to the user controlled hardware.