Accessing Google Directory API with NodeJs

Question

I am having trouble accessing the Google Directory API using node. What I am hoping to do is create and remove users from Groups (and create and list groups and their users). In testing I have managed to access most of the APIs without trouble but the Directory has been impossible.

Firstly, is what I am trying to do even possible?

Secondly, if it is possible, here is a sample of my code; what am I missing?

var google = require('googleapis');
var googleAuth = require('google-oauth-jwt');
var request = require('google-oauth-jwt').requestWithJWT();

request({
    url: 'https://www.googleapis.com/admin/directory/v1/groups?domain=mydomainname.com&customer=my_customer',
    jwt: {
        email: '[email protected]',
        keyFile: './MyPemFile.pem',
    scopes: [
        'https://www.googleapis.com/auth/admin.directory.orgunit',
        'https://www.googleapis.com/auth/admin.directory.device.chromeos',
        'https://www.googleapis.com/auth/admin.directory.user',
        'https://www.googleapis.com/auth/admin.directory.group',
        'https://www.googleapis.com/auth/drive.readonly'
    ]}
}, function (err, res, body) {
    if (err) console.log("Error", err);
    console.log("BODY", JSON.parse(body));
});

I have created a project in the Developer Console. I have created a new clientId (Service Account). I am then presented with a p12 file, which I use openSSL to convert to a pem file (file path for this given in keyFile setting above). The clientId email address created is used in the email setting above.

I have granted the project access to the Admin SDK. I have then gone into Admin Console and in Security -> Advanced -> Manage API client access, I have granted the Service Account access to all the scopes requested in the above code.

Hope, this makes sense, it is difficult to describe the full process. Please comment if you have any questions or need clarity on anything.

When running this code I always get a 403, "Not Authorized to access this resource/api".

Am I using the correct methodology? It is difficult to follow the Google Documentation as not all of help files match the current menu system.