Cannot get mTLS working on a Google Cloud Load Balancer using PrivateCA

Question

I am trying to get mTLS working on a global HTTPS load balancer in Google Cloud, but the client connections are just being rejected with the error "client_cert_validation_failed".

I followed the exact instructions here and here but it appears the trust config supplied just isn't working.

The trust config looks like this:

trustStores:
- trustAnchors:
  - pemCertificate: |
      -----BEGIN CERTIFICATE-----
      MIICWzCCAgKgAwIBAgIUANxN5VlXyQgGZp4uZJi7iXnIivEwCgYIKoZIzj0EAwIw
      NjERMA8GA1UEChMIVmlnaWxhbnQxITAfBgNVBAMTGGxvbmRvbi5hcGlzLnZpcHJv
      Lm9ubGluZTAeFw0yMzA4MjExMDA1NTZaFw0yMzExMTkxMDA1NTVaMDYxETAPBgNV
      BAoTCFZpZ2lsYW50MSEwHwYDVQQDExhsb25kb24uYXBpcy52aXByby5vbmxpbmUw
      WTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAT/H53qlrvXeBXtrPi8qnigM/YXKKKg
      tfssUk1Vxz+LoVor3DT4duqwyTm94naykqlmV25IzU5BAknff07TGVv5o4HtMIHq
      MA8GA1UdDwEB/wQFAwMH3oAwOwYDVR0lBDQwMgYIKwYBBQUHAwEGCCsGAQUFBwMC
      BggrBgEFBQcDAwYIKwYBBQUHAwQGCCsGAQUFBwMIMBIGA1UdEwEB/wQIMAYBAf8C
      AQUwHQYDVR0OBBYEFHo4y3dPvnRlI622Ytm5x/8WKZ1yMB8GA1UdIwQYMBaAFHo4
      y3dPvnRlI622Ytm5x/8WKZ1yMEYGA1UdEQQ/MD2CH2FnZW50cy5sb25kb24uYXBp
      cy52aXByby5vbmxpbmWCGm1jZC5hZ2VudHMuY2Eudmlwcm8ub25saW5lMAoGCCqG
      SM49BAMCA0cAMEQCIBmn2170sahTzA0iBYuULNeywX+r8fX8JucglsMQNBT8AiAP
      vJwqN9I7bmTgItfpjozMIExZiSeiTn45TfEdNLNsTQ==
      -----END CERTIFICATE-----

NOTE: PEM is definitely correct.

I have a TLS policy which uses this:

gcloud beta network-security server-tls-policies describe london-ips-tls-policy --location=global 

createTime: '2023-08-22T15:59:51.030272533Z'
mtlsPolicy:
  clientValidationMode: REJECT_INVALID
  clientValidationTrustConfig: projects/MY_PROJECT_NUM/locations/global/trustConfigs/london-ips-trust-config
name: projects/MY_PROJECT_ID/locations/global/serverTlsPolicies/london-ips-tls-policy
updateTime: '2023-08-22T16:28:04.303764856Z'

I then have a target-https-proxies which references this:

# ...
serverTlsPolicy: //networksecurity.googleapis.com/projects/MY_PROJECT_ID/locations/global/serverTlsPolicies/london-ips-tls-policy

However whenever I download one of the approved certificates like this one:

-----BEGIN CERTIFICATE-----
MIIDWzCCAwKgAwIBAgIUAKpLh7uR2I/q21WKu84Sg5VZadowCgYIKoZIzj0EAwIw
NjERMA8GA1UEChMIVmlnaWxhbnQxITAfBgNVBAMTGGxvbmRvbi5hcGlzLnZpcHJv
Lm9ubGluZTAeFw0yMzA4MjIxNjIwMTdaFw0yMzA5MjExNjIwMTZaMEAxPjA8BgNV
BAMTNXRvbXMtbWFjYm9vay1wcm8udGVzdC5hZ2VudHMubG9uZG9uLmFwaXMudmlw
cm8ub25saW5lMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAErfdb+fJlEUPn9tMD
zEaHkGik8mQ5Iys1IkhVBXbgUc3gPSj/BVGIaxXnu/nFMNQZ+QRInYC4MQ468+//
mXEAmqOCAeIwggHeMA4GA1UdDwEB/wQEAwIBsjAnBgNVHSUEIDAeBggrBgEFBQcD
AQYIKwYBBQUHAwIGCCsGAQUFBwMIMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFExS
GV/5N5qpQm83+J2P2QqtE+q7MB8GA1UdIwQYMBaAFHo4y3dPvnRlI622Ytm5x/8W
KZ1yMIGNBggrBgEFBQcBAQSBgDB+MHwGCCsGAQUFBzAChnBodHRwOi8vcHJpdmF0
ZWNhLWNvbnRlbnQtNjRkZjE5YTQtMDAwMC0yNDUyLWFlMDEtZDRmNTQ3ZjlmMjZj
LnN0b3JhZ2UuZ29vZ2xlYXBpcy5jb20vMWEwMzc3ODIwYzFmMjY2Y2Q1ZTMvY2Eu
Y3J0MEAGA1UdEQQ5MDeCNXRvbXMtbWFjYm9vay1wcm8udGVzdC5hZ2VudHMubG9u
ZG9uLmFwaXMudmlwcm8ub25saW5lMIGCBgNVHR8EezB5MHegdaBzhnFodHRwOi8v
cHJpdmF0ZWNhLWNvbnRlbnQtNjRkZjE5YTQtMDAwMC0yNDUyLWFlMDEtZDRmNTQ3
ZjlmMjZjLnN0b3JhZ2UuZ29vZ2xlYXBpcy5jb20vMWEwMzc3ODIwYzFmMjY2Y2Q1
ZTMvY3JsLmNybDAKBggqhkjOPQQDAgNHADBEAiBlVdt3+ccqFXgXRIQMoLvgbMpy
kC2s86eKXEhjU4t22QIgOHZaPGUzO3Zr9m1esHYn8T+cnSEZPaYaXigSxlil6cg=
-----END CERTIFICATE-----

Using the go code:

// load key/cert pair
    clientCertForTls, err := tls.LoadX509KeyPair(os.ExpandEnv(*clientCertPath), os.ExpandEnv(*privateKeyPath))
    if err != nil {
        log.Fatalf("unable to load x509 pair, %v", err)
    }

    // http client with client certs
    mTLSClient := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                ClientAuth:   tls.RequestClientCert,
                Certificates: []tls.Certificate{clientCertForTls},
            },
        },
    }

    // make request
    resp, err := mTLSClient.Get("https://OUR_URL.com/")
    if err != nil {
        log.Fatalf("unable to mtls ips, %v", err)
    }
    bodyDat, _ := io.ReadAll(resp.Body)
    log.Printf("body: %s", bodyDat)
    for k := range resp.Header {
        log.Printf("header[%s]: %s", k, resp.Header.Get(k))
    }
    if resp.StatusCode != http.StatusOK {
        log.Fatalf("failed to mtls ips, %s: %s", resp.Status)
    } else {
        log.Printf("ALL GOOD! :)")
    }

However the connection is always rejected by the server.

Using the logging query:

jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"

I can see the error "client_cert_validation_failed" in the rejection entry:

{
httpRequest: {
latency: "0s"
remoteIp: "REDACTED"
}
insertId: "REDACTED"
jsonPayload: {
@type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
backendTargetProjectNumber: "projects/MY_PROJECT_NUM"
remoteIp: "REDACTED"
statusDetails: "client_cert_validation_failed"
}
logName: "projects/MY_PROJECT_ID/logs/requests"
receiveTimestamp: "2023-08-22T17:00:55.441451401Z"
resource: {
labels: {
backend_service_name: ""
forwarding_rule_name: "lb-frontend-acme-protected"
project_id: "MY_PROJECT_ID"
target_proxy_name: ""
url_map_name: ""
zone: "global"
}
type: "http_load_balancer"
}
severity: "INFO"
timestamp: "2023-08-22T17:00:54.413317Z"
}

Why does this not work please?

If I change the clientValidationMode to ALLOW_INVALID_OR_MISSING_CLIENT_CERT it allows traffic through, so for some reason it's rejecting perfectly good client certificates which were issued by the Root CA (which is enabled).

UPDATE: I've also tried with cURL, but the connection is still reset, the same as the Go code: curl --key ~/.vipro-key.pem --cert ~/Downloads/client-cert.crt -v https://OUR_PROTECTED_ENDPOINT/

Answer 1

I believe the concern here is the leaf certificate. The leaf certificate cannot contain the timeStamping in the x509v3 Extended Key Usage which falls under the Certificate Requirement

You can check this by running this command “openssl x509 -in cert.pem -text”