How do I update Dotnet Serilog Logging Config to Deal with new Elasticsearch 8 security (https, username, password, tls cert)?

Question

I've upgraded to Elastic 8.x from 7.x. Now my elastic endpoint requires connecting using https with a username, password and a tls cert.

See examples here.

If I use this approach within my kubernetes cluster just to test connectivity, I can curl the Elastic service from my application's container. First, I have to export the tls cert and copy the cert into my container. Then I can curl the service (per the link above):

curl --cacert tls.crt -u elastic: https://elasticsearch-cluster-es-http.eck:9200

{
  "name" : "elasticsearch-cluster-es-default-1",
  "cluster_name" : "elasticsearch-cluster",
  "cluster_uuid" : "YqYl-gTpRd-URcoDhW5t1w",
  "version" : {
    "number" : "8.11.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "76013fa76dcbf144c886990c6290715f5dc2ae20",
    "build_date" : "2023-12-05T10:03:47.729926671Z",
    "build_snapshot" : false,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

How can I now update my dotnet logger configuration to handle the new https, username:password, and tls cert requirements? I've tried the following without success (also tried the fingerprint):

var elasticOptions = new ElasticsearchSinkOptions(new Uri($"https://{elasticServer}"))
            {
                AutoRegisterTemplate = true,
                IndexDecider = (@event, offset) =>
                    string.Format("{0}-{1}-{2:yyyy.MM.dd}", k8sNamespace, appName, offset),
                ModifyConnectionSettings = (settings) =>
                {
                    settings.EnableApiVersioningHeader();
                    settings.ClientCertificate(new X509Certificate2(crtBytes));
                    settings.BasicAuthentication("elastic", "<password>");
                    settings.DeadTimeout(TimeSpan.FromSeconds(300));
                    return settings;
                }
            };

I see the following errors in my app:

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot

dotnet 8 serilog.sinks.elasticsearch: 9.0.3 elasticsearch eck: 8.11.2

Answer 1

I solved this issue by grabbing the Elastic CA secret (name-es-http-ca-internal) out of Kubernetes and adding it to my ca-certificates.crt file in my applications docker file:

# add ca cert
COPY docker/ca-certs/elastic.crt /app/elastic.crt
RUN cat /app/elastic.crt >> /etc/ssl/certs/ca-certificates.crt

crtBytes above was obtained from the public crt secret value (name-es-http-certs-public)

byte[] crtBytes = Encoding.ASCII.GetBytes("-----BEGIN CERTIFICATE-----\nMIIEqDCCA5CgA.....");

However, I think I will take the approach of disabling tls all together so I don't have to manage these certs.

https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-tls-certificates.html#k8s-disable-tls