No Serilog logs are made from my asp.net core webapi to my dockerized elasticsearch

Question

No Serilog logs are made from my ASP.NET Core Web API to my dockerized elasticsearch. Why is that?

I have an ASP.NET Core Web API where I've installed

<PackageReference Include="Serilog.AspNetCore" Version="8.0.0" />
<PackageReference Include="Serilog.Sinks.Elasticsearch" Version="9.0.3" />

I've setup elasticsearch, logstash (not needed?) and kibana with docker-compose.

I get no data in Kibana and I think it's due to nothing landing in elasticsearch.

Listing indices in elasticsearch, I do not find the expected index, custom-index-{0:yyyy.MM}

Looking for indices:

docker exec -it elasticsearch bash

then

curl -XGET 'http://localhost:9200/_cat/indices'

I get

green open .geoip_databases                OPKgHw3hSNCmc4r0-ioqbw 1 0  41   0    39mb    39mb
green open .apm-custom-link                mrXi9BXyRC60MSlY94IorQ 1 0   0   0    226b    226b
green open .kibana_7.16.1_001              Dth8t0rHTt2xCSgqL5JLQg 1 0 296  13   2.3mb   2.3mb
green open .apm-agent-configuration        NgOE1PhPSlS3gDcHL8I1fw 1 0   0   0    226b    226b
green open .kibana_task_manager_7.16.1_001 YWW69XjBTQSr0mknnyKqoQ 1 0  17 795 177.8kb 177.8kb

---

docker-compose.yaml:

version: '3.4'

services:
  elasticsearch:
    image: elasticsearch:7.16.1
    container_name: elasticsearch
    environment:
      discovery.type: single-node
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
    ports:
      - "9200:9200"
      - "9300:9300"
    healthcheck:
      test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"]
      interval: 10s
      timeout: 10s
      retries: 3
    networks:
      - elastic

  logstash:
    image: logstash:7.16.1
    container_name: logstash
    environment:
      discovery.seed_hosts: logstash
      LS_JAVA_OPTS: "-Xms512m -Xmx512m"
    volumes:
      - ./logstash/pipeline/logstash-nginx.config:/usr/share/logstash/pipeline/logstash-nginx.config
      - ./logstash/nginx.log:/home/nginx.log
    ports:
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "5044:5044"
      - "9600:9600"
    depends_on:
      - elasticsearch
    networks:
      - elastic
    command: logstash -f /usr/share/logstash/pipeline/logstash-nginx.config

  kibana:
    image: kibana:7.16.1
    container_name: kibana
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - elastic
networks:
  elastic:
    driver: bridge

---

program.cs:

...
builder.Host.UseSerilog(
    (context, configuration) =>
        configuration.ReadFrom.Configuration(context.Configuration)
);

var app = builder.Build();
...

---

appsettings.json:

  "Serilog": {
    "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.ElasticSearch" ],
    "MinimumLevel": {
      "Default": "Verbose",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning"
      }
    },
    "WriteTo": [
      { "Name": "Console" },
      {
        "Name": "ElasticSearch",
        "Args": {
          "nodeUris": "http://elasticsearch:9200",
          "indexFormat": "custom-index-{0:yyyy.MM}",
          "autoRegisterTemplate": true
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId" ],
    "Properties": {
      "Application": "Drivers.WebApi"
    }

Answer 1

In my case I had "nodeUris": "http://localhost:9200" in appsettings.Development.json. This worked fine when running the app from Visual Studio but it didn't connect to Elasticsearch when both containers were running on Docker.

The solution was to override the environment variable like this:

myapi:
  environment:
    - Serilog__WriteTo__2__Args__nodeUris=http://elasticsearch:9200

The number indicates the position of Elasticsearch in the "WriteTo" array.

Something that was extremely useful to debug this problem was to enable Serilog SelfLog by adding this at the very beginning of the application:

Serilog.Debugging.SelfLog.Enable(msg =>
{
    try
    {
        if (!File.Exists("internallog.log"))
        {
            File.Create("internallog.log").Close();
        }

        File.AppendAllText("internallog.log", msg, Encoding.UTF8);
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.ToString());
    }
});