Metricbeat exits after a short period of time > "error fetching EC2 Identity Document: operation error ec2imd..."

Question

I'm trying to set up Metricbeat on Windows (self-managed). I'm following these steps (hopefully I'm not missing anything crucial): https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-installation-configuration.html

When I run Metricbeat:

PS C:\Program Files\Metricbeat> .\metricbeat.exe setup -e

It exits after a short period of time.

These are the most suspicious log lines:

{"log.level":"info","@timestamp":"2023-11-04T22:17:51.848+0200","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-11-04T22:17:52.876+0200","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":91},"message":"error fetching EC2 Identity Document: operation error ec2imds: GetInstanceIdentityDocument, exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/dynamic/instance-identity/document": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network..","service.name":"metricbeat","ecs.version":"1.6.0"}

The full log:

{"log.level":"info","@timestamp":"2023-11-04T22:17:48.728+0200","log.origin":{"file.name":"instance/beat.go","file.line":783},"message":"Home path: [C:\Program Files\Metricbeat] Config path: [C:\Program Files\Metricbeat] Data path: [C:\Program Files\Metricbeat\data] Logs path: [C:\Program Files\Metricbeat\logs]","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:48.729+0200","log.origin":{"file.name":"instance/beat.go","file.line":791},"message":"Beat ID: 098e1f15-5744-4b6d-bcbf-aec3ffd9dfa1","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:48.838+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1303},"message":"Beat info","service.name":"metricbeat","system_info":{"beat":{"path":{"config":"C:\Program Files\Metricbeat","data":"C:\Program Files\Metricbeat\data","home":"C:\Program Files\Metricbeat","logs":"C:\Program Files\Metricbeat\logs"},"type":"metricbeat","uuid":"098e1f15-5744-4b6d-bcbf-aec3ffd9dfa1"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-11-04T22:17:48.838+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1312},"message":"Build info","service.name":"metricbeat","system_info":{"build":{"commit":"10b198c985eb95c16405b979c63847881a199aba","libbeat":"8.10.4","time":"2023-10-11T19:25:38.000Z","version":"8.10.4"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-11-04T22:17:48.838+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1315},"message":"Go runtime info","service.name":"metricbeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":8,"version":"go1.20.8"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-11-04T22:17:48.873+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1321},"message":"Host info","service.name":"metricbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-11-04T12:11:48+02:00","name":"melshawaf","ip":["fe80::1b8e:4b8b:68e:430","169.254.195.57","fe80::e4f0:35a8:a4f7:8df3","172.28.0.1","fe80::f3fc:ac2c:5b83:7ef6","169.254.119.69","fe80::d94e:b18a:1022:5607","169.254.15.123","fde2:ff59:de69:8600:146:a034:128e:4e0","fde2:ff59:de69:8600:5574:b741:a839:8e4e","fe80::e47a:d8f7:1a7d:fa1f","192.168.8.103","fe80::97b8:d9dd:1375:6e34","169.254.120.209","::1","127.0.0.1","fe80::43fc:9ffa:6b3b:654c","192.168.208.1"],"kernel_version":"10.0.19041.3448 (WinBuild.160101.0800)","mac":["cc:48:3a:67:9f:87","00:15:5d:33:f5:17","2c:db:07:55:46:ab","2e:db:07:55:46:aa","2c:db:07:55:46:aa","2c:db:07:55:46:ae","00:15:5d:26:2d:f1"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"19045.3448"},"timezone":"EET","timezone_offset_sec":7200,"id":"f9911b9f-723b-4439-9003-88dda1936bad"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-11-04T22:17:48.873+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1350},"message":"Process info","service.name":"metricbeat","system_info":{"process":{"cwd":"C:\Program Files\Metricbeat","exe":"C:\Program Files\Metricbeat\metricbeat.exe","name":"metricbeat.exe","pid":30380,"ppid":17624,"start_time":"2023-11-04T22:17:41.010+0200"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-11-04T22:17:48.873+0200","log.origin":{"file.name":"instance/beat.go","file.line":329},"message":"Setup Beat: metricbeat; Version: 8.10.4","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-11-04T22:17:49.258+0200","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.259+0200","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://localhost:9200","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.259+0200","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: melshawaf","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.285+0200","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://localhost:9200","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.300+0200","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":179},"message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.300+0200","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":199},"message":"CA certificate matching 'ca_trusted_fingerprint' found, adding it to 'certificate_authorities'","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.369+0200","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.10.4","service.name":"metricbeat","ecs.version":"1.6.0"} Overwriting ILM policy is disabled. Set setup.ilm.overwrite: true for enabling.

{"log.level":"info","@timestamp":"2023-11-04T22:17:49.369+0200","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":230},"message":"Auto ILM enable success.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.372+0200","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy metricbeat exists already.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.373+0200","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":365},"message":"Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_primary_shard_size":"50gb"}}}}}}} as ILM is enabled.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.383+0200","log.logger":"template","log.origin":{"file.name":"template/load.go","file.line":245},"message":"Existing template will be overwritten, as overwrite is enabled.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.686+0200","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":159},"message":"Try loading template metricbeat-8.10.4 to Elasticsearch","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.745+0200","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":127},"message":"Template with name "metricbeat-8.10.4" loaded.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.747+0200","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":143},"message":"Data stream with name "metricbeat-8.10.4" already exists.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:49.747+0200","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":266},"message":"Loaded index template.","service.name":"metricbeat","ecs.version":"1.6.0"} Index setup finished. Loading dashboards (Kibana must be running and reachable) {"log.level":"info","@timestamp":"2023-11-04T22:17:49.747+0200","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":183},"message":"Kibana url: http://localhost:5601","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:50.150+0200","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":183},"message":"Kibana url: http://localhost:5601","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:17:51.848+0200","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-11-04T22:17:52.876+0200","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":91},"message":"error fetching EC2 Identity Document: operation error ec2imds: GetInstanceIdentityDocument, exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/dynamic/instance-identity/document": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network..","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-11-04T22:19:45.677+0200","log.origin":{"file.name":"instance/beat.go","file.line":996},"message":"Kibana dashboards successfully loaded.","service.name":"metricbeat","ecs.version":"1.6.0"} Loaded dashboards

Also here is my metricbeat.yml (comments removed):

###################### Metricbeat Configuration Example #######################
# =========================== Modules configuration ============================

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  
# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

# =================================== Kibana ===================================

setup.kibana:

  host: "localhost:5601"

# ================================== Outputs ===================================

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "metricbeat_user"
  password: "asdfjkl;"
  ssl:
    enabled: true
    ca_trusted_fingerprint: "caa8cfead8cd39a1e0fae0c199068cf7ee6f7dbc8fb7f7a94e8c3624c0ef7987"

# ================================= Processors =================================
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Answer 1

It seems that you have a network connection problem:

"error fetching EC2 Identity Document:"

and

"A socket operation was attempted to an unreachable network..","service.name":"metricbeat","ecs.version":"1.6.0"}"

why do you use https://localhost:9200 instead of http?