telegraf output to Elasticsearch: "health check timeout: no Elasticsearch node available"

Question

I'm having trouble connecting to an Elasticsearch instance with a Telegraf output plugin.

I created an Elasticsearch setup via the Elasticsearch service. I created a user and password (connected to a role) in Kibana for it.

Then I setup a Telegraf output for it:

  [[outputs.elasticsearch]]
  urls = [ "https://hostname:port" ] # required.

  timeout = "5s"

  enable_sniffer = false

  health_check_interval = "10s"

  ## HTTP basic authentication details.
  username = "my_username"
  password = "my_password"

  index_name = "device_logs" # required.

  insecure_skip_verify = true

  manage_template = true

  template_name = "telegraf"

  overwrite_template = false

But when I try to start Telegraf with this, it just gives the error,

[agent] Failed to connect to [outputs.elasticsearch], retrying in 15s, error was 'health check timeout: no Elasticsearch node available'

The connect fail seems to originate deep in the bowels of golang's net/http library, and I don't know how to get some more useful output at this point.

Things I've tried:

Thing #1: I tested cURL:

curl -u my_username:my_password -X POST "https://hostname:port/device_logs/_doc" -H 'Content-Type: application/json' -d'
    {
        "name": "John Doe"
    }'

This works fine.

Thing #2: I created a simple Go program to connect to elasticsearch from Go:

package main

import (
    "log"
    "time"
    "gopkg.in/olivere/elastic.v3"
)

func main() {
    // configure connection to ES
    client, err := elastic.NewClient(elastic.SetURL("https://hostname:port"))
    if err != nil {
      panic(err)
    }
    log.Printf("client.running? %v",client.IsRunning())
    if ! client.IsRunning() {
      panic("Could not make connection, not running")
    }
}

.. and it hits the first panic with the same "no Elasticsearch node available".

Thing #3: I tried running gdb on that Go program to debug into it. It jumps down to assembly as soon as I call NewClient, so I can't really learn what is happening in the bowels of net/http.

I've never used Go before, so I'm hoping to avoid hours of learning Go, spelunking, and debugging to get around what hopefully is a simple issue here.

Any ideas on how to get more info here or why this is failing? Are there build or runtime flags for Go that I can use? gdb-with-Go debugging tips so I can step down into the Go library code? Elasticsearch client know-how?

Answer 1

To answer my own question, the problem here turned out to be the roles permissions. The Telegraf output plugin for Elasticsearch needs both the monitor and the manage_index_templates permissions to be enabled, or else it'll fail to connect to the Elasticsearch server without printing any information about why.

BTW: to build golang code and be able to debug into the libraries it calls: go build -gcflags=all="-N -l"