Elasticsearch - cascading http inputs from Airflow API

Question

I would like to call an API (Airflow - for monitoring) from elasticsearch like in this example: https://www.elastic.co/guide/en/elasticsearch/reference/current/input-http.html#input-http-auth-basic-example I need to:

1. fetch a list of items from API (/dags) having the specific label and then for each DAG from this list
2. fetch the status of this DAG having dag_id (dags/{dag_id}/dagRuns) so that I know whether the execution succeeded or failed

Is it possible to do this cascading call in Elasticsearch http input? Or maybe there is a possibility (I don't see it) of fetching information about failing DAG in another way from Airflow?

Answer 1

You could achieve such cascading call via the Watcher chain input.

An Example code i took from the wiki:

    "input" : {
    "chain" : {
    "inputs" : [ 
      {
        "first" : {
          "simple" : { "path" : "/_search" }
        }
      },
      {
        "second" : {
          "http" : {
            "request" : {
              "host" : "localhost",
              "port" : 9200,
              "path" : "{{ctx.payload.first.path}}" 
            }
          }
        }
      }
    ]
  }
}

here ctx is your execution context. You can access previous http inputs like this: ctx.payload.<input-name>.<value>

Link to the current wiki: https://www.elastic.co/guide/en/elasticsearch/reference/current/input-chain.html