Elasticsearch - Authentication error when check to see if index exist

Question

I have my elastic search with cognito authentication enabled for kibana. This is working fine as expected.

In my python script I connect to elasticsearch by providing username/password in http_auth(), while creating the connection object. But when I attempt to check if an indices exist, am getting authentication error? Can someone please help. Heres the sample piece of code for your simulation please.

from __future__ import print_function
import json
import time
import urllib
import re
import sys
import requests
import base64
import time
from elasticsearch import Elasticsearch
from datetime import datetime

esEndpoint = ""
uname
pwd
indexName = 'index_1'

mappings_rds = {
    "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
    },
    "mappings": {
      "properties" : {
         "tableName": {
            "type": "keyword"
          },
         "tableRows": {
            "type": "integer"
          },        
         "updatedTime": {
            "type": "date",
            "format":"date_optional_time||yyyy-MM-dd'T'HH:mm:ss"
          },
        "created_timestamp":{
            "type": "date",
            "format":"date_optional_time||yyyy-MM-dd'T'HH:mm:ss"
          }
      }
    }
}

esClient = Elasticsearch([esEndPoint],http_auth=(uname,pwd))

try:
      res = esClient.indices.exists(indexName)
      print(res)
      if res is False:
          r = esClient.indices.create(indexName, body=mapping_rds, ignore=400)
      return 1
  except Exception as E:
          print("Unable to Create Index {0}".format(indexName))

Answer 1

Thank you all for your comments. I had a call with AWS Support team regarding this. The issue was found to be the username & password that I had used. I was using the user credentials created using AWS Cognito. The AWS support team confirmed that the said credentials is only meant for Kibana access, and it cannot be used for connecting to Elasticsearch from databricks / python script.

More info available here: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-request-signing.html

Once I generate the access/secret keys, and use it from my python script, am able to connect to elasticsearch to create index.

Sharing a sample code for future reference:

session=boto3.session.Session(aws_access_key_id=akey, aws_secret_access_key=skey, region_name=region)
credentials = session.get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
print(credentials.access_key, credentials.secret_key, credentials.token)
es = Elasticsearch(
    hosts = [{'host': esEndpoint, 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)

Thanks