How to integrate opensearch with logstash on the Kubernetes Cluster?

Question

I have a kubernetes cluster on which I have deployed a opensearch cluster and opensearch dashboard using Helm, I am also able to deploy logstash using helm successfully but I am confused on how to integrate those, I want to feed data to my Opensearch using logstash as my OBJECTIVE as I am not able to find much documentation on it as well. Any help is appreciated....Thanks in advance!

Deployed opensearch using Helm and logstash as well but unable to integrate them

Update here!!!

Have made a few changes to simplify the deployment and more control over the function,

I am testing deployment and service files this time, I will add the files below

Opensearch deployment file

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: logging
  name: opensearch
  labels:
    component: opensearch
spec:
  selector:
    matchLabels:
      component: opensearch
  replicas: 1
  serviceName: opensearch
  template:
    metadata:
      labels:
        component: opensearch
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox
          imagePullPolicy: IfNotPresent
          command:
            - sysctl
            - -w
            - vm.max_map_count=262144
          securityContext:
            privileged: true
      containers:
        - name: opensearch
          securityContext:
            capabilities:
              add:
                - IPC_LOCK
          image: opensearchproject/opensearch
          env:
            - name: KUBERNETES_CA_CERTIFICATE_FILE
              value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: "cluster.name"
              value: "opensearch-cluster"
            - name: "network.host"
              value: "0.0.0.0"
            - name: "discovery.seed_hosts"
              value: "[]"
            - name: discovery.type
              value: single-node              
            - name: OPENSEARCH_JAVA_OPTS
              value: -Xmx512M -Xms512M
            - name: "plugins.security.disabled"
              value: "false" 
          ports:
            - containerPort: 9200
              name: http
              protocol: TCP
            - containerPort: 9300
              name: transport
              protocol: TCP
          volumeMounts:
            - name: os-mount
              mountPath: /data
      volumes:
        - name: os-mount
          persistentVolumeClaim:
            claimName: nfs-pvc-os-logging

Opensearch svc file

---
apiVersion: v1
kind: Service
metadata:
  name: opensearch
  namespace: logging
  labels:
    service: opensearch
spec:
  type: ClusterIP
  selector:
    component: opensearch
  ports:
  - port: 9200
    targetPort: 9200

Opensearch dashboard deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-dash
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: open-dash
  template:
    metadata:
      labels:
        app: open-dash
    spec:
#      securityContext:
#      runAsUser: 0
            
      containers:
        - name: opensearch-dashboard
          image: opensearchproject/opensearch-dashboards:latest
          ports:
            - containerPort: 80
          env:
#            - name: ELASTICSEARCH_URL
#              value: https://opensearch.logging:9200
#            - name: "SERVER_HOST"
#              value: "localhost"
#            - name: "opensearch.hosts" 
#              value: https://opensearch.logging:9200
            - name: OPENSEARCH_HOSTS
              value:  '["https://opensearch.logging:9200"]'  

Opensearch Dashboard svc

---
apiVersion: v1
kind: Service
metadata:
  name: opensearch
  namespace: logging
  labels:
    service: opensearch
spec:
  type: ClusterIP
  selector:
    component: opensearch
  ports:
  - port: 9200
    targetPort: 9200

with the above configuration I am able to get the Dashboard UI open but in Dashboard pod logs I can see a 400 code logs can anyone please try to reproduce this issue, Also I need to integrate the logstash with this stack.

{"type":"response","@timestamp":"2023-02-20T05:05:34Z","tags":[],"pid":1,"method":"head","statusCode":400,"req":{"url":"/app/home","method":"head","headers":{"connection":"Keep-Alive","content-type":"application/json","host":"3.108.199.0:30406","user-agent":"Manticore 0.9.1","accept-encoding":"gzip,deflate","securitytenant":"user"},"remoteAddress":"10.244.1.1","userAgent":"Manticore 0.9.1"},"res":{"statusCode":400,"responseTime":2,"contentLength":9},"message":"HEAD /app/home 400 2ms - 9.0B

When deploying a logstash pod I get an error that

[WARN ] 2023-02-20 05:13:52.212 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://logstash:[email protected]:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://logstash:[email protected]:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond"}

Can somebody please try to help me with this puzzle

@Benla have made changes as per your recommendation to image now I am getting the following logs in logstash

[2023-02-20T05:18:43,028][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [2023-02-20T05:18:43,147][INFO ][org.reflections.Reflections] Reflections took 70 ms to scan 1 urls, producing 127 keys and 444 values [2023-02-20T05:18:43,351][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise. [2023-02-20T05:18:43,370][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x3bf49916@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:131 run>"} [2023-02-20T05:18:43,811][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.44} [2023-02-20T05:18:43,816][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"} [2023-02-20T05:18:43,821][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} [2023-02-20T05:18:43,835][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2023-02-20T05:18:43,869][INFO ][org.logstash.beats.Server][main][0710cad67e8f47667bc7612580d5b91f691dd8262a4187d9eca8cf87229d04aa] Starting server on port: 5044

I started getting these endless loop of logs

[WARN ] 2023-02-20 05:13:37.191 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://logstash:[email protected]:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://logstash:[email protected]:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond"}

Answer 1

I have not tested this with SSO please let us know suggestions on this

Opensearch, Opensearch Dashboard(Official Helm charts), Filebeat(Official Helm Charts) & Logstash Deployment(Official Helm Charts)

• Opensearch & Opensearch-dashboard https://github.com/opensearch-project/helm-charts
• Filebeat https://github.com/elastic/helm-charts/tree/main/filebeat
• Logstash [https://github.com/elastic/helm-charts/tree/main/logstash]

My ELK stack is divided into 3 Parts Each part deploys a separate components using Helm charts

• Opensearch(A)
• Dashboard(B)
• Filebeat
• Logstash

Recomended to create a seperate namespace for logging stack on k8s cluster

Opensearch Deployment(A) Navigate to "/opensearchhelm/opensearch/helm-charts/charts/opensearch" directory Inside opensearch directory you will find the values.yaml that contains the configuration for helm charts

Create persistent Volume for Opensearch you can then run the command to deploy the helm chart in the namespace of your choice

helm install opensearch opensearch/ -n namespace

Verify the deployment using below commands

kubectl get pods -n development

Exec into a existing pod that has curl installed to check connectivity between the cluster

kubectl exec -it -n namespace podname /bin/sh

Once inside pod curl the serviceName.namespace as below and that should return a json output in return

curl -X GET https://opensearch.logging:9200 -u 'admin:admin' --insecure

{
  "name" : "opensearch-cluster-master-0",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "nEFZv3l0RRyx3A0nGtQgug",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.4.0",
    "build_type" : "tar",
    "build_hash" : "744ca260b892d119be8164f48d92b8810bd7801c",
    "build_date" : "2022-11-15T04:42:29.671309257Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Once you get a similar output we can confirm that your opensearch is installed successfully

Opensearch Dashboard Deployment(B)

Navigate to "/opensearchhelm/opensearch/helm-charts/charts/opensearch-dashboard" in elk directory.

Inside opensearch directory you will find the values.yaml that contains the configuration for helm charts.

Replace the service name with your opensearch service name as below example to the "opensearchHosts" attribute in values.yaml

opensearchHosts: "https://opensearch-cluster-master:9200"

From opensearchhelm/opensearch/helm-charts/charts/ you can then run the command to deploy the helm chart in the namespace of your choice

helm install opensearchdash opensearch-dashboard/ -n namespace

This will deploy the Opensearch Dashboard that can be used to create dashboard from the Opensearch data

Filebeat Deployment

To get the data from opensearch to logstash we use filebeat, it scrapes the data from Opensearch and gives it to Logstash Filebeat does not use the Persistent Volume claims so we can go ahead with the deployment. Filebeat will automatically pick up the logs from the existing Opensearch in the namespace

*Filebeat configuration and other files for helm chart can be found in filebeat Navigate to directory containing filebeat

helm install filebeat filebeat -n namespace

This will deploy filebeat pods inside your namespace along with other resources.

The number of filebeat pods should be same as the number of your kubernetes Nodes(excluding namespace)

You can validate the deployment by checking logs of the pods

kubectl logs -f filebeat-filebeat-4tkk6 -n namespace

Querying logs should get you an output something like below

{"log.level":"info","@timestamp":"2023-03-22T07:55:12.614Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":104,"throttled":{"ns":506844664,"periods":5}}},"cpuacct":{"total":{"ns":1293495129}},"memory":{"mem":{"usage":{"bytes":150925312}}}},"cpu":{"system":{"ticks":3000,"time":{"ms":60}},"total":{"ticks":30650,"time":{"ms":630},"value":30650},"user":{"ticks":27650,"time":{"ms":570}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":18},"info":{"ephemeral_id":"c4c9fe7b-a469-4a98-a59a-9031a681c0a7","uptime":{"ms":1350147},"version":"8.5.1"},"memstats":{"gc_next":41205704,"memory_alloc":24577840,"memory_total":3591430176,"rss":123326464},"runtime":{"goroutines":90}},"filebeat":{"events":{"active":293,"added":415,"done":122},"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":420,"active":0,"batches":22,"total":420},"read":{"bytes":132},"write":{"bytes":88426}},"pipeline":{"clients":1,"events":{"active":0,"published":415,"total":415},"queue":{"acked":420}}},"registrar":{"states":{"current":454,"update":420},"writes":{"success":18,"total":18}},"system":{"load":{"1":0.06,"15":0.46,"5":0.22,"norm":{"1":0.0038,"15":0.0288,"5":0.0138}}}},"ecs.version":"1.6.0"}}

Logstash deployment

Logstash configuration and other files for helm chart can be found in logstash/

Navigate to logstash

Edit the values.yaml and update the configuration for your opensearch configuration and Index as shown below in example

vim values.yaml

Add the example configuration below

# Allows you to add any pipeline files in /usr/share/logstash/pipeline/
### ***warn*** there is a hardcoded logstash.conf in the image, override it first
logstashPipeline:
 logstash.conf: |
   input {
     beats {
       port => 5044
     }
   }
   output { 
   opensearch{ 
        hosts => "https://opensearch-cluster-master.logging"
        user => "admin"
        password => "admin"
        index => "myindex"
        ssl_certificate_verification => false } }

This will create an index that we will be able to query in the Opensearch Dashboard UI

Navigate to logstah directory and install the helm chart Run the helm command to deploy the Logstash tool

helm install logstash logstash/ -n namespace

To validate the working of the stack login to your Opensearch Dashboard

• Navigate to dev tools from side menu

• Run the below query if you are able to find the ondex you created

  GET myindex/_search?pretty

You should get a JSON output of the index if everything is working properly

Answer 2

I think there is not much difference in deploying logstash<->opensearch pipeline from deploying logstash<->elasticsearch in K8S, The only differences I've encountered are:

1. The docker or other container runtime image should be opensearchproject/logstash-oss-with-opensearch-output-plugin:latest rather than logstash.
2. The output plugin in your logstash config file sould be opensearch rather than elasticsearch.

Other than those you can go to every Logstash<->ElasticSearch in K8S tutorial, there are many.

Good Luck!

Answer 3

for opensearch sso(keycloak) please use below steps: opensearch:

1. Make a custom iamge for opensearch, for this make 2 files as below. i.config.yml(for opensearch security plugin) --- _meta: type: "config" config_version: 2

    config:
      dynamic:
        http:
          anonymous_auth_enabled: false
        authc:
          internal_auth:
            order: 0
            description: "HTTP basic authentication using the internal user database"
            http_enabled: true
            transport_enabled: true
            http_authenticator:
              type: basic
              challenge: false
            authentication_backend:
              type: internal
          openid_auth_domain:
            http_enabled: true
            transport_enabled: true
            order: 1
            http_authenticator:
              type: openid
              challenge: false
              config:
                subject_key: preferred_username
                roles_key: roles
                openid_connect_url: "https://keycloak-url/realms/realm-name/.well-known/openid-configuration"    
            authentication_backend:
              type: noop 
    ---
   

ii. log4j2.properties(this file will start logs in opensearch so we can see logs which are otherwise turned-off)

    ---
    logger.securityjwt.name = com.amazon.dlic.auth.http.jwt
    logger.securityjwt.level = trace
    ---

iii. Dockerfile

---
FROM opensearchproject/opensearch:2.5.0
RUN mkdir /usr/share/opensearch/plugins/opensearch-security/securityconfig
COPY config.yaml /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
COPY config.yaml /usr/share/opensearch/config/opensearch-security/config.yml
COPY log4j2.properties /usr/share/opensearch/config/log4j2.properties
---

2. Deploy opensearch with opensearch helm chart(change image with your customimage built using above configs). opensearch will deploy 3 pods.now go in each pod and fire belo command to start security plugin(do this only once for each pod of opensearch).

   ---

   /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh
   -cacert /usr/share/opensearch/config/root-ca.pem
   -cert /usr/share/opensearch/config/kirk.pem
   -key /usr/share/opensearch/config/kirk-key.pem
   -cd /usr/share/opensearch/config/opensearch-security
   -h localhost

   ---

   make sure all 3 pods are up and in ready state. opensearch-dashboard

3.Now we will configure opensearch-dashboard i. In values.yml of helm chart of opensearch-dashboard search for config

---    
config: 
    opensearch_dashboards.yml: |
        opensearch.hosts: [https://localhost:9200]
        opensearch.ssl.verificationMode: none
        opensearch.username: admin
        opensearch.password: admin
        opensearch.requestHeadersWhitelist: [authorization, securitytenant]
        opensearch_security.multitenancy.enabled: true
        opensearch_security.multitenancy.tenants.preferred: [Private, Global]
        opensearch_security.readonly_mode.roles: [kibana_read_only]
        opensearch_security.cookie.secure: false
        server.host: '0.0.0.0'

        opensearch_security.auth.type: "openid"
        opensearch_security.openid.connect_url: "https://keycloak-url/realms/realm-name/.well-known/openid-configuration"
        opensearch_security.openid.client_id: "admin"
        opensearch_security.openid.client_secret: "asgduasdjsadk"
        opensearch_security.openid.scope: "email openid"
        opensearch_security.openid.base_redirect_url: "https://opensearch_dashboards-url.io"
---

ii. deploy opensearch_dashboards.

Now once opensearch_dashboards is deployed and pod is in up and ready state you can go to https://opensearch_dashboards-url.io (your opensearch_dashboards url ) and you will see keycloak login form.