Can't filebeat use multiple input/output config files in one instance?

Question

Want to deploy filebeat with 3 log definations together. Send to different output targets.

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.10.0
          args: [
            "-c", "/etc/logs1.yml",
            "-c", "/etc/logs2.yml",
            "-c", "/etc/logs3.yml",
            "-e",
          ]
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config-logs1
              mountPath: /etc/logs1.yml
              subPath: filebeat-logs1.yml
              readOnly: true
            - name: config-logs2
              mountPath: /etc/logs2.yml
              subPath: logs2.yml
              readOnly: true
            - name: config-logs3
              mountPath: /etc/logs3.yml
              subPath: logs3.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: config-logs1
          configMap:
            defaultMode: 0600
            name: configmap-logs1
        - name: config-logs2
          configMap:
            defaultMode: 0600
            name: configmap-logs2
        - name: config-logs3
          configMap:
            defaultMode: 0600
            name: configmap-logs3
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate

logs1's configmap

data:
  filebeat-logs1.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/logs1.json

    output.logstash:
      hosts: ["logstash-logs1.default.svc.cluster.local:5044"]

logs2's configmap

data:
  filebeat-logs2.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/logs2.json

    output.logstash:
      hosts: ["logstash-logs2.default.svc.cluster.local:5044"]

logs3's configmap

data:
  filebeat-logs3.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/logs3.json

    output.logstash:
      hosts: ["logstash-logs3.default.svc.cluster.local:5044"]

When each log files changed, every time it only send to the third logs3's output logstash-logs3.default.svc.cluster.local:5044. But can get data all of three logs1.json/logs2.json/logs3.json files.

Can't filebeat use multiple output in this case on one machine?

Answer 1

You can have as many inputs as you want but you can only have one output, you will need to send your logs to a single logstash and from there you can send them to other places.

Answer 2

will sending data to Logstash server and to console output simultaneously work?

output.logstash:
   hosts: ["localhost:5044"]
output.console:

Something like this..

Answer 3

Filebeat does not support sending the same data to multiple logstash servers simultaneously. To achieve this you have to start multiple instances of Filebeat with different logstash server configurations.

This is the limitation of the Filebeat output plugin

To start multiple instances of filebeat in the same host, you can refer to this link