how to get the specfic deployment logs with fluent-bit in EKS

109 views Asked by At

I want to upload the specific deployment logs to s3 bucket with fluent-bit.

i used below mentioned configuration files: fluent-bit-daemonset.yaml:

kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
    spec:
      serviceAccountName: s3-access
      containers:
      - name: fluent-bit
        image: amazon/aws-for-fluent-bit:latest
        env:
        - name: FLUENT_BIT_CONFIG_FILE
          value: /fluent-bit/etc/fluent-bit.conf
        volumeMounts:
        - name: config
          mountPath: /fluent-bit/etc/
          readOnly: true
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 10
      volumes:
      - name: config
        configMap:
          name: fluent-bit-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containe

fluent-bit-configmap.yaml:

kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Daemon       Off
        Log_Level    info

    [INPUT]
        Name             tail
        Path             /var/log/containers/*.log
        #Path             /var/log/containers/qa-*_qa_*.log  # Focus only on 'qa' namespace and pods starting with 'qa'
        Parser           docker
        Tag              kube.*
        Refresh_Interval 5
        #Exclude_Path     /var/log/containers/kube-system_*,/var/log/containers/kube-public_*,/var/log/containers/default_*

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Merge_Log           On


    [OUTPUT]
        Name        s3
        Match       *
        bucket      logging-amazon-s3
        region      us-east-1
        s3_key_format /$TAG/%Y/%m/%d/%H/%M/%S.log
        use_put_object   On

with the above configuration fluent-bit working fine and all logs of EKS cluster are uploaded to s3.
Question: **Upload the specific deployment logs of qa namespace ** I used the below mentioned fluent-bit-configmap.yaml but unable to collect/upload the logs to s3.

kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Daemon       Off
        Log_Level    info

    [INPUT]
        Name             tail
        Path             /var/log/containers/*.log  
        Parser           docker
        Tag              kube.*
        Refresh_Interval 5

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Merge_Log           On

    [FILTER]
        Name                grep
        Match               kube.*
        # Filter for the 'qa' namespace
        Regex               kubernetes['namespace_name'] ^qa$
        # Filter for pods starting with 'qa'
        Regex               kubernetes['pod_name'] ^qa.*

    [OUTPUT]
        Name        s3
        Match       *
        bucket      logging-amazon-s3
        region      us-east-1
        s3_key_format /$TAG/%Y/%m/%d/%H/%M/%S.log
        use_put_object   On

when I used 2nd filer it's not working with the above configuration. please share the solution. I will be thankful.

0

There are 0 answers