I want to upload the specific deployment logs to s3 bucket with fluent-bit.
i used below mentioned configuration files: fluent-bit-daemonset.yaml:
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging
labels:
k8s-app: fluent-bit-logging
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
template:
metadata:
labels:
k8s-app: fluent-bit-logging
spec:
serviceAccountName: s3-access
containers:
- name: fluent-bit
image: amazon/aws-for-fluent-bit:latest
env:
- name: FLUENT_BIT_CONFIG_FILE
value: /fluent-bit/etc/fluent-bit.conf
volumeMounts:
- name: config
mountPath: /fluent-bit/etc/
readOnly: true
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 10
volumes:
- name: config
configMap:
name: fluent-bit-config
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containe
fluent-bit-configmap.yaml:
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Daemon Off
Log_Level info
[INPUT]
Name tail
Path /var/log/containers/*.log
#Path /var/log/containers/qa-*_qa_*.log # Focus only on 'qa' namespace and pods starting with 'qa'
Parser docker
Tag kube.*
Refresh_Interval 5
#Exclude_Path /var/log/containers/kube-system_*,/var/log/containers/kube-public_*,/var/log/containers/default_*
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
[OUTPUT]
Name s3
Match *
bucket logging-amazon-s3
region us-east-1
s3_key_format /$TAG/%Y/%m/%d/%H/%M/%S.log
use_put_object On
with the above configuration fluent-bit working fine and all logs of EKS cluster are uploaded to s3.
Question: **Upload the specific deployment logs of qa namespace **
I used the below mentioned fluent-bit-configmap.yaml but unable to collect/upload the logs to s3.
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Daemon Off
Log_Level info
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Refresh_Interval 5
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
[FILTER]
Name grep
Match kube.*
# Filter for the 'qa' namespace
Regex kubernetes['namespace_name'] ^qa$
# Filter for pods starting with 'qa'
Regex kubernetes['pod_name'] ^qa.*
[OUTPUT]
Name s3
Match *
bucket logging-amazon-s3
region us-east-1
s3_key_format /$TAG/%Y/%m/%d/%H/%M/%S.log
use_put_object On
when I used 2nd filer it's not working with the above configuration. please share the solution. I will be thankful.