I'm not receiving logs when some autoscaling event happens and in that time period all the logs are lost. Basically logs related to graceful shutdown.
I'm using Elastic Beanstalk with ECS and for my logging stack I'm using Grafana's Loki. So I have this app which does the following:
import signal
import sys
import time
# Define the handler function for SIGTERM
def sigterm_handler(signum, frame):
print("Received SIGTERM signal. Shutting down gracefully.")
start_time = time.time()
while time.time() - start_time < 150:
print("Graceful shutdown in progress...")
time.sleep(15)
print("Graceful shutdown complete.")
sys.exit(0)
# Register the handler for SIGTERM
signal.signal(signal.SIGTERM, sigterm_handler)
# Keep the program running until a SIGTERM is received
print("Program is running. Press Ctrl+C to send a SIGTERM signal.")
while True:
try:
time.sleep(1)
except KeyboardInterrupt:
print("SIGTERM signal sent.")
sys.exit(0)
Here's my Dockerrrun.aws.json
file:
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"essential": true,
"image": "amazon/aws-for-fluent-bit:stable",
"stopTimeout": 60,
"name": "log_router",
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"enable-ecs-log-metadata": "true",
"config-file-type": "file",
"config-file-value": "/fluent-bit/configs/minimize-log-loss.conf"
}
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "firelens-test",
"awslogs-region": "ap-south-1",
"awslogs-create-group": "true",
"awslogs-stream-prefix": "firelens"
}
},
"memoryReservation": 50
},
{
"name": "python-ecs-test-app",
"stopTimeout": 60,
"image": "1234567890.dkr.ecr.ap-south-1.amazonaws.com/test-service:latest",
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"Name": "loki",
"host": "loki.example.com",
"port": "443",
"tls": "on",
"remove_keys": "container_id,ecs_task_arn",
"labels": "job=firelens",
"label_keys": "$container_name,$ecs_task_definition,$source,$ecs_cluster"
}
},
"portMappings": [
{
"hostPort": 80,
"containerPort": 80
}
],
"memoryReservation": 125
}
]
}
Dockerfile
# Use the official Python base image
FROM python:3.9-slim
# Copy the Python script into the container
COPY main.py /app/sigterm_program.py
# Set the working directory
WORKDIR /app
# Define the command to run the script
ENTRYPOINT ["python","-u", "sigterm_program.py"]
Is there anything that I'm not doing correctly? Any help is appreciated. Thanks
I've tried blog by AWS. And all other possible solutions, i.e using Grafana's loki plugin with fluentbit.
But all of them are not able ship logs to Loki when the containers are in graceful shutdown.