Avoid email flood. in Python Alerting

Question

I have a script which will send email if the task is not in running status. This script runs every 5 min.

for task_list in status['tasks']:
        if task_list['state'] != 'RUNNING':
            alert_email(connector+'-task-'+str(task_list['id'])+'-status-'+task_list['state'])

What I am looking for is to avoid flooding email. Because something is failed may not be fixed in next 1 hour or so. Want to send only 1 email instead of 12 emails. I would appreciate any idea.

I was thinking of writing the alert in the sent directory before emailing the alert. next time the alert will be sent only if md5sum doesn't match with the alert written in sent directory or sent directory is empty.

Answer 1

Use a Boolean flag!

alert_sent = False
for task_list in status['tasks']:
    if task_list['state'] != 'RUNNING' and not alert_sent:
        alert_email(connector+'-task-'+str(task_list['id'])+'-status-'+task_list['state'])
        alert_sent = True

The alert_sent flag is initially set to False. When an alert email is sent, the flag is set to True, so that no further alerts will be sent for the rest of the task list.

UPDATE:

import os

sent_dir = 'sent'
if not os.path.exists(sent_dir):
    os.makedirs(sent_dir)

for task_list in status['tasks']:
    task_id = str(task_list['id'])
    state = task_list['state']
    filename = os.path.join(sent_dir, f"{task_id}-{state}.txt")
    if state != 'RUNNING' and not os.path.exists(filename):
        alert_email(connector+'-task-'+task_id+'-status-'+state)
        with open(filename, 'w') as f:
            f.write("Sent alert for task %s in state %s" % (task_id, state))

Now the script will check if a directory named sent exists. If it doesn't, the directory is created. Then, for each task in the list, the script generates a file name based on the task ID and state. If the task is not in a running state and the corresponding file doesn't exist, the alert is sent and a file is created to store the state of the alert.

If the task state changes (e.g., from "FAILED" to "RUNNING"), the corresponding file will be deleted and the next time the script runs, it will send another alert if the state is not "RUNNING".

UPDATE #2:

import time

alert_sent_time = 0
time_threshold = 300 # 5 minutes in seconds
for task_list in status['tasks']:
    if task_list['state'] != 'RUNNING' and time.time() - alert_sent_time > time_threshold:
        alert_email(connector+'-task-'+str(task_list['id'])+'-status-'+task_list['state'])
        alert_sent_time = time.time()

Now it will be reset automatically after 5 minutes (as you mentioned in your post)