How could I fully restart a lambda function runtime without having to make a new deployment?

Question

Currently I'm working on a database migration. We use cloudformation to handle our resources and we have some lambda functions which create direct connections to our current database. We use secrets manager to handle the database credentials (username, password, endpoint/host, port, etc...).

What we want to have done is that when I modify the, let's say, the endpoint/host on the secrets, the connection on all the lambda functions we have which make a direct connection to the database would be updated.

I have read this question and its answers and I have tried to force a cold-start using a script which executes the aws lambda update-function-configuration command for the lambdas that I need to refresh their runtime.

The issue with this approach is that it seems to not be enough to completely refresh the lambda runtime because the database connection is still behaving as before making changes on the values stored on the secrets.

We cannot afford the time to make a full deployment of the stacks responsible for the lambdas that we need to "restart".

I'm not sure if the UpdateFunctionCode API endpoint will be useful to me since some of our lamdbas use are image based and others are ZipFile based using a runtime.

Answer 1

Reading this helped me solve my issue. Since updating an environment variable forces the lambda runtime to be restarted, I can just retrieve the existing environment variables and add a new one to restart a given lambda's runtime.

I made a little script to cold-start the lambda functions listed on an array:

NOTE: I'm not the best at bash scripting; so, this could be optimized. Also, I did not have time to learn to handle json objects on bash; that's why I used a second script (python) for that.

#!/bin/bash

AWS_PROFILE="dev";
ENV="dev";

echo "Cold starting only lambdas from array";

declare -a LIST_OF_LAMBDAS=( \
    "lambda-name1-$ENV" \
    "lambda-name2-$ENV" \
    "lambda-name3-$ENV"
);

for lambda in ${LIST_OF_LAMBDAS[@]}; do
    echo "Cold-starting lambda: $lambda";
    aws lambda get-function-configuration \
        --function-name $lambda \
        --query '{revisionId: RevisionId, env: Environment}' > original_env.json;

    REVISION_ID=`cat original_env.json | grep revisionId | \
        sed 's/"revisionId": //g;s/ //g;s/://g;s/"//g;s/,//g'`;

    python add_new_env_var.py; # Uses original_env.json to creates a file, called updated_env.json, which contains the updated env. vars.

    aws lambda update-function-configuration \
        --function-name "$lambda" \
        --environment file://updated_env.json \
        --description "Restarting/cold-starting lambda..." \
        --revision-id "$REVISION_ID" \
        --profile $AWS_PROFILE > /dev/null;

    if [ $? -eq 0 ]; then
        echo "OK";
    else
        echo "FAIL";
        echo $lambda >> "lambdas_failed_to_cold_start.txt"
    fi
    rm original_env.json updated_env.json;
    printf "\n";
done

echo "Script finished - DONE"

The python script is pretty simple:

import json
import time


def add_new_env_var(env_dict):
    """This function takes a dictionary which should have the structure of the output produced by
    the aws cli command:
    
    aws lambda get-function-configuration \
        --function-name $lambda \ -> $lambda is a valid name for a lambda function on the given env.
        --query '{revisionId: RevisionId, env: Environment}'
    Args:
        env_dict (dict): Contains the environment variables which already existed on a given lambda.
    Returns:
        dict: Same dict as received, but with one more env. variable which stores the current
        timestamp
    """    
    current_timestamp = str(time.time())

    try:
        env_dict["env"]["Variables"]["COLD_START_TS"] = current_timestamp
    except TypeError:
        env_dict = {"env":{"Variables":{"COLD_START_TS": current_timestamp}}}

    return env_dict


def start_processing():
    original_env_dict = {}

    with open("original_env.json") as original_env_file:
        original_env_dict = json.load(original_env_file)

    updated_env_dict = add_new_env_var(original_env_dict)

    with open("updated_env.json", "w") as outfile:
        json.dump(updated_env_dict["env"], outfile)


if __name__ == "__main__":
    start_processing()

I know that this is kinda hacky, but I wanted to share since it may be useful for someone.