AWS SecretsManager password rotation does not work in Rails

Question

We are using AWS SecretsManager (SM) to store a rotating database password for our rails app. However, when SM rotates the password, we get "FATAL: password authentication failed for user".

The password is read in database.yml

password: '<%= AwsSecretService.new.get_db_pwd(ENV['DATABASE_USERNAME']) if Rails.env.production?   %>'

The problem appears to be that ActiveRecord is caching the database credentials from database.yml upon initialization and does not reparse database.yml.

My thought is to somehow trap the authentication failed error and reinitialize ActiveRecord. With the following command:

ActiveRecord::Base.establish_connection(::Rails.application.config.database_configuration[::Rails.env]) 

I'm not sure how to trap the error. Would I need to put an error handler in application_record.rb since all activerecord models inherit from it?

Or would it be better to use an Observer of some sort?

Ideas?

Answer 1

I'm working on this too and am encountering the same issue. It seems that Rails only reads in the password once at application startup. If the password is then rotated, some connections will succeed if they were already established, but new connections in the pool will fail because they'll try to use the now-outdated password.

If it were possible for Rails to be forced to create all the connections in the pool at startup, this would work. Alternatively, if it were possible for Rails to dynamically pull the password each time a new connection were established, that would also work.

However, I've not found the ability to do either thing.

Answer 2

First of all, you need to fetch new credentials from AWS:

lib/aws_secret_manager.rb:

class AwsSecretsManager
  class << self
    def database_config
      secret_json = fetch_secret_json
      return {} unless secret_json

      fetch_database_config(secret_json)
    end

    private

    def fetch_secret_json
      client = Aws::SecretsManager::Client.new(region: ENV.fetch('AWS_REGION', nil))
      secret_value_response = client.get_secret_value(secret_id: ENV.fetch('AWS_SECRET_ID', nil))
      secret_value_response.secret_string
    end

    def fetch_database_config(secret_json)
      secret_hash = JSON.parse(secret_json, symbolize_names: true)
      secret_hash.slice(:host, :port, :username, :password)
    end
  end
end

Rails uses ActiveRecord::DatabaseConfigurations::* classes to store database configurations and calls ActiveRecord::DatabaseConfigurations#resolve(config_or_env) method to get a configuration for a new connection. To apply a new password for a new connection you can override this method and prepend your changes to ActiveRecord::DatabaseConfigurations class.

lib/active_record/aws_database_configurations.rb:

module ActiveRecord
  module AwsDatabaseConfigurations
    def resolve(config)
      default_config = super(config)
      aws_config = AwsSecretsManager.database_config
      DatabaseConfigurations::HashConfig.new(
        default_config.env_name,
        default_config.name,
        default_config.configuration_hash.merge(aws_config)
      )
    end
  end
end

config/initializers/aws_credentials.rb:

require './lib/active_record/aws_database_configurations'

ActiveRecord::DatabaseConfigurations.prepend(ActiveRecord::AwsDatabaseConfigurations)