Has anybody tried to use AWS KMS on Heroku?
On one hand, Heroku runs on AWS, so presumably it should work.
On other hand, I haven't seen any references that it was used in Heroku.
My main goal is to be able to get an encryption key from a hardware secure module (vs hardcoding it in my code or putting it in an environmental variable).
There is no reason why you should not be able to call a KMS endpoint from anywhere you have internet connectivity.
You still need to have an access key and secret access key for the AWS account you are using and that account needs to have permissions on KMS.
You will need to distribute the encrypted stuff and the aws keys to your Heroku instance. You can then decrypt and use KMS from there.
One thing that it worth mentioning: when using KMS you never see the actual key KMS uses. You can create the key, you can encrypt and decrypt, and/or you can have permissions on the specific API operations, but you cannot get the plaintext key.
What you normally do is some sort of envelope encryption in which you generate your own key and encrypt it via KMS and send the encrypted key and the encrypted data to the destination.