How to pull parameters from AWS Paramstore from EMR Cluster

Question

We are using EMR. Before submitting a spark job, we are trying to pull config parameters from AWS Param Store (we have written a java program) and create a file and use it in spark-submit.

We are using aws-java-sdk-ssm API to pull the parameters from param store.

When we try to run this program on the master node, it expect's a token (.aws/configure/credentials) to connect to aws ssm to pull the parameters.

We are able to access S3 buckets without any credentials, but not the param store.

We are spinning up the EMR cluster from concourse pipeline.

The below is the method that we have written to pull the parameters from the paramstore:

private static List<Parameter> getParametersFromAWSParamStore(String path, String awsRegion){
        List<Parameter> paramList = new ArrayList<>();

        AWSSimpleSystemsManagement client = AWSSimpleSystemsManagementClientBuilder.standard().withRegion(awsRegion).build();
        GetParametersByPathRequest request = new GetParametersByPathRequest();
        request.withRecursive(true);
        request.withPath(path);
        request.setWithDecryption(true);
        GetParametersByPathResult result = null;
        do {
            result = client.getParametersByPath(request);
            paramList.addAll(result.getParameters());
        }while( result.getNextToken() !=null && !result.getNextToken().isEmpty());
        return paramList;
    }

Answer 1

We need to make sure that the IAM role with which this program is executed have ssm: getParametersByPath policy.

Earlier the policy was missing and thereby unable to pull the properties.