AWS Lambda function returns 403 ERROR The request could not be satisfied when sending a scan command to dynamodb

1.5k views Asked by At

I am attempting to send a Scan command to DynamoDB via API Gateway and a Lambda function written in Javascript (ES6 formatting). Every time I do so, I get 403 ERROR The request could not be satisfied. I have put the aws-sdk dependencies into a Lambda layer.

Lambda function:

import { DynamoDBClient, ScanCommand } from "@aws-sdk/client-dynamodb";
import { unmarshall } from '@aws-sdk/util-dynamodb';
const REGION = process.env.AWS_REGION;
const dynamo = new DynamoDBClient({region: REGION});
const tableName = process.env.MOVIE_TABLE;
//get table name from MOVIE_TABLE environment variable

/**
 *
 * Event doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format
 * @param {Object} event - API Gateway Lambda Proxy Input Format
 *
 * Context doc: https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html 
 * @param {Object} context
 *
 * Return doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
 * @returns {Object} object - API Gateway Lambda Proxy Output Format
 * 
 */

//test with command: sam local start-api --port 8080 --log-file logfile.txt
export const lambdaHandler = async (event, context) => {
    let respBody;
    let sCode = 200;
    if (event.httpMethod !== 'GET') {
        throw new Error(`GET method only accepts GET method, you tried: ${event.httpMethod}`);
    }
    //All log statements are written to CloudWatch
    console.info('received request for get:', event);
    console.info('received context:', context);
    try {
        const params = {
            TableName: tableName,
        };
        const command = new ScanCommand(params);
        console.info(`params Tablename: ${params.TableName}`);
        console.info(`Region: ${REGION}`);
        respBody = await dynamo.send(command);
        respBody = respBody.Items;
        respBody = respBody.map((i) => unmarshall(i));
    } catch (err) {
        sCode = err.statusCode;
        respBody = err.message;
        var stack = err.stack;
        //const { requestId, cfId, extendedRequestId } = err.$$metadata;
        console.info('Error stacktrace: \n');
        console.info(stack);
        //console.info('Error metdata: \n');
        //console.log({ requestId, cfId, extendedRequestId });
    } finally {
        respBody = JSON.stringify(respBody);
        console.info(`About to return status: ${sCode}, respBody: ${respBody}`);   
    }
    const response = {
        statusCode: sCode,
        body: respBody
    };
    return response;
};

template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  movie-crud-app
  
Globals:
  Function:
    Timeout: 3

Resources:
  GetItemsFunction:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: lambda-handlers/get-items/
      Handler: get-items.lambdaHandler
      Runtime: nodejs18.x
      Description: A simple function to get items
      Policies:
        #Give Create/Read/Update/Delete permissions to MovieTable
        - DynamoDBCrudPolicy:
            TableName: !Ref MovieTable
      Environment:
        Variables:
          #Make table name accessible as environment variable from function code during execution
          MOVIE_TABLE: !Ref MovieTable
      Architectures:
        - x86_64
      Layers:
        - !Ref DependenciesLayer
      Events:
        GetItems:
          Type: Api 
          Properties:
            Path: /items
            Method: get
            
  DependenciesLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
        LayerName: sam-app-dependencies
        Description: Dependencies for movie crud app (aws-sdk/client-dynamodb)
        ContentUri: dependencies/
        CompatibleRuntimes:
          - nodejs18.x
        LicenseInfo: 'MIT'
        RetentionPolicy: Retain
  
  MovieTable:
    Type: AWS::DynamoDB::Table
    Properties: 
      AttributeDefinitions: 
        - AttributeName: year
          AttributeType: N
        - AttributeName: title
          AttributeType: S
      KeySchema: 
        - AttributeName: year
          KeyType: HASH #Partition key
        - AttributeName: title
          KeyType: RANGE #Sort key
      ProvisionedThroughput: 
        ReadCapacityUnits: 10
        WriteCapacityUnits: 10
      TableName: "MovieTable"

I am 100% certain that the API Gateway is reaching the Lambda function. Logs from AWS Cloudwatch:

ERROR Invoke Error
{ "errorType": "Error", "errorMessage": "No valid endpoint provider available.", "stack": [ "Error: No valid endpoint provider available.",
" at /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/serializerMiddleware.js:10:15",
" at /var/runtime/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/baseCommand/DynamoDBDocumentClientCommand.js:11:20",
" at /opt/nodejs/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:5:28",
" at /var/runtime/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/commands/ScanCommand.js:28:28",
" at DynamoDBDocumentClient.send (/var/runtime/node_modules/@aws-sdk/smithy-client/dist-cjs/client.js:20:20)",
" at Runtime.lambdaHandler [as handler] (file:///var/task/get-items.mjs:60:29)",
" at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1085:29)"
]
}

I get an Internal Server Error if I use curl.

This app was deployed via AWS sam with: sam deploy --guided

I have tried sending a scan command to my dynamoDB table with a simple python script that uses boto3. That seems to have worked, so I thought that there might be an issue with permissions or roles that I may have missed.

Roles assigned to my lambda function:

JSON for GetItemsFunctionRolePolicy:

{
    "Statement": [
        {
            "Action": [
                "dynamodb:GetItem",
                "dynamodb:DeleteItem",
                "dynamodb:PutItem",
                "dynamodb:Scan",
                "dynamodb:Query",
                "dynamodb:UpdateItem",
                "dynamodb:BatchWriteItem",
                "dynamodb:BatchGetItem",
                "dynamodb:DescribeTable",
                "dynamodb:ConditionCheckItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:us-east-1:[mytablename]",
                "arn:aws:dynamodb:us-east-1:[mytablename]/index/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Perhaps the issue may have something to do with the integration request being LAMBDA_PROXY?

2023-03-06 Update: Apparently any Get or Scan commands sent to an empty data table will automatically give a "No valid endpoint provider" error. Once I filled the table with mock data, I no longer got that error. However, I still get the 403 Forbidden error if I try to call the API using Postman. If I use curl or a web browser, the function works. My next step is to fix the issue with Postman.

5

There are 5 answers

1
nisalap On

Assuming the template creates the resource correctly, try with this code.

const AWS = require("aws-sdk");
const dynamoDB = new AWS.DynamoDB.DocumentClient({
    region: process.env.AWS_REGION,
});


let REGION = process.env.AWS_REGION;
//get table name from MOVIE_TABLE environment variable in template.yaml
const tableName = process.env.MOVIE_TABLE;

export const lambdaHandler = async (event, context) => {
    let body;
    let statusCode = 200;
    if (event.httpMethod !== 'GET') {
        throw new Error(`GET method only accepts GET method, you tried: ${event.httpMethod}`);
    }
    console.info('received request for get:', event);
    console.info('received context:', context);
    try {
        var params = {
            TableName: tableName,
        };
        console.info(`Tablename: ${tableName}`);
        console.info(`DYNAMO: ${dynamo}`);
        console.info(`Region: ${REGION}`);
        body = await dynamoDB.scan(params).promise();
        body = body.Item;
    } catch (err) {
        StatusCode = err.statusCode;
        body = err.message;
        stack = err.stack;
        console.info('Error stacktrace: \n');
        console.info(stack);
    } finally {
        body = JSON.stringify(body);
        console.info(`About to return status: ${StatusCode}, body: ${body}`);
    }
    return {
        'statusCode': StatusCode,
        'body': body,
    };
};
0
nicktsan On

Apparently any Get or Scan commands sent to an empty data table will automatically give a "No valid endpoint provider" error. Once I filled the table with mock data, I no longer got that error. I was able to resolve the Postman issue by creating an API key and using it with Postman.

1
Phillip Harrington On

This error was happening to me, but the table I was using had data in it. When looking at the error call stack, I noticed that among the aws-sdk files coming from /var, there was one mid-stack that was coming from /opt, which is where lambda mounts code in layers.

So the culprit was my node_modules layer. I had created the layer without @aws-sdk dependencies in it (it was a devDependency for testing locally w/o sam) using --omit=dev but when building the layer zip file I noticed a lot of @aws-sdk files were still ending up in there.

It turned out another dependency I was using had peer dependencies from @aws-sdk and those got baked into the layer zip file. Adding --legacy-peer-deps to my npm command omitted these, and after creating a new version of the layer with the new zip file on AWS, the error was fixed.

My final npm command as part of my dependencies layer zip file build script is:

npm install --omit=dev --legacy-peer-deps

Seems simple/obvious, but I missed it and maybe this reply will help someone else.

2
Ashton Spina On

So this is the only place on the internet I see someone with the same issue.

I was following this example: https://github.com/aws-powertools/powertools-lambda-typescript/blob/main/examples/cdk/src/example-stack.ts and successfully deployed it.

  • I tested with items in the dynamodb, still same issue
    • tablename in the lambda execution matches the name of the dynamodb table in cloudformation (I'm assuming here this is a good thing, unless I'm missing something on naming convention)
    • policy on the lambda matches the ARN of the dynamodb table with correct actions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "xray:PutTraceSegments",
                "xray:PutTelemetryRecords"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "dynamodb:BatchGetItem",
                "dynamodb:GetRecords",
                "dynamodb:GetShardIterator",
                "dynamodb:Query",
                "dynamodb:GetItem",
                "dynamodb:Scan",
                "dynamodb:ConditionCheckItem",
                "dynamodb:BatchWriteItem",
                "dynamodb:PutItem",
                "dynamodb:UpdateItem",
                "dynamodb:DeleteItem",
                "dynamodb:DescribeTable"
            ],
            "Resource": [
                "arn:aws:dynamodb:eu-central-1:146425736795:table/real-estate-marketplace-prod-ListingsTableCCEBB5D2-CLP11ZEGR6Y4"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "ssm:DescribeParameters",
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:GetParameterHistory"
            ],
            "Resource": "arn:aws:ssm:eu-central-1:146425736795:parameter/app/uuid-api-url",
            "Effect": "Allow"
        }
    ]
}

  • I tested the stack output as suggested above but the layers look correct to me.

"message": "Error: No valid endpoint provider available.\n
at /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/serializerMiddleware.js:10:15\n
at /var/runtime/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/baseCommand/DynamoDBDocumentClientCommand.js:11:20\n
at /var/task/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:32\n
at /var/runtime/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/commands/ScanCommand.js:28:28\n
at DynamoDBDocumentClient.send (/var/runtime/node_modules/@aws-sdk/smithy-client/dist-cjs/client.js:20:20)\n
at getAllItemsHandler (/var/task/index.js:255:34)\n
at runRequest (/var/task/index.js:151:9)",

I've been stuck on this for so long, but can't figure out why I still get the error? It applies to any command I try to send on the dynamodb from the lambda using the dynamodb client.

I'm scanning like this

const docClient = tracer.captureAWSv3Client(new DynamoDBClient({ region: process.env.AWS_REGION }))
const data = await docClient.send(
    new ScanCommand({
      TableName: tableName,
    }),
  )

Has anyone encountered anything similar?

0
Ashton Spina On

It took me 3 days, but I figured this out.

In the example from github above there's layers that you bundle with the lambda.

const commonProps: Partial<NodejsFunctionProps> = {
  runtime: lambda.Runtime.NODEJS_18_X,
  timeout: Duration.seconds(60),
  logRetention: logs.RetentionDays.ONE_WEEK,
  memorySize: 256,
  tracing: Tracing.ACTIVE,
  environment: {
    NODE_OPTIONS: '--enable-source-maps', // see https://docs.aws.amazon.com/lambda/latest/dg/typescript-exceptions.html
    POWERTOOLS_SERVICE_NAME: 'store',
    POWERTOOLS_METRICS_NAMESPACE: 'PowertoolsExample',
    LOG_LEVEL: 'DEBUG',
  },
  bundling: {
    externalModules: [
      '@aws-lambda-powertools/commons',
      '@aws-lambda-powertools/logger',
      '@aws-lambda-powertools/tracer',
      '@aws-lambda-powertools/metrics'
    ],
  },
  layers: [],
}

I had originally changed this to include the nodeModules because I was getting errors about them not being available to

const commonProps: Partial<NodejsFunctionProps> = {
  runtime: lambda.Runtime.NODEJS_18_X,
  timeout: Duration.seconds(60),
  logRetention: logs.RetentionDays.ONE_WEEK,
  memorySize: 256,
  tracing: Tracing.ACTIVE,
  environment: {
    NODE_OPTIONS: '--enable-source-maps', // see https://docs.aws.amazon.com/lambda/latest/dg/typescript-exceptions.html
    POWERTOOLS_SERVICE_NAME: 'listings-store',
    POWERTOOLS_METRICS_NAMESPACE: 'KotijahtiListings',
    LOG_LEVEL: 'DEBUG',
  },
  bundling: {
    nodeModules: [
      '@aws-lambda-powertools/commons',
      '@aws-lambda-powertools/logger',
      '@aws-lambda-powertools/tracer',
      '@aws-lambda-powertools/metrics',
      '@aws-lambda-powertools/parameters'
    ],
  },
  layers: [],
}

Something in directly bundling that code rather than relying on the version bundled into the AWS Lambda default layer caused the issue. It works with the externalModules version now!