AWS API gateway returns 403 for some and 200 for others

Question

I have a apigateway endpoint that returns 200 for me, but when it's called by a third party they get 403.

I request via curl and python requests and get 200 for both

Bash:

curl -X POST -v --http1.1 https://939pd1ndql.execute-api.us-east-1.amazonaws.com/default/bitbucket-events

Python

requests.post('https://939pd1ndql.execute-api.us-east-1.amazonaws.com/default/bitbucket-events',

I get 200 response for each request.

However when a third party calls the endpoint they get

HTTPSConnectionPool(host='939pd1ndql.execute-api.us-east-1.amazonaws.com', port=443): Max retries exceeded with url: /default/bitbucket-events (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Forbidden',)))

The third part is bitbucket - I am trying to create bitbucket app (really just a JSON payload telling bitbucket to create a webhook):

I do not have control over how bitbucket performs the requests and the request is very opaque but I pointed it at ngrok and intercepted the request it makes:

POST /default/bitbucket-events HTTP/1.1
Host: 939pd1ndql.execute-api.us-east-1.amazonaws.com
User-Agent: python-requests/2.22.0
Content-Length: 2292
Accept: */*
Accept-Encoding: gzip, deflate
Content-Type: application/json
Sentry-Trace: 00-41043c2935294252aa25ac44716a2300-86324af91ef0493e-00
X-Forwarded-For: 104.192.142.247
X-Forwarded-Proto: https
X-Newrelic-Id: VwMGVVZSGwQJVFVXDwcPXg==
X-Newrelic-Transaction: PxQPB1daXQMHVwRWAQkDUQUIFB8EBw8RVU4aWl4JDVcDUgoEBVcLVlNXDkNKQQoBBlZRAAQHFTs=

{LOTS OF JSON HERE}

Nothing in the request that bitbucket sends looks like it could cause this problem.

The response I get from the curl command is:

*   Trying 3.84.56.177...
* TCP_NODELAY set
* Connected to 939pd1ndql.execute-api.us-east-1.amazonaws.com (3.84.56.177) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.execute-api.us-east-1.amazonaws.com
*  start date: Jul 22 00:00:00 2021 GMT
*  expire date: Aug 20 23:59:59 2022 GMT
*  subjectAltName: host "939pd1ndql.execute-api.us-east-1.amazonaws.com" matched cert's "*.execute-api.us-east-1.amazonaws.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> POST /default/bitbucket-events HTTP/1.1
> Host: 939pd1ndql.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 12 Apr 2022 22:00:39 GMT
< Content-Type: application/json
< Content-Length: 0
< Connection: keep-alive
< x-amzn-RequestId: 78585bb0-5db4-4273-9333-45ef8b44952d
< Access-Control-Allow-Origin: *
< x-amz-apigw-id: QfN1IHrSoAMFrMw=

I have now devolved the apigateway to be just a mock endpoint that returns 200 response:

and I have set the logging to be very loud:

But I only see log entries as a result of the curl and python request I make. The bitbucket request does not result in a log line.

Could this mean the bitbucket request is being rejected by AWS before my api gateway is handling the request? I have no WAF enabled

As you can tell I am running out of ideas.

Answer 1

I replicated your setup, but with my own API Gateway. I was able to install the app though, so strongly suspect it is something to with your API Gateway setup.

I am using the exact same app descriptor, with only the URL being different.

{
    "key": "codereview.doctor.staging",
    "name": "Code Review Doctor Staging",
    "description": "Target lambdas with 'staging' version alias",
    "vendor": {
        "name": "Code Review Doctor",
        "url": "https://codereview.doctor"
    },
    "baseUrl": "https://fj7987nlx3.execute-api.ap-southeast-1.amazonaws.com",
    "authentication": {
        "type": "jwt"
    },
    "lifecycle": {
        "installed": "/default/bitbucket-events",
        "uninstalled": "/default/bitbucket-events"
    },
    "modules": {
        "webhooks": [
            {
                "event": "pullrequest:created",
                "url": "/default/bitbucket-events"
            },
            {
                "event": "pullrequest:updated",
                "url": "/default/bitbucket-events"
            },
            {
                "event": "pullrequest:fulfilled",
                "url": "/default/bitbucket-events"
            }
        ]
    },
    "scopes": ["account", "repository", "pullrequest"],
    "contexts": ["account"]
}

My API GW POST configuration looks exactly like yours, so the difference may be somewhere else.

Note that I have deleted my API GW stage, so you will not be able to test using mine for now.

Answer 2

I found few possible reasons mentioned in AWS docs & one that is not. Probably it saved me 10-20 hours of troubleshooting

1. Forbidden -> Something wrong with Authorisation
2. Forbidden -> Wrong Stage name/not deployed
3. 3rd party tools have issues somehow connecting to "us-east-1" API Gateway endpoint. Deploying in another region issue solves it immediately. Haven't found any possible reasons. Postman ok, curl ok, 3rd party libs not ok (not even log traces; seems to cut of in flight)