Datastore writes failing with google.api_core.exceptions.PermissionDenied: 403 Received http2 header with status: 403

Question

Datastore writes failing with google.api_core.exceptions.PermissionDenied: 403 Received http2 header with status: 403

18 views Asked by Aleksa Tešić At 24 March 2024 at 16:51

My app is using Datastore as it's main DB, and for whatever reason, we just started getting:

google.api_core.exceptions.PermissionDenied: 403 Received http2 header with status: 403

errors, specifically on our Hetzner VM instances.

What I tried so far:

Tried writing into the database from local environment using the same service account key - works fine
Tried changing service account key - Got the same error. Both service accounts I tried have Owner permissions and Cloud Datastore Owner permissions

More detailed error is the following:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.PERMISSION_DENIED
    details = "Received http2 header with status: 403"
    debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-03-24T16:48:16.893337907+00:00", grpc_status:7, grpc_message:"Received http2 header with status: 403"}

What can I try next?

Original Q&A

There are 1 answers

**Aleksa Tešić** · Answer 1 · 2024-03-24T21:47:21+00:00

As this usually goes, I solved the problem literally 30mins after asking the question.

The Problem

Google has blocked some of the Hetzners IP addresses for some reason. As for the reason why, my best guess after looking around is that someone did something malicious and they just blocked the whole IP range.

How We Diagnosed the Problem

I've stumbled upon some reddit posts which suggested checking whether you could ping either www.google.com or www.googleapis.com

I logged in to the server in question and got 403 when trying to ping https://www.googleapis.com/oauth2/v1/certs with the following command:

curl --ipv4 https://www.googleapis.com/oauth2/v1/certs

In my case, response was the following:

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 403 (Forbidden)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>403.</b> <ins>That’s an error.</ins>
  <p>Your client does not have permission to get URL <code>/oauth2/v1/certs</code> from this server.  <ins>That’s all we know.</ins>

I've tried the same on another server that wasn't impacted and got the following response (which should be the same you get when opening link directly in the browser):

{
  "09bcf8028e06537d4d3ae4d84f5c5babcf2c0f0a": "-----BEGIN CERTIFICATE-----\nMIIDJzCCAg+gAwIBAgIJAIKgYSUb6hPfMA0GCSqGSIb3DQEBBQUAMDYxNDAyBgNV\nBAMMK2ZlZGVyYXRlZC1zaWdub24uc3lzdGVtLmdzZXJ2aWNlYWNjb3VudC5jb20w\nHhcNMjQwMzExMDQzODIwWhcNMjQwMzI3MTY1MzIwWjA2MTQwMgYDVQQDDCtmZWRl\ncmF0ZWQtc2lnbm9uLnN5c3RlbS5nc2VydmljZWFjY291bnQuY29tMIIBIjANBgkq\nhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvdtZ3cfuh44JlWkJRu+3yddVp58zxSHw\nsWiW/jpaXgpebo0an7qY2IEs3D7kC186Bwi0T7Km9mUcDbxod89IbtZuQQuhxlga\nXB+qX9GokNLdqg69rUaealXGrCdKOQ+rOBlNNGn3M4KywEC98KyQAKXe7prs7yGq\nI/434rrULaE7ZFmLAzsYNoZ/8l53SGDiRaUrZkhxXOEhlv1nolgYGIH2lkhEZ5Bl\nU53BfzwjO+bLeMwxJIZxSIOy8EBIMLP7eVu6AIkAr9MaDPJqeF7n7Cn8yv/qmy51\nbV+INRS+HKRVriSoUxhQQTbvDYYvJzHGYu/ciJ4oRYKkDEwxXztUewIDAQABozgw\nNjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDAWBgNVHSUBAf8EDDAKBggr\nBgEFBQcDAjANBgkqhkiG9w0BAQUFAAOCAQEAWW06gKHZpfxNG+0Bbrs4/LhPVLVY\n8KpsKkaQYtvcQOtA4BFwPyLMsGegPVfKIOMdqkWadHacvBE2ZqS74J+rObxwsvI5\nur6lbBX16jR7upi/xdMnLRZ4B52FSOpORQhukBBEwVeRasSBIqB+2SAoT4Nu7n58\n5KlsVeYyI5FzWJWwW9mun3cGauCLyLbP2IFnpTAqWdxcsH7lww9Amcd1vf4QFs5W\nYcOi1lSRWwTCkASQbiWFkS6hNR7euknhzqavUaMarm4FABb7oEPn4AJZCj+j+VFN\nxRza9bWYnMcOci2ncORWeB5CYwt6SEUIWXuKXdsA08mlYyKKg4gAJDoL9A==\n-----END CERTIFICATE-----\n",
  "adf5e710edfebecbefa9a61495654d03c0b8edf8": "-----BEGIN CERTIFICATE-----\nMIIDJjCCAg6gAwIBAgIIcPflSFYgkKgwDQYJKoZIhvcNAQEFBQAwNjE0MDIGA1UE\nAwwrZmVkZXJhdGVkLXNpZ25vbi5zeXN0ZW0uZ3NlcnZpY2VhY2NvdW50LmNvbTAe\nFw0yNDAzMTkwNDM4MjFaFw0yNDA0MDQxNjUzMjFaMDYxNDAyBgNVBAMMK2ZlZGVy\nYXRlZC1zaWdub24uc3lzdGVtLmdzZXJ2aWNlYWNjb3VudC5jb20wggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLjw3okH4AqrX5G/hKTAEANT6GnjNcdT41\nwJTDEP5pL2+vgShP2quOgn5iaRoSmr0Ga7XJdlkr+H/1BLfESw7xFz9ZK8hAIm0h\noNDPyHGTiVSNRCMwHhl/291OZ/wp2S0j20JgJu5JssLpN6CMMz2RuX0I2zeb3BJV\n47/klclALQmv46PsbP2y1ebQKjd/tELGsPTyQt6VdoCCWt3lN4aNkzJWILhzDWA9\nqPQKPqScOfxSXNH9a4GnOz+J6zkGtvNf7KG3RZ+eFel2mIY0xukpAPTgyY90wJir\nwQ0Xurn86Tt0+xWPW8cDq8Uad/uoTeyYPXnuzOyfOfQ15chtEye5AgMBAAGjODA2\nMAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/BAQDAgeAMBYGA1UdJQEB/wQMMAoGCCsG\nAQUFBwMCMA0GCSqGSIb3DQEBBQUAA4IBAQAVwFZiIo0SfDNW1VwmduKlKKPcZH6t\nGVcr9CP/aWxLyA+7rH77moI8H2v3EKqsIYdtcbYxKm2BOHkh9lksApe42tILuZ+F\nRUfY8EofyFqnerq9Z3Bdam9d2gvuF0yOpSskWyWXd/lcHC5G4AOtxtuXYlbNTjyX\nxDnI4bR7Q4jsNQ23bqorewyi586ZqNffYpm5OII1N9aaIm9UXzBaAU/bCBJVwMgw\nJisgUqxRX3FfUICgdEiGoB4bjMF3kq4UUGFJQ8HAtgDbSzPKrpPAVZzvgT1Du/Up\nkRltQSZ0TfIDwW4AbF6OYmswjrEBD5yz/9Nj7/mGyVBdBNKbNagsVxpe\n-----END CERTIFICATE-----\n"
}

The Solution

Go to Hetzner and just replace your IP, everything started working normally after that. Just keep in mind that Hetzner likes to assign the identical IP if you immediately delete and reassign.

In order to avoid that, first just detach the old IP, create new one, and after creating delete the old one.

Hope this will save someone else hours of pulling hair and debugging.

TechQA.

Datastore writes failing with google.api_core.exceptions.PermissionDenied: 403 Received http2 header with status: 403

There are 1 answers

The Problem

How We Diagnosed the Problem

The Solution

Related Questions in GOOGLE-CLOUD-PLATFORM

Related Questions in GOOGLE-CLOUD-FIRESTORE

Related Questions in DATASTORE

Popular Questions

Popular Tags

Trending Questions