"Token request failed." errors after installing provider-hosted add-in in new SharePoint site

Question

I have a provider-hosted SharePoint add-in, with a single instance of the add-in being used in multiple SharePoint tenants. It uses the Microsoft-provided TokenHelper.cs file to manage tokens/authentication.

Two times in the past week, I have had an issue where one or more users will encounter a "Token request failed." error, while other users are able to use the app fine. The issue seems to persist for some period of time (several hours ~ a day), and then eventually go away.

This issue seems to occur after someone installs the add-in in a SharePoint site where it was not previously installed. When this happens, several users within the SharePoint tenant where the add-in was installed see this error, but users in other tenants are unaffected.

Additional details:

• The issue persists if I open an incognito window and access the app from there using the same user account, or use that user account in a different browser
• The issue persists if I restart the app's website, clear session data, etc.

These details seem to suggest that the issue is persisting externally to the app itself, i.e. in SharePoint or some other Microsoft service. Given that the issue is tied to certain user accounts at any given time, this would suggest that the issue is with the refresh token obtained during the authentication redirect loop, as though the refresh token has been invalidated. But since the issue persists across browser sessions and app restarts, that would mean that SharePoint is continuing to serve up invalidated access tokens even after they are invalidated.

It is my understanding that other SharePoint add-in developers have been observing this same issue in their add-ins in the past week.

The line of code where the error occurs is this line in the GetAccessToken() method of TokenHelper.cs:

oauth2Response =
    client.Issue(AcsMetadataParser.GetStsUrl(targetRealm), oauth2Request) as OAuth2AccessTokenResponse;

I have captured the request and response for one of these failed token requests using Fiddler, and am including them below (with PII obfuscated).

Request:

POST https://accounts.accesscontrol.windows.net/5a41e9de-8fe8-4984-a108-************/tokens/OAuth/2 HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: accounts.accesscontrol.windows.net
Content-Length: 610
Expect: 100-continue

grant_type=refresh_token&client_id=ebb6ab2e-e627-4ffd-a95b-************%405a41e9de-8fe8-4984-a108-************&client_secret=*********&refresh_token=*******&resource=00000003-0000-0ff1-ce00-000000000000%2f*******sharepoint.com%405a41e9de-8fe8-4984-a108-************

Response:

HTTP/1.1 400 Bad Request
Cache-Control: no-store, no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
x-ms-request-id: 9610d5e7-3a98-4ec4-b8f1-2ffbedb0b100
x-ms-ests-server: 2.1.16522.6 - NCUS ProdSlices
X-XSS-Protection: 0
Set-Cookie: fpc=************; expires=Fri, 17-Nov-2023 10:28:46 GMT; path=/; secure; HttpOnly; SameSite=None
Set-Cookie: esctx=***********; domain=.accounts.accesscontrol.windows.net; path=/; secure; HttpOnly; SameSite=None
Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; httponly
Set-Cookie: stsservicecookie=estsfd; path=/; secure; httponly
Date: Wed, 18 Oct 2023 10:28:46 GMT
Content-Length: 478

{"error":"invalid_grant","error_description":"AADSTS70000: Provided grant is invalid or malformed.\r\nTrace ID: 9610d5e7-3a98-4ec4-b8f1-2ffbedb0b100\r\nCorrelation ID: ac485977-ec9e-4b73-8804-83b51db6937e\r\nTimestamp: 2023-10-18 10:28:46Z","error_codes":[70000],"timestamp":"2023-10-18 10:28:46Z","trace_id":"9610d5e7-3a98-4ec4-b8f1-2ffbedb0b100","correlation_id":"ac485977-ec9e-4b73-8804-83b51db6937e","error_uri":"https://accounts.accesscontrol.windows.net/error?code=70000"}

What could be causing this issue to occur, and then go away?

Is there any way to recover from this situation when it happens?

Steps to reproduce (use at your own risk)

1. Go to a SharePoint tenant where a provider-hosted add-in that uses ACS to call SharePoint has been in use for > 1 month
2. Launch the add-in and perform some action that would involve obtaining an access token to call SharePoint.
3. Go to a site collection within that tenant where the add-in has not previously been installed
4. Install the add-in there.
5. Attempt to use the add-in and perform some action that would involve obtaining an access token to call SharePoint.

Result: "Token request failed." error