How to check if SharePoint URL exists?

Question

Let's say we have the following SharePoint working URL:

https://mywebsite.sharepoint.com/_layouts/15/Test.aspx

The following code works for determining if the URL works or not unless the user is authenticated with Azure AD access token

var request = NetHelper.CreateWebRequest(url,
                    allowAutoRedirect: true,
                    method: WebRequestMethods.Http.Head);
......
// Authentication
......
bool exists = false;
using (var response = request.GetResponseWithRetry())
{
    if (response != null)
    {
        exists = response.StatusCode == HttpStatusCode.OK;
    }
}

This method works when we have the CookieContainer (username/password authentication) for authenticating. However, as per https://sharepoint.stackexchange.com/questions/248732/how-to-get-cookies-after-obtaining-azure-ad-access-token-to-sharepoint-online/248735 it's not possible to make WebRequest work when Azure AD authentication is being used and we apply HttpRequestHeader.Authorization header.

I tried to use CSOM library:

var csomFile = context.Web.GetFileByServerRelativeUrl(serverRelativeUrl);
context.Load(csomFile, f => f.Exists);
context.ExecuteQueryWithRetry();
bool exists = csomFile != null && csomFile.Exists;

However, this kind of code works only for actual files (I guess). It always returns False for the URL https://mywebsite.sharepoint.com/_layouts/15/Test.aspx.

So, my question is - Is there a way to determine if the URL exists using CSOM library, assuming we already have the authenticated ClientContext object (which can use either CookieContainer or HttpRequestHeader.Authorization header).

Answer 1

Sort of investigation why the scenario of using of Azure AD token is not possible to make "old-fashioned" calls (or, more precisely, explicitly disabled). As of my knowledge/findings, of course. I would be happy to be mistaking, but if not, then here's the situation.

The Microsoft's own applications (such Teams, Sharepoint Mobile, etc.) are using a special cookie to get web pages, and they get it using an undocumented method call, _api/SP.OAuth.NativeClient/Authenticate. When called with Azure AD token, this method returns a cookie that can be used for "classic" SharePoint web calls, the cookie returned is SPOIDCTL. Those clients are just including this cookie to the SharePoint "web" calls and can render views, for example, or get pages.

However, this functionality is only enabled for these specific Microsoft apps (detected by their "client IDs"). A third-party app will get 403 if it tries to call that API.

The motivation for this, as far as I understand, is that Microsoft is moving away from cookies auth, and it does not want to encourage third-party apps to use it or depend on it.

Here are some references:

• https://github.com/SharePoint/sp-dev-docs/issues/7743
• https://www.devjhorst.com/2020/04/