We have a website (mixed ASP.NET, .NET Core, and classic ASP) running in IIS on Windows Server 2016, with almost 20 "applications" (virtual directories) in the website. It is on a web farm of 2 servers, with a "sticky session" load-balancer before the farm. The website previously used on-premises LDAP authentication with a web form for sign-in. We changed to Azure AD (Microsoft Identity) authentication (using an AD setup from our corporate security group) several months ago.
For the Microsoft Identity code and configuration, we essentially used the code/configuration/NuGet-packages obtained from a new project created in Visual Studio 2022 if you select "Microsoft Identity" for the authentication choice, with a few minor changes to timeout values and similar.
There are typically about 15,000 - 20,000 page hits per day on the website. A very small number of these connections encounter one of a couple of different errors (that did not occur before we changed the authentication). This happens once or twice per day, sometimes several (5-10) times per day. There doesn't seem to be any common factors in the intermittent errors (different users, browsers, locations, etc.). It is commonly during a POST operation, but not always. If the user who encountered an error reloads the page, then everything works correctly again.
The (very occasional) errors with Microsoft Identity authentication are:
System.Web.HttpException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.
FormatException: The chunked cookie is incomplete. Only 0 of the expected 2 chunks were found, totaling 0 characters. A client size limit may have been exceeded.
The "machineKey" configuration exists and is identical in each of the applications (as well as the top-level) on the site - I have checked that multiple times. And, I have looked at the cookie values that are captured in the error log - there doesn't seem to be anything unexpected or excessive; there are the authentication and ASP.NET cookies, and we are not adding large cookie values in our code.
This is difficult to troubleshoot, and we can't seem to reproduce any of these errors in testing or on the staging server. And, we were not encountering any of these issues on the same website when it was using LDAP-based authenticaiton.
Everything on the servers is up-to-date (Windows updates, .NET versions, etc.), and all the NuGet packages are generally up-to-date in the Visual Studio projects.
Can you suggest what might be causing these rare errors, or how to investigate or troubleshoot the cause of the errors?
The details of the 2 different types of occasional errors that we see are as follows.
System.Web.HttpException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.
See http://go.microsoft.com/fwlink/?LinkID=314055 for more information.
---> System.Web.UI.ViewStateException: Invalid viewstate.
Client IP: [...]
Port: [...]
Referer: https:[...]
Path: /[...]
User-Agent: [...]
ViewState: [...]
at System.Web.UI.ViewStateException.ThrowError(System.Exception inner, System.String persistedState, System.String errorPageMessage, System.Boolean macValidationError) at offset 41
at System.Web.UI.ObjectStateFormatter.Deserialize(System.String inputString, System.Web.Security.Cryptography.Purpose purpose) at offset 238
at System.Web.UI.Util.DeserializeWithAssert(System.Web.UI.IStateFormatter2 formatter, System.String serializedState, System.Web.Security.Cryptography.Purpose purpose) at offset 0
at System.Web.UI.HiddenFieldPageStatePersister.Load() at offset 121
at System.Web.UI.Page.LoadPageStateFromPersistenceMedium() at offset 108
at System.Web.UI.Page.LoadAllState() at offset 7
at System.Web.UI.Page.ProcessRequestMain(System.Boolean includeStagesBeforeAsyncPoint, System.Boolean includeStagesAfterAsyncPoint) at offset 2404
at System.Web.UI.Page.ProcessRequest(System.Boolean includeStagesBeforeAsyncPoint, System.Boolean includeStagesAfterAsyncPoint) at offset 105
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at offset 397
at System.Web.HttpApplication.ExecuteStepImpl(System.Web.IExecutionStep step) at offset 100
at System.Web.HttpApplication.ExecuteStep(System.Web.IExecutionStep step, System.Boolean& completedSynchronously) at offset 21
And:
FormatException: The chunked cookie is incomplete. Only 0 of the expected 2 chunks were found, totaling 0 characters. A client size limit may have been exceeded.
at Microsoft.Owin.Host.SystemWeb.SystemWebChunkingCookieManager.GetRequestCookie(Microsoft.Owin.IOwinContext context, System.String key) at offset 266 in /_/src/Microsoft.Owin.Host.SystemWeb/SystemWebChunkingCookieManager.cs:line 108:col 29
at Microsoft.Owin.Security.Cookies.CookieAuthenticationHandler.<AuthenticateCoreAsync>d__12.MoveNext() at offset 1071 in /_/src/Microsoft.Owin.Security.Cookies/CookieAuthenticationHandler.cs:line 126:col 17
at Microsoft.Owin.Security.Infrastructure.AuthenticationHandler.<BaseInitializeAsync>d__31.MoveNext() at offset 318 in /_/src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs:line 70:col 17
at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__5.MoveNext() at offset 156 in /_/src/Microsoft.Owin.Security/Infrastructure/AuthenticationMiddleware.cs:line 28:col 13
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__7.MoveNext() at offset 115 in /_/src/Microsoft.Owin.Host.SystemWeb/IntegratedPipeline/IntegratedPipelineContextStage.cs:line 85:col 17
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.<DoFinalWork>d__12.MoveNext() at offset 116 in /_/src/Microsoft.Owin.Host.SystemWeb/IntegratedPipeline/IntegratedPipelineContext.cs:line 153:col 17
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(System.IAsyncResult ar) at offset 29 in /_/src/Microsoft.Owin.Host.SystemWeb/IntegratedPipeline/StageAsyncResult.cs:line 76:col 17
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at offset 215
at System.Web.HttpApplication.ExecuteStepImpl(System.Web.IExecutionStep step) at offset 100
at System.Web.HttpApplication.ExecuteStep(System.Web.IExecutionStep step, System.Boolean& completedSynchronously) at offset 55
EDIT (Nov-29): Note that I have read the article at http://go.microsoft.com/fwlink/?LinkID=314055 in the first error message above (and I have read similar articles), but this wasn't helpful. We already have a "machineKey" section in the top-level web.config and in each child application, and these sections are identical. The load balancer does have session affinity enabled. We are running IIS 10, not IIS 7, so the changes for IIS 7 aren't useful. And the "ViewStateUserKey" is set in all the applications.
EDIT (Nov-30): I have seen comments that the "Validation of viewstate MAC" error can happen when the IIS application pool (or app domain, or IIS process) is restarted. However, I have checked the server logs and IIS information, and there were not any restarts (of application pool, etc.) close to the times of these errors.
EDIT (Dec-1): Another possible cause that I have seen mentioned is that a user clicks on a "submit" button on the page before the page (specifically the event validation item, or perhaps the viewstate item) is fully loaded. However, I have reviewed the (browser) "page source" for the pages, and the eventvalidation content is almost immediately after the viewstate content and both are very early in the page. The pages don't typically render in the browser until they are fully loaded, so it would be very difficult to do any submit-type operation before the validation and viewstate are loaded. Also, I thought this was only an issue in early versions of ASP.NET and was corrected in a release like v3 or v4.0 of ASP.NET.