Word Automation Service BatchGetSyncJobStatus fails when requesting security token

Question

I'm running a SharePoint 2013 on-premise server on which I have deployed a simple WCF service as a farm solution. The service accepts simple Http post requests that contain single MS Word documents as payload and returns these files converted into PDFs. The service is accessible via Http to anonymous users. The WordAutomationService is running as Administration user account of the SharePoint server.

The service class creates an new instance of the Microsoft.Office.Word.Server.Conversions.SyncConverter and passes the proxy of the SharePoint's running WordAutomationService into the constructor (together with some ConversionJobSettings). Finally it calls the Convert method on the SyncConverter with the input stream (the Word document) and output stream (the web response which will contain the resulting PDF document produced by the WordAutomationService).

When creating the SyncConverter I don't set the UserToken property because the access to the service is by anonymous users. According to the remarks here https://msdn.microsoft.com/en-us/library/microsoft.office.word.server.conversions.syncconverter.usertoken.aspx this seems to be fine:

The default value for this property is a null reference (Nothing in Visual Basic), which is anonymous.

This setup works fine for small Word documents with a couple of pages and returns the expected PDF files. But as soon as the execution time of the WordAutomationService on the SharePoint exceeds a certain time threshold (around 5 seconds) the service fails because it never returns (which leads to a read timeout on the client). According to the logs it seems the reason for this is that after some time the synchronous conversion job moves the work into a background process:

Sync Stream job conversion takes too long. Don't wait anymore. Check its status later

It then polls the status of this job on a regular basis by calling ConversionServiceApplicationProxy.BatchGetSyncJobStatus. Unfortunately this call fails because internally it tries to create a new channel to talk to this process and for that asks for a security token. The SecurityTokenService however cannot complete the token request and throws an exception:

An unhandled exception has occurred. The security token request cannot be completed. System.InvalidOperationException: The security token request cannot be completed.    
at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForServiceContext(Uri contextUri)     
at Microsoft.SharePoint.SPChannelFactoryOperations.InternalCreateChannelActingAsLoggedOnUser[TChannel](ChannelFactory`1 factory, EndpointAddress address, Uri via)     
at Microsoft.Office.ConversionServices.Service.ConfigChannelFactory`1.CreateChannel(EndpointAddress address)     
at Microsoft.Office.ConversionServices.Service.ConversionServiceApplicationProxy.GetChannel(Uri uri)     
at Microsoft.Office.ConversionServices.Service.ConversionServiceApplicationProxy.ExecuteOnChannel(Uri endpointAddress, Action`1 action)     
at Microsoft.Office.ConversionServices.Service.ConversionServiceApplicationProxy.BatchGetSyncJobStatus(ICollection`1 ucids, Uri endpointAddress)     
at Microsoft.Office.ConversionServices.Service.BatchGetStatusPollingThread.Run()     
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)     
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)     
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)     
at System.Threading.ThreadHelper.ThreadStart() StackTrace:  
at onetnative.dll: (sig=37460b31-4453-4365-92f5-3a11c267be48|2|onetnative.pdb, offset=28F56) at onetnative.dll: (offset=15735)   

I'm at a loss now how to get rid of the token issue so that the system can create the necessary channel to poll the conversion job status. Any help is highly appreciated. Thanks!

(I can't post the full log because it registers as spam)

Answer 1

I’ve found that, if you were to install SharePoint 2013 on a Domain Controller (a topology that Microsoft said is only good for development but not for production), then the default anonymous user in IIS (IUSR) will not work reliably, and any WCF solution which is accessed via an IIS site that has Anonymous Access configured to use the IUSR account will fail when it attempts to access Security Token Service.

In this case the most expedient solution is to reconfigure IIS to use another anonymous identity, namely the identity tied to the Application Pool.

For example if your site is called NameOfSite, you can run this in an elevated PowerShell:

Set-WebConfigurationProperty `
  -Filter /system.WebServer/security/authentication/AnonymousAuthentication `
  -Name username `
  -Value "" `
  -location "NameOfSite"

This solves the immediate problem at hand which is that SecurityTokenForServiceContext fails. However, if you’ve installed SharePoint 2013 on Windows 2012 R2 as a Domain Controller, then it is not over: WordServerWorker actually will not start in this configuration.

I can also confirm, however, that if you were to install SharePoint 2013 on a standalone server (with <Setting Id="SERVERROLE" Value="SINGLESERVER"/> role in the unattended config file), then the entire solution works end-to-end, and WordServerWorker will actually start properly.

Previously, the most relevant (and unanswered) question on this must be this MSDN posting, “The security token request cannot be completed”. I would assume that in that case, the service was only in a meta-stable state, and one of the IIS workers would have previously obtained credentials via NTLM during local testing.

Answer 2

Usually when sharepoint service applications interact with each other, these services maintaining current user context trough wcf calls by using service application framework (SAF). Its allows these services to use SPContext.Current, preserve correlation id between call in logs and so on. When this context is lost, services stopping being able to communicate each other. For example this happens if we have a code that starts a new thread but didn't setup user for newly created thread context.

According to your description your service is anonymous and didn't use SAF to maintain user context, but uses some services that requires existence of that context

The possible solution would be is to use SAF(which is tricky configured WCF in a nutshell) instead of plain WCF services with no authentication

Edit:

One more possible solution may be is wrap your code with RunWithElevatedPrivileges to make your services connects sharepoint with application pool identity