Sitecore 9.1 Failed to start Marketing Automation Service

Dear Sitecore Family,

Hope you are enjoying working with the latest Sitecore 9.1 version that Sitecore just launched?

Luckily for us, we got a chance to upgrade our current Sitecore 8.1 instance to 9.1 and we are simply loving it! We have gone ahead with Migration upgrade instead of In place upgrade – because we feel it makes sense to start of clean!

Also, due to a lot of restrictions in our client’s environments we could not install Sitecore 9.1 using SIF – but instead we had to do so using Manual installs. We did a trick there, we installed Sitecore using SIF on VMs and then created zips/payloads and sent them over to the client environments and configured their environments.

You might have a question – how did we setup all the Windows Services for Sitecore Marketing Automation Service, Sitecore Processing Engine and Sitecore Search Indexer? Well, we executed the Windows command to add a Windows Service and it got added.

After we setup the Sitecore 9.1 instance on the client environments, both the Processing Engine and Search Indexer running as Windows Service in one go  but we found that the Marketing Automation Service didn’t start. And even if we manually went and tried to start it, it won’t start.

Challenge: Sitecore 9.1 Failed to start Marketing Automation Service

As usual, we went to the Sitecore Logs for the same, to check if something was wrong.

2019-03-06 08:13:43.074 -06:00 [Error] Error initializing XConnect client.
System.AggregateException: One or more errors occurred. —> Sitecore.XConnect.XdbCollectionUnavailableException: An error occurred while sending the request. —> System.InvalidOperationException: The certificate was not found. Store: My, Location: LocalMachine, FindType: FindByThumbprint, FindValue: 9AB68CA30395280B013ED4C6F781444EC1D482BC, InvalidAllowed: False.
at Sitecore.Xdb.Common.Web.CertificateHttpClientHandlerModifier.Process(HttpClientHandler handler)
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateRequestHandler()
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateClient()
at System.Lazy`1.CreateValue()
at System.Lazy`1.LazyInitValue()
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__41.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteGetAsync>d__36.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()
— End of inner exception stack trace —
at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.XConnect.Client.XConnectClientConfiguration.<InitializeAsync>d__32.MoveNext()
— End of inner exception stack trace —
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Sitecore.XConnect.Client.Configuration.XConnect.Extensions.ServiceCollectionExtensions.<>c__DisplayClass1_0.<UseXConnectClientConfiguration>b__0(IServiceProvider provider)
—> (Inner Exception #0) Sitecore.XConnect.XdbCollectionUnavailableException: An error occurred while sending the request. —> System.InvalidOperationException: The certificate was not found. Store: My, Location: LocalMachine, FindType: FindByThumbprint, FindValue: 9AB68CA30395280B013ED4C6F781444EC1D482BC, InvalidAllowed: False.
at Sitecore.Xdb.Common.Web.CertificateHttpClientHandlerModifier.Process(HttpClientHandler handler)
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateRequestHandler()
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateClient()
at System.Lazy`1.CreateValue()
at System.Lazy`1.LazyInitValue()
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__41.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteGetAsync>d__36.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()
— End of inner exception stack trace —
at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.XConnect.Client.XConnectClientConfiguration.<InitializeAsync>d__32.MoveNext()<—

From all the details – we following line caught our eye: The certificate was not found

Now, this was pretty standard, we had added the certificate to the XConnect Server and also added it to the Trusted Root, then why was the Certificate not found error seen?

We found the following link to be useful: https://community.sitecore.net/developers/f/5/t/10025

We thought of 2 things to be done:

Firstly, we thought, its good to follow the Powershell Steps mentioned by Rakesh Bindal.

hence, we did that

Check for any non-self-signed certificates, execute below PowerShell script to find out:

[code language=”powershell”]

Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}

[/code]

To our surprise, we had one result item there.

If you found any result item from the above powershell script then execute below PowerShell command to move these non-self-signed certificates into the Intermediate Certification Authorities store :

[code language=”powershell”]

Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Move-Item -Destination Cert:\LocalMachine\CA

[/code]

We again tried to start the Marketing Automation Service – and it didn’t start and we still faced the same error. Now, we thought, the certificate is definitely in there, but something with the Certificate isn’t set right, so we started checking the Thumbprint and that proved to be the culprit.

We went to the location: {Path-To-XConnect-Website}\App_Data\jobs\continuous\AutomationEngine\App_Config and opened the ConnectionStrings.Config

There we found that the Thumbprint was different there:

We changed the Thumbprint and tried starting the windows service for Marketing Automation and it worked.

Credits:

To my fellow team mates – Mahek Chaniyara, Yogini Zope and Greg Brown – for working with me and achieving success. 🙂

References:

The above trick worked for me, but I also found some useful links that can help anyone. Thought of giving mentioning it here.

• https://community.sitecore.net/developers/f/5/t/10020
• https://community.sitecore.net/developers/f/5/t/10025
• http://sitecore.skowronski.it/sitecore/install-sitecoreconfiguration-failed-to-start-service-sitecore-marketing-automation-engine/
• https://sitecore.stackexchange.com/questions/15310/warning-waiting-for-service-sc910-xconnect-marketingautomationservice
• https://community.sitecore.net/developers/f/5/p/10020/19721
• https://tothecore.sk/2018/08/01/cannot-start-sitecore-xconnect-search-indexer-windows-service/
• https://doc.sitecore.com/developers/90/platform-administration-and-architecture/en/configure-the-marketing-automation-engine.html

This is good to understand.

• https://doc.sitecore.com/developers/91/sitecore-experience-platform/en/marketing-automation-engine.html
• https://doc.sitecore.com/developers/91/platform-administration-and-architecture/en/marketing-automation-engine.html
• http://thebitsthatbyte.com/sitecore-9-xp1-xconnect-windows-services-marketing-automation-engine-and-xconnect-search-indexer/

Happy Sitecore Troubleshooting! 🙂