The Problem
I had installed Sitecore 9.0 Update 1 with no issue. But at Fishtank we use a trusted wildcard certificate for our Sitecore development environments. When I added this certificate, I immediately began receiving errors retrieving data from xConnect.
The analytics dashboard was blank:
And I could see 500 errors in the console:
Here is the head of the stacktrace:
Exception System.InvalidOperationException: Ensure definition type did not complete successfully. StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Cache-Control: private
Date: Mon, 11 Dec 2017 10:26:06 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 5149
Content-Type: text/html; charset=utf-8
}
at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetReportSegments()
...
The Solution
The underlying error in the xConnect log showed that the specific error was a 403.16 in IIS - HTTP 403.16 - Client certificate is untrusted or invalid.
That's the key in @nsgocev's terrific post on the issue that he has written (see the thanks below). Searching for that error takes us immediately to this post on Stackoverflow.
To my complete bewilderment, light is clearly shed on the topic:
...the Trusted Root Certification Authorities (i.e. Root) store can only contained certificates that are self-signed
So when I installed our trusted Fishtank wildcard certificate, I actually put a non-self-signed certificate (in this case signed by EssentialSSL) into the root store.
Luckily there is a PowerShell move we can do to easily fix this problem.
Firstly, this PowerShell command will show you all non-self-signed certificates in your root store:
Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}
In my case this produced:
Secondly, this PowerShell command moves the offending certificates above into the Intermediate Certification Authorities (i.e. CA) store:
Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Move-Item -Destination Cert:\LocalMachine\CA
Running this produced no output, but when I re-ran the command to show all non-self-signed certificates, nothing was returned.
Super promising! I log back into Sitecore 9.0 Update 1 and what do I find…
Success!
With Sitecore 9's xConnect we're all learning a lot more about security and certificates - Myself included. So beware if you're installing your own trusted, non-self-signed certificates.
Thank You
I would like to thank @nsgocev (Nikola Gotsev) for originally writing about his journey to solve this issue with xConnect and the mysterious 403 status code. I'm here to only add context, color and appreciation as this was one of the more vexing issues I've encountered.
Do you know Transformers: The Movie? Right before Ultra Magnus is killed by Galvatron(!), he tries to invoke the Matrix of Leadership to "light (the Autobots) darkest hour" but unfortunately for him, nothing happens.
That's what happened when I came across @Kamsar's amazing tome on xConnect and security. It was great and I thought it would save me. But instead, I became Ultra Magnus. :(
But there was one person who could save me and the Autobots. It was @nsgocev . In my darkest hour, he opened the Matrix of Leadership, transformed into Nsgocevus Prime and saved me from Sitecore 9 and IIS certificate despair. Must respect.
Thanks for reading.
