Enabling Individual CM Node Login in a Load Balanced CM Sitecore v9.3 Environment

Recently we upgraded a client from Sitecore v8.2 to v9.3.  They required 3 load balanced Content Management (CM) servers in the upgraded environment.  We stood up the new environment that included the following servers:

• CM-TEST-1 (CM Node 1)
• CM-TEST-2 (CM Node 2)
• CM-TEST-3 (CM Node 3)
• IDENTITY-TEST (Identity Server)

Login worked fine using the LB URL:  https://cm-test.client.com/sitecore. Sitecore redirects you to the Identity Server as it should and we can sign in there using a local Sitecore account or the user's Active Directory account.  (We had also integrated AD using ADFS)  Then they told us that they needed to be able to sign in to an individual CM node sometimes when they are troubleshooting issues and are trying to figure out if it is a problem with an individual server.

So I tried signing in to a single CM node using this URL https://cm-test-1.client.com/sitecore.  What I noticed was that after authenticating I was redirected back to the LB URL - https://cm-test.client.com/sitecore.  Hmmmm...  So I did a little research and found out that we had set the FederatedAuthentication.IdentityServer.CallbackAuthority setting in the \App_Config\Sitecore\Owin.Authentication.IdentityServer\Sitecore.Owin.Authentication.IdentityServer.config file like this:

<setting name="FederatedAuthentication.IdentityServer.CallbackAuthority" value="https://cm-test.client.com" />

This means that no matter where you start from when you want to sign in to Sitecore, you will always be redirected back to the LB URL after authenticating.  That's no good.  So I removed that.  Then I tried signing in to an individual node and I got an error on my Identity Server that said 

Sorry, there was an error: unauthorized_client

The solution to this was to go to the Identity Server and modify the following configuration file:  \Config\production\Sitecore.IdentityServer.Host.xml

In the AllowedCorsOrigins section I had to add entries for the individual nodes in a pipe delimited list like this:

<AllowedCorsOrigins>
    <AllowedCorsOriginsGroup1>https://cm-test.client.com|https://cm-test-1.client.com|https://cm-test-2.client.com|https://cm-test-3.client.com</AllowedCorsOriginsGroup1>
</AllowedCorsOrigins>

Once I saved that and restarted the Identity Server app pool then I could successfully sign in to an individual CM node.  However now when I tried to sign in to the LB URL something odd was happening.  I would attempt to sign in to https://cm-test.client.com/sitecore and after authenticating, Identity Server would try and POST back to http://cm-test.client.com/identity/signin.  NOTE that it was trying to post back using the HTTP protocol and not HTTPS.  Why was this happening?

I noticed that when you enter https://cm-test.client.com/sitecore in to your browser and hit enter that the system redirects you to Identity Server to sign in.  And in the querystring that gets sent to Identity Server there is a parameter called redirect_uri.  This is the URI that Identity Server will eventually send you back to after authenticating.  And sure enough it was set to the HTTP version of the URL and not HTTPS.  Why?

After much searching online and discussions with others I finally figured out that under the hood what was happening is that the load balancer is configured to talk to the individual CM nodes over HTTP.  So when I access the LB URL, the load balancer then issues an HTTP request to one of the individual nodes.  That node gets the request over HTTP and responds back with the redirect to Identity Server and includes the HTTP version of the redirect_uri because the request that it got was over HTTP.  There were two solutions to this.  One was to configure the load balancer to use HTTPS when talking to the individual nodes.  That seemed to be too much and not necessary.  The other solution was to create an outbound rewrite rule on each CM node that would modify the redirect_uri in the querystring to make sure it was HTTPS.

So I added the following rewrite rule to each CM node:

<system.webserver>
  <rewrite>
    <outboundrules>
	  <rule name="Change Redirect URI from HTTP to HTTPS" precondition="IsRedirect">
	    <match pattern="(.*)redirect_uri=http%3a(.*)" servervariable="RESPONSE_LOCATION">
		<action type="Rewrite" value="{R:1}redirect_uri=https%3a{R:2}">
	  </action></match></rule>
	  <preconditions>
		<precondition name="IsRedirect">
			<add input="{RESPONSE_STATUS}" pattern="3\d\d">
		</add></precondition>
	  </preconditions>
	</outboundrules>
  </rewrite>
</system>

What this does is it looks for any outbound traffic that is a redirect and looks for the string "redirect_uri=http%3a" and replaces it with "redirect_uri=https%3a".  The reason for including the "%3a" is that the information that Sitecore sends to Identity Server in the querystring is escaped.

Once I put this in place on all 3 CM nodes then sign in worked great from the LB URL and from each individual CM node also.

I hope this helps someone out there that has load balanced CMs in v9.1+ that uses Identity Server where HTTPS is off loaded to the load balancer.