Custom SSL validation when using WebSockets

[esardaya]

Describe the bug

It's possible to set custom SSL validation when communication happens using HttpClientHandler.
However, if we configure NetHttpBinding with WebSocketSettings.TransportUsage = WebSocketTransportUsage.Always, then we run into SSL validation issues that we can't work around.

To Reproduce
Steps to reproduce the behavior:

Create a binding like this one:

var binding = new NetHttpBinding(BasicHttpSecurityMode.Transport)
{
    MaxReceivedMessageSize = int.MaxValue,  
    WebSocketSettings = { TransportUsage = WebSocketTransportUsage.Always },
};

Try to disable SSL validation, which works when not using WebSockets:

....
serviceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.Custom;
serviceCertificate.Authentication.CustomCertificateValidator = new DisableCertificateValidation();
serviceCertificate.SslCertificateAuthentication = new X509ServiceCertificateAuthentication()
{
    CertificateValidationMode = X509CertificateValidationMode.None,
    RevocationMode = X509RevocationMode.NoCheck
};

Then try to call a service that uses a self-signed SSL certificate, or one that doesn't match the domain.

Expected behavior
The call should succeed.

Actual behavior
The call fails with the following errors:

An error occurred when creating the WebSocket with the factory of type 'CoreClrClientWebSocketFactory'. See the inner exception for details.

The SSL connection could not be established, see inner exception.

The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch

   at System.Net.WebSockets.WebSocketHandle.<ConnectAsync>d__13.MoveNext() in /_/src/libraries/System.Net.WebSockets.Client/src/System/Net/WebSockets/WebSocketHandle.Managed.cs:line 237
   at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__15.MoveNext() in /_/src/libraries/System.Net.WebSockets.Client/src/System/Net/WebSockets/ClientWebSocket.cs:line 92
   at System.ServiceModel.Channels.CoreClrClientWebSocketFactory.<CreateWebSocketAsync>d__0.MoveNext() in /_/src/System.ServiceModel.Http/src/System/ServiceModel/Channels/CoreClrClientWebSocketFactory.cs:line 34
   at System.ServiceModel.Channels.ClientWebSocketTransportDuplexSessionChannel.<CreateWebSocketWithFactoryAsync>d__16.MoveNext() in /_/src/System.ServiceModel.Http/src/System/ServiceModel/Channels/ClientWebSocketTransportDuplexSessionChannel.cs:line 207

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

Looking at the WebSocket factory: https://github.com/dotnet/wcf/blob/main/src/System.ServiceModel.Http/src/System/ServiceModel/Channels/CoreClrClientWebSocketFactory.cs

While ClientWebSocket has a way to provide custom SSL validation (via Options.RemoteCertificateValidationCallback), the factory currently has no support for it.
Please either make it so that any SSL validation we have already configured also works with web sockets (pass the X509CertificateValidator), or give us some other way we can specify the SSL validation for these types of connections.
For example, you could just add a property to the existing WebSocketTransportSettings class.

Unfortunately, since the WebSocket factories being used are internal, there is no way for us to even try to provide our own implementation and use reflection tricks.
Short of just copy-pasting the HttpsChannelFactory implementation and slightly modifying it, we don't seem to have any way to get custom validation for WebSockets.

[mconnew]

@esardaya, after the changes in this PR are released, you will need to set the SSL properties on ServiceCertificate.SslAuthentication which start as null. The Authentication property is used for non-HTTP protocols (WebSockets starts as HTTP). You will use code similar to this:

factory.Credentials.ServiceCertificate.SslCertificateAuthentication = new X509ServiceCertificateAuthentication();
factory.Credentials.ServiceCertificate.SslCertificateAuthentication.CertificateValidationMode = X509CertificateValidationMode.Custom;
MyX509CertificateValidator myX509CertificateValidator = new MyX509CertificateValidator(ScenarioTestHelpers.CertificateIssuerName);
factory.Credentials.ServiceCertificate.SslCertificateAuthentication.CustomCertificateValidator = myX509CertificateValidator;