Well, we have some special code in relation to SunMSCAPI, but it's not necessarily incompatible. Perhaps if we get some more information we can get this working.

I think SunMSCAPI private keys are only handles, i.e. they are not exportable; therefore any signature algorithm that you want to use with such a key has to be implemented by the SunMSCAPI provider itself. A reasonable guess for the cause of the above error is that BCJSSE is trying to instantiate a signature algorithm not supported by SunMSCAPI - or rather using a name for the signature algorithm that SunMSCAPI doesn't recognize/support.

Looking at the JDK source code, what jumps out at me is that SunMSCAPI supports "RSASSA-PSS", but apparently not "SHA256WITHRSAANDMGF1" and similar, which BCJSSE prefers to use for the TLS rsa_pss_... signature schemes.

To help check whether this is in fact the issue, please try using the bctls-debug jar. I would like to see more of the stack trace (to see where createStreamSigner is being called from) and ideally the name of the signature algorithm that was used to create the Signature instance (which would perhaps require you to use a debugger around the setup code).

P.S. To avoid problems from the KeyManager itself, please check that you are instantiating a "PKIX" KeyManager from the "BCJSSE" provider. A common configuration is to have BCJSSE high in the provider list (at least above other JSSE providers), and ssl.KeyManagerFactory.algorithm=PKIX set in java.security . BCJSSE is not generally compatible with KeyManagers from other providers.