We don't have a reproduction scenario, but we ran into an issue / limitation in BCJSSE in bctls-1.0.14.1, so we'd like to report that here.

(This is not blocking us anymore, since we have switched to a completely different endpoint which doesn't trigger this behavior.)

What follows is based on Wireshark traces and BCJSSE stack traces, but unfortunately I cannot share any of those.

The context is a Java 8 JVM where both the BCFIPS and BCJSSE providers are registered (and SunJSSE is not), not in approved-only mode and FIPS mode respectively, using bc-fips-1.0.2.3.jar and bctls-fips-1.0.14.1.jar. The JVM is the client, talking to a .NET endpoint that only supports TLS 1.2.

We observe that

• (A) the first SSL connection worked fine but raised warning(1) no_renegotiation(100) alert: Renegotiation not supported;
• (B) follow-up connections with that same session id, instead of raising that warning, fail with fatal alert unexpected_message(10).

For (A) that initial no-error connection, it seems the following is happening:

• The initial handshake must have completed successfully, including a session id in the ServerHello. (No session tickets play a role, it seems.)
• Some application data must have been sent to the .NET server.
• A TLS handshake message is received.
• In org.bouncycastle.tls.TlsClientProtocol.handleHandshakeMessage(short, HandshakeMessageInput) (line 762-777) this turns out to be a Hello request (0), which is different from a 'server hello' (2).
• Based on RFC 2246 7.4.1.1 and the fact that it has already sent application data, the BCJSSE client TlsClientProtocol looks to see if it must do renegotiation:

        case HandshakeType.hello_request:
        {
            assertEmpty(buf);

            /*
             * RFC 2246 7.4.1.1 Hello request This message will be ignored by the client if the
             * client is currently negotiating a session. This message may be ignored by the client
             * if it does not wish to renegotiate a session, or the client may, if it wishes,
             * respond with a no_renegotiation alert.
             */
            if (isApplicationDataReady())
            {
                handleRenegotiation();
            }
            break;
        }
   }

• And in the parent class's TlsProtocol.handleRenegotiation() it decides, for some reason, to refuseRenegotiation().

That results in the client sending the no_renegotiation(100) alert, logged as Client raised warning(1) no_renegotiation(100) alert: Renegotiation not supported.

So far so good, everything still working according to spec, as far as I can see.

Now compare this to (B) a failing TCP connection. So in a request where the client tries to resume the initial session using that earlier session id, again from logged stack traces and the bctls-fips-1.0.14.1.jar source code, it seems that that same Hello request (0) instead makes the communication fail:

• The initial handshake completes, including in ClientHello the previous ServerHello's session ID.
• Application data is sent to the .NET server.
• Two server handshake messages come in. One of them is probably a Hello (0).
• When that is received, slightly earlier in the interaction, the same method org.bouncycastle.tls.TlsClientProtocol.handleHandshakeMessage(short, HandshakeMessageInput) concludes that this is a resumed session, so it and it checks that it is in the 'just received Server Hello' state, and the handshake message is Finished (20):

        if (securityParameters.isResumedSession())
        {
            if (type != HandshakeType.finished || this.connection_state != CS_SERVER_HELLO)
            {
                throw new TlsFatalAlert(AlertDescription.unexpected_message);
            }

• That is not the case, and fatal unexpected_message(10) is raised.

From TLS 1.0 RFC 2246 mentioned in the code comment quoted above, and the actually relevant TLS 1.2 RFC 5246, it looks like BCJSSE 1.0.14.1, in the first part of org.bouncycastle.tls.TlsClientProtocol.handleHandshakeMessage(), assumes that in a 'resumed session' connection it will only ever receive a 'Finished (20)' handshake message, right after ServerHello.

And that assumption is, as far as I can see, violated by this .NET TLS server, which happily sends a Hello request (0) message.

Note that both RFCs say

7.4.1.1.  Hello Request

   When this message will be sent:

      The HelloRequest message MAY be sent by the server at any time.

so the fact that BCJSSE fails when this message is received at a time that it doesn't expect it, seems to be a bug?

And I don't yet see a workaround. Specifically, I haven't (yet) found way to disable session resume in the BCJSSE client. (That would be costly w.r.t. performance / TLS connection set-up time, but it should theoretically work: It would skip over the if (securityParameters.isResumedSession()) block in TlsClientProtocol.handleHandshakeMessage() where the unexpected_message(10) fatal alert is raised.)