CertPathValidatorException Certificate chaining error using Weblogic 11g and IBM JSSE

This case study describes the complete steps from root cause analysis to resolution of an SSL communication problem between a Weblogic 11g client application and remote Web Service provider that we faced recently when migrating our application from Weblogic 8.1 to Weblogic 11g.

It will also demonstrate the importance for proper understanding of the SSL and Java keystore principles as this is a common problem you may face with a Java EE environment using SSL to communication with your service providers / downstream systems.

Environment specifications

·         Java EE server (client side): Oracle Weblogic 11g

·         OS: AIX 5.3 TL12 64-bit

·         JDK: IBM JRE 1.6 SR9 64-bit

·         RDBMS: Oracle 10gr2

·         HTTP / HTTPS client API: Apache HTTP Client

·         JSSE / SSL provider: IBMJSSEProvider2

·         Platform type: Web Service and middle tier application

Monitoring and troubleshooting tools

·         Java / Javax SSL debug (logging / tracing purpose)

·         Java keytool command (Java keystore inspection)

·         JSP test page (logging / tracing purpose)

Problem overview

Problem type: The java.security.cert.CertPathValidatorException:

The certificate issued by <Root CA issuer> is not trusted; internal cause is: java.security.cert.CertPathValidatorException: Certificate chaining error

Our application backend calls to a remote Web Service provider were unable to complete proper SSL handshake due the above error. The problem was that the certificate of the downstream server was not trusted by the Weblogic 11g / JSSE provider.

Error detail

The error detail below did reveal an IBM JSSE2 component as the source of the SSL handshake failure: com.ibm.security.cert.CertPathUtil.findIssuer(CertPathUtil.java:298) and did reveal the following fact:

·         Fact #1: The failure is triggered during the SSL issuer verification check between the client (Weblogic 11g) list of trust Root CA’s and the remote server SSL certificate issuer found at the root of the certificate path

[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:

        java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:

        java.security.cert.CertPathValidatorException: Certificate chaining error when connecting to url="https://xyz.xyz.xyz.xyz/WebService"

javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:

        java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:

        java.security.cert.CertPathValidatorException: Certificate chaining error

        at com.ibm.jsse2.n.a(n.java:8)

        at com.ibm.jsse2.tc.a(tc.java:96)

        at com.ibm.jsse2.gb.a(gb.java:241)

        at com.ibm.jsse2.gb.a(gb.java:318)

        at com.ibm.jsse2.hb.a(hb.java:232)

        at com.ibm.jsse2.hb.a(hb.java:23)

        at com.ibm.jsse2.gb.n(gb.java:295)

        at com.ibm.jsse2.gb.a(gb.java:269)

        at com.ibm.jsse2.tc.a(tc.java:347)

        at com.ibm.jsse2.tc.g(tc.java:416)

        at com.ibm.jsse2.tc.a(tc.java:461)

        at com.ibm.jsse2.j.write(j.java:23)

        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:76)

        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:134)

        at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:502)

        at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1973)

        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:993)

        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)

        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)

        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)

        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)

        at com.org.app.HttpConnector.readResponseBody()

              .............................................

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

        at java.lang.reflect.Method.invoke(Method.java:611)

        at weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:92)

        at weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:74)

        at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:151)

        at com.sun.xml.ws.server.sei.EndpointMethodHandlerImpl.invoke(EndpointMethodHandlerImpl.java:265)

        at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:100)

        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:604)

        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:563)

        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:548)

        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:445)

        at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:373)

        at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:524)

        at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:255)

        at com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:140)

        at weblogic.wsee.jaxws.WLSServletAdapter.handle(WLSServletAdapter.java:208)

        at weblogic.wsee.jaxws.HttpServletAdapter$AuthorizedInvoke.run(HttpServletAdapter.java:310)

        at weblogic.wsee.jaxws.HttpServletAdapter.post(HttpServletAdapter.java:223)

        at weblogic.wsee.jaxws.JAXWSServlet.doPost(JAXWSServlet.java:124)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

        at weblogic.wsee.jaxws.JAXWSServlet.service(JAXWSServlet.java:79)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:183)

        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3686)

        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)

        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)

        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2268)

        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)

        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1446)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Caused by:

com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:

        java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:

        java.security.cert.CertPathValidatorException: Certificate chaining error

        at com.ibm.jsse2.util.e.b(e.java:55)

        at com.ibm.jsse2.util.e.b(e.java:77)

        at com.ibm.jsse2.util.d.a(d.java:13)

        at com.ibm.jsse2.hc.a(hc.java:80)

        at com.ibm.jsse2.hc.checkServerTrusted(hc.java:82)

        at com.ibm.jsse2.hc.b(hc.java:24)

        at com.ibm.jsse2.hb.a(hb.java:124)

        ... 66 more

Caused by:

java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:

        java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:

        java.security.cert.CertPathValidatorException: Certificate chaining error

        at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:411)

        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:258)

        at com.ibm.jsse2.util.e.b(e.java:112)

        ... 72 more

Caused by:

java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:

        java.security.cert.CertPathValidatorException: Certificate chaining error

        at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:111)

        at com.ibm.security.cert.PKIXCertPathValidatorImpl.engineValidate(PKIXCertPathValidatorImpl.java:176)

        at com.ibm.security.cert.PKIXCertPathBuilderImpl.myValidator(PKIXCertPathBuilderImpl.java:737)

        at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:649)

        at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:595)

        at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:595)

        at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:595)

        at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:357)

        ... 74 more

Caused by:

java.security.cert.CertPathValidatorException: Certificate chaining error

        at com.ibm.security.cert.CertPathUtil.findIssuer(CertPathUtil.java:298)

        at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:108)

        ... 81 more

SSL debug

SSL debug was enabled via the following argument in order to further troubleshoot the problem:

-Djavax.net.debug=ssl:verbose

-Dssl.debug=true

The additional traces below were generated which did confirm problem related to JSSE not trusting the received SSL certificate from the remote service provider.

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', SEND TLSv1 ALERT:  fatal, description = certificate_unknown

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', WRITE: TLSv1 Alert, length = 2

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called closeSocket()

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', handling exception: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:

       java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:

       java.security.cert.CertPathValidatorException: Certificate chaining error

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called close()

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called closeInternal(true)

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called close()

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called closeInternal(true)

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called close()

[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called closeInternal(true)

                    

Java trust keystore configuration verification

The SSL and Java keystore configuration was verified in the Weblogic 11g console which did not reveal any problem. By default, Weblogic is using DemoTrust and JRE trust (cacerts); which contain the ROOT CA signature for most SSL trusted provider such as Entrust which we use in our enterprise.

The remote server issuer was also verified against our Weblogic 11g JRE 1.6 trust keystore (cacerts) which did not reveal any problem and a perfect match.

Java trust keystore runtime verification

Given no clear indication on the root cause at this point, we decided to add additional tracing and verified the location of the trust keystore at runtime. The following code was executed from a test JSP page in order to verify each value at runtime:

//The loaded JSSE trust keystore location

System.getProperty("javax.net.ssl.trustStore"); // mismatch found!

//The Java Home of your Java VM running Weblogic 11g

System.getProperty("java.home");

//The loaded JSSE trust keystore type

System.getProperty("javax.net.ssl.trustStoreType");

//The JSSE trust keystore provider & encrypted password

System.getProperty("javax.net.ssl.trustStoreProvider");

System.getProperty("javax.net.ssl.trustStorePassword");

·         Fact #2: The additional tracing did reveal a mismatch of the location of the trust keystore since it was pointing to an application specific keystore containing only some custom self-signed CA’s and no Entrust ROOT CA

Root cause and resolution

Additional analysis of the application did reveal an override of the trust keystore location (javax.net.ssl.trustStore) which was leading to a SSL communication failure with our remote service provider since all Entrust ROOT CA’s were missing at runtime.

The application was updated and reconfigured in order to use the default JSSE trust keystore and ensure no override of the global Javax JSSE / SSL System properties.

Conclusion and recommendations

·         When facing “CertPathValidatorException: Certificate chaining error” problem with Weblogic and IBM JSSE, first verify your trust keystore configuration and ensure that your keystore does contain proper ROOT CA  entry of your trusted provider and matches your remote service provider SSL certificate issuer and signature

·         Please enable SSL debug in order to analyse the SSL handshake process and understand the source of the failure. These traces are really helpful and will help you pinpoint the root cause

·          Please consider adding code to print the runtime value of the Javax SSL specific System properties. This can really help you pinpoint any wrong SSL configuration or override of your expected SSL / JSSE configuration parameters

Posted in: JSSE,Oracle Weblogic,SSL