This case
study describes the complete steps from root cause analysis to resolution of an
SSL communication problem between a Weblogic 11g client application and remote
Web Service provider that we faced recently when migrating our application from
Weblogic 8.1 to Weblogic 11g.
Environment specifications
Monitoring and
troubleshooting tools
The SSL and Java keystore configuration was verified in the Weblogic 11g console which did not reveal any problem. By default, Weblogic is using DemoTrust and JRE trust (cacerts); which contain the ROOT CA signature for most SSL trusted provider such as Entrust which we use in our enterprise.
It will also demonstrate the importance for proper understanding
of the SSL and Java keystore principles as this is a common problem you may
face with a Java EE environment using SSL to communication with your service
providers / downstream systems.
Environment specifications
·
Java
EE server (client side): Oracle Weblogic 11g
·
OS:
AIX 5.3 TL12 64-bit
·
JDK:
IBM JRE 1.6 SR9 64-bit
·
RDBMS:
Oracle 10gr2
·
HTTP
/ HTTPS client API: Apache HTTP Client
·
JSSE
/ SSL provider: IBMJSSEProvider2
·
Platform
type: Web Service and middle tier application
·
Java
/ Javax SSL debug (logging / tracing
purpose)
·
Java
keytool command (Java keystore
inspection)
·
JSP
test page (logging / tracing purpose)
Problem overview
Problem type: The java.security.cert.CertPathValidatorException:
The certificate issued by <Root CA issuer> is not
trusted; internal cause is: java.security.cert.CertPathValidatorException:
Certificate chaining error
Our application backend calls to a remote Web Service
provider were unable to complete proper SSL handshake due the above error. The
problem was that the certificate of the downstream server was not trusted by
the Weblogic 11g / JSSE provider.
Error detail
The error detail below did reveal an IBM JSSE2 component
as the source of the SSL handshake failure: com.ibm.security.cert.CertPathUtil.findIssuer(CertPathUtil.java:298) and did reveal the
following fact:
·
Fact #1: The failure is
triggered during the SSL issuer verification check between the client (Weblogic
11g) list of trust Root
CA’s and the remote
server SSL certificate issuer found at the root of the certificate path
[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error when connecting to url="https://xyz.xyz.xyz.xyz/WebService"
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.jsse2.n.a(n.java:8)
at com.ibm.jsse2.tc.a(tc.java:96)
at com.ibm.jsse2.gb.a(gb.java:241)
at com.ibm.jsse2.gb.a(gb.java:318)
at com.ibm.jsse2.hb.a(hb.java:232)
at com.ibm.jsse2.hb.a(hb.java:23)
at com.ibm.jsse2.gb.n(gb.java:295)
at com.ibm.jsse2.gb.a(gb.java:269)
at com.ibm.jsse2.tc.a(tc.java:347)
at com.ibm.jsse2.tc.g(tc.java:416)
at com.ibm.jsse2.tc.a(tc.java:461)
at com.ibm.jsse2.j.write(j.java:23)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:76)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:134)
at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:502)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:993)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at com.org.app.HttpConnector.readResponseBody()
.............................................
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:92)
at weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:74)
at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:151)
at com.sun.xml.ws.server.sei.EndpointMethodHandlerImpl.invoke(EndpointMethodHandlerImpl.java:265)
at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:100)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:604)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:563)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:548)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:445)
at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:373)
at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:524)
at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:255)
at com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:140)
at weblogic.wsee.jaxws.WLSServletAdapter.handle(WLSServletAdapter.java:208)
at weblogic.wsee.jaxws.HttpServletAdapter$AuthorizedInvoke.run(HttpServletAdapter.java:310)
at weblogic.wsee.jaxws.HttpServletAdapter.post(HttpServletAdapter.java:223)
at weblogic.wsee.jaxws.JAXWSServlet.doPost(JAXWSServlet.java:124)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at weblogic.wsee.jaxws.JAXWSServlet.service(JAXWSServlet.java:79)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:183)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3686)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2268)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1446)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by:
com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.jsse2.util.e.b(e.java:55)
at com.ibm.jsse2.util.e.b(e.java:77)
at com.ibm.jsse2.util.d.a(d.java:13)
at com.ibm.jsse2.hc.a(hc.java:80)
at com.ibm.jsse2.hc.checkServerTrusted(hc.java:82)
at com.ibm.jsse2.hc.b(hc.java:24)
at com.ibm.jsse2.hb.a(hb.java:124)
... 66 more
Caused by:
java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:411)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:258)
at com.ibm.jsse2.util.e.b(e.java:112)
... 72 more
Caused by:
java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:111)
at com.ibm.security.cert.PKIXCertPathValidatorImpl.engineValidate(PKIXCertPathValidatorImpl.java:176)
at com.ibm.security.cert.PKIXCertPathBuilderImpl.myValidator(PKIXCertPathBuilderImpl.java:737)
at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:649)
at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:595)
at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:595)
at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:595)
at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:357)
... 74 more
Caused by:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.security.cert.CertPathUtil.findIssuer(CertPathUtil.java:298)
at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:108)
... 81 more
SSL debug
SSL debug was enabled via the following argument in order to further troubleshoot the problem:
-Djavax.net.debug=ssl:verbose
-Dssl.debug=true
The additional traces below were generated which did confirm problem related to JSSE not trusting the received SSL certificate from the remote service provider.
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', SEND TLSv1 ALERT: fatal, description = certificate_unknown
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', WRITE: TLSv1 Alert, length = 2
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called closeSocket()
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', handling exception: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by <Root CA issuer> is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called close()
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called closeInternal(true)
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called close()
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called closeInternal(true)
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called close()
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', called closeInternal(true)
Java trust keystore configuration verification
The SSL and Java keystore configuration was verified in the Weblogic 11g console which did not reveal any problem. By default, Weblogic is using DemoTrust and JRE trust (cacerts); which contain the ROOT CA signature for most SSL trusted provider such as Entrust which we use in our enterprise.
The remote server issuer was also verified against our Weblogic 11g JRE 1.6 trust keystore (cacerts) which did not reveal any problem and a perfect match.
Java trust keystore runtime verification
Given no clear indication on the root cause at this point, we decided to
add additional tracing and verified the location of the trust keystore at
runtime. The following code was executed from a test JSP page in order to
verify each value at runtime:
//The loaded JSSE trust keystore location
System.getProperty("javax.net.ssl.trustStore"); // mismatch found!
//The Java Home of your Java VM running Weblogic 11g
System.getProperty("java.home");
//The loaded JSSE trust keystore type
System.getProperty("javax.net.ssl.trustStoreType");
//The JSSE trust keystore provider & encrypted password
System.getProperty("javax.net.ssl.trustStoreProvider");
System.getProperty("javax.net.ssl.trustStorePassword");
·
Fact #2: The additional
tracing did reveal a mismatch of the location of the trust keystore since it
was pointing to an application specific keystore containing only some custom
self-signed CA’s and no Entrust ROOT CA
Root cause and resolution
Additional analysis of the application did reveal an
override of the trust keystore location (javax.net.ssl.trustStore) which was leading
to a SSL communication failure with our remote service provider since all
Entrust ROOT CA’s were missing at runtime.
The application was updated and reconfigured in order to
use the default JSSE trust keystore and ensure no override of the global Javax
JSSE / SSL System properties.
Conclusion and recommendations
·
When
facing “CertPathValidatorException: Certificate chaining error” problem with
Weblogic and IBM JSSE, first verify your trust keystore configuration and
ensure that your keystore does contain proper ROOT CA entry of your
trusted provider and matches your remote service provider SSL certificate
issuer and signature
·
Please
enable SSL debug in order to analyse the SSL handshake process and understand
the source of the failure. These traces are really helpful and will help you
pinpoint the root cause
·
Please consider adding code to print the
runtime value of the Javax SSL specific System properties. This can really help
you pinpoint any wrong SSL configuration or override of your expected SSL /
JSSE configuration parameters