Oracle Weblogic stuck thread detection

The following question will again test your knowledge of the Oracle Weblogic threading model. I’m looking forward for your comments and experience on the same.

If you are a Weblogic administrator, I’m certain that you heard of this common problem: stuck threads. This is one of the most common problems you will face when supporting a Weblogic production environment.

A Weblogic stuck thread simply means a thread performing the same request for a very long time and more than the configurable Stuck Thread Max Time.

Question:

How can you detect the presence of STUCK threads during and following a production incident?

Answer:

* An Oracle Weblogic 12c Thread Monitoring YouTube video is now available.

As we saw from our last article “Weblogic Thread Monitoring Tips”, Weblogic provides functionalities allowing us to closely monitor its internal self-tuning thread pool. It will also highlight you the presence of any stuck thread.

This monitoring view is very useful when you do a live analysis but what about after a production incident? The good news is that Oracle Weblogic will also log any detected stuck thread to the server log. Such information includes details on the request and more importantly, the thread stack trace. This data is crucial and will allow you to potentially better understand the root cause of any slowdown condition that occurred at a certain time.

<Sep 25, 2013 7:23:02 AM EST> <Error> <WebLogicServer> <Server1> <App1>

<[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'>

<BEA-000337> <[STUCK] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "608" seconds working on the request

"Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 608213 ms

POST /App1/jsp/test.jsp HTTP/1.1

Accept: application/x-ms-application...

Referer: http://..

Accept-Language: en-US

User-Agent: Mozilla/4.0 ..

Content-Type: application/x-www-form-urlencoded

Accept-Encoding: gzip, deflate

Content-Length: 539

Connection: Keep-Alive

Cache-Control: no-cache

Cookie: JSESSIONID=

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:

    <Application Execution Stack Trace>

    ...................................

               javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

               javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

               weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

               weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

               weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)

               weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184)

               weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction....

               weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run()

               weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

               weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)

               weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2281)

               weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2180)

               weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491)

               weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

               weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Here is one more tip: the generation and analysis of a JVM thread dump will also highlight you stuck threads. As we can see from the snapshot below, the Weblogic thread state is now updated to STUCK, which means that this particular request is being executed since at least 600 seconds or 10 minutes.

This is very useful information since the native thread state will typically remain to RUNNABLE. The native thread state will only get updated when dealing with BLOCKED threads etc. You have to keep in mind that RUNNABLE simply means that this thread is healthy from a JVM perspective. However, it does not mean that it truly is from a middleware or Java EE container perspective. This is why Oracle Weblogic has its own internal ExecuteThread state.

Finally, if your organization or client is using any commercial monitoring tool, I recommend that you enable some alerting around both hogging thread and stuck thread. This will allow your support team to take some pro-active actions before the affected Weblogic managed server(s) become fully unresponsive.

Posted in: