Weblogic Thread Monitoring Tips

If you are working as a middleware administrator or application support individual, you may have realized by now how crucial it is to have proper knowledge of the JVM along with a good understanding of the Java concurrency principles (yes you have to learn how to analyze thread dumps).

There is one principle I’m sure about: it is never too late to improve our knowledge and troubleshooting skills. Reaching a skill “plateau” is quite common and typically not due to our ability to learn but because of our fear and lack of willingness to embrace the challenges.

One of such challenges is possibly your ability to understand and assess the health of the JVM & middleware threads of the Java EE container you are responsible for such as Oracle Weblogic Server. If this is your situation then this post is for you.

Question:

How can you monitor the JVM threads in an efficient manner using the Weblogic admin console? Also, please elaborate how you can differentiate between healthy threads vs. slow running threads. Finally, what other tools can help you achieve this task?

Answer:

Please note that Weblogic Server 10.3.5 was used for the following example.

* An Oracle Weblogic 12c Thread Monitoring YouTube video is now available.

Oracle Weblogic Server is always installed with an admin console that provides you with out-of-the-box monitoring functions of the various Java EE resources exposed via the JMX API. Weblogic threads (created and assigned by the WLS kernel to the default self-tuning thread pool) are also fully exposed.

This monitoring page allows you to:

• Monitor the full list of all Java threads under Weblogic control.
• Correlate any slow running thread with your application, request and assigned Work Manager, if any.
• Generate a JVM Thread Dump of the Weblogic managed server directly from the page via the Dump Thread Stacks button.

Thread states - summary view

This section provides a summary of all different Weblogic threads and states.

Thread states - detailed view

The detailed view is much more interesting. This is where you will be spending most of your analysis time. Make sure that you add all proper columns including the associated Work Manager, application name etc.

The live Weblogic thread monitoring analysis process I usually follow is as per below. This approach is very useful for production environments when you are trying to determine the source of a performance slowdown or just to give you an idea of the health of the Weblogic threads.

• Refresh the page every 3-5 seconds.
• In between the refresh actions, identify the threads that are still executing the same request (slow running threads). This can be determined if you see the same Weblogic thread “Name” executing the same “Current Request” with the same “Total requests” value. Other criteria’s would be if Weblogic “promote” the affected thread(s) to Hogger or STUCK.
• Continue until you are done with your monitoring activity.
• As soon as one or a few slow running threads are found, identify the affected request(s) and application(s).
• Immediately after, generate a JVM Thread Dump using the Dump Thread Stacks button and copy/paste the output to a text editor for live or future analysis.

I also recommend that you use other tools to monitor the JVM and threads such as JVisualVM. JVisualVM will give a full view of all the threads, including GC related threads. It will also allow you to monitor the Java heap and correlate any finding with the health of the activity of the garbage collector.

Finally, if you suspect that you are dealing with a deeper thread concurrency problem such as thread lock contention or Java-level deadlock, you will need to generate a native thread dump (JVisualVM, kill -3 PID, jstack etc.) which will allow you to review the different monitor locks and locked ownable synchronizers.

Posted in: