/ Thread Dump analysis ~ Java EE Support Patterns

Thread Dump analysis

This page provides a consolidated of all articles and tutorials related to JVM Thread Dump analysis. Thread Dump analysis is a very important skill to master for any individual involved in Java EE production support and I highly recommend that you read all of these articles below.

Thread Dump analysis training plan

Java Deadlock


Training Videos

14 comments:

Hi,

Really good work. Very much appreciated.

This will helps me and other too.

Keep this good work.

Regards,
Peter Jerald

Thanks Peter,

I will release more thread dump related posts shortly, including problem pattern analysis.

Regards,
P-H

Very very nice and Very much Explanatory ...if you get chane please provide vedios...

Thanks
Narasimha

Thank you.

I will create more thread dump analysis shortly.

Regards,
P-H

very help full .. please keep up the good work

Thanks and Regards
Naveen Burman

very helpful .....thanks you...pierre

Thanks, I'm currently working on a Thread dump analysis tips and ticks article to be released in the next 1-2 weeks.

Thanks.
P-H

Hey Pierre,
Can you briefly look at this thread dump?
"AD Thread Pool-CSS9" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7097bbb8 WAITING

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

com.singularity.ee.util.de.c(de.java:523)

com.singularity.ee.util.de.take(de.java:516)

com.singularity.ee.util.ge.a(ge.java:487)

com.singularity.ee.util.sb.run(sb.java:691)

java.lang.Thread.run(Thread.java:662)

"Thread-45" waiting for lock com.comergent.dcm.cron.RunQueue@a908faf TIMED_WAITING

java.lang.Object.wait(Native Method)

com.comergent.dcm.cron.RunQueue.dequeue(RunQueue.java:231)

com.comergent.dcm.cron.RunQueue$RunRunnable.run(RunQueue.java:140)

"[STANDBY] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@18c92f94 WAITING

java.lang.Object.wait(Native Method)

java.lang.Object.wait(Object.java:485)

weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)

weblogic.work.ExecuteThread.run(ExecuteThread.java:183)

"AD Thread Pool-CSS8" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7097bbb8 WAITING

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

com.singularity.ee.util.de.c(de.java:523)

com.singularity.ee.util.de.take(de.java:516)

com.singularity.ee.util.ge.a(ge.java:487)

com.singularity.ee.util.sb.run(sb.java:691)

java.lang.Thread.run(Thread.java:662)

"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@58469848 WAITING

java.lang.Object.wait(Native Method)

java.lang.Object.wait(Object.java:485)

weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)

weblogic.work.ExecuteThread.run(ExecuteThread.java:183)

"[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@6655a179 WAITING

java.lang.Object.wait(Native Method)

java.lang.Object.wait(Object.java:485)

weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)

weblogic.work.ExecuteThread.run(ExecuteThread.java:183)
Guy

Hi Guy,

Not much going on this thread dump, were you facing any particular slowdown condition?

NOTE: this is a partial thread dump, i would recommend that you review each and every thread of WebLogic e.g. ExecuteThread: 'XYZ'

Please re-post further details, if any, from the WebLogic threads.

Regards,
P-H

hi, your articles were very nice and informative. I m new to analysis of thread dump and need you help with some trouble shooting. In our load environment we are seeing very low TPS and when we get the thread dumps
I noticed there are lot of thread in below state (like 900 odd threads). Does this mean anything or its
just an idle thread waiting for work to be assigned




"EventHandler-thread-352" prio=10 tid=0x00007fb5c8012800 nid=0x5e35 waiting on condition [0x00007fb2069af000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000648995a58> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

You are amazing, please keep up the good work.

hi ,can you look at this thread dump and suggest where is the problem... we are facing a slowness.
"RMI TCP Accept-0" daemon prio=6 tid=0x0000000000562d40 nid=0x934 runnable [0x000000000b6ef000..0x000000000b6ef9e0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x000000008548e400> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRM
IServerSocketFactory.java:31)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
at java.lang.Thread.run(Thread.java:595)
"RMI TCP Accept-10500" daemon prio=6 tid=0x0000000000562a90 nid=0x17e4 runnable
[0x000000000b5ef000..0x000000000b5efa60]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x000000008548e2b8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
at java.lang.Thread.run(Thread.java:595)

"RMI TCP Accept-0" daemon prio=6 tid=0x00000000005627e0 nid=0x9a4 runnable [0x00
0000000b4ef000..0x000000000b4efae0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x000000008548e7f0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
at java.lang.Thread.run(Thread.java:595)

"Timer-0" daemon prio=6 tid=0x0000000000562530 nid=0x13d8 in Object.wait() [0x00
0000000b36f000..0x000000000b36fb60]
at java.lang.Object.wait(Native Method)
- waiting on <0x000000008548d658> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x000000008548d658> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=6 tid=0x0000000000562280 nid=0xc88 runnable [0
x0000000000000000..0x0000000000000000]
"CompilerThread1" daemon prio=10 tid=0x0000000008f51810 nid=0x1724 waiting on co
ndition [0x0000000000000000..0x000000000986e9f0]
"CompilerThread0" daemon prio=10 tid=0x0000000008f51540 nid=0xafc waiting on con
dition [0x0000000000000000..0x000000000976ea40]
"AdapterThread" daemon prio=10 tid=0x0000000008f51270 nid=0x1100 waiting on cond
ition [0x0000000000000000..0x0000000000000000]
"Signal Dispatcher" daemon prio=10 tid=0x0000000000561fd0 nid=0x1048 waiting on
condition [0x0000000000000000..0x0000000000000000]
"Finalizer" daemon prio=8 tid=0x0000000000561d20 nid=0x770 in Object.wait() [0x0
00000000946f000..0x000000000946fa60]
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000854dd720> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
- locked <0x00000000854dd720> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

continuation of thread dump


"Reference Handler" daemon prio=10 tid=0x0000000000561a70 nid=0x958 in Object.wa
it() [0x000000000936f000..0x000000000936fae0]
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000853f0500> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00000000853f0500> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x000000000032d470 nid=0x1128 runnable

"GC task thread#0 (ParallelGC)" prio=6 tid=0x0000000000322f90 nid=0x10fc runnabl
e

"GC task thread#1 (ParallelGC)" prio=6 tid=0x0000000000323210 nid=0x11e0 runnabl
e

"GC task thread#2 (ParallelGC)" prio=6 tid=0x0000000000323490 nid=0xd0c runnable


"GC task thread#3 (ParallelGC)" prio=6 tid=0x0000000000323710 nid=0xcb4 runnable


"GC task thread#4 (ParallelGC)" prio=6 tid=0x0000000000323990 nid=0xfe8 runnable


"GC task thread#5 (ParallelGC)" prio=6 tid=0x0000000000323c10 nid=0xb10 runnable


"GC task thread#6 (ParallelGC)" prio=6 tid=0x0000000000323e90 nid=0x1428 runnabl
e

"GC task thread#7 (ParallelGC)" prio=6 tid=0x0000000000324110 nid=0xdf0 runnable


"GC task thread#8 (ParallelGC)" prio=6 tid=0x0000000000324390 nid=0xdd0 runnable


"GC task thread#9 (ParallelGC)" prio=6 tid=0x0000000000324610 nid=0x14f8 runnabl
e

"GC task thread#10 (ParallelGC)" prio=6 tid=0x0000000000324890 nid=0xd80 runnabl
e

"GC task thread#11 (ParallelGC)" prio=6 tid=0x0000000000324b10 nid=0x2bc runnabl
e

"GC task thread#12 (ParallelGC)" prio=6 tid=0x00000000005ffbf0 nid=0xde4 runnabl
e

"GC task thread#13 (ParallelGC)" prio=6 tid=0x00000000005ffe70 nid=0x228 runnabl
e

"GC task thread#14 (ParallelGC)" prio=6 tid=0x00000000006000f0 nid=0x1004 runnab
le

"GC task thread#15 (ParallelGC)" prio=6 tid=0x0000000000600370 nid=0x5cc runnabl
e

"GC task thread#16 (ParallelGC)" prio=6 tid=0x00000000006005f0 nid=0x15e8 runnab
le

"GC task thread#17 (ParallelGC)" prio=6 tid=0x0000000000600870 nid=0x12d0 runnab
le

"GC task thread#18 (ParallelGC)" prio=6 tid=0x0000000000600af0 nid=0xf5c runnabl
e

"GC task thread#19 (ParallelGC)" prio=6 tid=0x0000000000600d70 nid=0x1360 runnab
le

"GC task thread#20 (ParallelGC)" prio=6 tid=0x0000000000600ff0 nid=0x1538 runnab
le

"GC task thread#21 (ParallelGC)" prio=6 tid=0x0000000000601270 nid=0xd1c runnabl
e

"GC task thread#22 (ParallelGC)" prio=6 tid=0x00000000006014f0 nid=0x3c0 runnabl
e

"GC task thread#23 (ParallelGC)" prio=6 tid=0x0000000000601770 nid=0x14c0 runnab
le

"VM Periodic Task Thread" prio=10 tid=0x000000000032d7d0 nid=0x6b4 waiting on co
ndition

Post a Comment