/ HPROF – Memory leak analysis tutorial ~ Java EE Support Patterns

11.05.2011

HPROF – Memory leak analysis tutorial

This article will provide you with a tutorial on how you can analyze a JVM memory leak problem by generating and analyzing a Sun HotSpot JVM HPROF Heap Dump file.

A real life case study will be used for that purpose: Weblogic 9.2 memory leak affecting the Weblogic Admin server.

Environment specifications

·         Java EE server: Oracle Weblogic Server 9.2 MP1
·         Middleware OS: Solaris 10
·         Java VM: Sun HotSpot 1.5.0_22
·         Platform type: Middle tier

Monitoring and troubleshooting tools

·         Quest Foglight (JVM and garbage collection monitoring)
·          jmap (hprof / Heap Dump generation tool)
·         Memory Analyzer 1.1 via IBM support assistant (hprof Heap Dump analysis)
·         Platform type: Middle tier

Step #1 – WLS 9.2 Admin server JVM monitoring and leak confirmation

The Quest Foglight Java EE monitoring tool was quite useful to identify a Java Heap leak from our Weblogic Admin server. As you can see below, the Java Heap memory is growing over time.

If you are not using any monitoring tool for your Weblogic environment, my recommendation to you is to at least enable verbose:gc of your HotSpot VM. Please visit my Java 7 verbose:gc tutorial on this subject for more detailed instructions.


Step #2 – Generate a Heap Dump from your leaking JVM

Following the discovery of a JVM memory leak, the goal is to generate a Heap Dump file (binary format) by using the Sun JDK jmap utility.

** please note that jmap Heap Dump generation will cause your JVM to become unresponsive so please ensure that no more traffic is sent to your affected / leaking JVM before running the jmap utility **

<JDK HOME>/bin/jmap -heap:format=b <Java VM PID>


This command will generate a Heap Dump binary file (heap.bin) of your leaking JVM. The size of the file and elapsed time of the generation process will depend of your JVM size and machine specifications / speed.

For our case study, a binary Heap Dump file of ~ 2 GB was generated in about 1 hour elapsed time.

Sun HotSpot 1.5/1.6/1.7 Heap Dump file will also be generated automatically as a result of a OutOfMemoryError and by adding -XX:+HeapDumpOnOutOfMemoryError in your JVM start-up arguments.


Step #3 – Load your Heap Dump file in Memory Analyzer tool

It is now time to load your Heap Dump file in the Memory Analyzer tool. The loading process will take several minutes depending of the size of your Heap Dump and speed of your machine.




Step #4 – Analyze your Heap Dump

The Memory Analyzer provides you with many features, including a Leak Suspect report. For this case study, the Java Heap histogram was used as a starting point to analyze the leaking objects and the source.


For our case study, java.lang.String and char[] data were found as the leaking Objects. Now question is what is the source of the leak e.g. references of those leaking Objects. Simply right click over your leaking objects and select >> List Objects > with incoming references



As you can see, javax.management.ObjectName objects were found as the source of the leaking String & char[] data. The Weblogic Admin server is communicating and pulling stats from its managed servers via MBeans / JMX which create javax.management.ObjectName for any MBean object type. Now question is why Weblogic 9.2 is not releasing properly such Objects…

Root cause: Weblogic javax.management.ObjectName leak!

Following our Heap Dump analysis, a review of the Weblogic known issues was performed which did reveal the following Weblogic 9.2 bug below:

·         Weblogic Bug ID: CR327368
·         Description: Memory leak of javax.management.ObjectName objects on the Administration Server used to cause OutOfMemory error on the Administration Server.
·         Affected Weblogic version(s): WLS 9.2
·         Fixed in: WLS 10 MP1

 This finding was quite conclusive given the perfect match of our Heap Dump analysis, WLS version and this known problem description.

Conclusion

I hope this tutorial along with case study has helped you understand how you can pinpoint the source of a Java Heap leak using jmap and the Memory Analyzer tool.
Please don’t hesitate to post any comment or question.
I also provided free Java EE consultation so please simply email me and provide me with a download link of your Heap Dump file so I can analyze it for you and create an article on this Blog to describe your problem, root cause and resolution.

11 comments:

Hi P-H
Thanks for this article.

But I think To Read the Binary Heap Dump File we have another tool Jhat (But from Sun JDK 1.6 Onwards it is by default available inside “\bin” directory) is one of the Best Tool as well. If MAT Tool is not available thne we can get same output if we ran below commands.

jhat -J-mx1024m xyz.bin

one more update....
once we ran this jhat -J-mx1024m xyz.bin then we will have to open

url (http://localhost:7000 )
to get the result

Hi Sujit and thanks for your comments,

That is correct, JHat is also available from your JDK binaries to read HotSpot Heap Dump files.

However, MAT has much more built-in features such as out-of-the-box Leak Suspects report and much so I really recommend you download it and use this free tool for deep dive Heap Dump analysis.

If you only want a fast histogram report then JHat will do the job just fine.

Regards,
P-H

Hi P-H,

Thanks for your comment,
I have downloaded MAT and gone thru 2 heap dump bin files which i created locally to test and this is giving lots information about the class which is consuming more memory.
Is there anyway to upload phd file in this tools. I find only 2 formats (bin and hprof)are able to load to load and read.

Regards,
Sujit

Hi Sujit,

MAT also support IBM JRE generated Heap Dump format e.g. .phd files
Please have a look at this tutorial:

http://javaeesupportpatterns.blogspot.com/2011/02/ibm-sdk-heap-dump-httpsession-footprint.html

Thanks.
P-H

Hi P-H,

Thanks for the great tutorial.

Regards,
Shishir

Thanks Shishir for your comments.

P-H

Realy very good tutorial.
Thank you P-H

Hey P-H,
Just a question, I have this Reservation/POS program, and I was wondering i was looking at the Task Manager and found that at initial run, javaw.exe is using 60~70mb of memory. I'm unsure what is causing this or is this normal? I feel like it's taking too much memory than it should.

Hi Leaflet,

Please keep in mind that just firing a JVM process can by default use up to 100 MB of memory due to the default JVM settings etc.

60-70 MB, I don;t consider this excessive. The real question is how much of these JVM processes are you expecing to run concurrently from a single physical/virtual host?

Thanks.
P-H

Just one process. So 60-70mb is normal? But, the memory usage does increase and never actually releases it. And I think my program is suffering from a memory leak.

I can't post an image here so I'll make a link instead.

https://dl.dropbox.com/u/4074180/1.png
https://dl.dropbox.com/u/4074180/2.png

Post a Comment