A real life case study will be used for that purpose: Weblogic 9.2 memory leak affecting the Weblogic Admin server.
Environment specifications
· Java EE server: Oracle Weblogic Server 9.2 MP1
· Middleware OS: Solaris 10
· Java VM: Sun HotSpot 1.5.0_22
· Platform type: Middle tier
Monitoring and troubleshooting tools
· Quest Foglight (JVM and garbage collection monitoring)
· jmap (hprof / Heap Dump generation tool)
· Memory Analyzer 1.1 via IBM support assistant (hprof Heap Dump analysis)
· Platform type: Middle tier
Step #1 – WLS 9.2 Admin server JVM monitoring and leak confirmation
The Quest Foglight Java EE monitoring tool was quite useful to identify a Java Heap leak from our Weblogic Admin server. As you can see below, the Java Heap memory is growing over time.
If you are not using any monitoring tool for your Weblogic environment, my recommendation to you is to at least enable verbose:gc of your HotSpot VM. Please visit my Java 7 verbose:gc tutorial on this subject for more detailed instructions.
Step #2 – Generate a Heap Dump from your leaking JVM
Following the discovery of a JVM memory leak, the goal is to generate a Heap Dump file (binary format) by using the Sun JDK jmap utility.
** please note that jmap Heap Dump generation will cause your JVM to become unresponsive so please ensure that no more traffic is sent to your affected / leaking JVM before running the jmap utility **
<JDK HOME>/bin/jmap -heap:format=b <Java VM PID>
This command will generate a Heap Dump binary file (heap.bin) of your leaking JVM. The size of the file and elapsed time of the generation process will depend of your JVM size and machine specifications / speed.
For our case study, a binary Heap Dump file of ~ 2 GB was generated in about 1 hour elapsed time.
Step #3 – Load your Heap Dump file in Memory Analyzer tool
It is now time to load your Heap Dump file in the Memory Analyzer tool. The loading process will take several minutes depending of the size of your Heap Dump and speed of your machine.
Step #4 – Analyze your Heap Dump
The Memory Analyzer provides you with many features, including a Leak Suspect report. For this case study, the Java Heap histogram was used as a starting point to analyze the leaking objects and the source.
For our case study, java.lang.String and char[] data were found as the leaking Objects. Now question is what is the source of the leak e.g. references of those leaking Objects. Simply right click over your leaking objects and select >> List Objects > with incoming references
As you can see, javax.management.ObjectName objects were found as the source of the leaking String & char[] data. The Weblogic Admin server is communicating and pulling stats from its managed servers via MBeans / JMX which create javax.management.ObjectName for any MBean object type. Now question is why Weblogic 9.2 is not releasing properly such Objects…
Root cause: Weblogic javax.management.ObjectName leak!
Following our Heap Dump analysis, a review of the Weblogic known issues was performed which did reveal the following Weblogic 9.2 bug below:
· Weblogic Bug ID: CR327368
· Description: Memory leak of javax.management.ObjectName objects on the Administration Server used to cause OutOfMemory error on the Administration Server.
· Affected Weblogic version(s): WLS 9.2
· Fixed in: WLS 10 MP1
This finding was quite conclusive given the perfect match of our Heap Dump analysis, WLS version and this known problem description.
Conclusion
I hope this tutorial along with case study has helped you understand how you can pinpoint the source of a Java Heap leak using jmap and the Memory Analyzer tool.
Please don’t hesitate to post any comment or question.
I also provided free Java EE consultation so please simply email me and provide me with a download link of your Heap Dump file so I can analyze it for you and create an article on this Blog to describe your problem, root cause and resolution.
20 comments:
Hi P-H
Thanks for this article.
But I think To Read the Binary Heap Dump File we have another tool Jhat (But from Sun JDK 1.6 Onwards it is by default available inside “\bin” directory) is one of the Best Tool as well. If MAT Tool is not available thne we can get same output if we ran below commands.
jhat -J-mx1024m xyz.bin
one more update....
once we ran this jhat -J-mx1024m xyz.bin then we will have to open
url (http://localhost:7000 )
to get the result
Hi Sujit and thanks for your comments,
That is correct, JHat is also available from your JDK binaries to read HotSpot Heap Dump files.
However, MAT has much more built-in features such as out-of-the-box Leak Suspects report and much so I really recommend you download it and use this free tool for deep dive Heap Dump analysis.
If you only want a fast histogram report then JHat will do the job just fine.
Regards,
P-H
Hi P-H,
Thanks for your comment,
I have downloaded MAT and gone thru 2 heap dump bin files which i created locally to test and this is giving lots information about the class which is consuming more memory.
Is there anyway to upload phd file in this tools. I find only 2 formats (bin and hprof)are able to load to load and read.
Regards,
Sujit
Hi Sujit,
MAT also support IBM JRE generated Heap Dump format e.g. .phd files
Please have a look at this tutorial:
http://javaeesupportpatterns.blogspot.com/2011/02/ibm-sdk-heap-dump-httpsession-footprint.html
Thanks.
P-H
Hi P-H,
Thanks for the great tutorial.
Regards,
Shishir
Thanks Shishir for your comments.
P-H
Realy very good tutorial.
Thank you P-H
Hey P-H,
Just a question, I have this Reservation/POS program, and I was wondering i was looking at the Task Manager and found that at initial run, javaw.exe is using 60~70mb of memory. I'm unsure what is causing this or is this normal? I feel like it's taking too much memory than it should.
Hi Leaflet,
Please keep in mind that just firing a JVM process can by default use up to 100 MB of memory due to the default JVM settings etc.
60-70 MB, I don;t consider this excessive. The real question is how much of these JVM processes are you expecing to run concurrently from a single physical/virtual host?
Thanks.
P-H
Just one process. So 60-70mb is normal? But, the memory usage does increase and never actually releases it. And I think my program is suffering from a memory leak.
I can't post an image here so I'll make a link instead.
https://dl.dropbox.com/u/4074180/1.png
https://dl.dropbox.com/u/4074180/2.png
How was the Object being identified as leak source , do we have any pattern or search which we can follow to identify the leak
what is the difference between ".bin" and "hprof" formats in heap dump, when to use .bin or .hprof format while generating heap dump with jmap?
Hi J@mMUla,
Both .bin and hprof are actually the same format and heap dumps generated for the HotSpot JVM. JMAP will typically generate a default heap dump with ".bin" extension when you chose BINARY format.
In the event of OutOfMemoryError, the JVM will automatically (if option is enabled) generate .hprof Heap Dump file. Again, same binary format.
For IBM JVM, the format is different with extension of ".phd".
Regards,
P-H
Hello P-H,
By going through this article, i worked on few of the heap dumps i am having. But when i analyzed the heap dump with eclipse MAT, i see many List Objects > with incoming references. How can i analyze in this case. Can you please let me know.
Thanks,
Pranay.
It is always normal to see lot of objects in incoming references. Focus on incoming references that you truly care about. You can also use the GC root option.
P-H
Hi I m reading images in spring app.. using file reader.. The method I am using is taking huge time in 1500CU load. Can any one guess the mistake what I am doing.. Is there any better soluttion
Hi, can you please share a source code snippnet and what you are trying to achive with Spring and your File Reader.
Thx.
P-H
Hi P-H,
Great article. Thanks a lot.
Thanks,
Gaurav
Hi P-H
How did you arrive to a point that java.char /java.lang.string /javax.management are the objects that are leaking.
Regards
Shankar
Post a Comment