Java Heap Dump: Are you up to the task?

If you are as much enthusiasm as I am on Java performance, heap dump analysis should not be a mystery to you. If it is then the good news is that you have an opportunity to increase your Java troubleshooting skills and JVM knowledge.

The JVM has now evolve to a point that it is much easier today to generate and analyze a JVM heap dump vs. the old JDK 1.0 – JDK 1.4 days.

That being said, JVM heap dump analysis should not be seen as a replacement for profiling & JVM analysis tools such as JProfiler or Plumbr but complementary. It is particularly useful when troubleshooting Java heap memory leaks and java.lang.OutOfMemoryError problems.

This post will provide you with an overview of a JVM heap dump and what to expect out of it. It will also provide recommendations on how and when you should spend time analyzing a heap dump. Future articles will include tutorials on the analysis process itself.

Java Heap Dump overview

A JVM heap dump is basically a “snapshot” of the Java heap memory at a given time. It is quite different than a JVM thread dump which is a snapshot of the threads.

Such snapshot contains low level detail about the java objects and classes allocated on the Java heap such as:

• Java objects such as Class, fields, primitive values and references
• Classloader related data including static fields (important for classloader leak problems)
• Garbage collection roots or objects that are accessible from outside the heap (System classloader loaded resources such as rt.jar, JNI or native variables, Threads, Java Locals and more…)
• Thread related data & stacks (very useful for sudden Java heap increase problems, especially when combined with thread dump analysis)

Please note that it is usually recommended to generate a heap dump following a full GC in order to eliminate unnecessary “noise” from non-referenced objects.

Analysis reserved for the Elite?

One common misperception I have noticed over the last 10 years working with production support teams is the impression that deeper analysis tasks such as profiling, heap dump or thread dump analysis are reserved for the “elite” or the product vendor (Oracle, IBM…).

I could not disagree more.

As a Java developer, you write code potentially running in a highly concurrent thread environment, managing hundreds and hundreds of objects on the JVM. You do have to worry not only about concurrency issues but also on garbage collection and the memory footprint of your application(s). You are in the best position to perform this analysis since you are the expert of the application.

Find below typical questions you should be able to answer:

• How much concurrent threads are needed to run my application concurrently as per load forecast? How much memory each active thread is consuming before they complete their tasks?
• What is the static memory footprint of my application? (libraries, classloader footprint, in-memory cache data structures etc.)
• What is the dynamic memory footprint of my application under load? (sessions footprint etc.)
• Did you profile your application for any memory leak?

Load testing, profiling your application and analyzing Java heap dumps (ex: captured during a load test or production problem) will allow you to answer the above questions. You will then be in position to achieve the following goals:

• Reduce risk of performance problems post production implementation
• Add value to your work and your client by providing extra guidance & facts to the production and capacity management team; allowing them to take proper IT improvement actions
• Analyze the root cause of memory leak(s) or footprint problem(s) affecting your client IT production environment
• Increase your technical skills by learning these performance analysis principles and techniques
• Increase your JVM skills by improving your understanding of the JVM, garbage collection and Java object life cycles

 The last thing you want to reach is a skill “plateau”. If you are not comfortable with this type of analysis then my recommendations are as per below:

• Ask a more senior member of your team to perform the heap dump analysis and shadow his work and approach
• Once you are more comfortable, volunteer yourself to perform the same analysis (from a different problem case) and this time request a more experienced member to shadow your analysis work
• Eventually the student (you) will become the mentor

When to use

Analyzing JVM heap dumps should not be done every time you are facing a Java heap problem such as OutOfMemoryError. Since this can be a time consuming analysis process, I recommend this analysis for the scenarios below:

• The need to understand & tune your application and / or surrounding API or Java EE container itself memory footprint
• Java heap memory leak troubleshooting
• Java classloader memory leaks
• Sudden Java heap increase problems or trigger events (has to be combined with thread dump analysis as a starting point)

 Now find below some limitations associated with heap dump analysis:

• JVM heap dump generation is an intensive computing task which will hang your JVM until completed. Proper due diligence is required in order to reduce impact to your production environment
• Analyzing the heap dump will not give you the full Java process memory footprint e.g. native heap. For this purpose, you will need to rely on other tools and OS commands for that purpose
• You may face problems opening & parsing heap dumps generated from older version of JDK’s such as 1.4 or 1.5

Heap dump generation techniques

JVM heap dumps are typically generated as a result of 2 actions: 

• Auto-generated or triggered as a result of a java.lang.OutOfMemoryError (e.g. Java Heap, PermGen or native heap depletion)
• Manually generated via the usage of tools such as jmap, VisualVM (via JMX) or OS level command

# Auto-triggered heap dumps

If you are using the HotSpot Java VM 1.5+ or JRockit R28+ then you will need to add the following parameter below at your JVM start-up:

-XX:+HeapDumpOnOutOfMemoryError

The above parameter will enable to HotSpot VM to automatically generate a heap dump following an OOM event. The heap dump format for those JVM types is HPROF (*.hprof).

If you are using the IBM JVM 1.4.2+, heap dump generation as a result of an OOM event is enabled by default. The heap dump format for the IBM JVM is PHD (*.phd).

# Manually triggered heap dumps

Manual JVM heap dumps generation can be achieved as per below:

• Usage of jmap for HotSpot 1.5+
• Usage of VisualVM for HotSpot 1.6+ * recommended *

** Please do your proper due diligence for your production environment since JVM heap dump generation is an intrusive process which will hang your JVM process until completion **

If you are using the IBM JVM 1.4.2, you will need to add the following environment variables from your JVM start-up:

export IBM_HEAPDUMP=true

export IBM_HEAP_DUMP=true

For IBM JVM 1.5+ you will need to add the following arguments at the Java start-up:

-Xdump:heap

EX:

java -Xdump:none -Xdump:heap:events=vmstop,opts=PHD+CLASSIC
JVMDUMP006I Processing Dump Event "vmstop", detail "#00000000" - Please Wait.
JVMDUMP007I JVM Requesting Heap Dump using
 'C:\sdk\jre\bin\heapdump.20050323.142011.3272.phd'
JVMDUMP010I Heap Dump written to
 C:\sdk\jre\bin\heapdump.20050323.142011.3272.phd
JVMDUMP007I JVM Requesting Heap Dump using
 'C:\sdk\jre\bin\heapdump.20050323.142011.3272.txt'
JVMDUMP010I Heap Dump written to
 C:\sdk\jre\bin\heapdump.20050323.142011.3272.txt
JVMDUMP013I Processed Dump Event "vmstop", detail "#00000000".

Please review the Xdump documentation  for IBM JVM1.5+. 

For Linux and AIX®, the IBM JVM heap dump signal is sent via kill –QUIT or kill -3. This OS command will trigger JVM heap dump generation (PHD format).

I recommend that you review the MAT summary page on how to acquire JVM heap dump via various JVM & OS combinations.

Heap dump analysis tools

My primary recommended tool for opening and analyzing a JVM heap dump is Eclipse Memory Analyzer (MAT). This is by far the best tool out there with contributors such as SAP & IBM. The tool provides a rich interface and advanced heap dump analysis capabilities, including a “leak suspect” report. MAT also supports both HPROF & PHD heap dump formats.

I recommend my earlier post for a quick tutorial on how to use MAT and analyze your first JVM heap dump. I have also a few heap dump analysis case studies useful for your learning process.

Final words

I really hope that you will enjoy JVM heap dump analysis as much as I do. Future articles will provide you with generic tutorials on how to analyze a JVM heap dump and where to start. Please feel free to provide your comments.

Posted in: Heap Dump,Java,MAT,Performance