Java Heap Dump: Are you up to the task?

If you are as much enthusiasm as I am on Java performance, heap dump analysis should not be a mystery to you. If it is then the good news is that you have an opportunity to increase your Java troubleshooting skills and JVM knowledge.

The JVM has now evolve to a point that it is much easier today to generate and analyze a JVM heap dump vs. the old JDK 1.0 – JDK 1.4 days.

That being said, JVM heap dump analysis should not be seen as a replacement for profiling & JVM analysis tools such as JProfiler or Plumbr but complementary. It is particularly useful when troubleshooting Java heap memory leaks and java.lang.OutOfMemoryError problems.

This post will provide you with an overview of a JVM heap dump and what to expect out of it. It will also provide recommendations on how and when you should spend time analyzing a heap dump. Future articles will include tutorials on the analysis process itself.

Java Heap Dump overview

A JVM heap dump is basically a “snapshot” of the Java heap memory at a given time. It is quite different than a JVM thread dump which is a snapshot of the threads.

Such snapshot contains low level detail about the java objects and classes allocated on the Java heap such as:

• Java objects such as Class, fields, primitive values and references
• Classloader related data including static fields (important for classloader leak problems)
• Garbage collection roots or objects that are accessible from outside the heap (System classloader loaded resources such as rt.jar, JNI or native variables, Threads, Java Locals and more…)
• Thread related data & stacks (very useful for sudden Java heap increase problems, especially when combined with thread dump analysis)

Please note that it is usually recommended to generate a heap dump following a full GC in order to eliminate unnecessary “noise” from non-referenced objects.

Analysis reserved for the Elite?

One common misperception I have noticed over the last 10 years working with production support teams is the impression that deeper analysis tasks such as profiling, heap dump or thread dump analysis are reserved for the “elite” or the product vendor (Oracle, IBM…).

I could not disagree more.

As a Java developer, you write code potentially running in a highly concurrent thread environment, managing hundreds and hundreds of objects on the JVM. You do have to worry not only about concurrency issues but also on garbage collection and the memory footprint of your application(s). You are in the best position to perform this analysis since you are the expert of the application.

Find below typical questions you should be able to answer:

• How much concurrent threads are needed to run my application concurrently as per load forecast? How much memory each active thread is consuming before they complete their tasks?
• What is the static memory footprint of my application? (libraries, classloader footprint, in-memory cache data structures etc.)
• What is the dynamic memory footprint of my application under load? (sessions footprint etc.)
• Did you profile your application for any memory leak?

Load testing, profiling your application and analyzing Java heap dumps (ex: captured during a load test or production problem) will allow you to answer the above questions. You will then be in position to achieve the following goals:

• Reduce risk of performance problems post production implementation
• Add value to your work and your client by providing extra guidance & facts to the production and capacity management team; allowing them to take proper IT improvement actions
• Analyze the root cause of memory leak(s) or footprint problem(s) affecting your client IT production environment
• Increase your technical skills by learning these performance analysis principles and techniques
• Increase your JVM skills by improving your understanding of the JVM, garbage collection and Java object life cycles

 The last thing you want to reach is a skill “plateau”. If you are not comfortable with this type of analysis then my recommendations are as per below:

• Ask a more senior member of your team to perform the heap dump analysis and shadow his work and approach
• Once you are more comfortable, volunteer yourself to perform the same analysis (from a different problem case) and this time request a more experienced member to shadow your analysis work
• Eventually the student (you) will become the mentor

When to use

Analyzing JVM heap dumps should not be done every time you are facing a Java heap problem such as OutOfMemoryError. Since this can be a time consuming analysis process, I recommend this analysis for the scenarios below:

• The need to understand & tune your application and / or surrounding API or Java EE container itself memory footprint
• Java heap memory leak troubleshooting
• Java classloader memory leaks
• Sudden Java heap increase problems or trigger events (has to be combined with thread dump analysis as a starting point)

 Now find below some limitations associated with heap dump analysis:

• JVM heap dump generation is an intensive computing task which will hang your JVM until completed. Proper due diligence is required in order to reduce impact to your production environment
• Analyzing the heap dump will not give you the full Java process memory footprint e.g. native heap. For this purpose, you will need to rely on other tools and OS commands for that purpose
• You may face problems opening & parsing heap dumps generated from older version of JDK’s such as 1.4 or 1.5

Heap dump generation techniques

JVM heap dumps are typically generated as a result of 2 actions: 

• Auto-generated or triggered as a result of a java.lang.OutOfMemoryError (e.g. Java Heap, PermGen or native heap depletion)
• Manually generated via the usage of tools such as jmap, VisualVM (via JMX) or OS level command

# Auto-triggered heap dumps

If you are using the HotSpot Java VM 1.5+ or JRockit R28+ then you will need to add the following parameter below at your JVM start-up:

-XX:+HeapDumpOnOutOfMemoryError

The above parameter will enable to HotSpot VM to automatically generate a heap dump following an OOM event. The heap dump format for those JVM types is HPROF (*.hprof).

If you are using the IBM JVM 1.4.2+, heap dump generation as a result of an OOM event is enabled by default. The heap dump format for the IBM JVM is PHD (*.phd).

# Manually triggered heap dumps

Manual JVM heap dumps generation can be achieved as per below:

• Usage of jmap for HotSpot 1.5+
• Usage of VisualVM for HotSpot 1.6+ * recommended *

** Please do your proper due diligence for your production environment since JVM heap dump generation is an intrusive process which will hang your JVM process until completion **

If you are using the IBM JVM 1.4.2, you will need to add the following environment variables from your JVM start-up:

export IBM_HEAPDUMP=true

export IBM_HEAP_DUMP=true

For IBM JVM 1.5+ you will need to add the following arguments at the Java start-up:

-Xdump:heap

EX:

java -Xdump:none -Xdump:heap:events=vmstop,opts=PHD+CLASSIC
JVMDUMP006I Processing Dump Event "vmstop", detail "#00000000" - Please Wait.
JVMDUMP007I JVM Requesting Heap Dump using
 'C:\sdk\jre\bin\heapdump.20050323.142011.3272.phd'
JVMDUMP010I Heap Dump written to
 C:\sdk\jre\bin\heapdump.20050323.142011.3272.phd
JVMDUMP007I JVM Requesting Heap Dump using
 'C:\sdk\jre\bin\heapdump.20050323.142011.3272.txt'
JVMDUMP010I Heap Dump written to
 C:\sdk\jre\bin\heapdump.20050323.142011.3272.txt
JVMDUMP013I Processed Dump Event "vmstop", detail "#00000000".

Please review the Xdump documentation  for IBM JVM1.5+. 

For Linux and AIX®, the IBM JVM heap dump signal is sent via kill –QUIT or kill -3. This OS command will trigger JVM heap dump generation (PHD format).

I recommend that you review the MAT summary page on how to acquire JVM heap dump via various JVM & OS combinations.

Heap dump analysis tools

My primary recommended tool for opening and analyzing a JVM heap dump is Eclipse Memory Analyzer (MAT). This is by far the best tool out there with contributors such as SAP & IBM. The tool provides a rich interface and advanced heap dump analysis capabilities, including a “leak suspect” report. MAT also supports both HPROF & PHD heap dump formats.

I recommend my earlier post for a quick tutorial on how to use MAT and analyze your first JVM heap dump. I have also a few heap dump analysis case studies useful for your learning process.

Final words

I really hope that you will enjoy JVM heap dump analysis as much as I do. Future articles will provide you with generic tutorials on how to analyze a JVM heap dump and where to start. Please feel free to provide your comments.

Read More

IBM AIX: Java process size monitoring

This article will provide you with a quick reference guide on how to calculate the Java process size memory footprint for Java VM processes running on IBM AIX 5.3+ OS.

This is a complementary post to my original article on this subject: how to monitor the Java native memory on AIX. I highly recommend this read to any individual involved in production support or development of Java applications deployed on AIX.

Why is this knowledge important?

From my perspective, basic knowledge on how the OS is managing the memory allocation of your JVM processes is very important. We often overlook this monitoring aspect and only focus on the Java heap itself.

From my experience, most Java memory related problems are observed from the Java heap itself such as garbage collection problems, leaks etc. However, I’m confident that you will face situations in the future involving native memory problems or OS memory challenges. Proper knowledge of your OS and virtual memory management is crucial for proper root causes analysis, recommendations and solutions.

AIX memory vs. pages

As you may have seen from my earlier post, the AIX Virtual Memory Manager (VMM) is responsible to manage memory requests from the system and its applications.

The actual physical memory is converted and partitioned in units called pages; allocated either in physical RAM or stored on disk until it is needed. Each page can have a size of 4 KB (small page), 64 KB (medium page) or 16 MB (large page). Typically for a 64-bit Java process you will see a mix of all of the above.

What about the topas command?

The typical reflex when supporting applications on AIX is to run the topas command, similar to Solaris top. Find below an example of output from AIX 5.3:

As you can see, the topas command is not very helpful to get a clear view on the memory utilization since it is not providing the breakdown view that we need for our analysis. It is still useful to get a rough idea of the paging space utilization which can give you a quick idea of your top "paging space" consumer processes. Same can be achieved via the ps aux command.

AIX OS command to the rescue: svmon

The AIX svmon command is by far my preferred command to deep dive into the Java process memory utilization. This is a very powerful command, similar to Solaris pmap. It allows you to monitor the current memory “pages” allocation along with each segment e.g. Java Heap vs. native heap segments. Analyzing the svmon output will allow you to calculate the memory footprint for each page type (4 KB, 64 KB, and 16 MB).

Now find below a real example which will allow you to understand how the calculation is done:

# 64-bit JVM with -Xms2048m & -Xmx2048m (2 GB Java Heap)

# Command: svmon –P <Java PID>

As you can see, the total footprint of our Java process size was found at 2.2 GB which is aligned with current Java heap settings. You should be able to easily perform the same memory footprint analysis from your AIX environment

I hope this article has helped you to understand how to calculate the Java process size on AIX OS. Please feel free to post any comment or question.

Read More

java.lang.ClassNotFoundException: How to resolve

This article is intended for Java beginners facing java.lang.ClassNotFoundException challenges. It will provide you with an overview of this common Java exception, a sample Java program to support your learning process and resolution strategies.

If you are interested on more advanced class loader related problems, I recommended that you review my article series on java.lang.NoClassDefFoundError since these Java exceptions are closely related.

java.lang.ClassNotFoundException: Overview

As per the Oracle documentation, ClassNotFoundException is thrown following the failure of a class loading call, using its string name, as per below:

• The Class.forName method
• The ClassLoader.findSystemClass method
• The ClassLoader.loadClass method

In other words, it means that one particular Java class was not found or could not be loaded at “runtime” from your application current context class loader.

This problem can be particularly confusing for Java beginners. This is why I always recommend to Java developers to learn and refine their knowledge on Java class loaders. Unless you are involved in dynamic class loading and using the Java Reflection API, chances are that the ClassNotFoundException error you are getting is not from your application code but from a referencing API. Another common problem pattern is a wrong packaging of your application code. We will get back to the resolution strategies at the end of the article.

java.lang.ClassNotFoundException: Sample Java program

* A ClassNotFoundException YouTube video is now available.

Now find below a very simple Java program which simulates the 2 most common ClassNotFoundException scenarios via Class.forName() & ClassLoader.loadClass(). Please simply copy/paste and run the program with the IDE of your choice (Eclipse IDE was used for this example).

The Java program allows you to choose between problem scenario #1 or problem scenario #2 as per below. Simply change to 1 or 2 depending of the scenario you want to study.

# Class.forName()

private static final int PROBLEM_SCENARIO = 1;

# ClassLoader.loadClass()

private static final int PROBLEM_SCENARIO = 2;

# ClassNotFoundExceptionSimulator

package org.ph.javaee.training5;

/**

 * ClassNotFoundExceptionSimulator

 * @author Pierre-Hugues Charbonneau

 *

 */

public class ClassNotFoundExceptionSimulator {

      

       private static final String CLASS_TO_LOAD = "org.ph.javaee.training5.ClassA";

       private static final int PROBLEM_SCENARIO = 1;

      

       /**

        * @param args

        */

       public static void main(String[] args) {

            

             System.out.println("java.lang.ClassNotFoundException Simulator - Training 5");

             System.out.println("Author: Pierre-Hugues Charbonneau");

             System.out.println("http://javaeesupportpatterns.blogspot.com");

            

             switch(PROBLEM_SCENARIO) {

                   

                    // Scenario #1 - Class.forName()

                    case 1:

                          

                           System.out.println("\n** Problem scenario #1: Class.forName() **\n");

                           try {

                                 Class<?> newClass = Class.forName(CLASS_TO_LOAD);

                                

                                 System.out.println("Class "+newClass+" found successfully!");

                          

                           } catch (ClassNotFoundException ex) {

                                

                                 ex.printStackTrace();

                                

                                 System.out.println("Class "+CLASS_TO_LOAD+" not found!");

                          

                           } catch (Throwable any) {                           

                                 System.out.println("Unexpected error! "+any);

                           }

                          

                           break;

                          

                    // Scenario #2 - ClassLoader.loadClass()

                    case 2:

                          

                           System.out.println("\n** Problem scenario #2: ClassLoader.loadClass() **\n");                     

                           try {

                                 ClassLoader classLoader = Thread.currentThread().getContextClassLoader();            

                                 Class<?> callerClass = classLoader.loadClass(CLASS_TO_LOAD);

                                

                                 Object newClassAInstance = callerClass.newInstance();

                                

                                 System.out.println("SUCCESS!: "+newClassAInstance);

                           } catch (ClassNotFoundException ex) {

                                

                                 ex.printStackTrace();

                                

                                 System.out.println("Class "+CLASS_TO_LOAD+" not found!");

                          

                           } catch (Throwable any) {                           

                                 System.out.println("Unexpected error! "+any);

                           }

                          

                           break;

             }

            

             System.out.println("\nSimulator done!");

       }

}

# ClassA

package org.ph.javaee.training5;

/**

 * ClassA

 * @author Pierre-Hugues Charbonneau

 *

 */

public class ClassA {

      

private final static Class<ClassA> CLAZZ = ClassA.class;

      

       static {

             System.out.println("Class loading of "+CLAZZ+" from ClassLoader '"+CLAZZ.getClassLoader()+"' in progress...");

       }

      

       public ClassA() {

             System.out.println("Creating a new instance of "+ClassA.class.getName()+"...");

            

             doSomething();

       }

      

       private void doSomething() {           

             // Nothing to do...

       }

}

If you run the program as is, you will see the output as per below for each scenario:

#Scenario 1 output (baseline)

java.lang.ClassNotFoundException Simulator - Training 5

Author: Pierre-Hugues Charbonneau

http://javaeesupportpatterns.blogspot.com

** Problem scenario #1: Class.forName() **

Class loading of class org.ph.javaee.training5.ClassA from ClassLoader 'sun.misc.Launcher$AppClassLoader@bfbdb0' in progress...

Class class org.ph.javaee.training5.ClassA found successfully!

Simulator done!

#Scenario 2 output (baseline)

java.lang.ClassNotFoundException Simulator - Training 5

Author: Pierre-Hugues Charbonneau

http://javaeesupportpatterns.blogspot.com

** Problem scenario #2: ClassLoader.loadClass() **

Class loading of class org.ph.javaee.training5.ClassA from ClassLoader 'sun.misc.Launcher$AppClassLoader@2a340e' in progress...

Creating a new instance of org.ph.javaee.training5.ClassA...

SUCCESS!: org.ph.javaee.training5.ClassA@6eb38a

Simulator done!

For the “baseline” run, the Java program is able to load ClassA successfully.

Now let’s voluntary change the full name of ClassA and re-run the program for each scenario. The following output can be observed:

#ClassA changed to ClassB

private static final String CLASS_TO_LOAD = "org.ph.javaee.training5.ClassB";

#Scenario 1 output (problem replication)

java.lang.ClassNotFoundException Simulator - Training 5

Author: Pierre-Hugues Charbonneau

http://javaeesupportpatterns.blogspot.com

** Problem scenario #1: Class.forName() **

java.lang.ClassNotFoundException: org.ph.javaee.training5.ClassB

       at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

       at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

       at java.security.AccessController.doPrivileged(Native Method)

       at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:423)

       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

       at java.lang.Class.forName0(Native Method)

       at java.lang.Class.forName(Class.java:186)

       at org.ph.javaee.training5.ClassNotFoundExceptionSimulator.main(ClassNotFoundExceptionSimulator.java:29)

Class org.ph.javaee.training5.ClassB not found!

Simulator done!

#Scenario 2 output (problem replication)

java.lang.ClassNotFoundException Simulator - Training 5

Author: Pierre-Hugues Charbonneau

http://javaeesupportpatterns.blogspot.com

** Problem scenario #2: ClassLoader.loadClass() **

java.lang.ClassNotFoundException: org.ph.javaee.training5.ClassB

       at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

       at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

       at java.security.AccessController.doPrivileged(Native Method)

       at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:423)

       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

       at org.ph.javaee.training5.ClassNotFoundExceptionSimulator.main(ClassNotFoundExceptionSimulator.java:51)

Class org.ph.javaee.training5.ClassB not found!

Simulator done!

What happened? Well since we changed the full class name to org.ph.javaee.training5.ClassB, such class was not found at runtime (does not exist), causing both Class.forName() and ClassLoader.loadClass() calls to fail.

You can also replicate this problem by packaging each class of this program to its own JAR file and then omit the jar file containing ClassA.class from the main class path  Please try this and see the results for yourself…(hint: NoClassDefFoundError)

Now let’s jump to the resolution strategies.

java.lang.ClassNotFoundException: Resolution strategies

Now that you understand this problem, it is now time to resolve it. Resolution can be fairly simple or very complex depending of the root cause.

• Don’t jump on complex root causes too quickly, rule out the simplest causes first.
• First review the java.lang.ClassNotFoundException stack trace as per the above and determine which Java class was not loaded properly at runtime e.g. application code, third party API, Java EE container itself etc.
• Identify the caller e.g. Java class you see from the stack trace just before the Class.forName() or ClassLoader.loadClass() calls. This will help you understand if your application code is at fault vs. a third party API.
• Determine if your application code is not packaged properly e.g. missing JAR file(s) from your classpath
• If the missing Java class is not from your application code, then identify if it belongs to a third party API you are using as per of your Java application. Once you identify it, you will need to add the missing JAR file(s) to your runtime classpath or web application WAR/EAR file.
• If still struggling after multiple resolution attempts, this could means a more complex class loader hierarchy problem. In this case, please review my NoClassDefFoundError article series for more examples and resolution strategies

I hope this article has helped you to understand and revisit this common Java exception.

Please feel free to post any comment or question if you are still struggling with your java.lang.ClassNotFoundException problem.

Read More