ClassCastException and ClassLoader Puzzle

The following question and puzzle will test your knowledge on Java class loaders and more precisely on one of the Java language specifications. It will also help you better troubleshoot problems such as java.lang.NoClassDefFoundError.

I highly suggest that you do not look at the explanation and solution until you review the code and come up with your own explanation.

You can download the Java program source code and binaries (compiled with JDK 1.7) here. In order to run the program, simply use the following command:

<JDK 1.7 HOME>\bin\java -classpath MainProgram.jar org.ph.javaee.training9.ChildFirstCLPuzzle

** Make sure that you also download the 3 JAR files below before you run the program.

• MainProgram.jar contains the main program along with super class ProgrammingLanguage.
• ProgrammingLanguage.jar contains the super class ProgrammingLanguage.
• JavaLanguage.jar contains the implementation class JavaLanguage, which extends ProgrammingLanguage.

Question (puzzle)

Review closely the program source and packaging along with the diagram below reflecting the class loader delegation model used for this program.

Why can’t we cast (ChildFirstCLPuzzle.java @line 53) the Object javaLanguageInstance of type JavaLanguage, into ProgrammingLanguage?

...............................                   

// Finally, cast the object instance into ProgrammingLanguage

/** Question: why is the following code failing with ClassCastException given the fact JavaLanguage is indeed a ProgrammingLanguage?? **/

ProgrammingLanguage programmingLanguage = (ProgrammingLanguage)javaLanguageInstance;

............................... 

Propose a solution to allow the above cast to succeed without changing the original source code. Hint: look again at the class loader delegation model, packaging and diagram.

Answer & solution

The Java program is attempting to demonstrate a Java language specification rule related to class loaders: two classes loaded by different class loaders are considered to be distinct and hence incompatible.

If you review closely the program source, packaging and diagram, you will realize the following facts:

• Our main program is loaded to the parent class loader e.g. $AppClassLoader.
• The super class ProgrammingLanguage is also loaded to the parent class loader since it is referenced by our main program at line 53.
• The implementation class JavaLanguage is loaded to our child class loader e.g. ChildFirstClassLoader which is following a “child first” delegation model.
• Finally, the super class ProgrammingLanguage is also loaded to our child class loader.

The key point to understand is that the super class is loaded by 2 different class loaders. As per Java language specification, two classes loaded by different class loaders are considered to be distinct and hence incompatible. This means that ProgrammingLanguage loaded from the “child firs” class loader is different and not compatible with ProgrammingLanguage loaded from the parent classloader. This is why the cast attempted at line 53 failed with the error below:

ChildFirstCLPuzzle execution failed with ERROR: java.lang.ClassCastException: org.ph.javaee.training9.JavaLanguage cannot be cast to org.ph.javaee.training9.ProgrammingLanguage

Now how can we fix this problem without actually changing the source code? Well please keep in mind that we are using a “child first” delegation model here. This is why we end up with 2 versions of the same ProgrammingLanguage class. The solution can be visualized as per below.

In order to fix this problem, we can simply ensure that we have only one version of ProgrammingLanguage loaded. Since our main program is referencing ProgrammingLanguage directly, the solution is to remove the ProgrammingLanguage.jar file from the child class loader. This will force the child class loader to look for the class from the parent class loader, problem solved! In order to test the solution, simply remove the ProgrammingLanguage.jar from your testing folder and re-run the program.

I hope you appreciated this puzzle related to “child first” class loader delegation model and class loader rules. This understanding is especially important when you are dealing with complex Java EE deployments involving many class loaders, exposing you to this type of problems at runtime.

Please do not hesitate to post any comment or question on this puzzle.

Read More

Java 8: From PermGen to Metaspace

As you may be aware, the JDK 8 Early Access is now available for download. This allows Java developers to experiment with some of the new language and runtime features of Java 8.

One of these features is the complete removal of the Permanent Generation (PermGen) space which has been announced by Oracle since the release of JDK 7. Interned strings, for example, have already been removed from the PermGen space since JDK 7. The JDK 8 release finalizes its decommissioning.

This article will share the information that we found so far on the PermGen successor: Metaspace. We will also compare the runtime behavior of the HotSpot 1.7 vs. HotSpot 1.8 (b75) when executing a Java program “leaking” class metadata objects.

The final specifications, tuning flags and documentation around Metaspace should be available once Java 8 is officially released.

Metaspace: A new memory space is born

The JDK 8 HotSpot JVM is now using native memory for the representation of class metadata and is called Metaspace; similar to the Oracle JRockit and IBM JVM's.

The good news is that it means no more java.lang.OutOfMemoryError: PermGen space problems and no need for you to tune and monitor this memory space anymore…not so fast. While this change is invisible by default, we will show you next that you will still need to worry about the class metadata memory footprint. Please also keep in mind that this new feature does not magically eliminate class and classloader memory leaks. You will need to track down these problems using a different approach and by learning the new naming convention.

I recommend that you read the PermGen removal summary and comments from Jon on this subject. 

In summary:

PermGen space situation

• This memory space is completely removed.
• The PermSize and MaxPermSize JVM arguments are ignored and a warning is issued if present at start-up.

Metaspace memory allocation model

• Most allocations for the class metadata are now allocated out of native memory.
• The klasses that were used to describe class metadata have been removed.

Metaspace capacity

• By default class metadata allocation is limited by the amount of available native memory (capacity will of course depend if you use a 32-bit JVM vs. 64-bit along with OS virtual memory availability).
• A new flag is available (MaxMetaspaceSize), allowing you to limit the amount of native memory used for class metadata. If you don’t specify this flag, the Metaspace will dynamically re-size depending of the application demand at runtime.

Metaspace garbage collection

• Garbage collection of the dead classes and classloaders is triggered once the class metadata usage reaches the “MaxMetaspaceSize”.
• Proper monitoring & tuning of the Metaspace will obviously be required in order to limit the frequency or delay of such garbage collections. Excessive Metaspace garbage collections may be a symptom of classes, classloaders memory leak or inadequate sizing for your application.

Java heap space impact

• Some miscellaneous data has been moved to the Java heap space. This means you may observe an increase of the Java heap space following a future JDK 8 upgrade.

Metaspace monitoring

• Metaspace usage is available from the HotSpot 1.8 verbose GC log output.
• Jstat & JVisualVM have not been updated at this point based on our testing with b75 and the old PermGen space references are still present.

Enough theory now, let’s see this new memory space in action via our leaking Java program…

PermGen vs. Metaspace runtime comparison

In order to better understand the runtime behavior of the new Metaspace memory space, we created a class metadata leaking Java program. You can download the source here.

The following scenarios will be tested:

• Run the Java program using JDK 1.7 in order to monitor & deplete the PermGen memory space set at 128 MB.
• Run the Java program using JDK 1.8 (b75) in order to monitor the dynamic increase and garbage collection of the new Metaspace memory space.
• Run the Java program using JDK 1.8 (b75) in order to simulate the depletion of the Metaspace by setting the MaxMetaspaceSize value at 128 MB.

JDK 1.7 @64-bit – PermGen depletion

• Java program with 50K configured iterations
• Java heap space of 1024 MB
• Java PermGen space of 128 MB (-XX:MaxPermSize=128m)

As you can see form JVisualVM, the PermGen depletion was reached after loading about 30K+ classes. We can also see this depletion from the program and GC output.

Class metadata leak simulator

Author: Pierre-Hugues Charbonneau

http://javaeesupportpatterns.blogspot.com

ERROR: java.lang.OutOfMemoryError: PermGen space

Now let’s execute the program using the HotSpot JDK 1.8 JRE.

JDK 1.8 @64-bit – Metaspace dynamic re-size

• Java program with 50K configured iterations
• Java heap space of 1024 MB
• Java Metaspace space: unbounded (default)

As you can see from the verbose GC output, the JVM Metaspace did expand dynamically from 20 MB up to 328 MB of reserved native memory in order to honor the increased class metadata memory footprint from our Java program. We could also observe garbage collection events in the attempt by the JVM to destroy any dead class or classloader object. Since our Java program is leaking, the JVM had no choice but to dynamically expand the Metaspace memory space.

The program was able to run its 50K of iterations with no OOM event and loaded 50K+ Classes.

Let's move to our last testing scenario.

JDK 1.8 @64-bit – Metaspace depletion

• Java program with 50K configured iterations
• Java heap space of 1024 MB
• Java Metaspace space: 128 MB (-XX:MaxMetaspaceSize=128m)

As you can see form JVisualVM, the Metaspace depletion was reached after loading about 30K+ classes; very similar to the run with the JDK 1.7. We can also see this from the program and GC output. Another interesting observation is that the native memory footprint reserved was twice as much as the maximum size specified. This may indicate some opportunities to fine tune the Metaspace re-size policy, if possible, in order to avoid native memory waste.

Now find below the Exception we got from the Java program output.

Class metadata leak simulator

Author: Pierre-Hugues Charbonneau

http://javaeesupportpatterns.blogspot.com

ERROR: java.lang.OutOfMemoryError: Metadata space

Done!

As expected, capping the Metaspace at 128 MB like we did for the baseline run with JDK 1.7 did not allow us to complete the 50K iterations of our program. A new OOM error was thrown by the JVM. The above OOM event was thrown by the JVM from the Metaspace following a memory allocation failure.

#metaspace.cpp

Final words

I hope you appreciated this early analysis and experiment with the new Java 8 Metaspace. The current observations definitely indicate that proper monitoring & tuning will be required in order to stay away from problems such as excessive Metaspace GC or OOM conditions triggered from our last testing scenario. Future articles may include performance comparisons in order to identify potential performance improvements associated with this new feature.

Please feel free to provide any comment.

Read More

Java concurrency: the hidden thread deadlocks

Most Java programmers are familiar with the Java thread deadlock concept. It essentially involves 2 threads waiting forever for each other. This condition is often the result of flat (synchronized) or ReentrantLock (read or write) lock-ordering problems.

Found one Java-level deadlock:

=============================

"pool-1-thread-2":

  waiting to lock monitor 0x0237ada4 (object 0x272200e8, a java.lang.Object),

  which is held by "pool-1-thread-1"

"pool-1-thread-1":

  waiting to lock monitor 0x0237aa64 (object 0x272200f0, a java.lang.Object),

  which is held by "pool-1-thread-2"

The good news is that the HotSpot JVM is always able to detect this condition for you…or is it?

A recent thread deadlock problem affecting an Oracle Service Bus production environment has forced us to revisit this classic problem and identify the existence of “hidden” deadlock situations.

This article will demonstrate and replicate via a simple Java program a very special lock-ordering deadlock condition which is not detected by the latest HotSpot JVM 1.7. You will also find a video at the end of the article explaining you the Java sample program and the troubleshooting approach used.

The crime scene

I usually like to compare major Java concurrency problems to a crime scene where you play the lead investigator role. In this context, the “crime” is an actual production outage of your client IT environment. Your job is to:

• Collect all the evidences, hints & facts (thread dump, logs, business impact, load figures…)
• Interrogate the witnesses & domain experts (support team, delivery team, vendor, client…)

The next step of your investigation is to analyze the collected information and establish a potential list of one or many “suspects” along with clear proofs. Eventually, you want to narrow it down to a primary suspect or root cause. Obviously the law “innocent until proven guilty” does not apply here, exactly the opposite.

Lack of evidence can prevent you to achieve the above goal. What you will see next is that the lack of deadlock detection by the Hotspot JVM does not necessary prove that you are not dealing with this problem.

The suspect

In this troubleshooting context, the “suspect” is defined as the application or middleware code with the following problematic execution pattern.

• Usage of FLAT lock followed by the usage of ReentrantLock WRITE lock (execution path #1)
• Usage of ReentrantLock READ lock followed by the usage of FLAT lock (execution path #2)
• Concurrent execution performed by 2 Java threads but via a reversed execution order

The above lock-ordering deadlock criteria’s can be visualized as per below:

Now let’s replicate this problem via our sample Java program and look at the JVM thread dump output.

Sample Java program

This above deadlock conditions was first identified from our Oracle OSB problem case. We then re-created it via a simple Java program. You can download the entire source code of our program here.

The program is simply creating and firing 2 worker threads. Each of them execute a different execution path and attempt to acquire locks on shared objects but in different orders. We also created a deadlock detector thread for monitoring and logging purposes.

For now, find below the Java class implementing the 2 different execution paths.

package org.ph.javaee.training8;

import java.util.concurrent.locks.ReentrantReadWriteLock;

/**

 * A simple thread task representation

 * @author Pierre-Hugues Charbonneau

 *

 */

public class Task {

      

       // Object used for FLAT lock

       private final Object sharedObject = new Object();

       // ReentrantReadWriteLock used for WRITE & READ locks

       private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();

      

       /**

        *  Execution pattern #1

        */

       public void executeTask1() {

            

             // 1. Attempt to acquire a ReentrantReadWriteLock READ lock

             lock.readLock().lock();

            

             // Wait 2 seconds to simulate some work...

             try { Thread.sleep(2000);}catch (Throwable any) {}

            

             try {              

                    // 2. Attempt to acquire a Flat lock...

                    synchronized (sharedObject) {}

             }

             // Remove the READ lock

             finally {

                    lock.readLock().unlock();

             }           

            

             System.out.println("executeTask1() :: Work Done!");

       }

      

       /**

        *  Execution pattern #2

        */

       public void executeTask2() {

            

             // 1. Attempt to acquire a Flat lock

             synchronized (sharedObject) {                 

                   

                    // Wait 2 seconds to simulate some work...

                    try { Thread.sleep(2000);}catch (Throwable any) {}

                   

                    // 2. Attempt to acquire a WRITE lock                   

                    lock.writeLock().lock();

                   

                    try {

                           // Do nothing

                    }

                   

                    // Remove the WRITE lock

                    finally {

                           lock.writeLock().unlock();

                    }

             }

            

             System.out.println("executeTask2() :: Work Done!");

       }

      

       public ReentrantReadWriteLock getReentrantReadWriteLock() {

             return lock;

       }

}

As soon ad the deadlock situation was triggered, a JVM thread dump was generated using JVisualVM.

As you can see from the Java thread dump sample. The JVM did not detect this deadlock condition (e.g. no presence of Found one Java-level deadlock) but it is clear these 2 threads are in deadlock state.

Root cause: ReetrantLock READ lock behavior

The main explanation we found at this point is associated with the usage of the ReetrantLock READ lock. The read locks are normally not designed to have a notion of ownership. Since there is not a record of which thread holds a read lock, this appears to prevent the HotSpot JVM deadlock detector logic to detect deadlock involving read locks.

Some improvements were implemented since then but we can see that the JVM still cannot detect this special deadlock scenario.

Now if we replace the read lock (execution pattern #1) in our program by a write lock, the JVM will finally detect the deadlock condition but why?

Found one Java-level deadlock:

=============================

"pool-1-thread-2":

  waiting for ownable synchronizer 0x272239c0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),

  which is held by "pool-1-thread-1"

"pool-1-thread-1":

  waiting to lock monitor 0x025cad3c (object 0x272236d0, a java.lang.Object),

  which is held by "pool-1-thread-2"

Java stack information for the threads listed above:

===================================================

"pool-1-thread-2":

       at sun.misc.Unsafe.park(Native Method)

       - parking to wait for  <0x272239c0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)

       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)

       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)

       at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)

       at org.ph.javaee.training8.Task.executeTask2(Task.java:54)

       - locked <0x272236d0> (a java.lang.Object)

       at org.ph.javaee.training8.WorkerThread2.run(WorkerThread2.java:29)

       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

       at java.lang.Thread.run(Thread.java:722)

"pool-1-thread-1":

       at org.ph.javaee.training8.Task.executeTask1(Task.java:31)

       - waiting to lock <0x272236d0> (a java.lang.Object)

       at org.ph.javaee.training8.WorkerThread1.run(WorkerThread1.java:29)

       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

       at java.lang.Thread.run(Thread.java:722)

This is because write locks are tracked by the JVM similar to flat locks. This means the HotSpot JVM deadlock detector appears to be currently designed to detect:

• Deadlock on Object monitors involving FLAT locks
• Deadlock involving Locked ownable synchronizers associated with WRITE locks

The lack of read lock per-thread tracking appears to prevent deadlock detection for this scenario and significantly increase the troubleshooting complexity.

I suggest that you read Doug Lea’s comments on this whole issue since concerns were raised back in 2005 regarding the possibility to add per-thread read-hold tracking due to some potential lock overhead.

Find below my troubleshooting recommendations if you suspect a hidden deadlock condition involving read locks:

• Analyze closely the thread call stack trace, it may reveal some code potentially acquiring read locks and preventing other threads to acquire write locks.
• If you are the owner of the code, keep track of the read lock count via the usage of the lock.getReadLockCount() method

I’m looking forward for your feedback, especially from individuals with experience on this type of deadlock involving read locks.

Finally, find below a video explaining such findings via the execution and monitoring of our sample Java program.

Read More

QOTD: Java Thread vs. Java Heap Space

The following question is quite common and is related to OutOfMemoryError: unable to create new native thread problems during the JVM thread creation process and the JVM thread capacity. 

This is also a typical interview question I ask to new technical candidates (senior role). I recommend that you attempt to provide your own response before looking at the answer.

Question:

Why can’t you increase the JVM thread capacity (total # of threads) by expanding the Java heap space capacity via -Xmx?

Answer:

The Java thread creation process requires native memory to be available for the JVM process. Expanding the Java heap space via the –Xmx argument will actually reduce your Java thread capacity since this memory will be “stolen” from the native memory space.

• For a 32-bit JVM, the Java heap space is in a race with the native heap, including the thread capacity
• For a 64-bit JVM, the thread capacity will mainly depend of your OS physical & virtual memory availability along with your current OS process related tuning parameters

In order to better understand this limitation, I now propose to you the following video tutorial.

You can also download the sample Java program from the link below:
https://docs.google.com/file/d/0B6UjfNcYT7yGazg5aWxCWGtvbm8/edit

Read More

Java Verbose GC - tutorial video

This is my first tutorial video which will provide you with technical detail on how to enable and analyze the verbose:gc output data of your JVM process.

You can also download the Java sample program from the link below. Please make sure that you configure your Java runtime with a heap space of only 1 GB (-Xmx1024m).
https://docs.google.com/file/d/0B6UjfNcYT7yGenZCU3FfTHFfNnc/edit

Future videos may also include scripts in order for the non-English audience to perform proper language translation.

I’m looking forward for your feedback and suggestions on topics and video format you would like to see.

Read More