This
article will provide the complete root cause analysis details and resolution of
a Java heap memory leak (Apache OpenJPA leak) affecting an Oracle Weblogic
server 10.0 production environment.
This post
will also demonstrate the importance to follow the Java Persistence API best
practices when managing the javax.persistence.EntityManagerFactory lifecycle.
Environment specifications
- Java EE server:
Oracle Weblogic Portal 10.0
- OS:
Solaris 10
- JDK:
Oracle/Sun HotSpot JVM 1.5 32-bit @2 GB capacity
- Java Persistence API: Apache
OpenJPA 1.0.x (JPA 1.0 specifications)
- RDBMS:
Oracle 10g
- Platform type: Web
Portal
Troubleshooting tools
- Quest Foglight
for Java (Java heap monitoring)
- MAT (Java heap dump analysis)
Problem description & observations
The
problem was initially reported by our Weblogic production support team
following production outages. An initial root cause analysis exercise did
reveal the following facts and observations:
- Production outages were observed on regular basis
after ~2 weeks of traffic.
- The failures were due to Java heap (OldGen) depletion
e.g. OutOfMemoryError: Java heap space error found in the Weblogic logs.
- A Java heap memory leak was confirmed after
reviewing the Java heap OldGen space utilization over time from Foglight
monitoring tool along with the Java verbose GC historical data.
Following
the discovery of the above problems, the decision was taken to move to the next
phase of the RCA and perform a JVM heap dump analysis of the affected Weblogic (JVM)
instances.
JVM heap dump analysis
In order
to generate a JVM heap dump, the supported team did use the HotSpot 1.5 jmap
utility which generated a heap dump file (heap.bin) of about ~1.5 GB. The heap
dump file was then analyzed using the Eclipse Memory Analyzer Tool. Now let’s
review the heap dump analysis so we can understand the source of the OldGen
memory leak.
MAT
provides an initial Leak Suspects
report which can be very useful to highlight your high memory contributors. For
our problem case, MAT was able to identify a leak suspect contributing to almost
600 MB or 40% of the total OldGen space capacity.
At this
point we found one instance of java.util.LinkedList using almost 600 MB of memory
and loaded to one of our application parent class loader (@ 0x7e12b708). The
next step was to understand the leaking objects along with the source of
retention.
MAT allows
you to inspect any class loader instance of your application, providing you with
capabilities to inspect the loaded classes & instances. Simply search for
the desired object by providing the address e.g. 0x7e12b708 and then inspect
the loaded classes & instances by selecting List Objects > with outgoing
references.
As you can
see from the above snapshot, the analysis was quite revealing. What we found
was one instance of org.apache.openjpa.enhance.PCRegistry at the source of the
memory retention; more precisely the culprit was the _listeners field implemented as a LinkedList.
For your
reference, the Apache OpenJPA PCRegistry is used internally to track the registered
persistence-capable classes. Find below a snippet of the PCRegistry source code
from Apache OpenJPA version 1.0.4 exposing the _listeners field.
/**
*
Tracks registered persistence-capable classes.
*
* @since 0.4.0
* @author Abe White
*/
public class PCRegistry {
// DO NOT ADD ADDITIONAL DEPENDENCIES TO THIS
CLASS
private static final Localizer
_loc = Localizer.forPackage
(PCRegistry.class);
// map of pc classes to meta structs;
weak so the VM can GC classes
private static final Map _metas = new ConcurrentReferenceHashMap
(ReferenceMap.WEAK, ReferenceMap.HARD);
// register class listeners
private static final Collection
_listeners = new LinkedList();
……………………………………………………………………………………
Now the
question is why is the memory footprint of this internal data structure so big
and potentially leaking over time? The next step was to deep dive into the
_listeners LinkedLink instance in order to review the leaking objects.
We finally
found that the leaking objects were actually the JDBC & SQL mapping
definitions (metadata) used by our application in order to execute various
queries against our Oracle database. A review of the JPA specifications, OpenJPA
documentation and source did confirm that the root cause was associated with a
wrong usage of the javax.persistence.EntityManagerFactory
such of lack of closure of a newly created EntityManagerFactory instance.
If you
look closely at the above code snapshot, you will realize that the close() method
is indeed responsible to cleanup any recently used metadata repository instance.
It did also raise another concern, why are we creating such Factory instances
over and over…
The next
step of the investigation was to perform a code walkthrough of our application
code, especially around the life cycle management of the JPA EntityManagerFactory
and EntityManager objects.
Root cause and solution
A code
walkthrough of the application code did reveal that the application was creating
a new instance of EntityManagerFactory on each single request and not closing
it properly.
public class Application
{
@Resource
private UserTransaction utx = null;
// Initialized on each application request and not closed!
@PersistenceUnit(unitName
= "UnitName")
private EntityManagerFactory emf = Persistence.createEntityManagerFactory("PersistenceUnit");
public EntityManager getEntityManager() {
return this.emf.createEntityManager();
}
public void businessMethod() {
// Create a new EntityManager instance via from
the newly created EntityManagerFactory instance
// Do something...
// Close the EntityManager instance
}
}
This code
defect and improver use of JPA EntityManagerFactory was causing a leak or
accumulation of metadata repository instances within the OpenJPA _listeners
data structure demonstrated from the earlier JVM heap dump analysis.
The solution
of the problem was to centralize the management & life cycle of the thread
safe javax.persistence.EntityManagerFactory via the Singleton pattern. The
final solution was implemented as per below:
- Create and maintain only one static instance of javax.persistence.EntityManagerFactory
per application class loader and implemented via the Singleton Pattern.
- Create and dispose new instances of EntityManager for
each application request.
Following
the implementation of the solution to our production environment, no more Java
heap OldGen memory leak is observed.
Please
feel free to provide your comments and share your experience on the same.
