A Java memory problem such as java.lang.OutOfMemoryError: PermGen space is one of the most frequent and complex problems a Java EE application support person can face with a production system. This article is part #1 of a series of posts that will focus on a particular OOM flavour: PermGen space depletion of a Java HotSpot VM.
Part #1 primary objective is to revisit the fundamentals of the permanent generation space and to teach you how to identify a particular pattern of PermGen space problem and possible causes of PermGen memory leak.
Please refer to HotSpot PermGen space for a high level overview of this special Java HotSpot VM memory storage.
A PermGen Troubleshooting Guide video is also available from our YouTube channel.
A PermGen Troubleshooting Guide video is also available from our YouTube channel.
java.lang.OutOfMemoryError: PermGen space patterns
Find below some of the most common patterns of OutOfMemoryError due to the depletion of the PermGen space.
Pattern
|
Symptoms
|
Possible root cause scenarios
|
Resolution
|
OOM observed during or after a migration of a Java EE server to newer version
|
- OOM may be observed on server start-up at deployment time
- OOM may be observed very shortly after server start-up and after 1 or 2+ hours of production traffic
|
- Higher PermGen capacity is often required due to increased Java EE server vendor code and libraries
|
- Increase your PermGen space capacity via
-XX:MaxPermSize
|
OOM observed after a certain period of time
|
- OOM observed after a longer but consistent period of time (days)
- PermGen space monitoring will show hourly or daily increase during your application business hours
|
- There are many possible causes of PermGen space memory leak. The most common is a class loader leak: increasing number of Class objects overtime
- Improper JVM arguments like usage of the Xnoclassgc flag (turn OFF Class garbage collection)
|
- Review your JVM HotSpot start-up arguments for any obvious problem like Xnoclassgc flag
- Analyse the JVM HotSpot Heap Dump as it can provides some hints on the source of a class loader leak
- Investigate any third party API you are using for any potential class loader leak defect
- Investigate your application code for any improper use of Reflection API and / or dynamic class loading
|
OOM observed following a redeploy of your application code (EAR, WAR files...)
|
- OOM may be observed during or shortly after your application redeploy process
|
- Unloading and reloading of your application code can lead to PermGen leak (class loader leak) and deplete your PermGen space fairly quickly
|
- Open a ticket with your Java EE vendor for any known class loader leak issue
- Shutdown and restart your server (JVM) post deployment to cleanup any class loader leak
|
Conclusion
I hope this short article has helped you understand the role of the JVM HotSpot permanent generation space and recognize some common OutOfMemoryError patterns.
The part #2 will focus on the diagnostic and deep dive analysis of your PermGen space problem. You can also refer to OutOfMemoryError PermGen space for a real case study on that problem.
3 comments:
Hi PH ,
Thanks for the article .
COuld you please let me know if there is similar kind of article for the OOM in Java Heap space and Native space.
Hi, PH, thanks for this write-up.
We have an issue that is probably causing PermGen OutOfMemoryError and I was wondering if you have had any experienced what I am going explain hereafter.
While analyzing a heap dump using Yourkit we have observed many instances of DelegatingClassLoader created under application running in JBoss environemnt.
It is allegedly used for some sort of reflection optimization mechanism according to the url at http://www-01.ibm.com/support/docview.wss?uid=swg21566549.
Becuase this is what seems to be eating up our memory allocated for PermGen, as the number of the DelegatingClassLoader is gradually increased and ends up with OutOfMemoryError, we have tried setting sun.reflect.inflationThreshold = 0 as suggested thereby.
However it does not seem to be working as we are still seeing DelegatingClassLoaders getting generated.
We also scrutinized the source code corresponds to where the property is set and the value is retrieved.
http://javasourcecode.org/html/open-source/jdk/jdk-6u23/sun/reflect/ReflectionFactory.java.html
and
http://javasourcecode.org/html/open-source/jdk/jdk-6u23/sun/reflect/NativeMethodAccessorImpl.java.html
Looks to me that specifying 0 will not have any effect on suppressing the generation of the DelegatingClassLoaders. The value apparently needs to something very big and
Integer.MAX_VALUE is something we want to use in order to turn that feature off forever.
Have you by any chance had struggled with the same issue? We are now looking for some answers by professionals that can help us determine what to do with this issue.
Thanks.
Hi Haruhiko and thanks for posting your comments,
You problem is showing symptoms of PermGen memory leak; unless you get OOM under peak load only which would indicate PermGen capacity issue vs. slowly growing leak.
PermGen memory leak is not an uncommon problem and I have seen many occurrences when working with my IT clients over the last years. The most common problem is offending code (application or Java EE container itself) not releasing the ClassLoader instances and/or objets as a result of Reflection calls.
The IBM article describes one possible source of the problem but I would first recommend that you focus also on other common causes.
If you need a second eye to look at your generated Heap Dump then please feel free to contact me at phcharbonneau@hotmail.com
That being said find below my recommendations:
- Confirm if PermGen space depletion (OOM) is triggered as a result of slow memory leak vs. peak load e.g. too many ClassLoader instances created at the same time vs. PermGen capacity
- Review your HotSpot JVM args and make sure -Xnoclassgc is not enabled (disable Class garbage collection)
- Perform a deeper dive Heap Dump analysis and identify the code creating/referencing the leaking ClassLoader instances
Regards,
P-H
Post a Comment