/ GC overhead limit exceeded - Problem and analysis approach ~ Java EE Support Patterns

8.22.2011

GC overhead limit exceeded - Problem and analysis approach

This short article will provide you with a description of this new JVM 1.6 HotSpot OutOfMemoryError error message and how you should attack this problem until its resolution.

You can also refer to this post for a real case study on a Java Heap problem (OutOfMemoryError: GC overhead limit exceeded) affecting a JBoss production system.

Please also feel free to post any comment or question if you need help with your problem.

java.lang.OutOfMemoryError: GC overhead limit exceeded – what is it?

Everyone involved in Java EE production support is familiar with OutOfMemoryError problems since they are one of the most common problem type you can face. However, if your environment recently upgraded to Java HotSpot 1.6 VM, you may have observed this error message in the logs: java.lang.OutOfMemoryError: GC overhead limit exceeded.

GC overhead limit exceeded is a new policy that was added by default for the Java HotSpot VM 1.6 only. It basically allows the VM to detect potential OutOfMemoryError conditions earlier and before it runs out of Java Heap space; allowing the JVM to abort the current Thread(s) processing with this OOM error.

The policy criterias are based on the elapsed time and frequency of your VM GC collections e.g. GC elapsed time too high, too many Full GC iterations or too much time spent in GC can trigger this error.

The official Sun statement is a par below:
The parallel / concurrent collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line.

Is it useful for Java EE production systems?

I have found on most of my problem cases that this new policy is useful at some level since it is preventing a full JVM hang and allowing you to take some actions such as data collection, JVM Heap Dump, JVM Thread Dump etc. before the whole JVM becomes unresponsive.

But don’t expect this new feature to fix your Java Heap problem, it is meant to prevent a full JVM hang and to abort some big memory allocation etc. you must still perform your own analysis and due diligence.

Is there any scenario where it can cause more harm than good?

Yes, Java applications dealing with large memory allocations / chunks could see much more frequent OOM due to GC overhead limit exceeded. Some applications dealing with a long GC elapsed time but healthy overall memory usage could also be affected.

In the above scenarios, you may want to consider turning OFF this policy and see if it’s helping your environment stability.

java.lang.OutOfMemoryError: GC overhead limit exceeded – can I disable it?

Yes, you can disable this default policy by simply adding this parameter at your JVM start-up:

-XX:-UseGCOverheadLimit

Please keep in mind that this error is very likely to be a symptom of a JVM Heap / tuning problem so my recommendation to you is always to focus on the root cause as opposed to the symptom.

java.lang.OutOfMemoryError: GC overhead limit exceeded – how can I fix it?

You should not worry too much about the GC overhead limit error itself since it’s very likely just a symptom / hint. What you must focus on is on your potential Java Heap problem(s) such as Java Heap leak,  improper Java Heap tuning etc. Find below a list of high level steps to troubleshoot further:

·         If not done already, enabled verbose GC >> -verbose:gc
·         Analyze the verbose GC output and determine the memory footprint of the Java Heap; including the ratio of Young Gen vs. Old Gen.. Having an old gen footprint too high will lead to too many frequent Full GC and ultimately to the OOM :  GC overhead limit exceeded
·         Analyze the verbose GC output or use a tool like JConsole to determine if your Java Heap is leaking over time. This can be observed via monitoring of the HotSpot old gen space.
·         Look at your young Gen requirement as well, if you application generates a lot of short live objects then your Java Heap space must be big enough in order for the VM to allocate a bigger Young Gen space
·         If facing a Java Heap leak and / or if you have concern on your Old Gen footprint then add the following parameter to your start-up JVM arguments: -XX:+HeapDumpOnOutOfMemoryError . This will generate a Heap Dump (hprof format) on OOM event that you can analyze using a tool like Memory Analyzer or JHat.

16 comments:

To disable shouldn't you add "-XX:-UseGCOverheadLimit" to JVM command, instead of "-XX:+UseGCOverheadLimit"?

Thanks anonymous to point that out. You are correct. There was a typo that I just corrected and yes -XX:-UseGCOverheadLimit with a "minus" sign is the proper way to disable it.

Regards,
P-H

[...] if more than 98% of the total time is spent in garbage collection [...]

Do you know what is meant with 98% of total time? Does that mean the GC is active for 98% of wall clock time. Or does it mean that 98% of the CPU time of the JVM is spend in GC?

Hi Anonymous,

This normally means the JVM time not CPU. Obviously a JVM spending so much time in GC will be consuming a large amount of CPU.

A healthy JVM should spend minimal time in GC and most of its time processing concurrent requests. Increased Full GC means increased CPU (especially for concurrent collector) but more importantly increased contention within JVM and performance degradation.

From my experience, this error is a symptom of either JVM tuning problem (improper Java Heap size etc.), Java Heap leak or capacity problem (too much load over a single JVM process etc.).

The error allows you a last chance to take some action and/or data gathering before the JVM goes in full OutOfMemoryError & hang state.

Regards,
P-H

My application front end is not loadinf because of this error in logs , plz suggest how and what help i need from sysadmin .

This is the rerror in logs

ErrorMsg: AwE-5015 Database connection error (1/20/13 12:03 PM)
Details: jdbc:oracle:thin:@FTDCDSSDB013.FTDC.CUMMINS.COM:1527:edwawx1
java.lang.OutOfMemoryError: GC overhead limit exceeded

ErrorMsg: AwE-5001 Database Query Error (1/20/13 12:03 PM)
Details: getConnection
null
java.sql.SQLException
at com.appworx.server.sql.AxConnectionPool.A(AxConnectionPool.java:398)
at com.appworx.server.sql.DBAccess.getConnection(DBAccess.java:1502)
at com.appworx.server.sql.CommonDB.A(CommonDB.java:564)
at com.appworx.server.sql.CommonDB.A(CommonDB.java:540)
at com.appworx.server.sql.CommonDB.getSqlData(CommonDB.java:182)
at com.appworx.server.data.Master.H(Master.java:749)
at com.appworx.server.data.Master.D(Master.java:90)
at com.appworx.server.data.Master$1.run(Master.java:709)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Hi Randhir,

Can you please describe what your database query does and how many data rows it is trying to retrieve from the database?

If too much data is retrieved from the database vs your Java Heap capacity, this will cause the Java heap space to grow very quickly and resulting in this Exception from the HotSpot VM.

Please also share your JVM start-up arguments such as -Xms & -xmx.

One more question, is this problem occuring when trying just one request or after a certain period / days of operation?

Regards,
P-H

UseGCOverheadLimit
The official Sun statement is a par below for:...This feature is designed to prevent applications from running for an extended period of time...

Any idea what exactly a JVM does besides reporting logging an error
OutOfMemoryError: GC overhead limit exceeded

The above sun documentation suggest the feature is designed to prevent application from running for an extended period of time, what does that mean does JVM kill a particular thread or transaction that is responsible for using large chunk of memory? I doubt it as I don't see any message suggesting that JVM is killing thread:N due to the above issue or I don't see any drop in memory usage. In my case sometimes i notice things becomes normal after some time and on occasion this error is followed by real OOM error

Hi Anonymous,

What this feature simply does is thrown a preventive OOM before the actual JVM is in full hang & depletion status. It it thrown when it detects the JVM spends too much time in garbage collection.

In practice, it will only allows you to take some actions such as generating extra data points from your monitoring tool, thread dump etc.

If you are getting this error, it is a clear symptom that you are dealing with a Java Heap memory problem such as:

- Memory leak
- Sudden event / high memory thread consumers
- Improper Java Heap size tuning or GC policy etc.

I suggest my other post so you can actually see how the JVM native implementation is handling this logic.

http://javaeesupportpatterns.blogspot.com/2012/01/gc-overhead-limit-exceeded-understand.html

Regards,
P-H

Please can anyone provide me suggestion regarding an error that i am facing right now. I am trying to execute a JAVA application in a JPPF grid. Initially, the application tends to give correct results. But after some time i get the following error:
Exception in thread "Receiver@192.168.1.36" java.lang.OutOfMemoryError: GC overhead limit exceeded

Hi Curious,

The problem very often indicates that your application is using too much Java heap memory vs. allocated capacity, causing too frequent garbage collections. Did you assess the utilization of the Java heap and health of the garbage collection?

Please share your JVM start-up arguments such as -Xms and -Xmx

P-H

Hi Admin,
Sometime our application become non responsive, it does not respond to user query. When we enquire in the log file we found the that application has thrown below exception and after restarting the tomcat it started behaving normal :

2014-04-06 06:44:21,041 [Multicast keep-alive Heartbeat Receiver thread-517] ERROR net.sf.ehcache.distribution.MulticastRMICacheManagerPeerProvider- Unable to lookup remote cache peer for //130.29.28.40:59890/oracle.acomp31.cos.agilent.com.1521.ciapvw1.plat.com.unica.manager.security.datamodel.impl.RoleImpl.roleRoleMap. Cause was not due to an IOException or NotBoundException which will occur in normal operation: GC overhead limit exceeded

Is there any workaround to overcome this problem.

Hi Pradeep,

It is very likely that your Java heap is running very close to out of memory. Please check any monitoring tool that your application is using and look at the Java heap history. I would recommend you to enable verbose:Gc.

http://javaeesupportpatterns.blogspot.com/2011/10/verbosegc-output-tutorial-java-7.html

The first step is really to get a good diagnostic of your Java heap.

Thanks.
P-H

even in 64 bit machines is this required to define??

Hi Imdath,

Yes, even on 64-bit JVM, you can have this problem. Again, before disabling the GC overhead detection, please ensure that your JVM and Java Heap is healthy and not facing with other issues such as memory leaks.

Thanks.
P-H

Post a Comment