This post provides a step by step tutorial on how you can monitor the native memory utilization of an IBM Java VM using AIX OS and IBM SDK. I will first provide you with some basic background on the AIX memory manager followed by a segment breakdown of a 32 bit Java process and the key commands you must use.
This post should help you going forward in your troubleshooting of a native OutOfMemoryError problem and/or in your capacity planning analysis of the native memory footprint of your Java application.
The Native Heap can easily be monitored using the AIX svmon command. This provides you both the current memory footprint and a breakdown view of your Java process; including the segments reserved for your application Java Heap.
Native Heap should be monitored on a regular basis on production systems, including in capacity planning exercises such as load testing in order to determine any native memory problem such as memory leak.
This post should help you going forward in your troubleshooting of a native OutOfMemoryError problem and/or in your capacity planning analysis of the native memory footprint of your Java application.
Background
Most of the focus these days is spent on the Java Heap footprint analysis and tuning. We very often overlook to monitor and look for possible issues with the native memory portion of the Java process.
If you are supporting a Java EE application running on the IBM SDK and AIX, it is important to acquire this monitoring skillet before you are facing a production incident situation such as OutOfMemoryError condition.
AIX VMM high level overview
The AIX Virtual Memory Manager (VMM) is responsible to manage memory requests from the system and its applications.
Physical and virtual memory segments are transformed and partitioned in units called pages (Paging Space); allocated either in physical RAM or stored on disk until it is needed. AIX also uses virtual memory in order to address more memory than is physically available in the system.
It is important to note that each of these pages are created in 4096 byes (4KB) block segment.
AIX 5.3 – IBM 32-bit Java VM breakdown
The memory space available to a 32-bit program is broken up by AIX into 16 segments of 256 MB each. The AIX memory model being used by a program determines how those segments can be used by the program. In particular, the memory model determines how many segments are available to the native heap and how many segments are available for other purposes
The AIX memory model is controlled by the LDR_CNTRL=MAXDATA environment variable. In most modern Java EE environment, the most common is a large memory allocation such as 0XA0000000@DSA that allows the 32 bit process to use up to allocate up to 2.25G for the Java Heap.
Example for LDR_CNTRL=MAXDATA=0XA0000000@DSA
Segment #
|
Description
|
0
|
AIX Kernel only
|
1
|
Java program code only (Java executable)
|
2
|
Primordial Stack (main program thread stack)
|
3 (3-C)
|
10 segments (3-C) available for the native heap (via malloc). However, only segment #3 is initially reserved
|
4-C
|
9 segments (4-C) available for the Java Heap (via mmap)
|
D
|
Shared library code
|
E
|
Memory Mapped space (mmap / shmat)
|
F
|
Shared library data
|
Java heap vs. native heap
As you can see in the above tab. The Java heap and native heap are in a race. When using the IBM SDK, there is no parameter to explicitly control the native memory allocation. You can control your native memory capacity by controlling the Java Heap capacity (via -Xmx) along with the LDR_CNTRL environment variable.
It is important to note that you should always define increment or decrement your JVM memory arguments by 256 MB memory block to prevent reserving and losing a certain % of unused memory. A common mistake is to round the –Xmx value. Let’s say we configure 1800m as –Xmx. To satisfy this request, the IBM VM will need to allocate 8 segments of 256 MB for the mmap operation; losing almost 1 full segment. A better approach is to configure to 1792m; which is exactly 7 segments of 256 MB, leaving one full segment up for grab for the native heap.
As a general rule:
- The bigger is your Java Heap allocation via –Xmx, the smaller is your native heap capacity and vice-versa
Ex: let’s say your application requires ~1.5G of Java Heap
· Java Heap memory settings should be: 6 segments x 256 MB = 1536 MB
· The native heap capacity will be: 4 segments x 256 MB = 1024 MB
Native heap monitoring
I will now show how you can validate your configuration and monitor the native heap utilization using an example from a real AIX 5.3 environment using a 32-bit IBM SDK 1.5.
Environment specifications
· Java EE server: Weblogic Integration 9.2
· OS: AIX 5.3 TL9
· JDK: 32-bit J2RE 1.5.0 IBM J9 SR6 (-Xms1792m -Xmx1792m)
## Full segment breakdown view command
svmon -P <java process pid> -m -r –i
It is also useful to filter depending on which memory segments you are interesting in:
## Native Heap memory segments command
svmon -P <java process pid> -m -r | grep 'working storage'
## Java Heap memory segments command
svmon -P <java process pid> -m -r | grep ‘mmap’
The above snapshots are clearly showing a total capacity of 7 Java Heap segments of 256 MB for total of 1792 MB.
The current utilization native heap allocation is split over 2 segments for total of total 424 MB. This means that we still have 2 full segments available for the Native Heap, if needed.
** Please note that the Java Heap snapshot corresponds to the Java Heap capacity, not utilization. The actual real Java Heap utilization must be captured via verbose GC or other monitoring tools **
Conclusion
Native Heap should be monitored on a regular basis on production systems, including in capacity planning exercises such as load testing in order to determine any native memory problem such as memory leak.
