WAS频繁内存溢出问题求助
8台应用服务器，系统为 Linux x64( Redhat5.7) 每台服务器16G内存，每台服务器WAS部署为4节点集群，每节点JVM内存分配1024-2048.
web服务器使用负载均衡RAIDWARE代替。近期频繁随机出现内存溢出，随机性某一台服务器会产生javacore,headdump文件。
分析了多天的JAVACORE和heapdump日志，报错信息基本都一样，但是却看不出问题出在哪里
JAVACORE信息
***WARNING*** Java heap is almost exhausted : 12% free Java heap Please enable verbosegc trace and use IBM Pattern Modeling and Analysis Tool(http://www.alphaworks.ibm.com/tech/pmat) to analyze garbage collection activities. If heapdumps are generated at the same time, please use IBM HeapAnalyzer(http://www.alphaworks.ibm.com/tech/heapanalyzer) to analyze Java heap.
File name : G:error235JAVAjavacore.20140128.111440.18991.0005.txt
Cause of thread dump : Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Java heap space" received
Date: 2014/01/28 at 11:15:16
Process ID : Not available
Operating System : Linux 2.6.18-274.el5
Processor Architecture : amd64
Number of Processors : 8
Java version : JRE 1.6.0 Linux amd64-64 build jvmxa6460sr15-20131016_170922 (pxa6460sr15ifix-20131203_01(SR15+IX90131+IX90132+IV52621))
Virtual machine version : VM build 20131016_170922
Just-In-Time(JIT) compiler switch, Ahead-Of-Time (AOT) compiler switch, Compiler version : r9_20130920_46510ifx2
Garbage collector version : GC - GA24_Java6_SR15_20131016_1337_B170922_CMPRSS
Java Heap Information
Maximum Java heap size : 3076m
Initial Java heap size : 1024m
Free Java heap size: 376.61 MB
Allocated Java heap size: 3 GB
Number of loaded classes in Java heap : 25,874
Number of classloaders in Java heap : 1,861
Last Garbage Collection Detail Nursery Area Free : 0 bytes Total : 0 bytes Tenured Area Free : 398,143,152 bytes Total : 3,225,419,776 bytes 12% free Global Garbage Collector Counter : 215
heapdump显示信息
附件中有完整的javacore文件
web服务器使用负载均衡RAIDWARE代替。近期频繁随机出现内存溢出，随机性某一台服务器会产生javacore,headdump文件。
分析了多天的JAVACORE和heapdump日志，报错信息基本都一样，但是却看不出问题出在哪里
JAVACORE信息
***WARNING*** Java heap is almost exhausted : 12% free Java heap Please enable verbosegc trace and use IBM Pattern Modeling and Analysis Tool(http://www.alphaworks.ibm.com/tech/pmat) to analyze garbage collection activities. If heapdumps are generated at the same time, please use IBM HeapAnalyzer(http://www.alphaworks.ibm.com/tech/heapanalyzer) to analyze Java heap.
File name : G:error235JAVAjavacore.20140128.111440.18991.0005.txt
Cause of thread dump : Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Java heap space" received
Date: 2014/01/28 at 11:15:16
Process ID : Not available
Operating System : Linux 2.6.18-274.el5
Processor Architecture : amd64
Number of Processors : 8
Java version : JRE 1.6.0 Linux amd64-64 build jvmxa6460sr15-20131016_170922 (pxa6460sr15ifix-20131203_01(SR15+IX90131+IX90132+IV52621))
Virtual machine version : VM build 20131016_170922
Just-In-Time(JIT) compiler switch, Ahead-Of-Time (AOT) compiler switch, Compiler version : r9_20130920_46510ifx2
Garbage collector version : GC - GA24_Java6_SR15_20131016_1337_B170922_CMPRSS
Java Heap Information
Maximum Java heap size : 3076m
Initial Java heap size : 1024m
Free Java heap size: 376.61 MB
Allocated Java heap size: 3 GB
Number of loaded classes in Java heap : 25,874
Number of classloaders in Java heap : 1,861
Last Garbage Collection Detail Nursery Area Free : 0 bytes Total : 0 bytes Tenured Area Free : 398,143,152 bytes Total : 3,225,419,776 bytes 12% free Global Garbage Collector Counter : 215
heapdump显示信息
附件中有完整的javacore文件
附件：
javacore.20140128.111440.18991.0005.zip (472.44 KB)
javacore.20140128.111440.18991.0005.zip (472.44 KB)
11同行回答
貌似在com/tmri/framework/axis/util/TmriXmlUtil.multiBean2XMLUTF8(TmriXmlUtil.java:172(Compiled Code))方法中往StringBuffer中放了一个很大的字符串（超过400M）。收起
浏览1909
com/tmri/pub/bean/DrvLicense，这个应该是框架里的某个持久化的类吧，改一改这个类，在里面写点输出日志，比如返回的String的size超过800K的就输出，或者length>2000的就输出，这样就能追踪到哪块代码出了问题。
还有一个就是临时结果集，也需要注意，是较容易造成OOM的祸首的。收起
还有一个就是临时结果集，也需要注意，是较容易造成OOM的祸首的。收起
浏览2110
这是OOM，从你的图中看出有78%的内存空间被Object com/tmri/pub/bean/DrvLicense占用，需要检查应用程序哪些地方会调用生成这么多对象；
还有需要分析一下GC日志，看一下是否存在内存泄露情况，如果存在泄露对象com/tmri/pub/bean/DrvLicense就有可能是一个泄露点，可以修改代码和定时重启sever来解决和缓解该问题。收起
还有需要分析一下GC日志，看一下是否存在内存泄露情况，如果存在泄露对象com/tmri/pub/bean/DrvLicense就有可能是一个泄露点，可以修改代码和定时重启sever来解决和缓解该问题。收起
浏览2093
从以上几个信息可以看出来。通过javacore分析，线程有很多等待的线程。导致jvm崩溃的原因是内存溢出。通过分析heapdump文件。可以看出内存中存在大量的lang行的对象。占到了内存的78%。看第3和第5图。上边说明怀疑的泄漏点是com/tmri/pub/bean/DriLicense。所以建议程序员查看这个类。什么方法产生了如此多的lang行对象。这是我的个人意见。仅供参考。呵呵收起
浏览2165