软件开发WASWAS宕机

was宕掉了,日志如下

NULL           ------------------------------------------------------------------------0SECTION       TITLE subcomponent dump routineNULL           ==...显示全部
NULL           ------------------------------------------------------------------------
0SECTION       TITLE subcomponent dump routine
NULL           ===============================
1TISIGINFO     Dump Event "gpf" (00002000) received
1TIDATETIME    Date:                 2013/11/13 at 06:41:20
1TIFILENAME    Javacore filename:    /home/was/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20131113.064118.6754.0003.txt
NULL           ------------------------------------------------------------------------
0SECTION       GPINFO subcomponent dump routine
NULL           ================================
2XHOSLEVEL     OS Level         : Linux 2.6.32-131.0.15.el6.x86_64
2XHCPUS        Processors -
3XHCPUARCH       Architecture   : amd64
3XHNUMCPUS       How Many       : 48
3XHNUMASUP       NUMA support enabled
NULL           
1XHEXCPCODE    J9Generic_Signal_Number: 0000000000000004
1XHEXCPCODE    Signal_Number: 000000000000000B
1XHEXCPCODE    Error_Value: 0000000000000000
1XHEXCPCODE    Signal_Code: 0000000000000001
1XHEXCPCODE    Handler1: 00007F975F69C780
1XHEXCPCODE    Handler2: 00007F975F35FF20
1XHEXCPCODE    InaccessibleAddress: 0000000000000020
NULL           
1XHEXCPMODULE  Module: /home/was/IBM/WebSphere/AppServer/java/jre/lib/amd64/compressedrefs/libj9jit24.so
1XHEXCPMODULE  Module_base_address: 00007F975E5EC000




NULL           ------------------------------------------------------------------------
0SECTION       ENVINFO subcomponent dump routine
NULL           =================================
1CIJAVAVERSION J2RE 6.0 IBM J9 2.4 Linux amd64-64 build jvmxa6460-20080816_22093
1CIVMVERSION   VM build 20080816_022093_LHdSMr
1CIJITVERSION  JIT enabled, AOT enabled - r9_20080721_1330ifx2
1CIGCVERSION   GC - 20080724_AA_CMPRSS
1CIRUNNINGAS   Running as a standalone JVM
1CICMDLINE     /home/was/IBM/WebSphere/AppServer/java/bin/java -Declipse.security -Dwas.status.socket=54843 -Dosgi.install.area=/home/was/IBM/WebSphere/AppServer -Dosgi.configuration.area=/home/was/IBM/WebSphere/AppServer/profiles/AppSrv01/configuration -Djava.awt.headless=true -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M -Xbootclasspath/p:/home/was/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/home/was/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -classpath /home/was/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/home/was/IBM/WebSphere/AppServer/properties:/home/was/IBM/WebSphere/AppServer/lib/startup.jar:/home/was/IBM/WebSphere/AppServer/lib/bootstrap.jar:/home/was/IBM/WebSphere/AppServer/lib/jsf-nls.jar:/home/was/IBM/WebSphere/AppServer/lib/lmproxy.jar:/home/was/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/home/was/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/home/was/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/home/was/IBM/WebSphere/AppServer/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -Xms50m -Xmx256m -Xcompressedrefs -Dws.ext.dirs=/home/was/IBM/WebSphere/AppServer/java/lib:/home/was/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/home/was/IBM/WebSphere/AppServer/classes:/home/was/IBM/WebSphere/AppServer/lib:/home/was/IBM/WebSphere/AppServer/installedChannels:/home/was/IBM/WebSphere/AppServer/lib/ext:/home/was/IBM/WebSphere/AppServer/web/help:/home/was/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/home/was/IBM/WebSphere/AppServer/derby -Dcom.ibm.itp.location=/home/was/IBM/WebSphere/AppServer/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/home/was/IBM/WebSphere/AppServer/profiles/AppSrv01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=/home/was/IBM/WebSphere/AppServer -Dpython.cachedir=/home/was/IBM/WebSphere/AppServer/profiles/AppSrv01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/home/was/IBM/WebSphere/AppServer/profiles/AppSrv01 -Dcom.ibm.security.jgss.debug=off -Dcom.ibm.security.krb5.Krb5Debug=off -Djava.security.auth.login.config=/home/was/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/wsjaas.conf -Djava.security.policy=/home/was/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/server.policy com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /home/was/IBM/WebSphere/AppServer/profiles/AppSrv01/config localhostCell01 localhostNode01 nodeagent
1CIJAVAHOMEDIR Java Home Dir:   /home/was/IBM/WebSphere/AppServer/java/jre
1CIJAVADLLDIR  Java DLL Dir:    /home/was/IBM/WebSphere/AppServer/java/jre/bin
1CISYSCP       Sys Classpath:   /home/was/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/amd64/compressedrefs/jclSC160/vm.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/annotation.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/beans.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/java.util.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/jndi.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/logging.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/security.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/sql.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmorb.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmorbapi.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmcfw.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/rt.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/charsets.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/resources.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmpkcs.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmcertpathfw.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmjgssfw.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmjssefw.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmsaslfw.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmjcefw.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmjgssprovider.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmjsseprovider2.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmcertpathprovider.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/ibmxmlcrypto.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/management-agent.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/xml.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/jlm.jar;/home/was/IBM/WebSphere/AppServer/java/jre/lib/javascript.jar;
1CIUSERARGS    UserArgs:




NULL           ------------------------------------------------------------------------
0SECTION       MEMINFO subcomponent dump routine
NULL           =================================
1STHEAPFREE    Bytes of Heap Space Free: 131B350
1STHEAPALLOC   Bytes of Heap Space Allocated: 3814C00
NULL      




NULL           
1STSEGTYPE     JIT Code Cache
NULL           segment  start    alloc    end       type     bytes
1STSEGMENT     00007F97588C9460 00007F962A810000 00007F962B010000 00007F962B010000  00000068 800000
1STSEGMENT     00007F97588C93A0 00007F975D2B4000 00007F975DAB4000 00007F975DAB4000  00000068 800000
NULL           
1STSEGTYPE     JIT Data Cache
NULL           segment  start    alloc    end       type     bytes
1STSEGMENT     00007F97588CDDE0 00007F975CA92010 00007F975CF1ABC8 00007F975D292010  00000048 800000
NULL           
1STGCHTYPE     GC History  
3STHSTTYPE     22:39:56:240364000 GMT j9mm.134 -   Allocation failure end: newspace=0/0 oldspace=21695152/58805248 loa=201184/587776
3STHSTTYPE     22:39:56:240362000 GMT j9mm.139 -   Reference count end: weak=11819 soft=362 phantom=807 threshold=11 maxThreshold=32
3STHSTTYPE     22:39:56:240234000 GMT j9mm.91 -   GlobalGC end: workstackoverflow=0 overflowcount=0 weakrefs=11819 soft=362 threshold=11 phantom=807 finalizers=650 newspace=0/0 oldspace=21711912/58805248 loa=201184/587776
3STHSTTYPE     22:39:56:240228000 GMT j9mm.90 -   GlobalGC collect complete
3STHSTTYPE     22:39:56:240135000 GMT j9mm.57 -   Sweep end
3STHSTTYPE     22:39:56:237601000 GMT j9mm.56 -   Sweep start
3STHSTTYPE     22:39:56:237599000 GMT j9mm.55 -   Mark end
3STHSTTYPE     22:39:56:186298000 GMT j9mm.54 -   Mark start





NULL           
NULL           ------------------------------------------------------------------------
0SECTION       LOCKS subcomponent dump routine
NULL           ===============================
NULL           
1LKPOOLINFO    Monitor pool info:
2LKPOOLTOTAL     Current total number of monitors: 154
NULL           
1LKMONPOOLDUMP Monitor Pool Dump (flat & inflated object-monitors):
2LKMONINUSE      sys_mon_t:0x00007F9758FAE8D0 infl_mon_t: 0x00007F9758FAE940:
3LKMONOBJECT       org/eclipse/osgi/framework/eventmgr/EventManager$EventThread@000000000324F978/000000000324F984:
3LKNOTIFYQ            Waiting to be notified:
3LKWAITNOTIFY            "Framework Event Dispatcher" (0x0000000012373300)
2LKMONINUSE      sys_mon_t:0x00007F968000D920 infl_mon_t: 0x00007F968000D990:
3LKMONOBJECT       org/eclipse/osgi/framework/eventmgr/EventManager$EventThread@00000000032AE868/00000000032AE874:
3LKNOTIFYQ            Waiting to be notified:
3LKWAITNOTIFY            "Start Level Event Dispatcher" (0x0000000012374300)
2LKMONINUSE      sys_mon_t:0x00007F968000E0A0 infl_mon_t: 0x00007F968000E110:
3LKMONOBJECT       org/eclipse/core/internal/jobs/WorkerPool@0000000004E8EE58/0000000004E8EE64:
3LKNOTIFYQ            Waiting to be notified:
3LKWAITNOTIFY            "Worker-0" (0x00000000124E9E00)


NULL           
NULL           ------------------------------------------------------------------------
0SECTION       THREADS subcomponent dump routine
NULL           =================================
NULL            
1XMCURTHDINFO  Current Thread Details
NULL           ----------------------
3XMTHREADINFO      "StatusCacheTimedThread:true" TID:0x00000000134D6900, j9thread_t:0x00007F95AC0C3560, state:R, prio=5
3XMTHREADINFO1            (native thread ID:0x1AE4, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE          at com/ibm/ws/management/status/NodeAgentStatusCache$StatusCacheTimedThread.run(NodeAgentStatusCache.java:812)
NULL           
1XMTHDINFO     All Thread Details
NULL           ------------------
NULL           
2XMFULLTHDDUMP Full thread dump J9 VM (J2RE 6.0 IBM J9 2.4 Linux amd64-64 build jvmxa6460-20080816_2209320080816_022093_LHdSMr, native threads):
3XMTHREADINFO      "P=682503:O=0:CT" TID:0x00000000121AD100, j9thread_t:0x00007F97580067A0, state:CW, prio=5
3XMTHREADINFO1            (native thread ID:0x1A63, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE          at java/lang/Thread.sleep(Native Method)
4XESTACKTRACE          at java/lang/Thread.sleep(Thread.java:850)
4XESTACKTRACE          at com/ibm/ws/runtime/WsServerImpl.main(WsServerImpl.java:679)
4XESTACKTRACE          at com/ibm/ws/runtime/WsServer.main(WsServer.java:59)
4XESTACKTRACE          at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
4XESTACKTRACE          at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:45(Compiled Code))
4XESTACKTRACE          at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
4XESTACKTRACE          at java/lang/reflect/Method.invoke(Method.java:599(Compiled Code))
4XESTACKTRACE          at com/ibm/wsspi/bootstrap/WSLauncher.launchMain(WSLauncher.java:213)
4XESTACKTRACE          at com/ibm/wsspi/bootstrap/WSLauncher.main(WSLauncher.java:93)
4XESTACKTRACE          at com/ibm/wsspi/bootstrap/WSLauncher.run(WSLauncher.java:74)
4XESTACKTRACE          at org/eclipse/core/internal/runtime/PlatformActivator$1.run(PlatformActivator.java:78)
4XESTACKTRACE          at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.runApplication(EclipseAppLauncher.java:92)
4XESTACKTRACE          at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.start(EclipseAppLauncher.java:68)
4XESTACKTRACE          at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStarter.java:400)
4XESTACKTRACE          at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStarter.java:177)
4XESTACKTRACE          at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
4XESTACKTRACE          at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:45)
4XESTACKTRACE          at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
4XESTACKTRACE          at java/lang/reflect/Method.invoke(Method.java:599)
4XESTACKTRACE          at org/eclipse/core/launcher/Main.invokeFramework(Main.java:340)
4XESTACKTRACE          at org/eclipse/core/launcher/Main.basicRun(Main.java:282)
4XESTACKTRACE          at org/eclipse/core/launcher/Main.run(Main.java:981)
4XESTACKTRACE          at com/ibm/wsspi/bootstrap/WSPreLauncher.launchEclipse(WSPreLauncher.java:330)
4XESTACKTRACE          at com/ibm/wsspi/bootstrap/WSPreLauncher.main(WSPreLauncher.java:108)
3XMTHREADINFO      "JIT Compilation Thread" TID:0x00000000121AF100, j9thread_t:0x00007F9758007180, state:CW, prio=10
3XMTHREADINFO1            (native thread ID:0x1A66, native priority:0xB, native policy:UNKNOWN)
3XMTHREADINFO      "Signal Dispatcher" TID:0x0000000012240300, j9thread_t:0x00007F97589F1800, state:R, prio=5
3XMTHREADINFO1            (native thread ID:0x1A68, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE          at com/ibm/misc/SignalDispatcher.waitForSignal(Native Method)
4XESTACKTRACE          at com/ibm/misc/SignalDispatcher.run(SignalDispatcher.java:54)



3XMTHREADINFO      "[SSL: ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8878]]" TID:0x0000000013798D00, j9thread_t:0x00007F9759CCDBC0, state:R, prio=5
3XMTHREADINFO1            (native thread ID:0x15E4, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE          at java/net/PlainSocketImpl.socketAccept(Native Method)
4XESTACKTRACE          at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:414(Compiled Code))
4XESTACKTRACE          at java/net/ServerSocket.implAccept(ServerSocket.java:458(Compiled Code))
4XESTACKTRACE          at com/ibm/jsse2/qc.accept(qc.java:67(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/http/HttpTransport.run(HttpTransport.java:430)
4XESTACKTRACE          at java/lang/Thread.run(Thread.java:735(Compiled Code))
3XMTHREADINFO      "Non-deferrable Alarm : 4" TID:0x0000000012D8B200, j9thread_t:0x00007F9598004FD0, state:CW, prio=5
3XMTHREADINFO1            (native thread ID:0x781A, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE          at java/lang/Object.wait(Native Method)
4XESTACKTRACE          at java/lang/Object.wait(Object.java:196(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/util/BoundedBuffer.waitGet_(BoundedBuffer.java:175(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/util/BoundedBuffer.poll(BoundedBuffer.java:595(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/util/ThreadPool.getTask(ThreadPool.java:849(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1534(Compiled Code))
3XMTHREADINFO      "TCPChannel.DCS : 3" TID:0x00000000137B1700, j9thread_t:0x00007F9660004110, state:CW, prio=5
3XMTHREADINFO1            (native thread ID:0x7855, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE          at java/lang/Object.wait(Native Method)
4XESTACKTRACE          at java/lang/Object.wait(Object.java:196(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/util/BoundedBuffer.waitGet_(BoundedBuffer.java:175(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/util/BoundedBuffer.take(BoundedBuffer.java:519(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/util/ThreadPool.getTask(ThreadPool.java:849(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1534(Compiled Code))
3XMTHREADINFO      "UpstreamPing" TID:0x00000000133E2100, j9thread_t:0x00007F96000CE610, state:CW, prio=5
3XMTHREADINFO1            (native thread ID:0x79F5, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE          at java/lang/Thread.sleep(Native Method)
4XESTACKTRACE          at java/lang/Thread.sleep(Thread.java:850(Compiled Code))
4XESTACKTRACE          at com/ibm/ws/management/RoutingTable$PingThread.run(RoutingTable.java:1355)
3XMTHREADINFO      "StatusCacheTimedThread:true" TID:0x0000000012EFD500, j9thread_t:0x00007F9759CCE5A0, state:B, prio=5
3XMTHREADINFO1            (native thread ID:0x79FB, native priority:0x5, native policy:UNKNOWN)收起
参与13

查看其它 9 个回答peter1407的回答

peter1407peter1407数据库运维工程师NRI
问题真的解决了?我觉得原因不对,而且也不用工具吧。
1 javacore里的 1CICMDLINE 中是明确显示:-Xms50m -Xmx256m  ;
  最小堆为50M,最大堆为256M。可是再仔细看一下,是nodeagent 。
server的JVM肯定要调整,64位的一般2G或3G,可nodeagent 默认的堆一般都足够了。
2 再看  1STHEAPFREE    Bytes of Heap Space Free: 131B350            (换算成十进制为19M)
            1STHEAPALLOC   Bytes of Heap Space Allocated: 3814C00  (换算成十进制为56M)
  就是说,还没用到最大堆,还在最小堆56M的阶段,而且剩余19M,当时没有JVM内存不足。
3 看javacore的开头信息, “Dump Event "gpf" (00002000) received
   如果是内存不足,一般显示OOM(Out Of Memory)
     而gpf一般是产品本身的问题
所以,肯定不是JVM内存的问题,个人觉得:如果不是jvm和was的版本不匹配造成的,就可能是was本身的bug。
这种问题重启基本都能恢复,如果是bug再次触发也不太容易,如果不是特别重要的系统,暂时可以先用着。重要的生产系统,则要考虑集群,保证高可用性。
银行 · 2015-02-05
浏览2602

回答者

peter1407
数据库运维工程师NRI
擅长领域: 双活应用服务器存储

peter1407 最近回答过的问题

回答状态

  • 发布时间:2015-02-05
  • 关注会员:1 人
  • 回答浏览:2602
  • X社区推广