银行中间件
服务突挂起，Exception in thread "StatusCacheTimedThread:false
问题：服务挂起，无法访问
部署模式：垂直集群，其中一个server无法访问
平台+版本：AIX6.1 Power7 + WAS6.1.0.31
Systemerr日志：
[11-7-29 17:46:59:867 GMT+08:00] 0000080d SystemErr R Exception in thread "StatusCacheTimedThread:false" Exception in thread "StatusCacheTimedThread:true" Exception in thread "Timer-7" java.lang.OutOfMemoryError
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R java.lang.OutOfMemoryError
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at sun.nio.cs.StreamDecoder$ConverterSD.(StreamDecoder.java:293)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at java.lang.Throwable.fillInStackTrace(Native Method)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at sun.nio.cs.StreamDecoder$ConverterSD.(StreamDecoder.java:278)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:140)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at java.io.InputStreamReader.(InputStreamReader.java:125)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at java.lang.Throwable.(Throwable.java:56)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at nc.bs.logging.FileLoggerConfigManager$GB2312Properties.load(FileLoggerConfigManager.java:331)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at java.lang.Throwable.(Throwable.java:67)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at nc.bs.logging.FileLoggerConfigManager.loadModuleLoggerConfigurations(FileLoggerConfigManager.java:156)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.util.xml.XMLJavaMappingRegistry.querySerializer(XMLJavaMappingRegistry.java:157)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at nc.bs.logging.config.LoggingConfigMonitor$LogConfigFileMonitorTask.run(LoggingConfigMonitor.java:64)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.AdminServiceSMR.querySerializer(AdminServiceSMR.java:101)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at java.util.TimerThread.mainLoop(Timer.java:537)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at java.util.TimerThread.run(Timer.java:487)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.encoding.soapenc.ParameterSerializer.marshall(ParameterSerializer.java:105)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.rpc.RPCMessage.serializeParams(RPCMessage.java:333)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.rpc.RPCMessage.marshall(RPCMessage.java:307)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.Body.marshall(Body.java:145)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.Envelope.marshall(Envelope.java:195)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:368)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at org.apache.soap.rpc.Call.WASinvoke(Call.java:433)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.SOAPConnectorClient$8.run(SOAPConnectorClient.java:720)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:713)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:582)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:409)
[11-7-29 17:47:20:301 GMT+08:00] 0000080d SystemErr R java.lang.OutOfMemoryError
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at $Proxy37.invoke(Unknown Source)
[11-7-29 17:47:20:301 GMT+08:00] 0000080d SystemErr R at java.lang.Thread.sleep(Native Method)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.AdminClientImpl.invoke(AdminClientImpl.java:205)
[11-7-29 17:47:20:301 GMT+08:00] 0000080d SystemErr R at java.lang.Thread.sleep(Thread.java:939)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.status.AppServerStatusCache$StatusCacheTimedThread.run(AppServerStatusCache.java:707)
[11-7-29 17:47:20:301 GMT+08:00] 0000080d SystemErr R at com.ibm.ws.management.status.AppServerStatusCache$StatusCacheTimedThread.run(AppServerStatusCache.java:647)
[11-7-29 17:47:20:302 GMT+08:00] 0000080c SystemErr R Exception in thread "LT=0:P=757127:O=0:port=46564" java.lang.OutOfMemoryError
[11-7-29 17:47:20:302 GMT+08:00] 0000080c SystemErr R at java.lang.Throwable.fillInStackTrace(Native Method)
[11-7-29 17:47:20:302 GMT+08:00] 00000810 SystemErr R Exception in thread "LT=1:P=757127:O=0:port=2813" java.lang.OutOfMemoryError
[11-7-29 17:47:46:865 GMT+08:00] 00000810 SystemErr R at java.net.PlainSocketImpl.socketAccept(Native Method)
[11-7-29 17:47:46:866 GMT+08:00] 00000810 SystemErr R at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:457)
[11-7-29 17:47:46:866 GMT+08:00] 00000810 SystemErr R at java.net.ServerSocket.implAccept(ServerSocket.java:473)
[11-7-29 17:47:46:866 GMT+08:00] 00000810 SystemErr R at java.net.ServerSocket.accept(ServerSocket.java:444)
[11-7-29 17:47:46:866 GMT+08:00] 00000810 SystemErr R at com.ibm.rmi.transport.ListenerThread.run(ListenerThread.java:170)
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.lang.Throwable.(Throwable.java:56)
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.lang.Throwable.(Throwable.java:67)
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.io.InterruptedIOException.(InterruptedIOException.java:60)
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.net.SocketTimeoutException.(SocketTimeoutException.java:39)
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.net.PlainSocketImpl.socketAccept(Native Method)
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:457)
[11-7-29 17:47:46:867 GMT+08:00] 0000080c SystemErr R at java.net.ServerSocket.implAccept(ServerSocket.java:473)
[11-7-29 17:47:46:867 GMT+08:00] 0000080c SystemErr R at java.net.ServerSocket.accept(ServerSocket.java:444)
[11-7-29 17:47:46:867 GMT+08:00] 0000080c SystemErr R at com.ibm.rmi.transport.ListenerThread.run(ListenerThread.java:170)
[11-7-29 17:47:51:034 GMT+08:00] 00000811 SystemErr R Exception in thread "ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8884]" java.lang.OutOfMemoryError
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.net.PlainSocketImpl.socketAccept(Native Method)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:457)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.net.ServerSocket.implAccept(ServerSocket.java:473)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.net.ServerSocket.accept(ServerSocket.java:444)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at com.ibm.ws.http.HttpTransport.run(HttpTransport.java:456)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.lang.Thread.run(Thread.java:811)
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R Exception in thread "AIO Timer Thread 1" java.lang.OutOfMemoryError
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R at com.ibm.io.async.TimeSlot.(TimeSlot.java:39)
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R at com.ibm.io.async.Timer.insertSlotAtEnd(Timer.java:362)
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R at com.ibm.io.async.Timer.insertWorkItem(Timer.java:323)
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R at com.ibm.io.async.Timer.run(Timer.java:233)
部署模式：垂直集群，其中一个server无法访问
平台+版本：AIX6.1 Power7 + WAS6.1.0.31
Systemerr日志：
[11-7-29 17:46:59:867 GMT+08:00] 0000080d SystemErr R Exception in thread "StatusCacheTimedThread:false" Exception in thread "StatusCacheTimedThread:true" Exception in thread "Timer-7" java.lang.OutOfMemoryError
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R java.lang.OutOfMemoryError
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at sun.nio.cs.StreamDecoder$ConverterSD.
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at java.lang.Throwable.fillInStackTrace(Native Method)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at sun.nio.cs.StreamDecoder$ConverterSD.
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:140)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at java.io.InputStreamReader.
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at java.lang.Throwable.
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at nc.bs.logging.FileLoggerConfigManager$GB2312Properties.load(FileLoggerConfigManager.java:331)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at java.lang.Throwable.
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at nc.bs.logging.FileLoggerConfigManager.loadModuleLoggerConfigurations(FileLoggerConfigManager.java:156)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.util.xml.XMLJavaMappingRegistry.querySerializer(XMLJavaMappingRegistry.java:157)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at nc.bs.logging.config.LoggingConfigMonitor$LogConfigFileMonitorTask.run(LoggingConfigMonitor.java:64)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.AdminServiceSMR.querySerializer(AdminServiceSMR.java:101)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at java.util.TimerThread.mainLoop(Timer.java:537)
[11-7-29 17:47:20:300 GMT+08:00] 0000080e SystemErr R at java.util.TimerThread.run(Timer.java:487)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.encoding.soapenc.ParameterSerializer.marshall(ParameterSerializer.java:105)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.rpc.RPCMessage.serializeParams(RPCMessage.java:333)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.rpc.RPCMessage.marshall(RPCMessage.java:307)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.Body.marshall(Body.java:145)
[11-7-29 17:47:20:300 GMT+08:00] 0000080f SystemErr R at org.apache.soap.Envelope.marshall(Envelope.java:195)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:368)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at org.apache.soap.rpc.Call.WASinvoke(Call.java:433)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.SOAPConnectorClient$8.run(SOAPConnectorClient.java:720)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:713)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:582)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:409)
[11-7-29 17:47:20:301 GMT+08:00] 0000080d SystemErr R java.lang.OutOfMemoryError
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at $Proxy37.invoke(Unknown Source)
[11-7-29 17:47:20:301 GMT+08:00] 0000080d SystemErr R at java.lang.Thread.sleep(Native Method)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.AdminClientImpl.invoke(AdminClientImpl.java:205)
[11-7-29 17:47:20:301 GMT+08:00] 0000080d SystemErr R at java.lang.Thread.sleep(Thread.java:939)
[11-7-29 17:47:20:301 GMT+08:00] 0000080f SystemErr R at com.ibm.ws.management.status.AppServerStatusCache$StatusCacheTimedThread.run(AppServerStatusCache.java:707)
[11-7-29 17:47:20:301 GMT+08:00] 0000080d SystemErr R at com.ibm.ws.management.status.AppServerStatusCache$StatusCacheTimedThread.run(AppServerStatusCache.java:647)
[11-7-29 17:47:20:302 GMT+08:00] 0000080c SystemErr R Exception in thread "LT=0:P=757127:O=0:port=46564" java.lang.OutOfMemoryError
[11-7-29 17:47:20:302 GMT+08:00] 0000080c SystemErr R at java.lang.Throwable.fillInStackTrace(Native Method)
[11-7-29 17:47:20:302 GMT+08:00] 00000810 SystemErr R Exception in thread "LT=1:P=757127:O=0:port=2813" java.lang.OutOfMemoryError
[11-7-29 17:47:46:865 GMT+08:00] 00000810 SystemErr R at java.net.PlainSocketImpl.socketAccept(Native Method)
[11-7-29 17:47:46:866 GMT+08:00] 00000810 SystemErr R at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:457)
[11-7-29 17:47:46:866 GMT+08:00] 00000810 SystemErr R at java.net.ServerSocket.implAccept(ServerSocket.java:473)
[11-7-29 17:47:46:866 GMT+08:00] 00000810 SystemErr R at java.net.ServerSocket.accept(ServerSocket.java:444)
[11-7-29 17:47:46:866 GMT+08:00] 00000810 SystemErr R at com.ibm.rmi.transport.ListenerThread.run(ListenerThread.java:170)
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.lang.Throwable.
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.lang.Throwable.
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.io.InterruptedIOException.
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.net.SocketTimeoutException.
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.net.PlainSocketImpl.socketAccept(Native Method)
[11-7-29 17:47:46:866 GMT+08:00] 0000080c SystemErr R at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:457)
[11-7-29 17:47:46:867 GMT+08:00] 0000080c SystemErr R at java.net.ServerSocket.implAccept(ServerSocket.java:473)
[11-7-29 17:47:46:867 GMT+08:00] 0000080c SystemErr R at java.net.ServerSocket.accept(ServerSocket.java:444)
[11-7-29 17:47:46:867 GMT+08:00] 0000080c SystemErr R at com.ibm.rmi.transport.ListenerThread.run(ListenerThread.java:170)
[11-7-29 17:47:51:034 GMT+08:00] 00000811 SystemErr R Exception in thread "ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8884]" java.lang.OutOfMemoryError
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.net.PlainSocketImpl.socketAccept(Native Method)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:457)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.net.ServerSocket.implAccept(ServerSocket.java:473)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.net.ServerSocket.accept(ServerSocket.java:444)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at com.ibm.ws.http.HttpTransport.run(HttpTransport.java:456)
[11-7-29 17:47:51:035 GMT+08:00] 00000811 SystemErr R at java.lang.Thread.run(Thread.java:811)
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R Exception in thread "AIO Timer Thread 1" java.lang.OutOfMemoryError
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R at com.ibm.io.async.TimeSlot.
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R at com.ibm.io.async.Timer.insertSlotAtEnd(Timer.java:362)
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R at com.ibm.io.async.Timer.insertWorkItem(Timer.java:323)
[11-7-29 17:51:57:978 GMT+08:00] 00000815 SystemErr R at com.ibm.io.async.Timer.run(Timer.java:233)
10同行回答
看javacore和heapdump文件啊。收起
浏览1087
明显是“java.lang.OutOfMemoryError”JVM 内存溢出
建议：
1、定义是那块的内存溢出
2、修改JVM参数（不知道上述环境的JDK是那个厂商的？位数是否与所应用的匹配）
3、也可通过抓heap dump或采用工具（jprofiler）分析下应用程序问题，哪个类或方法占用资源收起
建议：
1、定义是那块的内存溢出
2、修改JVM参数（不知道上述环境的JDK是那个厂商的？位数是否与所应用的匹配）
3、也可通过抓heap dump或采用工具（jprofiler）分析下应用程序问题，哪个类或方法占用资源收起
浏览1071
程序访问量大，压力大造成的吧收起
浏览1045
内存泄露，请检查是native泄露还是heap 泄露，在native 日志中，检查是否有ACTION=6收起
浏览1102
和我的问题一样哎。。。。
等赐教收起
等赐教收起
浏览1077
java.lang.OutOfMemoryError, 不挂才不正常收起
浏览1144