急救,两个集群的所以server突然全部自动重启

本系统为由F5做负载的两个集群(分别4节点和8节点),同一时间全部无法访问,三方监控软件显示也是所有节点都shut down了,过会后又自动重启恢复了

[5/22/13 8:03:18:629 GMT-08:00] 00000488 SystemOut     O 05-22 08:03:18 WARN  [com.talkweb.ecm.core.filter.UserLoginFilter] May 22, 2013 8:03:18 AM用户(email:shenggang@chinaunicom.cn;IP:10.3.80.137;sessionId:wAfYGt5bkXLjdlZOkPC0I4t)访问地址:/WEB-INF/business/perform/performer/selfOrgMenu.jsp。具体参数信息:具体参数信息:{"type":["user"]}
[5/22/13 8:03:18:928 GMT-08:00] 00000498 SystemOut     O 05-22 08:03:18 WARN  [com.talkweb.ecm.core.filter.UserLoginFilter] May 22, 2013 8:03:18 AM用户(email:shenggang@chinaunicom.cn;IP:10.3.80.137;sessionId:wAfYGt5bkXLjdlZOkPC0I4t)访问地址:/core/getSubNodes.action。具体参数信息:具体参数信息:{"type":["1"],"_":["1369181124495"],"scope":["1"],"pId":["1000"]}
[5/22/13 8:03:19:369 GMT-08:00] 000004bc TCPChannel    I   TCPC0002I: TCP Channel TCP_4 has stopped listening on host *  (IPv6) port 9443.
[5/22/13 8:03:19:395 GMT-08:00] 000004bc TCPChannel    I   TCPC0002I: TCP Channel TCP_1 has stopped listening on host *  (IPv6) port 9061.
[5/22/13 8:03:19:411 GMT-08:00] 000004bc TCPChannel    I   TCPC0002I: TCP Channel TCP_2 has stopped listening on host *  (IPv6) port 9080.
[5/22/13 8:03:19:425 GMT-08:00] 000004bc TCPChannel    I   TCPC0002I: TCP Channel TCP_3 has stopped listening on host *  (IPv6) port 9044.
[5/22/13 8:03:19:437 GMT-08:00] 00000497 WebContainer  E   Servlet.has.become.temporarily.unavailable.for.service
[5/22/13 8:03:19:437 GMT-08:00] 00000483 WebContainer  E   Servlet.has.become.temporarily.unavailable.for.service
[5/22/13 8:03:19:437 GMT-08:00] 00000498 WebContainer  E   Servlet.has.become.temporarily.unavailable.for.service
[5/22/13 8:03:19:438 GMT-08:00] 00000488 WebContainer  E   Servlet.has.become.temporarily.unavailable.for.service
[5/22/13 8:03:19:438 GMT-08:00] 000004bc ApplicationMg A   WSVR0217I: Stopping application: ecm_war
[5/22/13 8:03:19:445 GMT-08:00] 0000048e WebContainer  E   Servlet.has.become.temporarily.unavailable.for.service
[5/22/13 8:03:19:446 GMT-08:00] 00000480 WebContainer  E   Servlet.has.become.temporarily.unavailable.for.service
[5/22/13 8:03:19:448 GMT-08:00] 000004a1 WebContainer  E   Servlet.has.become.temporarily.unavailable.for.service
[5/22/13 8:03:19:462 GMT-08:00] 0000043b WebContainer  E   Servlet.has.become.temporarily.unavailable.for.service
[5/22/13 8:03:19:515 GMT-08:00] 000004bc ServletWrappe I   SRVE0253I: [ecm_war] [/ecm] [/WEB-INF/backstage/addiPaymentBack.jsp]: Destroy successful.
。。。。。。
[5/22/13 8:06:36:669 GMT-08:00] 000004bc ApplicationMg A   WSVR0220I: Application stopped: ecm_sync_war
[5/22/13 8:06:36:701 GMT-08:00] 000004bc SchedulerServ I   SCHD0040I: The Scheduler Service is stopping.
[5/22/13 8:06:36:715 GMT-08:00] 000004bc SchedulerServ I   SCHD0002I: The Scheduler Service has stopped.
[5/22/13 8:06:36:735 GMT-08:00] 000004bc AppProfileCom I   ACIN0009I: The application profiling service is stopping.
[5/22/13 8:06:36:745 GMT-08:00] 000004bc ActivitySessi I   WACS0049I: The ActivitySession service is stopping.
[5/22/13 8:06:36:810 GMT-08:00] 000004bc DCSStackImpl  I   DCSV8011I: DCS Stack DefaultCoreGroup.app5cluster at Member ecmapp501EcmApp501Node01app04: Member received a request for shutdown.
[5/22/13 8:06:36:830 GMT-08:00] 000004bc DCSStackImpl  I   DCSV8011I: DCS Stack DefaultCoreGroup at Member ecmapp501EcmApp501Node01app04: Member received a request for shutdown.
[5/22/13 8:11:01:982 GMT-08:00] 00000018 ThreadMonitor W   WSVR0605W: Thread "WebContainer : 908" (0000049a) has been active for 658090 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:155)

at oracle.net.ns.Packet.receive(Unknown Source)

at oracle.net.ns.NSProtocol.connect(Unknown Source)

at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:844)

at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)

at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:414)

at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:165)

at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)

at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)

at java.sql.DriverManager.getConnection(DriverManager.java:572)

at java.sql.DriverManager.getConnection(DriverManager.java:165)

at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:174)

at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:165)

at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:149)

at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)

at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)

at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)

at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:577)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:678)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:710)

at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:776)

at com.unicom.mss.jdbc.impl.ImportPersonInfoDaoImpl.isAlowToSendToEcm(ImportPersonInfoDaoImpl.java:194)

at com.unicom.mss.jdbc.PersonTools.isAlowToSendToEcm(PersonTools.java:1328)

at com.unicom.mss.SB_EIP_EIP_ImportEmpInfoSrv.SBEIPEIPImportEmpInfoSrvImpl.process(SBEIPEIPImportEmpInfoSrvImpl.java:122)

at sun.reflect.GeneratedMethodAccessor4446.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:618)

at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:166)

at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:82)

at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:55)

at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:68)

at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:432)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284)

at java.util.concurrent.FutureTask.run(FutureTask.java:138)

at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)

at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:98)

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)

at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:104)

at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)

at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:452)

at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:196)

at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:220)

at org.apache.cxf.transport.servlet.AbstractCXFServlet.doPost(AbstractCXFServlet.java:153)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)

at org.apache.cxf.transport.servlet.AbstractCXFServlet.service(AbstractCXFServlet.java:211)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1213)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:658)

at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:526)

at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3574)

at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:269)

at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:831)

at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1478)

at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:133)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:457)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:300)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:271)

at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)

at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)

at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)

at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)

at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)

at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136)

at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:196)

at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:751)

at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881)

at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1551)
[5/22/13 8:17:02:014 GMT-08:00] 0000003c ThreadMonitor W   WSVR0605W: Thread "SoapConnectorThreadPool : 1" (0000002e) has been active for 711358 milliseconds and may be hung.  There is/are 2 thread(s) in total in the server that may be hung.

at java.net.InetAddress.getLocalHost(InetAddress.java:1439)

at org.apache.soap.util.mime.MimeUtils.getUniqueValue(MimeUtils.java:85)

at org.apache.soap.rpc.SOAPContext.setRootPart(SOAPContext.java:383)

at org.apache.soap.rpc.SOAPContext.setRootPart(SOAPContext.java:426)

at org.apache.soap.rpc.SOAPContext.setRootPart(SOAPContext.java:401)

at org.apache.soap.transport.TransportMessage.save(TransportMessage.java:303)

at com.ibm.ws.management.connector.soap.SOAPResponse.writeSOAPResponse(SOAPResponse.java:87)

at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:296)

at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:64)

at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:742)

at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:528)

at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1551)
[5/22/13 8:40:49:436 GMT-08:00] 000004bc FailureScopeC A   WTRN0105I: The transaction service has shutdown successfully with no transactions requiring recovery.
[5/22/13 8:40:49:463 GMT-08:00] 000004bc TCPChannel    I   TCPC0002I: TCP Channel TCP_5 has stopped listening on host *  (IPv6) port 9354.
[5/22/13 8:40:49:498 GMT-08:00] 000004bc ServerCollabo A   WSVR0024I: Server app04 stopped
************ Start Display Current Environment ************
WebSphere Platform 6.1 [ND 6.1.0.39 cf391124.02]  running with process name ecmapp501EcmApp501Node01app04 and process id 339984
Detailed IFix information: Please use the versionInfo command to view this information
Host Operating System is AIX, version 5.3
Java version = 1.5.0, Java Compiler = j9jit23, Java VM name = IBM J9 VM
was.install.root = /usr/IBM/WebSphere/AppServer
user.install.root = /usr/IBM/WebSphere/AppServer/profiles/AppSrv01
Java Home = /usr/IBM/WebSphere/AppServer/java/jre
ws.ext.dirs = /usr/IBM/WebSphere/AppServer/java/lib:/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/usr/IBM/WebSphere/AppServer/classes:/usr/IBM/WebSphere/AppServer/lib:/usr/IBM/WebSphere/AppServer/installedChannels:/usr/IBM/WebSphere/AppServer/lib/ext:/usr/IBM/WebSphere/AppServer/web/help:/usr/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/usr/IBM/WebSphere/AppServer/properties:/usr/IBM/WebSphere/AppServer/lib/startup.jar:/usr/IBM/WebSphere/AppServer/lib/bootstrap.jar:/usr/IBM/WebSphere/AppServer/lib/j2ee.jar:/usr/IBM/WebSphere/AppServer/lib/lmproxy.jar:/usr/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/usr/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/usr/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/usr/IBM/WebSphere/AppServer/java/lib/tools.jar
Java Library path = /usr/IBM/WebSphere/AppServer/java/jre/bin:/usr/IBM/WebSphere/AppServer/java/jre/bin:/usr/IBM/WebSphere/AppServer/java/jre/bin/classic:/usr/IBM/WebSphere/AppServer/java/jre/bin:/usr/IBM/WebSphere/AppServer/bin:/usr/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/usr/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/usr/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/usr/lib
************* End Display Current Environment *************




请各位大侠帮忙看看
参与3

2同行回答

db2176db2176项目经理中国电信湖南分公司
回复 2# huangdos 数据库链接获取之后,线程为什么就挂起了呢?原因还是不明确?显示全部
回复 2# huangdos

数据库链接获取之后,线程为什么就挂起了呢?原因还是不明确?收起
电信运营商 · 2013-06-03
浏览1335
huangdoshuangdosCTO深圳乐凯撒股份有限公司
看日志是,数据库连接获取之后,线程挂起,导致服务宕机。显示全部
看日志是,数据库连接获取之后,线程挂起,导致服务宕机。收起
互联网服务 · 2013-05-29
浏览1657

提问者

db2176
项目经理中国电信湖南分公司

相关问题

相关资料

问题状态

  • 发布时间:2013-05-29
  • 关注会员:1 人
  • 问题浏览:6373
  • 最近回答:2013-06-03
  • X社区推广