liberty 应用总是自动停掉问题
问题现象: 我们当前有两套一样的环境,部署相同的应用的和使用同样的os/应用版本(配置基本相同)。其中一套A是好的。但是一套B就进程出现宕机的情况
系统B的应用总是在正常运行一段时间后,应用自动停掉。然后日志中反复出现如下的错误,尝试n次后,进程停掉了。
日志:
在ffdc中的日志
Exception = javax.resource.spi.ResourceAllocationException
Source = com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection
probeid = 299
Stack Dump = javax.resource.spi.ResourceAllocationException: Failed preTestConnection. Pool requests blocked for xxx/jdbc/WorklightDS until the test connection thread is successful.
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2628)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2333)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1285)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:852)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:354)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:325)
at com.worklight.server.database.api.WorklightDataSource.getConnection(WorklightDataSource.java:247)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:210)
at com.sun.proxy.$Proxy32.getAutoCommit(Unknown Source)
at org.apache.openjpa.lib.jdbc.DelegatingConnection.getAutoCommit(DelegatingConnection.java:202)
at org.apache.openjpa.lib.jdbc.DelegatingConnection.getAutoCommit(DelegatingConnection.java:202)
at org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator$ConfiguringConnection.(ConfiguringConnectionDecorator.java:109)
at org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator.decorate(ConfiguringConnectionDecorator.java:93)
at org.apache.openjpa.lib.jdbc.DecoratingDataSource.decorate(DecoratingDataSource.java:100)
at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:88)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:945)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:930)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.retainConnection(JDBCStoreManager.java:216)
at org.apache.openjpa.kernel.DelegatingStoreManager.retainConnection(DelegatingStoreManager.java:163)
at org.apache.openjpa.kernel.BrokerImpl.retainConnection(BrokerImpl.java:3734)
at org.apache.openjpa.kernel.BrokerImpl.beginStoreManagerTransaction(BrokerImpl.java:1302)
at org.apache.openjpa.kernel.BrokerImpl.beginStore(BrokerImpl.java:1287)
at org.apache.openjpa.kernel.DelegatingBroker.beginStore(DelegatingBroker.java:1013)
at org.apache.openjpa.persistence.EntityManagerImpl.beginStore(EntityManagerImpl.java:614)
at org.springframework.orm.jpa.vendor.OpenJpaDialect.beginTransaction(OpenJpaDialect.java:55)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:330)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374)
at com.worklight.core.util.RssBrokerUtils.beginTransaction(RssBrokerUtils.java:276)
at com.worklight.core.util.RssBrokerUtils.doInTransaction(RssBrokerUtils.java:130)
at com.worklight.gadgets.bean.WidgetServiceBean.syncApplicationsFromDBTransaction(WidgetServiceBean.java:475)
at com.worklight.gadgets.bean.ClusterDataSynchronizationTask.step(ClusterDataSynchronizationTask.java:35)
at com.worklight.core.tasks.TaskThread.run(TaskThread.java:111)
Dump of callerThis
Object type = com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource
strings[0] = "_______________________________________________________________________"
strings[1] = ""
strings[2] = " First Failure Data Capture information for"
strings[3] = " com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource@2eaa6b21"
strings[4] = "_______________________________________________________________________"
strings[5] = ""
strings[6] = "ResourceRefInfo:"
strings[7] = "
[Resource-ref CMConfigData key items]
res-sharing-scope: 0 (SHAREABLE)
res-isolation-level: 0 (TRANSACTION_NONE)
res-auth: 1 (APPLICATION)
commitPriority 0
branchCoupling -1
loginConfigurationName: null
loginConfigProperties: null
Resource ref name: not set
"
strings[8] = "Current Configuration ID: 1
"
strings[9] = "Connection Manager:"
strings[10] = " [ConnectionManager]@550d7440
JNDI Name
shareable
"
strings[11] = "_______________________________________________________________________"
strings[12] = ""
strings[13] = " First Failure Data Capture information for"
strings[14] = " com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl@56b0f870"
strings[15] = "_______________________________________________________________________"
strings[16] = ""
strings[17] = "com.ibm.ws.rsadapter.DSConfig@7c86ad86"
strings[18] = "[beginTranForResultSetScrollingAPIs, true, beginTranForVendorAPIs, true, commitOrRollbackOnCleanup, rollback, connectionSharing, 0, isolationLevel, -1, jndiName, xxx/jdbc/WorklightDS, queryTimeout, null, statementCacheSize, 10, supplementalJDBCTrace, null, syncQueryTimeoutWithTransactionTimeout, false, transactional, false]"
strings[19] = "[com.ibm.ws.classloading.internal.AppClassLoader@53feb3c6, true, ExceptionChecking, com.ibm.websphere.rsadapter.DB2UniversalDataStoreHelper@6dea7848, com.ibm.ws.rsadapter.spi.InternalDB2UniversalDataStoreHelper@3c7ef758, false, interface javax.sql.ConnectionPoolDataSource]"
strings[20] = "{serverName=xxxxxxx, traceLevel=0, user=wluser, databaseName=xxxxxxx, retrieveMessagesFromServerOnGetMessage=true, driverType=4, dataSourceClass=com.ibm.db2.jcc.DB2ConnectionPoolDataSource, currentSchema=xxxxxxx, portNumber=50000, deferPrepares=true}"
strings[21] = "Resource Adapter:"
strings[22] = " null
"
strings[23] = "Database Type:"
strings[24] = " null
"
strings[25] = "Description:"
strings[26] = " null
"
strings[27] = "Log Writer:"
strings[28] = " null
"
strings[29] = "Transaction Branches are set to be Loosely Coupled: "
strings[30] = " false
"
strings[31] = "Counter of fatal connection errors on ManagedConnections created by this MCF:"
strings[32] = " 0
"
strings[33] = "Validate existing connections on cleanup after a fatal connection error is detected?"
strings[34] = " true
"
strings[35] = "ResetConnectionByBackendDatabase: "
strings[36] = " false
"
strings[37] = "Backend id checking is: "
strings[38] = " true
"
strings[39] = "JMSOnePhaseOptimization: "
strings[40] = " false
"
strings[41] = "Reauthentication: "
strings[42] = " false
"
strings[43] = "newDBConnectionValidationEnabled: "
strings[44] = " false
"
strings[45] = "connectionRetriesDuringDBFailover: "
strings[46] = " 100
"
strings[47] = "connRetryDurationDuringDBFailover: "
strings[48] = " 3000
"
strings[49] = "Failover count: "
strings[50] = " 0
"
strings[51] = "OracleImplicitConnectionPooling enabled: "
strings[52] = " false
"
strings[53] = ""
strings[54] = "_______________________________________________________________________"
strings[55] = ""
strings[56] = " First Failure Data Capture information for"
strings[57] = " com.ibm.ws.rsadapter.spi.WSRdbDataSource@136cd251"
strings[58] = "_______________________________________________________________________"
strings[59] = ""
strings[60] = "DataSource Implementation Class Name:"
strings[61] = " com.ibm.db2.jcc.DB2ConnectionPoolDataSource
"
strings[62] = "DataSource Name:"
strings[63] = " null
"
strings[64] = "DataStoreHelper:"
strings[65] = " com.ibm.websphere.rsadapter.DB2UniversalDataStoreHelper@6dea7848
"
strings[66] = "DataSource properties:"
strings[67] = " {serverName=xxxxxxx, traceLevel=0, user=xxxxxxx, databaseName=xxxxxxx, retrieveMessagesFromServerOnGetMessage=true, driverType=4, dataSourceClass=com.ibm.db2.jcc.DB2ConnectionPoolDataSource, currentSchema=xxxxxxx, portNumber=50000, deferPrepares=true}
"
strings[68] = "ONE PHASE ENABLED
"
strings[69] = "Underlying DataSource Object: com.ibm.db2.jcc.DB2ConnectionPoolDataSource@4d63cdcd"
strings[70] = " com.ibm.db2.jcc.DB2ConnectionPoolDataSource@4d63cdcd
"
strings[71] = "Counter of unique, modified DataSource configurations: 0
"
strings[72] = "Failover servers list: "
strings[73] = " [Ljava.lang.Object;@535c756c
从日志中看是由于测试连接数据库失败导致的连接中断的。从dump也没看出有啥问题。
另外, liberty不像其他的App那样子配置jdbc的时候设置很多的连接属性,比如超时时间或者连接数等的。
在发生问题的时候,测试数据库连接也是正常的。所以我也怀疑可能是配置问题,比如超过连接数等。但是不知道如何验证。
哪位伙伴遇到上述问题,还请帮忙看看?