背景: kettle 工具提数从源数据库到目标数据库。
但是 由于kettle 启动的时候,上面的job都是并发的,感觉服务器的物理内存都被吃掉了,基本上是99%满了。报错日志如下:
ERROR 21-10 04:43:15,507 - 表输入TABLE A - An error occurred, processing will be stopped:
Error occured while trying to connect to the database
Error connecting to database: (using class oracle.jdbc.driver.OracleDriver)
无法从套接字读取更多的数据
ERROR 21-10 04:43:15,507 - TABLE B表输入 - An error occurred, processing will be stopped:
Error occured while trying to connect to the database
Error connecting to database: (using class oracle.jdbc.driver.OracleDriver)
无法从套接字读取更多的数据
ERROR 21-10 04:43:15,507 - TABLE B表输出 - An error occurred intialising this step:
Error occured while trying to connect to the database
Error connecting to database: (using class oracle.jdbc.driver.OracleDriver)
无法从套接字读取更多的数据
ERROR 21-10 04:43:15,507 - 表输出TABLE C 4 - An error occurred intialising this step:
Error occured while trying to connect to the database
Error connecting to database: (using class oracle.jdbc.driver.OracleDriver)
无法从套接字读取更多的数据
ERROR 21-10 04:43:15,507 - TABLE B表输出 - 错误初始化步骤[TABLE B表输出]
ERROR 21-10 04:43:15,507 - 表输入TABLE A - 错误初始化步骤[表输入TABLE A]
ERROR 21-10 04:43:15,507 - TABLE B表输入 - 错误初始化步骤[TABLE B表输入]
ERROR 21-10 04:43:15,507 - 表输出TABLE C 4 - 错误初始化步骤[表输出TABLE C 4]
ERROR 21-10 04:43:15,508 - TABLE B - 步骤 [TABLE B表输入.0] 初始化失败!
ERROR 21-10 04:43:15,508 - TABLE C - 步骤 [表输入TABLE A.0] 初始化失败!
ERROR 21-10 04:43:15,508 - TABLE B - 步骤 [TABLE B表输出.0] 初始化失败!
ERROR 21-10 04:43:15,508 - TABLE C - 步骤 [表输出TABLE C 4.0] 初始化失败!
INFO 21-10 04:43:15,508 - TABLE B表输入 - Finished reading query, closing connection.
INFO 21-10 04:43:15,508 - 表输入TABLE D - Finished reading query, closing connection.
ERROR 21-10 04:43:15,509 - TABLE B表输出 - Unexpected error rolling back the database connection.
INFO 21-10 04:43:15,511 - 表输入TABLE E - Finished reading query, closing connection.
INFO 21-10 04:43:15,515 - 表输入TABLE G - Finished reading query, closing connection.
INFO 21-10 04:43:15,518 - 表输入TABLE A - Finished reading query, closing connection.
ERROR 21-10 04:43:15,518 - 表输出TABLE C 4 - Unexpected error rolling back the database connection.
ERROR 21-10 04:43:15,585 - 表输出TABLE C 4 - org.pentaho.di.core.exception.KettleDatabaseException:
Unable to get database metadata from this database connection
at java.lang.Thread.run (Thread.java:745)
at org.pentaho.di.job.entries.job.JobEntryJobRunner.run (JobEntryJobRunner.java:63)
at org.pentaho.di.job.Job.execute (Job.java:420)
at org.pentaho.di.job.Job.execute (Job.java:642)
at org.pentaho.di.job.Job.execute (Job.java:642)
at org.pentaho.di.job.Job.execute (Job.java:503)
at org.pentaho.di.job.entries.trans.JobEntryTrans.execute (JobEntryTrans.java:1024)
at org.pentaho.di.trans.Trans.execute (Trans.java:364)
at org.pentaho.di.trans.Trans.prepareExecution (Trans.java:766)
at org.pentaho.di.trans.steps.tableoutput.TableOutput.dispose (TableOutput.java:697)
at org.pentaho.di.core.database.Database.rollback (Database.java:717)
at org.pentaho.di.core.database.Database.rollback (Database.java:728)
at org.pentaho.di.core.database.Database.getDatabaseMetaData (Database.java:3139)
at org.pentaho.di.core.database.Database.getDatabaseMetaData(Database.java:3143)
at org.pentaho.di.core.database.Database.rollback(Database.java:728)
at org.pentaho.di.core.database.Database.rollback(Database.java:717)
at org.pentaho.di.trans.steps.tableoutput.TableOutput.dispose(TableOutput.java:697)
at org.pentaho.di.trans.Trans.prepareExecution(Trans.java:766)
at org.pentaho.di.trans.Trans.execute(Trans.java:364)
at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:1024)
at org.pentaho.di.job.Job.execute(Job.java:503)
at org.pentaho.di.job.Job.execute(Job.java:642)
at org.pentaho.di.job.Job.execute(Job.java:642)
at org.pentaho.di.job.Job.execute(Job.java:420)
at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:63)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.pentaho.di.core.database.Database.getDatabaseMetaData(Database.java:3139)
... 12 more
ERROR 21-10 04:43:15,585 - TABLE B表输出 - org.pentaho.di.core.exception.KettleDatabaseException:
Unable to get database metadata from this database connection
at java.lang.Thread.run (Thread.java:745)
at org.pentaho.di.job.entries.job.JobEntryJobRunner.run (JobEntryJobRunner.java:63)
at org.pentaho.di.job.Job.execute (Job.java:420)
at org.pentaho.di.job.Job.execute (Job.java:642)
at org.pentaho.di.job.Job.execute (Job.java:642)
at org.pentaho.di.job.Job.execute (Job.java:503)
at org.pentaho.di.job.entries.trans.JobEntryTrans.execute (JobEntryTrans.java:1024)
at org.pentaho.di.trans.Trans.execute (Trans.java:364)
at org.pentaho.di.trans.Trans.prepareExecution (Trans.java:766)
at org.pentaho.di.trans.steps.tableoutput.TableOutput.dispose (TableOutput.java:697)
at org.pentaho.di.core.database.Database.rollback (Database.java:717)
at org.pentaho.di.core.database.Database.rollback (Database.java:728)
at org.pentaho.di.core.database.Database.getDatabaseMetaData (Database.java:3139)
at org.pentaho.di.core.database.Database.getDatabaseMetaData(Database.java:3143)
at org.pentaho.di.core.database.Database.rollback(Database.java:728)
at org.pentaho.di.core.database.Database.rollback(Database.java:717)
at org.pentaho.di.trans.steps.tableoutput.TableOutput.dispose(TableOutput.java:697)
at org.pentaho.di.trans.Trans.prepareExecution(Trans.java:766)
at org.pentaho.di.trans.Trans.execute(Trans.java:364)
at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:1024)
at org.pentaho.di.job.Job.execute(Job.java:503)
at org.pentaho.di.job.Job.execute(Job.java:642)
at org.pentaho.di.job.Job.execute(Job.java:642)
at org.pentaho.di.job.Job.execute(Job.java:420)
at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:63)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.pentaho.di.core.database.Database.getDatabaseMetaData(Database.java:3139)
... 12 more
ERROR 21-10 04:43:15,727 - TRUNCATE TABLE F - 执行该作业项时发生了一个错误:
Error occured while trying to connect to the database
另外,我发现一个问题,为什么手动执行kettle 提数非常快,而且不报错,但是服务器上做了个定时任务启动kette就经常报错那?即便是手动执行和服务器上去执行内存基本上都是99%以上,但是手动执行就是不报错而且还很快,难道手动执行 不往日志表里面写日志这个节省时间了吗?
求大家集思广益,求帮忙?还是去想办法优化sql那。