DIA8336C Requested too many semaphores
最近遇到一DB2问题报信号量不足，没有资源创建进程或线程，请各位大牛帮忙看一下
前台表现：
诊断日志中内容为：
操作系统及DB2版本信息如下：
在查找这个问题时发现其与DB2: Severe system error occurs with SQLO_NORES, DIA8336C and DIA8532C messages in db2diag.log类似，但我们的操作系统是rhel，而且内核参数在问题出现前后没有任何变化，一直使用好好的，突然就这样了，实在不知道从何入手。
前台表现：
[db2inst1@BK1983T2 ~]$ db2 "connect to sample" SQL0902C A system error (reason code = "") occurred. Subsequent SQL statements cannot be processed. SQLSTATE=58005
诊断日志中内容为：
2014-09-10-10.17.40.015607+480 I82974821E510 LEVEL: Severe PID : 3011 TID : 47852378122560PROC : db2sysc INSTANCE: db2inst1 NODE : 000 DB : SAMPLE APPHDL : 0-9 APPID: 127.0.0.1.35025.140910021738 AUTHID : sample EDUID : 28 EDUNAME: db2agent (sample) FUNCTION: DB2 UDB, global services, sqzEDUObj::StartEDU, probe:10 RETCODE : ZRC=0x870F00F2=-2029059854=SQLO_NORES "no resources to create process or thread" 2014-09-10-10.47.49.303211+480 I83006314E438 LEVEL: Severe PID : 3011 TID : 47852436842816PROC : db2sysc INSTANCE: db2inst1 NODE : 000 EDUID : 14 EDUNAME: db2ipccm FUNCTION: DB2 UDB, common communication, sqlccGetIPCs, probe:50 RETCODE : ZRC=0x850F0081=-2062614399=SQLO_SSEM_EXCEED_MAX "Requesting too many semaphores" DIA8336C Requested too many semaphores.
操作系统及DB2版本信息如下：
操作系统版本：Red Hat Enterprise Linux Server release 5.3 (Tikanga) 数据库版本：DB2 v9.7.0.0
在查找这个问题时发现其与DB2: Severe system error occurs with SQLO_NORES, DIA8336C and DIA8532C messages in db2diag.log类似，但我们的操作系统是rhel，而且内核参数在问题出现前后没有任何变化，一直使用好好的，突然就这样了，实在不知道从何入手。
附件：
db2trc.tar.gz (140.84 KB)
3同行回答
IPC配置信息：
------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 16777216 max total shared memory (kbytes) = 33554432 min seg size (bytes) = 1 ------ Semaphore Limits -------- max number of arrays = 1024 max semaphores per array = 500 max semaphores system wide = 256000 max ops per semop call = 250 semaphore max value = 32767 ------ Messages: Limits -------- max queues system wide = 16384 max size of message (bytes) = 65536 default max size of queue (bytes) = 65536收起
浏览2810
3375 0.053937000 | | | | pdIsDiagLevelOk exit 3376 0.053939000 | | | | pdLogInternal entry [eduid 14 eduname db2ipccm] 3377 0.053940000 | | | | | pdLogMustIgnoreFODCLogging entry [eduid 14 eduname db2ipccm] 3378 0.053941000 | | | | | pdLogMustIgnoreFODCLogging exit 3379 0.053943000 | | | | | sqltGetDiagPath entry [eduid 14 eduname db2ipccm] 3380 0.053946000 | | | | | sqltGetDiagPath data [probe 10] 3381 0.053947000 | | | | | sqltGetDiagPath exit 3382 0.053951000 | | | | pdLogInternal data [probe 130] 3383 0.053961000 | | | | | sqloGetInstanceOwnerName entry [eduid 14 eduname db2ipccm] 3384 0.053962000 | | | | | sqloGetInstanceOwnerName exit 3385 0.053964000 | | | | | sqloGetProgramName entry [eduid 14 eduname db2ipccm] 3386 0.053966000 | | | | | sqloGetProgramName exit 3387 0.053967000 | | | | | sqloGetName entry [eduid 14 eduname db2ipccm] 3388 0.053968000 | | | | | sqloGetName exit 3389 0.053970000 | | | | | sqltgets entry [eduid 14 eduname db2ipccm] 3390 0.053972000 | | | | | | sqloGetName entry [eduid 14 eduname db2ipccm] 3391 0.053972000 | | | | | | sqloGetName exit 3392 0.053974000 | | | | | sqltgets exit 3393 0.053976000 | | | | | sqloSigMask entry [eduid 14 eduname db2ipccm] 3394 0.053978000 | | | | | sqloSigMask exit 3395 0.053981000 | | | | pdLogInternal data [probe 110] 3396 0.053982000 | | | | | sqloSigMask entry [eduid 14 eduname db2ipccm] 3397 0.053983000 | | | | | sqloSigMask exit 3398 0.053986000 | | | | | sqloSigMask entry [eduid 14 eduname db2ipccm] 3399 0.053987000 | | | | | sqloSigMask exit 3400 0.056852000 | | | | | sqloopenp entry [eduid 14 eduname db2ipccm] 3401 0.056871000 | | | | | sqloopenp mbt [Marker:PD_SQO_OPEN_WRITE_FOR_SHARED_LOCK ] 3402 0.056887000 | | | | | | sqloflock entry [eduid 14 eduname db2ipccm] 3403 0.056891000 | | | | | | sqloflock exit 3404 0.056893000 | | | | | sqloopenp exit [rc = 0x070F00CB = 118423755 = SQLO_FREV] 3405 0.056894000 | | | | | sqloSigMask entry [eduid 14 eduname db2ipccm] 3406 0.056896000 | | | | | sqloSigMask exit 3407 0.056898000 | | | | | sqlochgfileptr entry [eduid 14 eduname db2ipccm] 3408 0.056902000 | | | | | sqlochgfileptr exit 3409 0.056967000 | | | | pdLogInternal data [probe 100] 3410 0.056976000 | | | | | sqlzGetZRCInfo entry [eduid 14 eduname db2ipccm] 3411 0.056977000 | | | | | sqlzGetZRCInfo exit [rc = 0xFFFFFC7A = -902] 3412 0.056979000 | | | | | sqlogmsg_noconv entry [eduid 14 eduname db2ipccm] 3413 0.056980000 | | | | | | sqlocptrs entry [eduid 14 eduname db2ipccm] 3414 0.056982000 | | | | | | sqlocptrs exit 3415 0.056983000 | | | | | | sqlnlsGetRegVars entry [eduid 14 eduname db2ipccm] 3416 0.056988000 | | | | | | | sqloGetEnvInternal entry [eduid 14 eduname db2ipccm] 3417 0.056989000 | | | | | | | sqloGetEnvInternal exit [rc = 0x870F0104 = -2029059836 = RC_ENV_NOT_FOUND] 3418 0.056990000 | | | | | | | sqloGetEnvInternal entry [eduid 14 eduname db2ipccm] 3419 0.056991000 | | | | | | | sqloGetEnvInternal exit [rc = 0x870F0104 = -2029059836 = RC_ENV_NOT_FOUND] 3420 0.056992000 | | | | | | | sqloGetEnvInternal entry [eduid 14 eduname db2ipccm] 3421 0.056993000 | | | | | | | sqloGetEnvInternal exit [rc = 0x870F0104 = -2029059836 = RC_ENV_NOT_FOUND] 3422 0.056994000 | | | | | | | sqloGetEnvInternal entry [eduid 14 eduname db2ipccm] 3423 0.056995000 | | | | | | | sqloGetEnvInternal exit [rc = 0x870F0104 = -2029059836 = RC_ENV_NOT_FOUND] 3424 0.056996000 | | | | | | | sqloGetEnvInternal entry [eduid 14 eduname db2ipccm] 3425 0.056997000 | | | | | | | sqloGetEnvInternal exit [rc = 0x870F0104 = -2029059836 = RC_ENV_NOT_FOUND] 3426 0.056998000 | | | | | | | sqloGetEnvInternal entry [eduid 14 eduname db2ipccm] 3427 0.056999000 | | | | | | | sqloGetEnvInternal exit [rc = 0x870F0104 = -2029059836 = RC_ENV_NOT_FOUND] 3428 0.057002000 | | | | | | sqlnlsGetRegVars data [probe 90] 3429 0.057005000 | | | | | | sqlnlsGetRegVars exit 3430 0.057012000 | | | | | | sqlnlsgmsg_cpcv entry 3431 0.057013000 | | | | | | | sqlnlsMessage entry 3432 0.057015000 | | | | | | | sqlnlsMessage data [probe 10] 3433 0.057022000 | | | | | | | | sqlnlscmsg entry 3434 0.057022000 | | | | | | | | sqlnlscmsg data [probe 10] 3435 0.057025000 | | | | | | | | sqlnlscmsg data [probe 20] 3436 0.057027000 | | | | | | | | | sqlnlsmapcc entry 3437 0.057028000 | | | | | | | | | sqlnlsmapcc data [probe 1] 3438 0.057034000 | | | | | | | | | sqlnlsmapcc data [probe 2] 3439 0.057037000 | | | | | | | | | sqlnlsmapcc exit 3440 0.057039000 | | | | | | | | sqlnlscmsg data [probe 30] 3441 0.057041000 | | | | | | | | sqlnlscmsg data [probe 40] 3442 0.057042000 | | | | | | | | sqlnlscmsg data [probe 100] 3443 0.057043000 | | | | | | | | sqlnlscmsg exit 3444 0.057044000 | | | | | | | sqlnlsMessage data [probe 30] 3445 0.057045000 | | | | | | | sqlnlsMessage data [probe 40] 3446 0.057047000 | | | | | | | sqlnlsMessage data [probe 50] 3447 0.057049000 | | | | | | | sqlnlsMessage data [probe 60] 3448 0.057051000 | | | | | | | sqlnlsMessage data [probe 90] 3449 0.057074000 | | | | | | | sqlnlsMessage data [probe 150] 3450 0.057075000 | | | | | | | sqlnlsMessage data [probe 170] 3451 0.057077000 | | | | | | | sqlnlsMessage exit 3452 0.057080000 | | | | | | sqlnlsgmsg_cpcv exit [rc = 0x00000028 = 40] 3453 0.057082000 | | | | | sqlogmsg_noconv exit [rc = 0x00000028 = 40] 3454 0.057088000 | | | | | sqlowrite entry [eduid 14 eduname db2ipccm] 3455 0.057134000 | | | | | sqlowrite exit 3456 0.057138000 | sqlccGetIPCs data [probe 50] 3457 0.057143000 | | | | | sqloclose entry [eduid 14 eduname db2ipccm] 3569 0.065174000 | | | | | sqloclose exit 3570 0.065178000 | | | | | sqloSigMask entry [eduid 14 eduname db2ipccm] 3571 0.065180000 | | | | | sqloSigMask exit 3572 0.065181000 | | | | | sqloSigMask entry [eduid 14 eduname db2ipccm] 3573 0.065183000 | | | | | sqloSigMask exit 3574 0.065184000 | | | | pdLogInternal exit 3575 0.065185000 | | | pdLogRC exit 3576 0.065186000 | | sqlt_logerr_zrc exit 3577 0.065188000 | | sqlt_logerr_data (secondary logging function) entry [eduid 14 eduname db2ipccm] 3578 0.065190000 | | sqlt_logerr_data (secondary logging function) data [probe 10] 3579 0.065192000 | | sqlt_logerr_data (secondary logging function) data [probe 20]
更加详细的db2trc内容见收起
浏览2753
我做了一个trace，格式化后内容如下：【fmt文件内容】
【flw文件内容】
3456 data DB2 UDB common communication sqlccGetIPCs fnc (3.3.43.114.0.50) pid 3011 tid 47852436842816 cpid 792 node 0 sec 0 nsec 57138000 probe 50 bytes 447 Data1 (PD_TYPE_DIAG_LOG_REC,439) Diagnostic log record: 2014-09-10-14.09.03.257667+480 I363256E438 LEVEL: Severe PID : 3011 TID : 47852436842816PROC : db2sysc INSTANCE: db2inst1 NODE : 000 EDUID : 14 EDUNAME: db2ipccm FUNCTION: DB2 UDB, common communication, sqlccGetIPCs, probe:50 RETCODE : ZRC=0x850F0081=-2062614399=SQLO_SSEM_EXCEED_MAX "Requesting too many semaphores" DIA8336C Requested too many semaphores.
【flw文件内容】
pid = 3011 tid = 47852436842816 node = 0 3324 0.053060000 sqlccipc_process_conn entry [eduid 14 eduname db2ipccm] 3325 0.053075000 | sqlccGetIPCs entry [eduid 14 eduname db2ipccm] 3326 0.053688000 | | sqlogmshr entry [eduid 14 eduname db2ipccm] 3327 0.053705000 | | sqlogmshr data [probe 10] 3328 0.053709000 | | sqlogmshr data [probe 30] 3329 0.053711000 | | sqlogmshr data [probe 132] 3330 0.053714000 | | | sqloGetEnvInternal entry [eduid 14 eduname db2ipccm] 3331 0.053731000 | | | sqloGetEnvInternal exit [rc = 0x870F0104 = -2029059836 = RC_ENV_NOT_FOUND] 3332 0.053735000 | | sqlogmshr data [probe 142] 3333 0.053738000 | | | SqloMemController::registerConsumer entry [eduid 14 eduname db2ipccm] 3334 0.053742000 | | | | sqlogmblkEx entry [eduid 14 eduname db2ipccm] 3335 0.053769000 | | | | sqlogmblkEx mbt [Marker:PD_OSS_ALLOCATED_MEMORY ] 3336 0.053770000 | | | | sqlogmblkEx exit 3337 0.053774000 | | | | SqloMemController::requestMemory entry [eduid 14 eduname db2ipccm] 3338 0.053781000 | | | | SqloMemController::requestMemory exit 3339 0.053782000 | | | SqloMemController::registerConsumer exit 3340 0.053784000 | | | sqloAllocateSharedMemory entry [eduid 14 eduname db2ipccm] 3341 0.053785000 | | | sqloAllocateSharedMemory data [probe 110] 3342 0.053788000 | | | | sqloGetSharedMemoryFromOs entry [eduid 14 eduname db2ipccm] 3343 0.053790000 | | | | | sqloMemCreateSingleSegment entry [eduid 14 eduname db2ipccm] 3344 0.053800000 | | | | | sqloMemCreateSingleSegment exit 3345 0.053801000 | | | | sqloGetSharedMemoryFromOs data [probe 2020] 3346 0.053803000 | | | | | sqloMemAttachToSegments entry [eduid 14 eduname db2ipccm] 3347 0.053809000 | | | | | sqloMemAttachToSegments exit 3348 0.053812000 | | | | sqloGetSharedMemoryFromOs exit 3349 0.053815000 | | | sqloAllocateSharedMemory exit 3350 0.053816000 | | sqlogmshr data [probe 250] 3351 0.053817000 | | sqlogmshr exit 3352 0.053821000 | | sqloashm entry [eduid 14 eduname db2ipccm] 3353 0.053824000 | | sqloashm data [probe 10] 3354 0.053828000 | | | sqloGetUserAttribById entry [eduid 14 eduname db2ipccm] 3355 0.053834000 | | | | sqloxltc_app entry [eduid 14 eduname db2ipccm] 3356 0.053835000 | | | | sqloxltc_app exit 3357 0.053838000 | | | sqloGetUserAttribById data [probe 62] 3358 0.053895000 | | | sqloGetUserAttribById data [probe 66] 3359 0.053898000 | | | | sqloxult_app entry [eduid 14 eduname db2ipccm] 3360 0.053900000 | | | | sqloxult_app exit 3361 0.053902000 | | | sqloGetUserAttribById exit 3362 0.053904000 | | sqloashm exit 3363 0.053906000 | | sqloSSemSetCreate entry [eduid 14 eduname db2ipccm] 3364 0.053911000 | | sqloSSemSetCreate exit [rc = 0x850F0081 = -2062614399 = SQLO_SSEM_EXCEED_MAX] 3365 0.053917000 | sqlccGetIPCs error [probe 40] 3366 0.053919000 | | sqlt_logerr_zrc entry [eduid 14 eduname db2ipccm] 3367 0.053920000 | | sqlt_logerr_zrc data [probe 10] 3368 0.053922000 | | | sqlzGetZRCInfo entry [eduid 14 eduname db2ipccm] 3369 0.053923000 | | | sqlzGetZRCInfo exit [rc = 0xFFFFFC7A = -902] 3370 0.053930000 | | | pdLogRC entry [eduid 14 eduname db2ipccm] 3371 0.053932000 | | | | pdIsDiagLevelOk entry [eduid 14 eduname db2ipccm] 3372 0.053934000 | | | | pdIsDiagLevelOk data [probe 10] 3373 0.053935000 | | | | pdIsDiagLevelOk data [probe 20] 3374 0.053936000 | | | | pdIsDiagLevelOk data [probe 500] 3375 0.053937000 | | | | pdIsDiagLevelOk exit收起
浏览2940