双机切换后备机宕库
对数据库进行双机切换,备机接管成功后,过几分钟报断言宕库,触发双机切换,备机能正常拉起,当前HDR关系处于on状态。
备机日志:
00:44:00 Maximum server connections 0
00:45:59 Booting Language from module <>
00:45:59 Loading Module
01:00:05 Checkpoint Completed: duration was 0 seconds.
01:00:05 Checkpoint loguniq 22990, logpos 0xbea32c, timestamp: 0x62af3d5d
01:00:05 Maximum server connections 9
01:08:20 Checkpoint Completed: duration was 2 seconds.
01:08:20 Checkpoint loguniq 22991, logpos 0x832c, timestamp: 0x62af447d
01:08:20 Maximum server connections 9
01:09:40 Checkpoint Completed: duration was 0 seconds.
01:09:40 Checkpoint loguniq 22991, logpos 0x12018, timestamp: 0x62af4572
01:09:40 Maximum server connections 9
01:09:41 DR: Receive error in thread dr_secrcv [id: 167]
01:09:41 ASF Echo-Thread Server: asfcode = -25582: oserr = 0: errstr = : Network connection is broken.
01:09:42 DR: Turned off on secondary server
01:09:50 Logical Recovery has reached the transaction cleanup phase.
01:09:50 Logical Recovery Complete.
109642 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks
01:09:50 Logical Recovery Complete.
01:09:51 Quiescent Mode
01:09:51 Logical Log 22990 Complete, timestamp: 0x62af4575.
01:09:52 Checkpoint Completed: duration was 0 seconds.
01:09:52 Checkpoint loguniq 22991, logpos 0x13018, timestamp: 0x62af4583
01:09:52 Maximum server connections 9
01:09:52 DR: Reservation of the last logical log for log backup turned on
01:09:52 DR: new type = standard
01:09:52 On-Line Mode
01:09:53 DR: new type = primary, secondary server name = gzvc3_server1_net
01:09:53 DR: Trying to connect to secondary server = gzvc3_server1_net
01:09:54 DR: Cannot connect to secondary server
01:09:54 DR: Turned off on primary server
01:09:56 (116) connection rejected - no calls allowed for cdraccept
01:10:26 (126) connection rejected - no calls allowed for cdraccept
01:11:12 (135) connection rejected - no calls allowed for cdraccept
01:12:11 (150) connection rejected - no calls allowed for cdraccept
01:13:27 (163) connection rejected - no calls allowed for cdraccept
01:14:03 DR: Primary server connected
01:14:03 DR: Secondary server needs failure recovery
01:14:05 DR: Sending log 22991 (current), size 100000 pages, 0.04 percent used
01:14:06 DR: Sending Logical Logs Completed
01:14:07 DR: Primary server operational
01:14:09 Checkpoint Completed: duration was 2 seconds.
01:14:09 Checkpoint loguniq 22991, logpos 0x2f2b4, timestamp: 0x62af55bc
01:14:09 Maximum server connections 11
01:14:28 Assert Failed: semop: errno = 36
01:14:28 IBM Informix Dynamic Server Version 9.40.FC7XF
01:14:28 Who: Session(0, @, 0, 0000000000000000)
Thread(0, idle, 0, 14)
File: mt.c Line: 1345
01:14:28 stack trace for pid 5391 written to /tellinshare/informix_tmp/af.ea2f3
01:14:28 Assert Failed: semop: errno = 36
01:14:28 IBM Informix Dynamic Server Version 9.40.FC7XF
01:14:28 Who: Session(0, @, 0, 0000000000000000)
Thread(0, idle, 0, 16)
File: mt.c Line: 1345
01:14:28 stack trace for pid 5393 written to /tellinshare/informix_tmp/af.10a2f3
01:14:28 Out of virtual shared memory, cannot grow exception stack
01:14:28 ERROR: Exception recursion in afhandler (thread 0).
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 semctl: errno = 22
01:14:28 IBM Informix Dynamic Server Version 9.40.FC7XF Software Serial Number AAA#B000000
01:14:28 Assert Failed: semop: errno = 36
01:14:28 Who: Session(0, @, 0, 0000000000000000)
Thread(0, idle, 0, 16)
File: mt.c Line: 1345
01:14:28 Stack for thread: 0 idle
base: 0x800000010022c7d8
len: 32768
pc: 0x0000000000000000
tos: 0x800000010022db20
state: running
vp: 16
af.10a2f3:
( 0) 0x40000000008a7408 legacy_hp_afstack + 0x258 [/opt/informix/bin/oninit]
( 1) 0x40000000008a6a5c afstack + 0x5c [/opt/informix/bin/oninit]
( 2) 0x40000000008a5cec afhandler + 0x984 [/opt/informix/bin/oninit]
( 3) 0x40000000008a5354 afcrash_interface + 0x4c [/opt/informix/bin/oninit]
( 4) 0x400000000086cfb8 P + 0x128 [/opt/informix/bin/oninit]
( 5) 0x4000000000870914 idle_processor + 0x5ac [/opt/informix/bin/oninit]
( 6) 0x400000000087e334 startup + 0xd4 [/opt/informix/bin/oninit]
( 7) 0x400000000087e11c resume + 0x10c [/opt/informix/bin/oninit]
base: 0x800000010022c7d8
len: 32768
pc: 0x0000000000000000
tos: 0x800000010022db20
state: running
vp: 16
现在怀疑是初始共享内存太小,造成informix后续处理中共享内存不够,但运行了好几年平时切换没出现这个问题;各位大侠有啥见解?谢谢!