网络中断导致HACMP已经node关闭日志，请大侠们讲讲日志过程

问题就是网络造成的收起

Unexpected termination of clstrmgrES.
如何找到被异常终止的原因。收起

关键信息
LABEL:          GS_DOM_MERGE_ER
IDENTIFIER:     9DEC29E1

Date/Time:       Sun Nov 22 16:36:34 CST 2009
Sequence Number: 1562
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           O
Type:            PERM
Resource Name:   grpsvcs         

Description
Group Services daemon exit to merge domains

Probable Causes
Network between two node groups has repaired

Failure Causes
Network communication has been blocked.
Topology Services has been partitioned.

        Recommended Actions
        Check the network connection.
Check the Topology Services.
Verify that Group Services daemon has been restarted
Call IBM Service if problem persists

Detail Data
DETECTING MODULE
RSCT,NS.C,1.107.1.35,4370                     
ERROR ID
6Vb0vR0mnP09/olj0C4.2g0...................
REFERENCE CODE
                                          
DIAGNOSTIC EXPLANATION
The master requests to dissolve my domain because of the merge with other domain 1.10


以下是errpt日志截图

LABEL:          ERRLOG_ON
IDENTIFIER:     9DBCFDEE

Date/Time:       Mon Nov 23 09:12:20 CST 2009
Sequence Number: 1568
Machine Id:      00CF884D4C00
Node Id:         localhost
Class:           O
Type:            TEMP
Resource Name:   errdemon        

Description
ERROR LOGGING TURNED ON

Probable Causes
ERRDEMON STARTED AUTOMATICALLY

User Causes
/USR/LIB/ERRDEMON COMMAND

        Recommended Actions
        NONE

---------------------------------------------------------------------------
LABEL:          OPMSG
IDENTIFIER:     AA8AB241

Date/Time:       Sun Nov 22 16:36:35 CST 2009
Sequence Number: 1567
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           O
Type:            TEMP
Resource Name:   OPERATOR        

Description
OPERATOR NOTIFICATION

User Causes
ERRLOGGER COMMAND

        Recommended Actions
        REVIEW DETAILED DATA

Detail Data
MESSAGE FROM ERRLOGGER COMMAND
clexit.rc : Unexpected termination of clstrmgrES
---------------------------------------------------------------------------
LABEL:          SRC_SVKO
IDENTIFIER:     BC3BE5A3

Date/Time:       Sun Nov 22 16:36:34 CST 2009
Sequence Number: 1566
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           S
Type:            PERM
Resource Name:   SRC            

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
        1024
SOFTWARE ERROR CODE
       -9017
ERROR CODE
           0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
clstrmgrES
---------------------------------------------------------------------------
LABEL:          SRC_RSTRT
IDENTIFIER:     BA431EB7

Date/Time:       Sun Nov 22 16:36:34 CST 2009
Sequence Number: 1565
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           S
Type:            PERM
Resource Name:   SRC            

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        VERIFY SUBSYSTEM RESTARTED AUTOMATICALLY

Detail Data
SYMPTOM CODE
           0
SOFTWARE ERROR CODE
       -9035
ERROR CODE
           0
DETECTING MODULE
'srchevn.c'@line:'217'
FAILING MODULE
emsvcs
---------------------------------------------------------------------------
LABEL:          SRC_SVKO
IDENTIFIER:     BC3BE5A3

Date/Time:       Sun Nov 22 16:36:34 CST 2009
Sequence Number: 1564
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           S
Type:            PERM
Resource Name:   SRC            

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
        2560
SOFTWARE ERROR CODE
       -9017
ERROR CODE
           0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
grpsvcs
---------------------------------------------------------------------------
LABEL:          HA002_ER
IDENTIFIER:     12081DC6

Date/Time:       Sun Nov 22 16:36:34 CST 2009
Sequence Number: 1563
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           S
Type:            PERM
Resource Name:   haemd           

Description
SOFTWARE PROGRAM ERROR

Probable Causes
SUBSYSTEM

Failure Causes
SUBSYSTEM

        Recommended Actions
        REPORT DETAILED DATA
        CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
DETECTING MODULE
LPP=PSSP,Fn=emd_gsi.c,SID=1.4.1.36,L#=1361,                                    
DIAGNOSTIC EXPLANATION
haemd: 2521-032 Cannot dispatch group services (1).

---------------------------------------------------------------------------
LABEL:          GS_DOM_MERGE_ER
IDENTIFIER:     9DEC29E1

Date/Time:       Sun Nov 22 16:36:34 CST 2009
Sequence Number: 1562
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           O
Type:            PERM
Resource Name:   grpsvcs         

Description
Group Services daemon exit to merge domains

Probable Causes
Network between two node groups has repaired

Failure Causes
Network communication has been blocked.
Topology Services has been partitioned.

        Recommended Actions
        Check the network connection.
Check the Topology Services.
Verify that Group Services daemon has been restarted
Call IBM Service if problem persists

Detail Data
DETECTING MODULE
RSCT,NS.C,1.107.1.35,4370                     
ERROR ID
6Vb0vR0mnP09/olj0C4.2g0...................
REFERENCE CODE
                                          
DIAGNOSTIC EXPLANATION
The master requests to dissolve my domain because of the merge with other domain 1.10
---------------------------------------------------------------------------
LABEL:          GOENT_RCVRY_EXIT
IDENTIFIER:     F3931284

Date/Time:       Sun Nov 22 16:36:28 CST 2009
Sequence Number: 1561
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           H
Type:            INFO
Resource Name:   ent5            
Resource Class:  adapter
Resource Type:   14106902
Location:        U5791.001.99B0FGG-P2-C06-T1
VPD:            
        Product Specific.(  ).......10/100/1000 Base-TX PCI-X Adapter
        Part Number.................03N6524
        FRU Number..................03N6524
        EC Level....................H14006
        Manufacture ID..............YL1021
        Network Address.............001125C070AF
        ROM Level.(alterable).......GOL021

Description
ETHERNET NETWORK RECOVERY MODE

        Recommended Actions
        PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
FILE NAME
line: 204 file: goent_intr.c
PCI ETHERNET STATISTICS
00C4 44C2 0063 0853 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 087E 3F51 0000 0003 3E47 3923 0000 0000 0805 323A 003C 57D5 070D C054
0000 0000 013C DCFB 0000 0000 B4FF 8862 0000 0000 0000 0000 0000 0001 0000 1493
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0018 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 BB83 00F0 0068 0C00 0000 0000 01A0 0000 0000
0000 0000 0000 0000 0000 C9E1 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000
DEVICE DRIVER INTERNAL STATE
5555 5555 0000 0000 0000 0000
SOURCE ADDRESS
0011 25C0 70AF
---------------------------------------------------------------------------
LABEL:          GOENT_RCVRY_EXIT
IDENTIFIER:     F3931284

Date/Time:       Sun Nov 22 16:36:28 CST 2009
Sequence Number: 1560
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           H
Type:            INFO
Resource Name:   ent3            
Resource Class:  adapter
Resource Type:   14106902
Location:        U5791.001.99B0FK5-P2-C06-T1
VPD:            
        Product Specific.(  ).......10/100/1000 Base-TX PCI-X Adapter
        Part Number.................03N6524
        FRU Number..................03N6524
        EC Level....................H14006
        Manufacture ID..............YL1021
        Network Address.............001125C092B9
        ROM Level.(alterable).......GOL021

Description
ETHERNET NETWORK RECOVERY MODE

        Recommended Actions
        PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
FILE NAME
line: 204 file: goent_intr.c
PCI ETHERNET STATISTICS
00C4 44C2 0063 0853 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 336D C341 0000 001C 0E4E 0779 0000 0000 306D EEAB 003C 57D4 070D BF56
0000 0000 2BA6 E691 0000 0083 FE0A 110B 0000 0000 0000 0000 0000 0001 0000 1591
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0124 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 BB83 00F0 0068 0C00 0000 0000 01A0 0000 0000
0000 0000 0000 0000 0000 C9E1 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000
DEVICE DRIVER INTERNAL STATE
5555 5555 0000 0000 0000 0000
SOURCE ADDRESS
0011 25C0 92B9
---------------------------------------------------------------------------
LABEL:          TS_LOC_DOWN_ST
IDENTIFIER:     173C787F

Date/Time:       Sun Nov 22 16:24:02 CST 2009
Sequence Number: 1559
Machine Id:      00CF884D4C00
Node Id:         node83f884d
Class:           S
Type:            INFO
Resource Name:   topsvcs         

Description
Possible malfunction on local adapter

Probable Causes
Local adapter mal-functioned
Local adapter lost connection to network
Local adapter mis-configured

Failure Causes
Local adapter mal-functioned
Local adapter lost connection to network
Local adapter mis-configured

        Recommended Actions
        Verify adapter configuration
        Verify network connectivity

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.10,4865            
ERROR ID
6zV5DL.0cP09/Qqp/C4.2g0...................
REFERENCE CODE
                                          
Adapter interface name
en5
Adapter offset
           1
Adapter IP address
192.168.100.1收起

重点是这个announcementCb: Called, state=ST_UNSTABLE, provider
token 1收起

以上日志应该是导致宕机的原因。那位老大给讲讲啊收起

不过我先在看出点门道了，贴上来大家给分析一下。
Nov 22 16:36:33 node83f884d user:notice HACMP for AIX: EVENT START: join_standby node83f884d 192.168.100.1
Nov 22 16:36:33 node83f884d user:notice HACMP for AIX: EVENT COMPLETED: join_standby node83f884d 192.168.100.1 0
Nov 22 16:36:34 node83f884d local0:crit clstrmgrES[909374]: Sun Nov 22 16:36:34 announcementCb: Called, state=ST_UNSTABLE, provider
token 1
Nov 22 16:36:34 node83f884d daemon:err|error haemd[1179652]: LPP=PSSP,Fn=emd_gsi.c,SID=1.4.1.36,L#=1361,                           
         haemd: 2521-032 Cannot dispatch group services (1).
Nov 22 16:36:34 node83f884d local0:crit clstrmgrES[909374]: Sun Nov 22 16:36:34 announcementCb: GsToken 2, AdapterToken 3, rm_GsToke
n 1
Nov 22 16:36:34 node83f884d local0:crit clstrmgrES[909374]: Sun Nov 22 16:36:34 announcementCb: GRPSVCS announcment code=512; exitin
g
Nov 22 16:36:34 node83f884d local0:crit clstrmgrES[909374]: Sun Nov 22 16:36:34  CHECK FOR FAILURE OF RSCT SUBSYSTEMS (topsvcs or gr
psvcs)
Nov 22 16:36:35 node83f884d user:notice HACMP for AIX: clexit.rc : Unexpected termination of clstrmgrES.
Nov 22 16:36:35 node83f884d user:notice HACMP for AIX: clexit.rc : Halting system immediately!!!
Nov 23 09:12:41 node83f884d daemon:notice RMCdaemon[921796]: (Recorded using libct_ffdc.a cv 2):::Error ID: 6eKora0dNe09/yXE0C4.2g0.
..................:::Reference ID:  :::Template ID: a6df45aa:::Details File:  :::Location: RSCT,rmcd.c,1.51,209                     
     :::RMCD_INFO_0_ST The daemon is started.收起

开始还真不知道。对不起了各位老大
日志太大了，没有办法啊。
还请原谅了。收起

故障原因是由于网络中断，导致一个节点的lpar关闭。问题已经解决。网络恢复后，重启hacmp服务，启动数据库，应用就正常了。
主要是我刚刚接触hacmp。没有环境装。所以不太明白网络中断后hacmp服务都在做什么，贴出日志请大侠们给讲讲。该怎么看日志。
这个应用上面做的是oracle 9i的rac。收起

我也不是很明白这个日志：
不过里面提到了资源组的问题。如下：
:process_resources[2134] eval JOB_TYPE=ERROR RESOURCE_GROUPS="iprg2"
:process_resources[2134] JOB_TYPE=ERROR RESOURCE_GROUPS=iprg2

你针对看看，能不能发现点什么？收起