HA服务down掉

2台P720,HA6.1,HA服务突然down了,资源也没有切换,查看了cluster.log,显示如下:说是由于lv不完整导致了vg不能vary on,但是什么原因导致了lv不完整呢?还需要看哪些信息能确定原因呢?谢谢了
Dec 10 16:05:14 zz_boss2 user:notice HACMP for AIX: cl_mirrorset: Unable to vary on volume group informixvg because logical volume lv_tmpchunk1 is incomplete
Dec 10 16:05:15 zz_boss2 user:notice last message repeated 3 times
Dec 10 16:05:18 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move_release zz_boss2 1
Dec 10 16:05:18 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move zz_boss2 1 RELEASE
Dec 10 16:05:18 zz_boss2 user:notice HACMP for AIX: EVENT START: node_down_local
Dec 10 16:05:18 zz_boss2 user:notice HACMP for AIX: cl_mirrorset: Unable to vary on volume group informixvg because logical volume lv_tmpchunk1 is incomplete
Dec 10 16:05:18 zz_boss2 user:notice HACMP for AIX: EVENT START: stop_server informix_server
Dec 10 16:05:19 zz_boss2 user:notice HACMP for AIX: cl_mirrorset: Unable to vary on volume group informixvg because logical volume lv_tmpchunk1 is incomplete
Dec 10 16:05:34 zz_boss2 user:notice last message repeated 19 times
Dec 10 16:05:39 zz_boss2 user:notice HACMP for AIX: cl_mirrorset: Unable to vary on volume group informixvg because logical volume lv_tmpchunk1 is incomplete
Dec 10 16:07:12 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: stop_server informix_server 0
Dec 10 16:07:12 zz_boss2 user:notice HACMP for AIX: EVENT START: release_vg_fs ALL informixvg informixdatavg
Dec 10 16:07:17 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: release_vg_fs ALL informixvg informixdatavg 0
Dec 10 16:07:17 zz_boss2 user:notice HACMP for AIX: EVENT START: release_service_addr serv_b
Dec 10 16:07:19 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: release_service_addr serv_b 0
Dec 10 16:07:19 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: node_down_local 0
Dec 10 16:07:19 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move zz_boss2 1 RELEASE 0
Dec 10 16:07:19 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move_release zz_boss2 1 0
Dec 10 16:07:21 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move_fence zz_boss2 1
Dec 10 16:07:21 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move_fence zz_boss2 1 0
Dec 10 16:07:21 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move_acquire zz_boss2 1
Dec 10 16:07:21 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move zz_boss2 1 ACQUIRE
Dec 10 16:07:21 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move zz_boss2 1 ACQUIRE 0
Dec 10 16:07:22 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move_acquire zz_boss2 1 0
Dec 10 16:07:27 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move_complete zz_boss2 1
Dec 10 16:07:27 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move_complete zz_boss2 1 0
Dec 10 16:07:30 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move_release zz_boss1 1
Dec 10 16:07:30 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move zz_boss1 1 RELEASE
Dec 10 16:07:30 zz_boss2 user:notice HACMP for AIX: EVENT START: node_down_local
Dec 10 16:07:30 zz_boss2 user:notice HACMP for AIX: EVENT START: stop_server informix_server
Dec 10 16:07:31 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: stop_server informix_server 0
Dec 10 16:07:31 zz_boss2 user:notice HACMP for AIX: EVENT START: release_vg_fs ALL informixvg informixdatavg
Dec 10 16:07:32 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: release_vg_fs ALL informixvg informixdatavg 0
Dec 10 16:07:32 zz_boss2 user:notice HACMP for AIX: EVENT START: release_service_addr serv_b
Dec 10 16:07:32 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: release_service_addr serv_b 0
Dec 10 16:07:32 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: node_down_local 0
Dec 10 16:07:32 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move zz_boss1 1 RELEASE 0
Dec 10 16:07:32 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move_release zz_boss1 1 0
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move_fence zz_boss1 1
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move_fence zz_boss1 1 0
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move_acquire zz_boss1 1
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move zz_boss1 1 ACQUIRE
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move zz_boss1 1 ACQUIRE 0
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move_acquire zz_boss1 1 0
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT START: rg_move_complete zz_boss1 1
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT START: node_up_remote_complete zz_boss1
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: node_up_remote_complete zz_boss1 0
Dec 10 16:07:36 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: rg_move_complete zz_boss1 1 0
Dec 10 16:19:31 zz_boss2 daemon:notice telnetd[24969318]: telnet from ::ffff:10.68.0.198 on /dev/pts/0
Dec 10 16:19:40 zz_boss2 daemon:notice telnetd[8585282]: telnet from ::ffff:10.68.0.118 on /dev/pts/1
Dec 10 16:23:39 zz_boss2 daemon:notice telnetd[10420420]: telnet from ::ffff:10.68.0.118 on /dev/pts/1
Dec 10 16:30:31 zz_boss2 daemon:notice telnetd[45285440]: telnet from ::ffff:10.68.0.199 on /dev/pts/2
Dec 10 16:31:25 zz_boss2 daemon:notice telnetd[9764902]: telnet from ::ffff:10.68.0.93 on /dev/pts/3
Dec 10 16:33:21 zz_boss2 daemon:notice telnetd[11469010]: telnet from ::ffff:10.68.0.199 on /dev/pts/4
Dec 10 16:37:08 zz_boss2 user:notice HACMP for AIX: EVENT START: node_down zz_boss2 forced
Dec 10 16:37:08 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: node_down zz_boss2 forced 0
Dec 10 16:37:09 zz_boss2 user:notice HACMP for AIX: EVENT START: node_down_complete zz_boss2 forced
Dec 10 16:37:09 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: node_down_complete zz_boss2 forced 0
Dec 10 16:37:10 zz_boss2 local0:crit clstrmgrES[6422556]: Wed Dec 10 16:37:10 approvalCb: Checking previous BEING_FORCE_DOWN flag
Dec 10 16:37:10 zz_boss2 local0:crit clstrmgrES[6422556]: Wed Dec 10 16:37:10 SendRmInfoBcast: ha_gs_send_message() failed rc=8
Dec 10 16:42:00 zz_boss2 daemon:notice xntpd[4915350]: 3.4y
Dec 10 16:42:00 zz_boss2 daemon:notice xntpd[4915350]: tickadj = 1000, tick = 10000, tvu_maxslew = 99000
Dec 10 16:42:08 zz_boss2 daemon:notice RMCdaemon[8061174]: (Recorded using libct_ffdc.a cv 2):::Error ID: 6eKora0UR.WI/0.m102E./....................:::Reference ID:  :::Template ID: a6df45aa:::Details File:  :::Location: RSCT,rmcd.c,1.94,239                          :::RMCD_INFO_0_ST The daemon is started.
Dec 10 16:42:12 zz_boss2 daemon:notice telnetd[7405818]: telnet from ::ffff:10.68.0.198 on /dev/pts/0
Dec 10 16:42:34 zz_boss2 daemon:notice telnetd[8388658]: telnet from serv_a on /dev/pts/1
Dec 10 16:43:36 zz_boss2 daemon:notice telnetd[8716320]: telnet from ::ffff:10.68.0.93 on /dev/pts/2
Dec 10 16:44:29 zz_boss2 local0:crit clstrmgrES[6553610]: Wed Dec 10 16:44:29 HACMP: clstrmgrES: VRMF fix level in product ODM = 12
Dec 10 16:44:29 zz_boss2 local0:crit clstrmgrES[6553610]: Wed Dec 10 16:44:29 CLSTR_JOIN_AUTO_START - This is the normal start request
Dec 10 16:44:29 zz_boss2 daemon:notice topsvcs[18284704]: (Recorded using libct_ffdc.a cv 2):::Error ID: 6UpNEL0hT.WI/KsG102E./....................:::Reference ID:  :::Template ID: 97419d60:::Details File:  :::Location: rsct,bootstrp.C,1.215.1.13,4956               :::TS_START_ST Topology Services daemon started Topology Services daemon started by: SRC Topology Services daemon log file location /var/ha/log/topsvcs.10.164429.zz_boss_cluster.en_US Topology Services daemon run directory /var/ha/run/topsvcs.zz_boss_cluster/
Dec 10 16:44:31 zz_boss2 daemon:notice grpsvcs[17956936]: (Recorded using libct_ffdc.a cv 2):::Error ID: 63Y7ej0jT.WI/YIo002E./....................:::Reference ID:  :::Template ID: afa89905:::Details File:  :::Location: RSCT,pgsd.C,1.62.1.27,710                     :::GS_START_ST Group Services daemon started DIAGNOSTIC EXPLANATION HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_trace.
Dec 10 16:44:55 zz_boss2 user:notice HACMP for AIX: EVENT START: node_up zz_boss2
Dec 10 16:44:57 zz_boss2 user:notice HACMP for AIX: EVENT START: acquire_service_addr
Dec 10 16:44:58 zz_boss2 user:notice HACMP for AIX: EVENT START: acquire_aconn_service en6 net_ether_01
Dec 10 16:44:58 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: acquire_aconn_service en6 net_ether_01 0
Dec 10 16:44:58 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: acquire_service_addr 0
Dec 10 16:45:05 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: node_up zz_boss2 0
Dec 10 16:45:05 zz_boss2 user:notice HACMP for AIX: EVENT START: node_up_complete zz_boss2
Dec 10 16:45:05 zz_boss2 user:notice HACMP for AIX: Warning: syncvg can take considerable amount of time, depending on data size and network bandwidth.
Dec 10 16:45:05 zz_boss2 user:notice HACMP for AIX: EVENT START: start_server informix_server
Dec 10 16:45:05 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: start_server informix_server 0
Dec 10 16:45:06 zz_boss2 user:notice HACMP for AIX: EVENT COMPLETED: node_up_complete zz_boss2 0
Dec 10 16:45:16 zz_boss2 user:notice HACMP for AIX: syncvg completed successfully
Dec 10 16:45:16 zz_boss2 user:notice HACMP for AIX: Warning: syncvg can take considerable amount of time, depending on data size and network bandwidth.
Dec 10 16:45:24 zz_boss2 daemon:notice telnetd[9240664]: telnet from ::ffff:10.68.0.198 on /dev/pts/3
Dec 10 16:45:27 zz_boss2 user:notice HACMP for AIX: syncvg completed successfully
Dec 10 16:45:40 zz_boss2 daemon:notice telnetd[4522074]: telnet from ::ffff:10.68.0.118 on /dev/pts/4
参与2

1同行回答

zwz99999zwz99999系统工程师dcits
如果是lv不完整的话,应该是两边的vg信息不一致导致的,共享vg重新做导出导入,ha重新同步一下即可!显示全部
如果是lv不完整的话,应该是两边的vg信息不一致导致的,共享vg重新做导出导入,ha重新同步一下即可!收起
系统集成 · 2014-12-11
浏览1577

提问者

青铜1989
系统运维工程师北京银信长远科技股份有限公司
擅长领域: 服务器存储分布式系统

相关问题

相关资料

问题状态

  • 发布时间:2014-12-10
  • 关注会员:1 人
  • 问题浏览:4217
  • 最近回答:2014-12-11
  • X社区推广