互联网服务故障诊断
aix 5.3hacm5.4双机自动切换/usr文件系统满
大家好，
双机环境 操作系统版本5300-07 hacmp 5.4
在10号发生双机切换，看日志是双机监控进程异常导致切换资源组，但未找到具体原因，请各位给分析一下，多谢。
Jan 10 00:56:22 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:56:24 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:27 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:56:32 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 11) (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 77) (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 23) (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 11) (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 15) (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 78) (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 76) (127.0.0.1+38839+7)
Jan 10 00:56:39 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:39 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:39 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 64) (127.0.0.1+38839+7)
Jan 10 00:56:39 1 user:notice HACMP for AIX: EVENT START: server_down infox1 12 ora_app_mnt
Jan 10 00:56:39 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 100) (127.0.0.1+38839+7)
Jan 10 00:56:39 1 user:notice HACMP for AIX: EVENT COMPLETED: server_down infox1 12 ora_app_mnt 0
Jan 10 00:56:39 1 user:notice HACMP for AIX: EVENT START: server_down_complete infox1 12 ora_app_mnt
Jan 10 00:56:39 1 user:notice HACMP for AIX: EVENT COMPLETED: server_down_complete infox1 12 ora_app_mnt 0
Jan 10 00:56:41 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:41 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 98) (127.0.0.1+38839+7)
Jan 10 00:56:42 1 user:notice HACMP for AIX: EVENT START: rg_move_release infox1 2
Jan 10 00:56:42 1 user:notice HACMP for AIX: EVENT START: rg_move infox1 2 RELEASE
Jan 10 00:56:42 1 user:notice HACMP for AIX: EVENT START: node_down_local
Jan 10 00:56:42 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:42 1 user:notice HACMP for AIX: EVENT START: stop_server ora_app
Jan 10 00:56:43 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:43 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 23) (127.0.0.1+38839+7)
Jan 10 00:56:43 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:43 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 77) (127.0.0.1+38839+7)
Jan 10 00:56:48 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:52 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:52 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:56:54 1 user:notice HACMP for AIX: EVENT COMPLETED: stop_server ora_app 0
Jan 10 00:56:54 1 user:notice HACMP for AIX: EVENT START: release_vg_fs ALL webvg backupvg1 backupvg2 expvg vgora
Jan 10 00:56:5Jan 10 00:57:51 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:57:511 daemon:err|error snmpd[266370]: EXCEPTIONS: cannot write to log file /usr/tmp/snmpd.log: There is not enough space in the file system.
Jan 10 00:57:531 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:57:531 daemon:err|error snmpd[266370]: EXCEPTIONS: cannot write to log file /usr/tmp/snmpd.log: There is not enough space in the file system.
Jan 10 00:57:541 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:57:541 daemon:err|error snmpd[266370]: EXCEPTIONS: cannot write to log file /usr/tmp/snmpd.log: There is not enough space in the file system.
双机环境 操作系统版本5300-07 hacmp 5.4
在10号发生双机切换，看日志是双机监控进程异常导致切换资源组，但未找到具体原因，请各位给分析一下，多谢。
Jan 10 00:56:22 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:56:24 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:27 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:56:32 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 11) (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 77) (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:37 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 23) (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 11) (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 15) (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 78) (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:38 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 76) (127.0.0.1+38839+7)
Jan 10 00:56:39 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:39 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:39 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 64) (127.0.0.1+38839+7)
Jan 10 00:56:39 1 user:notice HACMP for AIX: EVENT START: server_down infox1 12 ora_app_mnt
Jan 10 00:56:39 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 100) (127.0.0.1+38839+7)
Jan 10 00:56:39 1 user:notice HACMP for AIX: EVENT COMPLETED: server_down infox1 12 ora_app_mnt 0
Jan 10 00:56:39 1 user:notice HACMP for AIX: EVENT START: server_down_complete infox1 12 ora_app_mnt
Jan 10 00:56:39 1 user:notice HACMP for AIX: EVENT COMPLETED: server_down_complete infox1 12 ora_app_mnt 0
Jan 10 00:56:41 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:41 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 98) (127.0.0.1+38839+7)
Jan 10 00:56:42 1 user:notice HACMP for AIX: EVENT START: rg_move_release infox1 2
Jan 10 00:56:42 1 user:notice HACMP for AIX: EVENT START: rg_move infox1 2 RELEASE
Jan 10 00:56:42 1 user:notice HACMP for AIX: EVENT START: node_down_local
Jan 10 00:56:42 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:42 1 user:notice HACMP for AIX: EVENT START: stop_server ora_app
Jan 10 00:56:43 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:43 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 23) (127.0.0.1+38839+7)
Jan 10 00:56:43 1 daemon:notice snmpd[266370]: NOTICE: SMUX packet from (127.0.0.1+38839+7)
Jan 10 00:56:43 1 daemon:notice snmpd[266370]: NOTICE: SMUX trap: (6 77) (127.0.0.1+38839+7)
Jan 10 00:56:48 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:52 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:56:52 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:56:54 1 user:notice HACMP for AIX: EVENT COMPLETED: stop_server ora_app 0
Jan 10 00:56:54 1 user:notice HACMP for AIX: EVENT START: release_vg_fs ALL webvg backupvg1 backupvg2 expvg vgora
Jan 10 00:56:5Jan 10 00:57:51 1 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:57:511 daemon:err|error snmpd[266370]: EXCEPTIONS: cannot write to log file /usr/tmp/snmpd.log: There is not enough space in the file system.
Jan 10 00:57:531 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: aa@1
Jan 10 00:57:531 daemon:err|error snmpd[266370]: EXCEPTIONS: cannot write to log file /usr/tmp/snmpd.log: There is not enough space in the file system.
Jan 10 00:57:541 daemon:err|error snmpd[266370]: EXCEPTIONS: authentication error: invalid community name: public
Jan 10 00:57:541 daemon:err|error snmpd[266370]: EXCEPTIONS: cannot write to log file /usr/tmp/snmpd.log: There is not enough space in the file system.
14同行回答
浏览1506
/usr文件系统现在是正常的，它会自动恢复到正常的文件系统使用率
# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 15.00 10.82 28% 3874 1% /
/dev/hd2 5.00 2.83 44% 51016 8% /usr
/dev/hd9var 4.00 1.97 51% 1813 1% /var
/dev/hd3 4.00 3.37 16% 23021 3% /tmp收起
# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 15.00 10.82 28% 3874 1% /
/dev/hd2 5.00 2.83 44% 51016 8% /usr
/dev/hd9var 4.00 1.97 51% 1813 1% /var
/dev/hd3 4.00 3.37 16% 23021 3% /tmp收起
浏览1190
先把/usr 文件系统空间解决一下吧，安排时间再进行测试。。。 工作要一步一步做啊收起
浏览1219