db2start的时候出现的问题
最近新搭建了一套测试库，系统是SUSE 11sp2的，db2版本是9.7.0.10的都设置完后，启动数据库的时候报如下错误：db2inst1@db2_admt_1:~> db2start07/16/2015 20:01:03 0 0 SQL1026N The database manager is already ... 显示全部
查看其它 8 个回答15236260698 的回答
参考一下如下这个内容：
HACMP failover以后无法db2start
相关内容：HACMP，DPF，rc.db2pe
背景介绍：
AIX 5.3
BCU 210
DB2 V9 GA
主要系统为server1，server2，server3，server4。其中server1为primary partition 0，server2为standby partition 0，server3为primary partition 1-8，server4为standby partition 1-8。
系统发生故障导致HACMP发生failover，从server1,3转移至server2,4。但是db2start无法启动。错误信息：
08/24/2007 13:12:52 0 0 SQL6048N A communication error occurred during
START or STOP DATABASE MANAGER processing.
08/24/2007 13:12:53 1 0 SQL6048N A communication error occurred during
START or STOP DATABASE MANAGER processing.
08/24/2007 13:12:55 7 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:55 4 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:55 2 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:55 8 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:56 5 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:56 3 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:56 6 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
SQL1032N No start database manager command was issued. SQLSTATE=57019
分析思路与解决步骤：
遇到这种生产系统down的问题，大家首先需要做的就是保持冷静。在心里对自己默说三遍：be cool, be cool, be cool。
好了，既然我们冷静下来，就可以开始着手解决问题了。
首先，我们知道，一个配置好的系统不可能无缘无故地down掉无法启动，而问题诊断的目的就是找到错误的根本。
一般来说，遇到一个给定的错误信息，我们首先需要做的就是检查错误信息的内容。这里的错误信息为：SQL6048N与SQL6030N rc=19。
因此，当我们使用db2 ? SQL6048N我们得到：
SQL6048N A communication error occurred during START or STOP
DATABASE MANAGER processing.
Explanation:
A TCP/IP communication error occurred while the START or STOP
DATABASE MANAGER command was trying to establish connection with
all the nodes defined in the sqllib/db2nodes.cfg file.
User Response:
Do the following:
o Ensure that the node has the proper authorization defined in
the .rhosts or the host.equiv files.
o Ensure that the application is not using more than (500 +
(1995 - 2 * total_number_of_nodes)) file descriptors at the
same time.
o Ensure all the Enterprise Server Edition environment
variables are defined in the profile file.
o Ensure the profile file is written in the Korn Shell script
format.
o Ensure that all the host names defined in the db2nodes.cfg
file in the sqllib directory are defined on the network and
are running.
而运行db2 ? SQL6030N我们得到
SQL6030N START or STOP DATABASE MANAGER failed. Reason code
"".
Explanation:
The reason code indicates the error. The statement cannot be
processed.
……
(19) Unable to create the UNIX socket directory
/tmp/db2__ /$DB2INSTANCE.
……
(19) Check the permission of the /tmp filesystem to make sure
all intermediate directories of /tmp/db2__/
$DB2INSTANCE can be created.
因此，我们很容易联想到，SQL6048N是由于其他分区产生了SQL6030N导致，因此，目前我们的探索方向应当指向SQL6030N。
根据SQL6030N，我们首先检查/tmp的权限：
drwxrwxrwt 151 bin bin 344576 Aug 29 09:30 tmp
看起来一切都好。当我们进入tmp进行ls时发现tmp目录为空。根据RC=19的描述，db2应当是在创建socket时发生问题，所以我们试验手工创建db2_09_01发现没有任何错误。这里看起来好像并非权限问题，因此我们需要进一步分析db2diag.log。
删除db2diag.log，重新db2start得到错误信息，分析最新生成的db2diag.log，第一个映入眼帘的错误信息为：
2007-08-24-17.06.23.632545-240 E459117A403 LEVEL: Error (OS)
PID : 239460 TID : 1 PROC : db2fcmr 5
INSTANCE: db2inst1 NODE : 005
FUNCTION: DB2 UDB, oper system services, sqloPdbBindSocket, probe:20
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
CALLED : OS, -, bind
OSERR : EADDRNOTAVAIL (68) "Can't assign requested address"
2007-08-24-17.06.24.546516-240 I459521A384 LEVEL: Severe
PID : 239460 TID : 1 PROC : db2fcmr 5
INSTANCE: db2inst1 NODE : 005
FUNCTION: DB2 UDB, fast comm manager,
sqkfTcpServices::allocRemoteLinkResource, probe:100
RETCODE : ZRC=0x8159006D=-2124873619=SQLKF_SOCKDIR_NOT_CREATED
"FCM socket directory not created successfully"
从这里我们可以看出什么？首先这个错误级别为Error与Severe，说明是严重错误。而Error (OS)可以看出真正的错误信息是由OS发出的，接收到的进程为db2fcmr分区5，实例为db2inst1。
发生错误的db2函数为sqloPdbBindSocket。可能大家不清楚这个函数的具体作用，但是根据函数名可以推测出，这个函数是对套接字作bind()操作，而CALLED: OS, -, bind也可以看出问题的出现是在OS函数的bind()，返回的错误信息为EADDRNOTAVAIL，errno为68（/usr/include/errno.h中存在#define EADDRNOTAVAIL 68 /* Can't assign requested address */）。
什么原因可能导致这个问题呢？这个时候我们需要google一下bind函数，发现其可能出现的错误为：
The bind() function shall fail if:
[EADDRINUSE]
The specified address is already in use.
[EADDRNOTAVAIL]
The specified address is not available from the local machine.
[EAFNOSUPPORT]
The specified address is not a valid address for the address family of the specified socket.
[EBADF]
The socket argument is not a valid file descriptor.
[EINVAL]
The socket is already bound to an address, and the protocol does not support binding to a new address; or the socket has been shut down.
[ENOTSOCK]
The socket argument does not refer to a socket.
[EOPNOTSUPP]
The socket type of the specified socket does not support binding to an address.
If the address family of the socket is AF_UNIX, then bind() shall fail if:
[EACCES]
A component of the path prefix denies search permission, or the requested name requires writing in a directory with a mode that denies write permission.
[EDESTADDRREQ] or [EISDIR]
The address argument is a null pointer.
[EIO]
An I/O error occurred.
[ELOOP]
A loop exists in symbolic links encountered during resolution of the pathname in address.
[ENAMETOOLONG]
A component of a pathname exceeded {NAME_MAX} characters, or an entire pathname exceeded {PATH_MAX} characters.
[ENOENT]
A component of the pathname does not name an existing file or the pathname is an empty string.
[ENOTDIR]
A component of the path prefix of the pathname in address is not a directory.
[EROFS]
The name would reside on a read-only file system.
The bind() function may fail if:
[EACCES]
The specified address is protected and the current user does not have permission to bind to it.
[EINVAL]
The address_len argument is not a valid length for the address family.
[EISCONN]
The socket is already connected.
[ELOOP]
More than {SYMLOOP_MAX} symbolic links were encountered during resolution of the pathname in address.
[ENAMETOOLONG]
Pathname resolution of a symbolic link produced an intermediate result whose length exceeds {PATH_MAX}.
[ENOBUFS]
Insufficient resources were available to complete the call.
其中对EADDRNOTAVAIL的解释为：指定的地址对本机不可用。
呵呵，到这里相信诸位读者已经知道下一步的步骤了。没错，就是db2nodes.cfg。
打开db2nodes.cfg，当前的信息为：
0 server02_mgmt 0 server 02_fcm
1 server04_mgmt 0 server04_fcm
2 server03_mgmt 1 server03_fcm
3 server03_mgmt 2 server03_fcm
4 server03_mgmt 3 server03_fcm
5 server03_mgmt 4 server03_fcm
6 server03_mgmt 5 server03_fcm
7 server03_mgmt 6 server03_fcm
8 server03_mgmt 0 server03_fcm
根据我们之前对系统架构的理解，以上的db2nodes.cfg看起来很不对劲。对比我们得到的错误信息，发现分区0，1的服务器名正确，而其他2-8的服务器名依然指向failover之前的server3。
现在的当务之急自然是把数据库救活。因此，立刻备份当前db2nodes.cfg，手工修改其为：
0 server02_mgmt 0 server 02_fcm
1 server04_mgmt 0 server04_fcm
2 server04_mgmt 1 server04_fcm
3 server04_mgmt 2 server04_fcm
4 server04_mgmt 3 server04_fcm
5 server04_mgmt 4 server04_fcm
6 server04_mgmt 5 server04_fcm
7 server04_mgmt 6 server04_fcm
8 server04_mgmt 7 server04_fcm
再次执行db2start，返回结果成功。尝试连接数据库一切正常，说明问题已经解决。
到这里，既然生产系统可以继续运行，我们也就可以长出一口气，暂时不用担心被炒鱿鱼了。下面要做的就是分析问题的产生原因，以避免问题再次出现。
这里我们需要思考，到底是什么导致了db2nodes.cfg出现问题。
熟悉DPF的读者一定知道，在db2引擎中db2nodes.cfg是只读的。当db2运行时该文件会被锁死，没有任何进程可以对该文件进行修改。
因此，既然问题不可能在db2运行时产生，那么错误则缩小至Failover发生的过程当中。
检查failover时产生的日志hacmp.out，希望找出什么步骤改变的db2nodes.cfg。
经过漫长的搜索，其中关于rc.db2pe的错误信息映入眼帘：
rc.db2pe - server02_mgmt: The NFS Mount Point for db2inst1 with homedir /db2home/db2inst1 is /db2home
rc.db2pe - server02_mgmt: starting at Fri Aug 24 11:02:24 EDT 2007
server02_mgmt - rc.db2pe: Using ethernet as host in db2nodes.cfg
rc.db2pe - server02_mgmt: PRE server01 server02
rc.db2pe - server02_mgmt: POST server01 server02
rc.db2pe - server02_mgmt: NAME server02
rc.db2pe - server02_mgmt: START 1
rc.db2pe - server02_mgmt: STOP 0
rc.db2pe - server02_mgmt: NODEUP 1
rc.db2pe - server02_mgmt: NODEDOWN 0
rc.db2pe - server02_mgmt: NODEPRE 2
rc.db2pe - server02_mgmt: NODEPOST 2
rc.db2pe - server02_mgmt: HOT STANDBY 1
rc.db2pe - server02_mgmt: NFS_RETRIES 20
rc.db2pe - server02_mgmt: START RETRIES 4
rc.db2pe - server02_mgmt: STOP RETRIES 6
rc.db2pe - server02_mgmt: INSTANCE DIRECTORY /db2home/db2inst1
rc.db2pe - server02_mgmt: NFS DIRECTORY /db2home
rc.db2pe - server02_mgmt: TAKEOVER STARTING PORT 1
rc.db2pe - server02_mgmt: NETWORK INTERFACE USED BY DBPARTITION 0 IS server02_fcm
rc.db2pe - server02_mgmt: EXECUTING IN STATE START_DATABASE
rc.db2pe - server02_mgmt: ATTEMPTING TO MOUNT /db2home
rc.db2pe - server02_mgmt: verify_nfs_auto EXITING with RC FriAug24110224 0 0
rc.db2pe - server02_mgmt: ATTEMPTING TO MOUNT /db2home
rc.db2pe - server02_mgmt: CREATED FILE /db2home/db2inst1/NO_RECOVERY.0
rc.db2pe - server02_mgmt: DB2 Not Running - Returning.
rc.db2pe - server02_mgmt: Checking if db2sysc LN are present for db2inst1
rc.db2pe - server02_mgmt: cleaning up db2 and shared memory
/db2home/db2inst1/sqllib/bin/ipclean: Removing DB2 engine and client's IPC resources for db2inst1.
rc.db2pe - server02_mgmt: Start DB2 Server instance db2inst1 LN 0
rc.db2pe - server02_mgmt: 08-24-07 11:02:37 Attempting to start LN 0
rc.db2pe - server02_mgmt: db2start nodenum 0 restart hostname server02_mgmt port 0 netname server02_fcm
08/24/2007 11:03:36 0 0 SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
rc.db2pe - server02_mgmt: Start Unsuccessful. Will retry in 8 sec.
rc.db2pe - server02_mgmt: Retrying start of LN 0
rc.db2pe - server02_mgmt: db2start nodenum 0 restart hostname server02_mgmt port 0 netname server02_fcm
08/24/2007 11:03:46 0 0 SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
rc.db2pe - server02_mgmt: Start Unsuccessful. Will retry in 8 sec.
rc.db2pe - server02_mgmt: Retrying start of LN 0
rc.db2pe - server02_mgmt: db2start nodenum 0 restart hostname server02_mgmt port 0 netname server02_fcm
08/24/2007 11:03:56 0 0 SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
rc.db2pe - server02_mgmt: Start Unsuccessful. Will retry in 8 sec.
rc.db2pe - server02_mgmt: Retrying start of LN 0
rc.db2pe - server02_mgmt: db2start nodenum 0 restart hostname server02_mgmt port 0 netname server02_fcm
08/24/2007 11:04:07 0 0 SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
rc.db2pe - server02_mgmt: DB2 PROCESS START FAILED on node server02_mgmt
rc.db2driver - server02_mgmt: 08-24-07 11:04:07 End of script (start)
曾经实现过HACMP的读者可能知道，rc.db2pe.eee是DB2提供的failover脚本，存在于~ /sqllib/samples/hacmp/es/rc.db2pe。
检查相关部分的程序代码：
################################################################################
#This function starts a db2 logical node reliably in an HACMP config. Its inputs:<<<<
#the logical node number and port. Recall the instance was supplied
#as input to the rc.db2pe script ($DB2user).<
#This function on takeover starts a logical node number LL via the command
#db2start nodenum LL restart hostname (hostname) port (port) netname (netname).
#If start is by db2 proc restart or not takeover then db2start nodenum LL is used.<<<<<<<<<<<<<<<<<<<<<<<
#If the start is not successful, as indicated by SQL1063N in the output of the<
#start, the start will be retried up to $START_RETRIES times.<
#If running in restart (db2_proc_restart) and $FAILOVER = YES then failover HACMP
#if start is not successful.
#If the start is unsuccessful, STATUS file is set for later use.<<<<<<<<<<<<<
#If the start is the first LN of DB2user, clean ipc for NFS failover/cleanup<
#################################################################################
function start_db2
{
parmlist=$1
pt=$2
lnnlist=`echo $parmlist | sed 's/,/ /g'`
if [ "$pt" = "RESTART" ] ; then
restart=1
TAG="RESTART"
pt=-1
else
TAG="NORESTART"
restart=0
fi
echo "$PROGID - $HOST: Checking if db2sysc LN are present for $DB2user"
SYSC_RUNNING=0
ps -aef | awk '$1 = "$DB2user"' | while read JUNK PID PPID line; do
HIT=0
for i in $line;do
if [ $HIT -eq 1 ] ; then
stripi=`echo $i | awk '{ print $1 }'`
if [ -n "$stripi" -a "$PIDNAME" = "db2sysc" ] ; then
SYSC_RUNNING=1
echo "$PROGID - $HOST: db2sysc LN are present for $DB2user"
HIT=0
break
fi
fi
test=`echo $i | grep db2`
if [ -n "$test" ] ; then
PIDNAME=$test
HIT=1
else
HIT=0
fi
done
done
if [ $SYSC_RUNNING -eq 0 ] ; then
echo "$PROGID - $HOST: cleaning up db2 and shared memory"
clean_db2
fi
for It in $lnnlist; do<<<
nochange=0
su $DB2user -c "cat $lnndir/sqllib/db2nodes.cfg" | while read a b c d e;do
if [ "$a" = "$It" -a "$b" = "$hnn" -a "$c" = "$pt" -a "$d" = "$snn" ] ; then
nochange=1
break
else
nochange=0
fi
done
echo "$PROGID - $HOST: Start DB2 Server instance $DB2user LN $It"
DBERR="YYY"
FSYSE="YYY"
retrycount=1
while [ -n "$DBERR" -o -n "$FSYSE" -a -z "$DBST" ] ; do
if [ $retrycount -eq 1 ] ; then
echo "$PROGID - $HOST: Attempting to start LN $It"
else
echo "$PROGID - $HOST: Start Unsuccessful. Will retry in 8 sec."
sleep 8
echo "$PROGID - $HOST: Retrying start of LN $It"
fi
if [ $restart -ne 1 -a $nochange -ne 1 ] ; then
echo "$PROGID - $HOST: db2start nodenum $It restart hostname $hnn port $pt netname $snn"
su - $DB2user -c $lnndir/sqllib/adm/db2start nodenum $It restart hostname $hnn port $pt netname $snn 2>&1 > /tmp/db2pe.$DB2user.start.$It.$TAG
else
echo "$PROGID - $HOST: db2start nodenum $It"
su - $DB2user -c $lnndir/sqllib/adm/db2start nodenum $It 2>&1 > /tmp/db2pe.$DB2user.start.$It.$TAG
fi
DBERR=`cat /tmp/db2pe.$DB2user.start.$It.$TAG | grep SQL | grep -v SQL1063N | grep -v SQL1026N | grep -v SQL8011W`
DBST=`cat /tmp/db2pe.$DB2user.start.$It.$TAG | grep SQL | grep SQL1026N`
FSYSE=`cat /tmp/db2pe.$DB2user.start.$It.$TAG | grep /usr/guest`
cat /tmp/db2pe.$DB2user.start.$It.$TAG<
retrycount=`expr $retrycount + 1`
if [ $retrycount -gt $START_RETRIES ] ; then
echo "$PROGID - $HOST: DB2 PROCESS START FAILED on node $HOST"
/usr/sbin/wall "$PROGID - $HOST: DB2 START FAILED on node $HOST"
/usr/bin/db2_update_events HAIND OFF
/usr/bin/db2_update_events HA ON
/usr/bin/db2_update_events DB2P ON
if [ $restart -eq 1 -a "$FAILOVER" = "YES" ] ; then
echo "$PROGID - $HOST: FAILOVER SPECIFIED, FAILING OVER HACMP"
/usr/sbin/wall "FAILOVER SPECIFIED, FAILING OVER HACMP"
/usr/bin/db2_update_events HAIND OFF
/usr/bin/db2_update_events HA ON
/usr/sbin/cluster/utilities/clstop -grsy
exit<
else
echo 99 > /tmp/STATUS.result.$It
/usr/bin/db2_update_events HAIND OFF
/usr/bin/db2_update_events HA ON
exit 99<<<<<<<
fi
fi
done
pt=`expr $pt \+ 1`
done
}
很明显，错误是在su - $DB2user -c $lnndir/sqllib/adm/db2start nodenum $It restart hostname $hnn port $pt netname $snn 2>&1 > /tmp/db2pe.$DB2user.start.$It.$TAG发生的。
检查infocenter发现这一行的作用就是更改db2nodes.cfg中相关分区的信息。因此，现在整个问题的雏形应该已经在我们的脑海中形成：
1） 系统发生错误导致failover
2） failover脚本自动调用rc.db2pe.eee更新db2nodes.cfg
3） 其间发生错误导致db2nodes.cfg损坏
4） db2start发生错误
但是现在的问题就是，为什么rc.db2pe.eee发生错误呢？回过头来继续看hacmp.out得到failover时间，映射到db2diag.log中有如下信息：
2007-08-24-11.02.38.830220-240 I34075074A446 LEVEL: Event
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:1100
DATA #1 : String, 81 bytes
DB2NODE=0 DB2LPORT=0 /db2home/db2inst1/sqllib/adm/db2rem db2profile
CHECKNODE 0 0
DATA #2 : Hexdump, 4 bytes
0x0FFFFFFFFFFF9EFC : 0000 0033 ...3
2007-08-24-11.03.35.293256-240 I34075521A428 LEVEL: Event
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:1100
DATA #1 : String, 63 bytes
/db2home/db2inst1/sqllib/adm/db2rstar db2profile SN RESTART 0 0
DATA #2 : Hexdump, 4 bytes
0x0FFFFFFFFFFF9EFC : 0000 0012 ....
2007-08-24-11.03.36.308742-240 E34075950A600 LEVEL: Error
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand,
probe:110
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 204 bytes
The remote shell program terminated prematurely. The most likely causes
are either that the DB2RSHCMD registry variable is set to an invalid
setting, or the remote command program failed to authenticate.
DATA #2 : String, 12 bytes
/usr/bin/rsh
2007-08-24-11.03.36.310952-240 E34076551A496 LEVEL: Error
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand,
probe:200
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 13 bytes
server 02_mgmt
DATA #2 : String, 13 bytes
server02_mgmt
DATA #3 : String, 60 bytes
[files]: 3004-610 You are required to change your password.
2007-08-24-11.03.36.312332-240 I34077048A279 LEVEL: Event
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:80
DATA #1 : signed integer, 4 bytes
-6048
2007-08-24-11.11.49.855438-240 I34077328A301 LEVEL: Info
PID : 251358 TID : 1 PROC : db2loggr
(ACEDW) 7
INSTANCE: db2inst1 NODE : 007
FUNCTION: DB2 UDB, data protection services, sqlpghck, probe:1780
MESSAGE : DB2 is waiting for log files to be archived.
2007-08-24-11.03.44.505647-240 I34077630A446 LEVEL: Event
PID : 1179672 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:1100
DATA #1 : String, 81 bytes
DB2NODE=0 DB2LPORT=0 /db2home/db2inst1/sqllib/adm/db2rem db2profile
CHECKNODE 0 0
DATA #2 : Hexdump, 4 bytes
0x0FFFFFFFFFFF9EFC : 0000 0033 ...3
2007-08-24-11.03.45.522321-240 E34078077A600 LEVEL: Error
PID : 1179672 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand,
probe:110
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 204 bytes
The remote shell program terminated prematurely. The most likely causes
are either that the DB2RSHCMD registry variable is set to an invalid
setting, or the remote command program failed to authenticate.
DATA #2 : String, 12 bytes
/usr/bin/rsh
2007-08-24-11.03.45.524112-240 E34078678A496 LEVEL: Error
PID : 1179672 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand,
probe:200
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 13 bytes
server 02_mgmt
DATA #2 : String, 13 bytes
server 02_mgmt
DATA #3 : String, 60 bytes
[files]: 3004-610 You are required to change your password.
到此为止，问题的产生应当已经非常明显了。在sqloPdbInitializeRemoteCommand的时候，从/usr/bin/rsh得到错误，从操作系统返回的错误信息为3004-610，也就是[files]: 3004-610 You are required to change your password.。
换句话说，当系统进行failover的时候正好碰上系统强制更改密码。而DBA在登录系统解决failover问题时已经修改过密码，因此后来我们的db2start在修改了db2nodes.cfg后成功，而从hacmp.out与db2diag.log中可以分析出，本次failover失败的根本原因在于操作系统的安全政策设置。只要更改安全政策至实例拥有者密码永不过期则可以避免问题的再次发生。
HACMP failover以后无法db2start
相关内容：HACMP，DPF，rc.db2pe
背景介绍：
AIX 5.3
BCU 210
DB2 V9 GA
主要系统为server1，server2，server3，server4。其中server1为primary partition 0，server2为standby partition 0，server3为primary partition 1-8，server4为standby partition 1-8。
系统发生故障导致HACMP发生failover，从server1,3转移至server2,4。但是db2start无法启动。错误信息：
08/24/2007 13:12:52 0 0 SQL6048N A communication error occurred during
START or STOP DATABASE MANAGER processing.
08/24/2007 13:12:53 1 0 SQL6048N A communication error occurred during
START or STOP DATABASE MANAGER processing.
08/24/2007 13:12:55 7 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:55 4 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:55 2 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:55 8 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:56 5 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:56 3 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
08/24/2007 13:12:56 6 0 SQL6030N START or STOP DATABASE MANAGER failed.
Reason code "19".
SQL1032N No start database manager command was issued. SQLSTATE=57019
分析思路与解决步骤：
遇到这种生产系统down的问题，大家首先需要做的就是保持冷静。在心里对自己默说三遍：be cool, be cool, be cool。
好了，既然我们冷静下来，就可以开始着手解决问题了。
首先，我们知道，一个配置好的系统不可能无缘无故地down掉无法启动，而问题诊断的目的就是找到错误的根本。
一般来说，遇到一个给定的错误信息，我们首先需要做的就是检查错误信息的内容。这里的错误信息为：SQL6048N与SQL6030N rc=19。
因此，当我们使用db2 ? SQL6048N我们得到：
SQL6048N A communication error occurred during START or STOP
DATABASE MANAGER processing.
Explanation:
A TCP/IP communication error occurred while the START or STOP
DATABASE MANAGER command was trying to establish connection with
all the nodes defined in the sqllib/db2nodes.cfg file.
User Response:
Do the following:
o Ensure that the node has the proper authorization defined in
the .rhosts or the host.equiv files.
o Ensure that the application is not using more than (500 +
(1995 - 2 * total_number_of_nodes)) file descriptors at the
same time.
o Ensure all the Enterprise Server Edition environment
variables are defined in the profile file.
o Ensure the profile file is written in the Korn Shell script
format.
o Ensure that all the host names defined in the db2nodes.cfg
file in the sqllib directory are defined on the network and
are running.
而运行db2 ? SQL6030N我们得到
SQL6030N START or STOP DATABASE MANAGER failed. Reason code
"
Explanation:
The reason code indicates the error. The statement cannot be
processed.
……
(19) Unable to create the UNIX socket directory
/tmp/db2_
……
(19) Check the permission of the /tmp filesystem to make sure
all intermediate directories of /tmp/db2_
$DB2INSTANCE can be created.
因此，我们很容易联想到，SQL6048N是由于其他分区产生了SQL6030N导致，因此，目前我们的探索方向应当指向SQL6030N。
根据SQL6030N，我们首先检查/tmp的权限：
drwxrwxrwt 151 bin bin 344576 Aug 29 09:30 tmp
看起来一切都好。当我们进入tmp进行ls时发现tmp目录为空。根据RC=19的描述，db2应当是在创建socket时发生问题，所以我们试验手工创建db2_09_01发现没有任何错误。这里看起来好像并非权限问题，因此我们需要进一步分析db2diag.log。
删除db2diag.log，重新db2start得到错误信息，分析最新生成的db2diag.log，第一个映入眼帘的错误信息为：
2007-08-24-17.06.23.632545-240 E459117A403 LEVEL: Error (OS)
PID : 239460 TID : 1 PROC : db2fcmr 5
INSTANCE: db2inst1 NODE : 005
FUNCTION: DB2 UDB, oper system services, sqloPdbBindSocket, probe:20
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
CALLED : OS, -, bind
OSERR : EADDRNOTAVAIL (68) "Can't assign requested address"
2007-08-24-17.06.24.546516-240 I459521A384 LEVEL: Severe
PID : 239460 TID : 1 PROC : db2fcmr 5
INSTANCE: db2inst1 NODE : 005
FUNCTION: DB2 UDB, fast comm manager,
sqkfTcpServices::allocRemoteLinkResource, probe:100
RETCODE : ZRC=0x8159006D=-2124873619=SQLKF_SOCKDIR_NOT_CREATED
"FCM socket directory not created successfully"
从这里我们可以看出什么？首先这个错误级别为Error与Severe，说明是严重错误。而Error (OS)可以看出真正的错误信息是由OS发出的，接收到的进程为db2fcmr分区5，实例为db2inst1。
发生错误的db2函数为sqloPdbBindSocket。可能大家不清楚这个函数的具体作用，但是根据函数名可以推测出，这个函数是对套接字作bind()操作，而CALLED: OS, -, bind也可以看出问题的出现是在OS函数的bind()，返回的错误信息为EADDRNOTAVAIL，errno为68（/usr/include/errno.h中存在#define EADDRNOTAVAIL 68 /* Can't assign requested address */）。
什么原因可能导致这个问题呢？这个时候我们需要google一下bind函数，发现其可能出现的错误为：
The bind() function shall fail if:
[EADDRINUSE]
The specified address is already in use.
[EADDRNOTAVAIL]
The specified address is not available from the local machine.
[EAFNOSUPPORT]
The specified address is not a valid address for the address family of the specified socket.
[EBADF]
The socket argument is not a valid file descriptor.
[EINVAL]
The socket is already bound to an address, and the protocol does not support binding to a new address; or the socket has been shut down.
[ENOTSOCK]
The socket argument does not refer to a socket.
[EOPNOTSUPP]
The socket type of the specified socket does not support binding to an address.
If the address family of the socket is AF_UNIX, then bind() shall fail if:
[EACCES]
A component of the path prefix denies search permission, or the requested name requires writing in a directory with a mode that denies write permission.
[EDESTADDRREQ] or [EISDIR]
The address argument is a null pointer.
[EIO]
An I/O error occurred.
[ELOOP]
A loop exists in symbolic links encountered during resolution of the pathname in address.
[ENAMETOOLONG]
A component of a pathname exceeded {NAME_MAX} characters, or an entire pathname exceeded {PATH_MAX} characters.
[ENOENT]
A component of the pathname does not name an existing file or the pathname is an empty string.
[ENOTDIR]
A component of the path prefix of the pathname in address is not a directory.
[EROFS]
The name would reside on a read-only file system.
The bind() function may fail if:
[EACCES]
The specified address is protected and the current user does not have permission to bind to it.
[EINVAL]
The address_len argument is not a valid length for the address family.
[EISCONN]
The socket is already connected.
[ELOOP]
More than {SYMLOOP_MAX} symbolic links were encountered during resolution of the pathname in address.
[ENAMETOOLONG]
Pathname resolution of a symbolic link produced an intermediate result whose length exceeds {PATH_MAX}.
[ENOBUFS]
Insufficient resources were available to complete the call.
其中对EADDRNOTAVAIL的解释为：指定的地址对本机不可用。
呵呵，到这里相信诸位读者已经知道下一步的步骤了。没错，就是db2nodes.cfg。
打开db2nodes.cfg，当前的信息为：
0 server02_mgmt 0 server 02_fcm
1 server04_mgmt 0 server04_fcm
2 server03_mgmt 1 server03_fcm
3 server03_mgmt 2 server03_fcm
4 server03_mgmt 3 server03_fcm
5 server03_mgmt 4 server03_fcm
6 server03_mgmt 5 server03_fcm
7 server03_mgmt 6 server03_fcm
8 server03_mgmt 0 server03_fcm
根据我们之前对系统架构的理解，以上的db2nodes.cfg看起来很不对劲。对比我们得到的错误信息，发现分区0，1的服务器名正确，而其他2-8的服务器名依然指向failover之前的server3。
现在的当务之急自然是把数据库救活。因此，立刻备份当前db2nodes.cfg，手工修改其为：
0 server02_mgmt 0 server 02_fcm
1 server04_mgmt 0 server04_fcm
2 server04_mgmt 1 server04_fcm
3 server04_mgmt 2 server04_fcm
4 server04_mgmt 3 server04_fcm
5 server04_mgmt 4 server04_fcm
6 server04_mgmt 5 server04_fcm
7 server04_mgmt 6 server04_fcm
8 server04_mgmt 7 server04_fcm
再次执行db2start，返回结果成功。尝试连接数据库一切正常，说明问题已经解决。
到这里，既然生产系统可以继续运行，我们也就可以长出一口气，暂时不用担心被炒鱿鱼了。下面要做的就是分析问题的产生原因，以避免问题再次出现。
这里我们需要思考，到底是什么导致了db2nodes.cfg出现问题。
熟悉DPF的读者一定知道，在db2引擎中db2nodes.cfg是只读的。当db2运行时该文件会被锁死，没有任何进程可以对该文件进行修改。
因此，既然问题不可能在db2运行时产生，那么错误则缩小至Failover发生的过程当中。
检查failover时产生的日志hacmp.out，希望找出什么步骤改变的db2nodes.cfg。
经过漫长的搜索，其中关于rc.db2pe的错误信息映入眼帘：
rc.db2pe - server02_mgmt: The NFS Mount Point for db2inst1 with homedir /db2home/db2inst1 is /db2home
rc.db2pe - server02_mgmt: starting at Fri Aug 24 11:02:24 EDT 2007
server02_mgmt - rc.db2pe: Using ethernet as host in db2nodes.cfg
rc.db2pe - server02_mgmt: PRE server01 server02
rc.db2pe - server02_mgmt: POST server01 server02
rc.db2pe - server02_mgmt: NAME server02
rc.db2pe - server02_mgmt: START 1
rc.db2pe - server02_mgmt: STOP 0
rc.db2pe - server02_mgmt: NODEUP 1
rc.db2pe - server02_mgmt: NODEDOWN 0
rc.db2pe - server02_mgmt: NODEPRE 2
rc.db2pe - server02_mgmt: NODEPOST 2
rc.db2pe - server02_mgmt: HOT STANDBY 1
rc.db2pe - server02_mgmt: NFS_RETRIES 20
rc.db2pe - server02_mgmt: START RETRIES 4
rc.db2pe - server02_mgmt: STOP RETRIES 6
rc.db2pe - server02_mgmt: INSTANCE DIRECTORY /db2home/db2inst1
rc.db2pe - server02_mgmt: NFS DIRECTORY /db2home
rc.db2pe - server02_mgmt: TAKEOVER STARTING PORT 1
rc.db2pe - server02_mgmt: NETWORK INTERFACE USED BY DBPARTITION 0 IS server02_fcm
rc.db2pe - server02_mgmt: EXECUTING IN STATE START_DATABASE
rc.db2pe - server02_mgmt: ATTEMPTING TO MOUNT /db2home
rc.db2pe - server02_mgmt: verify_nfs_auto EXITING with RC FriAug24110224 0 0
rc.db2pe - server02_mgmt: ATTEMPTING TO MOUNT /db2home
rc.db2pe - server02_mgmt: CREATED FILE /db2home/db2inst1/NO_RECOVERY.0
rc.db2pe - server02_mgmt: DB2 Not Running - Returning.
rc.db2pe - server02_mgmt: Checking if db2sysc LN are present for db2inst1
rc.db2pe - server02_mgmt: cleaning up db2 and shared memory
/db2home/db2inst1/sqllib/bin/ipclean: Removing DB2 engine and client's IPC resources for db2inst1.
rc.db2pe - server02_mgmt: Start DB2 Server instance db2inst1 LN 0
rc.db2pe - server02_mgmt: 08-24-07 11:02:37 Attempting to start LN 0
rc.db2pe - server02_mgmt: db2start nodenum 0 restart hostname server02_mgmt port 0 netname server02_fcm
08/24/2007 11:03:36 0 0 SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
rc.db2pe - server02_mgmt: Start Unsuccessful. Will retry in 8 sec.
rc.db2pe - server02_mgmt: Retrying start of LN 0
rc.db2pe - server02_mgmt: db2start nodenum 0 restart hostname server02_mgmt port 0 netname server02_fcm
08/24/2007 11:03:46 0 0 SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
rc.db2pe - server02_mgmt: Start Unsuccessful. Will retry in 8 sec.
rc.db2pe - server02_mgmt: Retrying start of LN 0
rc.db2pe - server02_mgmt: db2start nodenum 0 restart hostname server02_mgmt port 0 netname server02_fcm
08/24/2007 11:03:56 0 0 SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
rc.db2pe - server02_mgmt: Start Unsuccessful. Will retry in 8 sec.
rc.db2pe - server02_mgmt: Retrying start of LN 0
rc.db2pe - server02_mgmt: db2start nodenum 0 restart hostname server02_mgmt port 0 netname server02_fcm
08/24/2007 11:04:07 0 0 SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
rc.db2pe - server02_mgmt: DB2 PROCESS START FAILED on node server02_mgmt
rc.db2driver - server02_mgmt: 08-24-07 11:04:07 End of script (start)
曾经实现过HACMP的读者可能知道，rc.db2pe.eee是DB2提供的failover脚本，存在于~ /sqllib/samples/hacmp/es/rc.db2pe。
检查相关部分的程序代码：
################################################################################
#This function starts a db2 logical node reliably in an HACMP config. Its inputs:<<<<
#the logical node number and port. Recall the instance was supplied
#as input to the rc.db2pe script ($DB2user).<
#This function on takeover starts a logical node number LL via the command
#db2start nodenum LL restart hostname (hostname) port (port) netname (netname).
#If start is by db2 proc restart or not takeover then db2start nodenum LL is used.<<<<<<<<<<<<<<<<<<<<<<<
#If the start is not successful, as indicated by SQL1063N in the output of the<
#start, the start will be retried up to $START_RETRIES times.<
#If running in restart (db2_proc_restart) and $FAILOVER = YES then failover HACMP
#if start is not successful.
#If the start is unsuccessful, STATUS file is set for later use.<<<<<<<<<<<<<
#If the start is the first LN of DB2user, clean ipc for NFS failover/cleanup<
#################################################################################
function start_db2
{
parmlist=$1
pt=$2
lnnlist=`echo $parmlist | sed 's/,/ /g'`
if [ "$pt" = "RESTART" ] ; then
restart=1
TAG="RESTART"
pt=-1
else
TAG="NORESTART"
restart=0
fi
echo "$PROGID - $HOST: Checking if db2sysc LN are present for $DB2user"
SYSC_RUNNING=0
ps -aef | awk '$1 = "$DB2user"' | while read JUNK PID PPID line; do
HIT=0
for i in $line;do
if [ $HIT -eq 1 ] ; then
stripi=`echo $i | awk '{ print $1 }'`
if [ -n "$stripi" -a "$PIDNAME" = "db2sysc" ] ; then
SYSC_RUNNING=1
echo "$PROGID - $HOST: db2sysc LN are present for $DB2user"
HIT=0
break
fi
fi
test=`echo $i | grep db2`
if [ -n "$test" ] ; then
PIDNAME=$test
HIT=1
else
HIT=0
fi
done
done
if [ $SYSC_RUNNING -eq 0 ] ; then
echo "$PROGID - $HOST: cleaning up db2 and shared memory"
clean_db2
fi
for It in $lnnlist; do<<<
nochange=0
su $DB2user -c "cat $lnndir/sqllib/db2nodes.cfg" | while read a b c d e;do
if [ "$a" = "$It" -a "$b" = "$hnn" -a "$c" = "$pt" -a "$d" = "$snn" ] ; then
nochange=1
break
else
nochange=0
fi
done
echo "$PROGID - $HOST: Start DB2 Server instance $DB2user LN $It"
DBERR="YYY"
FSYSE="YYY"
retrycount=1
while [ -n "$DBERR" -o -n "$FSYSE" -a -z "$DBST" ] ; do
if [ $retrycount -eq 1 ] ; then
echo "$PROGID - $HOST: Attempting to start LN $It"
else
echo "$PROGID - $HOST: Start Unsuccessful. Will retry in 8 sec."
sleep 8
echo "$PROGID - $HOST: Retrying start of LN $It"
fi
if [ $restart -ne 1 -a $nochange -ne 1 ] ; then
echo "$PROGID - $HOST: db2start nodenum $It restart hostname $hnn port $pt netname $snn"
su - $DB2user -c $lnndir/sqllib/adm/db2start nodenum $It restart hostname $hnn port $pt netname $snn 2>&1 > /tmp/db2pe.$DB2user.start.$It.$TAG
else
echo "$PROGID - $HOST: db2start nodenum $It"
su - $DB2user -c $lnndir/sqllib/adm/db2start nodenum $It 2>&1 > /tmp/db2pe.$DB2user.start.$It.$TAG
fi
DBERR=`cat /tmp/db2pe.$DB2user.start.$It.$TAG | grep SQL | grep -v SQL1063N | grep -v SQL1026N | grep -v SQL8011W`
DBST=`cat /tmp/db2pe.$DB2user.start.$It.$TAG | grep SQL | grep SQL1026N`
FSYSE=`cat /tmp/db2pe.$DB2user.start.$It.$TAG | grep /usr/guest`
cat /tmp/db2pe.$DB2user.start.$It.$TAG<
retrycount=`expr $retrycount + 1`
if [ $retrycount -gt $START_RETRIES ] ; then
echo "$PROGID - $HOST: DB2 PROCESS START FAILED on node $HOST"
/usr/sbin/wall "$PROGID - $HOST: DB2 START FAILED on node $HOST"
/usr/bin/db2_update_events HAIND OFF
/usr/bin/db2_update_events HA ON
/usr/bin/db2_update_events DB2P ON
if [ $restart -eq 1 -a "$FAILOVER" = "YES" ] ; then
echo "$PROGID - $HOST: FAILOVER SPECIFIED, FAILING OVER HACMP"
/usr/sbin/wall "FAILOVER SPECIFIED, FAILING OVER HACMP"
/usr/bin/db2_update_events HAIND OFF
/usr/bin/db2_update_events HA ON
/usr/sbin/cluster/utilities/clstop -grsy
exit<
else
echo 99 > /tmp/STATUS.result.$It
/usr/bin/db2_update_events HAIND OFF
/usr/bin/db2_update_events HA ON
exit 99<<<<<<<
fi
fi
done
pt=`expr $pt \+ 1`
done
}
很明显，错误是在su - $DB2user -c $lnndir/sqllib/adm/db2start nodenum $It restart hostname $hnn port $pt netname $snn 2>&1 > /tmp/db2pe.$DB2user.start.$It.$TAG发生的。
检查infocenter发现这一行的作用就是更改db2nodes.cfg中相关分区的信息。因此，现在整个问题的雏形应该已经在我们的脑海中形成：
1） 系统发生错误导致failover
2） failover脚本自动调用rc.db2pe.eee更新db2nodes.cfg
3） 其间发生错误导致db2nodes.cfg损坏
4） db2start发生错误
但是现在的问题就是，为什么rc.db2pe.eee发生错误呢？回过头来继续看hacmp.out得到failover时间，映射到db2diag.log中有如下信息：
2007-08-24-11.02.38.830220-240 I34075074A446 LEVEL: Event
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:1100
DATA #1 : String, 81 bytes
DB2NODE=0 DB2LPORT=0 /db2home/db2inst1/sqllib/adm/db2rem db2profile
CHECKNODE 0 0
DATA #2 : Hexdump, 4 bytes
0x0FFFFFFFFFFF9EFC : 0000 0033 ...3
2007-08-24-11.03.35.293256-240 I34075521A428 LEVEL: Event
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:1100
DATA #1 : String, 63 bytes
/db2home/db2inst1/sqllib/adm/db2rstar db2profile SN RESTART 0 0
DATA #2 : Hexdump, 4 bytes
0x0FFFFFFFFFFF9EFC : 0000 0012 ....
2007-08-24-11.03.36.308742-240 E34075950A600 LEVEL: Error
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand,
probe:110
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 204 bytes
The remote shell program terminated prematurely. The most likely causes
are either that the DB2RSHCMD registry variable is set to an invalid
setting, or the remote command program failed to authenticate.
DATA #2 : String, 12 bytes
/usr/bin/rsh
2007-08-24-11.03.36.310952-240 E34076551A496 LEVEL: Error
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand,
probe:200
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 13 bytes
server 02_mgmt
DATA #2 : String, 13 bytes
server02_mgmt
DATA #3 : String, 60 bytes
[files]: 3004-610 You are required to change your password.
2007-08-24-11.03.36.312332-240 I34077048A279 LEVEL: Event
PID : 1179658 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:80
DATA #1 : signed integer, 4 bytes
-6048
2007-08-24-11.11.49.855438-240 I34077328A301 LEVEL: Info
PID : 251358 TID : 1 PROC : db2loggr
(ACEDW) 7
INSTANCE: db2inst1 NODE : 007
FUNCTION: DB2 UDB, data protection services, sqlpghck, probe:1780
MESSAGE : DB2 is waiting for log files to be archived.
2007-08-24-11.03.44.505647-240 I34077630A446 LEVEL: Event
PID : 1179672 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:1100
DATA #1 : String, 81 bytes
DB2NODE=0 DB2LPORT=0 /db2home/db2inst1/sqllib/adm/db2rem db2profile
CHECKNODE 0 0
DATA #2 : Hexdump, 4 bytes
0x0FFFFFFFFFFF9EFC : 0000 0033 ...3
2007-08-24-11.03.45.522321-240 E34078077A600 LEVEL: Error
PID : 1179672 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand,
probe:110
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 204 bytes
The remote shell program terminated prematurely. The most likely causes
are either that the DB2RSHCMD registry variable is set to an invalid
setting, or the remote command program failed to authenticate.
DATA #2 : String, 12 bytes
/usr/bin/rsh
2007-08-24-11.03.45.524112-240 E34078678A496 LEVEL: Error
PID : 1179672 TID : 1 PROC : db2start
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand,
probe:200
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 13 bytes
server 02_mgmt
DATA #2 : String, 13 bytes
server 02_mgmt
DATA #3 : String, 60 bytes
[files]: 3004-610 You are required to change your password.
到此为止，问题的产生应当已经非常明显了。在sqloPdbInitializeRemoteCommand的时候，从/usr/bin/rsh得到错误，从操作系统返回的错误信息为3004-610，也就是[files]: 3004-610 You are required to change your password.。
换句话说，当系统进行failover的时候正好碰上系统强制更改密码。而DBA在登录系统解决failover问题时已经修改过密码，因此后来我们的db2start在修改了db2nodes.cfg后成功，而从hacmp.out与db2diag.log中可以分析出，本次failover失败的根本原因在于操作系统的安全政策设置。只要更改安全政策至实例拥有者密码永不过期则可以避免问题的再次发生。