AIX mount: 0506-351 错误处理
故障现象：1、主库日志报错 Sun Nov 27 12:31:34 2011 Errors in file /home/oracle/admin/zjtv/bdump/zjtv1_arc0_803242.trc: ORA-16055: FAL request rejected ARCH: FAL archive failed. Archiver continuing ...显示全部
故障现象：
1、主库日志报错
Sun Nov 27 12:31:34 2011
Errors in file /home/oracle/admin/zjtv/bdump/zjtv1_arc0_803242.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
Sun Nov 27 12:31:34 2011
ORACLE Instance zjtv1 - Archival Error. Archiver continuing.
Sun Nov 27 12:36:34 2011
Errors in file /home/oracle/admin/zjtv/bdump/zjtv1_arc2_709010.trc:
ORA-00270: error creating archive log
Sun Nov 27 12:36:34 2011
FAL[server, ARC2]: FAL archive failed, see trace file.
2、从库日志草错，显示归档文件没有权限创建
Sun Nov 27 13:26:59 2011
Errors in file /home/oracle/admin/standby/udump/standby_rfs_450640.trc:
ORA-00270: error creating archive log /archivelog/arch_2_22493_614088933.arc
ORA-19504: failed to create file "/archivelog/arch_2_22493_614088933.arc"
ORA-27040: file create error, unable to create file
IBM AIX RISC System/6000 Error: 13: Permission denied
故障分析：
该从库上周刚刚重新调整归档目录/archivelog条带化到hdisk0hdisk1， 故障发生前从库刚刚进行过重启动作，重启后就出现如上错误。
怀疑是没有设置自动MOUNT，根据如下思路有如下具体排查经过：
1、首先查看目录是否存在
P560A:/archivelog#df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 2.00 0.56 73% 2135 2% /
/dev/hd2 4.00 1.66 59% 50260 12% /usr
/dev/hd9var 1.00 0.27 74% 3219 5% /var
/dev/hd3 10.00 5.62 44% 141687 10% /tmp
/dev/hd1 1.00 0.54 47% 865 1% /home
/proc - - - - - /proc
/dev/hd10opt 2.00 1.53 24% 7221 2% /opt
/dev/weblogiclv 10.00 9.47 6% 47 1% /weblogic
/dev/weblogic9lv 10.00 1.85 82% 93440 18% /weblogic9
/dev/tsmdb 7.50 2.92 62% 7918 2% /tsmdb
/dev/oraclelv 10.00 2.08 80% 27758 6% /home/oracle
/dev/oradata_lv 300.00 138.46 54% 94 1% /oradata
/dev/fslv00 150.00 26.45 83% 25 1% /backup
发现/archivelog未挂载
2、尝试挂载，出现错误
P560A:/#mount /archivelog
mount: 0506-351 The log device and fs device must be in the same volume group.
显然原因是日志设备和你要挂载的文件系统不在一个卷组里
3、查看archive_lv在哪个VG上，结果是rootvg
P560A:/archivelog#lslv archive_lv
LOGICAL VOLUME: archive_lv VOLUME GROUP: rootvg
LV IDENTIFIER: 00c3ee9e00004c000000011c74979ab0.16 PERMISSION: read/write
VG STATE: active/complete LV STATE: opened/syncd
TYPE: jfs2 WRITE VERIFY: off
MAX LPs: 512 PP SIZE: 256 megabyte(s)
COPIES: 1 SCHED POLICY: striped
LPs: 400 PPs: 400
STALE PPs: 0 BB POLICY: relocatable
INTER-POLICY: maximum RELOCATABLE: no
INTRA-POLICY: middle UPPER BOUND: 2
MOUNT POINT: /archivelog LABEL: /archivelog
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes (superstrict)
Serialize IO ?: NO
STRIPE WIDTH: 2
STRIPE SIZE: 128K
P560A:/archivelog#
4、查看rootvg的日志设备jfs2log在hd8上
P560A:/archivelog#lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/syncd N/A
hd6 paging 32 64 2 open/syncd N/A
hd8 jfs2log 1 2 2 open/syncd N/A
hd4 jfs2 8 16 2 open/syncd /
hd2 jfs2 16 32 2 open/syncd /usr
hd9var jfs2 4 8 2 open/syncd /var
hd3 jfs2 40 80 2 open/syncd /tmp
hd1 jfs2 4 8 2 open/syncd /home
hd10opt jfs2 8 16 2 open/syncd /opt
lg_dumplv sysdump 8 8 1 open/syncd N/A
fwdump jfs2 2 2 1 closed/syncd /var/adm/ras/platform
oraclelv jfs2 40 40 1 open/syncd /home/oracle
tsmdb jfs2 30 60 2 open/syncd /tsmdb
archive_lv jfs2 400 400 2 closed/syncd /archivelog
5、查看/etc/filesystems文件的/archivelog的LOG属性在/dev/loglv00，而正常的应该是/dev/hd8
6、修改/archivelog的LOG属性
vi /etc/filesystems
/oradata:
dev = /dev/oradata_lv
vfs = jfs2
log = /dev/loglv00
mount = true
options = rw
account = false
/archivelog:
dev = /dev/archive_lv
vfs = jfs2
log = /dev/loglv00 --> hd8
mount = true
options = rw
account = false
7、mount成功
mount /archivelog
8、P560A:/archivelog#df -g发现已经挂载
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 2.00 0.56 73% 2135 2% /
/dev/hd2 4.00 1.66 59% 50260 12% /usr
/dev/hd9var 1.00 0.27 74% 3219 5% /var
/dev/hd3 10.00 5.62 44% 141687 10% /tmp
/dev/hd1 1.00 0.54 47% 865 1% /home
/proc - - - - - /proc
/dev/hd10opt 2.00 1.53 24% 7221 2% /opt
/dev/weblogiclv 10.00 9.47 6% 47 1% /weblogic
/dev/weblogic9lv 10.00 1.85 82% 93440 18% /weblogic9
/dev/tsmdb 7.50 2.92 62% 7918 2% /tsmdb
/dev/oraclelv 10.00 2.08 80% 27758 6% /home/oracle
/dev/oradata_lv 300.00 138.46 54% 94 1% /oradata
/dev/fslv00 150.00 26.45 83% 25 1% /backup
/dev/archive_lv 100.00 80.91 20% 59 1% /archivelog收起
1、主库日志报错
Sun Nov 27 12:31:34 2011
Errors in file /home/oracle/admin/zjtv/bdump/zjtv1_arc0_803242.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
Sun Nov 27 12:31:34 2011
ORACLE Instance zjtv1 - Archival Error. Archiver continuing.
Sun Nov 27 12:36:34 2011
Errors in file /home/oracle/admin/zjtv/bdump/zjtv1_arc2_709010.trc:
ORA-00270: error creating archive log
Sun Nov 27 12:36:34 2011
FAL[server, ARC2]: FAL archive failed, see trace file.
2、从库日志草错，显示归档文件没有权限创建
Sun Nov 27 13:26:59 2011
Errors in file /home/oracle/admin/standby/udump/standby_rfs_450640.trc:
ORA-00270: error creating archive log /archivelog/arch_2_22493_614088933.arc
ORA-19504: failed to create file "/archivelog/arch_2_22493_614088933.arc"
ORA-27040: file create error, unable to create file
IBM AIX RISC System/6000 Error: 13: Permission denied
故障分析：
该从库上周刚刚重新调整归档目录/archivelog条带化到hdisk0hdisk1， 故障发生前从库刚刚进行过重启动作，重启后就出现如上错误。
怀疑是没有设置自动MOUNT，根据如下思路有如下具体排查经过：
1、首先查看目录是否存在
P560A:/archivelog#df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 2.00 0.56 73% 2135 2% /
/dev/hd2 4.00 1.66 59% 50260 12% /usr
/dev/hd9var 1.00 0.27 74% 3219 5% /var
/dev/hd3 10.00 5.62 44% 141687 10% /tmp
/dev/hd1 1.00 0.54 47% 865 1% /home
/proc - - - - - /proc
/dev/hd10opt 2.00 1.53 24% 7221 2% /opt
/dev/weblogiclv 10.00 9.47 6% 47 1% /weblogic
/dev/weblogic9lv 10.00 1.85 82% 93440 18% /weblogic9
/dev/tsmdb 7.50 2.92 62% 7918 2% /tsmdb
/dev/oraclelv 10.00 2.08 80% 27758 6% /home/oracle
/dev/oradata_lv 300.00 138.46 54% 94 1% /oradata
/dev/fslv00 150.00 26.45 83% 25 1% /backup
发现/archivelog未挂载
2、尝试挂载，出现错误
P560A:/#mount /archivelog
mount: 0506-351 The log device and fs device must be in the same volume group.
显然原因是日志设备和你要挂载的文件系统不在一个卷组里
3、查看archive_lv在哪个VG上，结果是rootvg
P560A:/archivelog#lslv archive_lv
LOGICAL VOLUME: archive_lv VOLUME GROUP: rootvg
LV IDENTIFIER: 00c3ee9e00004c000000011c74979ab0.16 PERMISSION: read/write
VG STATE: active/complete LV STATE: opened/syncd
TYPE: jfs2 WRITE VERIFY: off
MAX LPs: 512 PP SIZE: 256 megabyte(s)
COPIES: 1 SCHED POLICY: striped
LPs: 400 PPs: 400
STALE PPs: 0 BB POLICY: relocatable
INTER-POLICY: maximum RELOCATABLE: no
INTRA-POLICY: middle UPPER BOUND: 2
MOUNT POINT: /archivelog LABEL: /archivelog
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes (superstrict)
Serialize IO ?: NO
STRIPE WIDTH: 2
STRIPE SIZE: 128K
P560A:/archivelog#
4、查看rootvg的日志设备jfs2log在hd8上
P560A:/archivelog#lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/syncd N/A
hd6 paging 32 64 2 open/syncd N/A
hd8 jfs2log 1 2 2 open/syncd N/A
hd4 jfs2 8 16 2 open/syncd /
hd2 jfs2 16 32 2 open/syncd /usr
hd9var jfs2 4 8 2 open/syncd /var
hd3 jfs2 40 80 2 open/syncd /tmp
hd1 jfs2 4 8 2 open/syncd /home
hd10opt jfs2 8 16 2 open/syncd /opt
lg_dumplv sysdump 8 8 1 open/syncd N/A
fwdump jfs2 2 2 1 closed/syncd /var/adm/ras/platform
oraclelv jfs2 40 40 1 open/syncd /home/oracle
tsmdb jfs2 30 60 2 open/syncd /tsmdb
archive_lv jfs2 400 400 2 closed/syncd /archivelog
5、查看/etc/filesystems文件的/archivelog的LOG属性在/dev/loglv00，而正常的应该是/dev/hd8
6、修改/archivelog的LOG属性
vi /etc/filesystems
/oradata:
dev = /dev/oradata_lv
vfs = jfs2
log = /dev/loglv00
mount = true
options = rw
account = false
/archivelog:
dev = /dev/archive_lv
vfs = jfs2
log = /dev/loglv00 --> hd8
mount = true
options = rw
account = false
7、mount成功
mount /archivelog
8、P560A:/archivelog#df -g发现已经挂载
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 2.00 0.56 73% 2135 2% /
/dev/hd2 4.00 1.66 59% 50260 12% /usr
/dev/hd9var 1.00 0.27 74% 3219 5% /var
/dev/hd3 10.00 5.62 44% 141687 10% /tmp
/dev/hd1 1.00 0.54 47% 865 1% /home
/proc - - - - - /proc
/dev/hd10opt 2.00 1.53 24% 7221 2% /opt
/dev/weblogiclv 10.00 9.47 6% 47 1% /weblogic
/dev/weblogic9lv 10.00 1.85 82% 93440 18% /weblogic9
/dev/tsmdb 7.50 2.92 62% 7918 2% /tsmdb
/dev/oraclelv 10.00 2.08 80% 27758 6% /home/oracle
/dev/oradata_lv 300.00 138.46 54% 94 1% /oradata
/dev/fslv00 150.00 26.45 83% 25 1% /backup
/dev/archive_lv 100.00 80.91 20% 59 1% /archivelog收起