政府机关磁盘
难缠的磁盘问题
AIX:6.1
阵列IBM DS8300
主机：两台P690其中各一个分区，这里就已P690A，P690B作为区别
问题如下：
root@P690A7:/>#pcmpath query device
Total Dual Active and Active/Asymmetrc Devices : 4
DEV#: 2 DEVICE NAME: hdisk2 TYPE: 2107900 ALGORITHM: Load Balance
SERIAL: 75DNGC10B03
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 0
1 fscsi0/path1 CLOSE NORMAL 0 0
2 fscsi1/path2 CLOSE NORMAL 0 0
3 fscsi1/path3 CLOSE NORMAL 0 0
DEV#: 3 DEVICE NAME: hdisk3 TYPE: 2107900 ALGORITHM: Load Balance
SERIAL: 75DNGC11025
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 0
1 fscsi0/path1 CLOSE NORMAL 0 0
2 fscsi1/path2 CLOSE NORMAL 0 0
3 fscsi1/path3 CLOSE NORMAL 0 0
DEV#: 4 DEVICE NAME: hdisk4 TYPE: 2107900 ALGORITHM: Load Balance
SERIAL: 75DNGC1111F
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 0
1 fscsi0/path1 CLOSE NORMAL 0 0
2 fscsi1/path2 CLOSE NORMAL 0 0
3 fscsi1/path3 CLOSE NORMAL 0 0
DEV#: 5 DEVICE NAME: hdisk5 TYPE: 2107900 ALGORITHM: Load Balance
SERIAL: 75DNGC11400
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 0
1 fscsi0/path1 CLOSE NORMAL 0 0
2 fscsi1/path2 CLOSE NORMAL 0 0
3 fscsi1/path3 CLOSE NORMAL 0 0
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
hdisk2 00c80dbdb4e6a8f3 None
hdisk3 0032958ebf797b5b None
hdisk4 0032958ebf17babd None
hdisk5 none None
root@P690A7:/tmp>#chdev -l hdisk5 -a pv=yes
Method error (/etc/methods/chgdisk):
0514-047 Cannot access a device.
问题可以看到PVID加不上
最直接的方法reboot
在reboot之后使用lspv后PVID还是none老样子。
那不行，先查下ODM库了，
odmget -q "name=hdisk5 and attribute=pvid" CuAt
还是报错，报错的内容是在ODM库中找不到hdisk5,郁闷了.来点更狠的,删光纤卡了
root@P690A7:/tmp>#lsdev -p fscsi0
hdisk2 Available 3Y-08-02 IBM MPIO FC 2107
hdisk3 Available 3Y-08-02 IBM MPIO FC 2107
hdisk4 Available 3Y-08-02 IBM MPIO FC 2107
hdisk5 Available 3Y-08-02 IBM MPIO FC 2107
rmt1 Available 3Y-08-02 Other FC SCSI Tape Drive
rmt2 Available 3Y-08-02 Other FC SCSI Tape Drive
sfwcomm0 Available 3Y-08-02-FF Fiber Channel Storage Framework Comm
root@P690A7:/tmp>#rmdev -dl hdisk2
hdisk2 deleted
root@P690A7:/tmp>#rmdev -dl hdisk3
hdisk3 deleted
root@P690A7:/tmp>#rmdev -dl hdisk4
hdisk4 deleted
root@P690A7:/tmp>#rmdev -dl rmt1
rmt1 deleted
root@P690A7:/tmp>#rmdev -dl rmt2
rmt2 deleted
root@P690A7:/tmp>#rmdev -dl hdisk5
hdisk5 deleted
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
root@P690A7:/tmp>#lsdev -p fscsi0
sfwcomm0 Available 3Y-08-02-FF Fiber Channel Storage Framework Comm
root@P690A7:/tmp>#rmdev -dl sfwcomm0
sfwcomm0 deleted
root@P690A7:/tmp>#lsdev -p fcs0
fcnet0 Defined 3Y-08-01 Fibre Channel Network Protocol Device
fscsi0 Available 3Y-08-02 FC SCSI I/O Controller Protocol Device
root@P690A7:/tmp>#rmdev -dl fcnet0
fcnet0 deleted
root@P690A7:/tmp>#rmdev -dl fscsi0
fscsi0 deleted
root@P690A7:/tmp>#rmdev -dl fcs0
fcs0 deleted
root@P690A7:/tmp>#lsdev -p fscsi1
rmt3 Available 84-08-01 Other FC SCSI Tape Drive
rmt4 Available 84-08-01 Other FC SCSI Tape Drive
rmt5 Available 84-08-01 Other FC SCSI Tape Drive
rmt6 Available 84-08-01 Other FC SCSI Tape Drive
rmt7 Available 84-08-01 Other FC SCSI Tape Drive
rmt8 Available 84-08-01 Other FC SCSI Tape Drive
rmt9 Available 84-08-01 Other FC SCSI Tape Drive
rmt10 Available 84-08-01 Other FC SCSI Tape Drive
sfwcomm1 Available 84-08-01-FF Fiber Channel Storage Framework Comm
root@P690A7:/tmp>#rmdev -dl rmt3
rmt3 deleted
root@P690A7:/tmp>#rmdev -dl rmt4
rmt4 deleted
root@P690A7:/tmp>#rmdev -dl rmt5
rmt5 deleted
root@P690A7:/tmp>#rmdev -dl rmt6
rmt6 deleted
root@P690A7:/tmp>#rmdev -dl rmt7
rmt7 deleted
root@P690A7:/tmp>#rmdev -dl rmt8
rmt8 deleted
root@P690A7:/tmp>#rmdev -dl rmt9
rmt9 deleted
root@P690A7:/tmp>#rmdev -dl rmt10
rmt10 deleted
root@P690A7:/tmp>#rmdev -dl sfwcomm1
sfwcomm1 deleted
root@P690A7:/tmp>#lsdev -p fcs1
fcnet1 Defined 84-08-02 Fibre Channel Network Protocol Device
fscsi1 Available 84-08-01 FC SCSI I/O Controller Protocol Device
root@P690A7:/tmp>#rmdev -dl fcnet1
fcnet1 deleted
root@P690A7:/tmp>#rmdev -dl fscsi1
fscsi1 deleted
root@P690A7:/tmp>#rmdev -dl fcs1
fcs1 deleted
root@P690A7:/tmp>#lsdev -Cc adapter
ent0 Available 3M-08 10/100/1000 Base-TX PCI-X Adapter (14106902)
ent1 Available 6F-08 10/100/1000 Base-TX PCI-X Adapter (14106902)
ent2 Available 6k-08 10/100 Mbps Ethernet PCI Adapter II (1410ff01)
ent3 Available 7s-08 10/100/1000 Base-TX PCI-X Adapter (14106902)
sa0 Available 5J-08 IBM 8-Port EIA-232/RS-422A (PCI) Adapter
scsi0 Available 67-08 Wide/Ultra-3 SCSI I/O Controller
scsi1 Available 7b-08 Wide/Ultra-3 SCSI I/O Controller
sisscsia0 Available 3A-08 PCI-X Dual Channel Ultra320 SCSI Adapter
vsa0 Available LPAR Virtual Serial Adapter
重认设备，还不行。
root@P690A7:/tmp>#cfgmgr
cfgmgr: 0514-621 WARNING: The following device packages are required for
device support but are not currently installed.
devices.fcp.changer
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
hdisk2 00c80dbdb4e6a8f3 oradatavg
hdisk3 0032958ebf797b5b oradatavg
hdisk4 0032958ebf17babd oradatavg
hdisk5 none None
root@P690A7:/tmp>#chdev -l hdisk5 -a pv=yes
Method error (/etc/methods/chgdisk):
0514-047 Cannot access a device.
好像有点没辙了，想到会不会磁盘上存在锁的问题，先查看看
root@P690A7:/tmp>#pcmquerypr -vh /dev/hdisk5
3332393538653036
果然，这个hdisk5上存在有磁盘锁，所以磁盘不然访问。
但是在A机和B机上uname -a查看下，该不会是B机做成的锁吧
P690B7:/>#uname -a
AIX P690B7 1 6 0032958E4C00
root@P690A7:/>#uname -a
AIX P690A7 1 6 003295EE4C00
因为如果是B机做成的锁pr key应该和uname -a的值是相对应，看看又不像，因为这个hdisk5所在的LUN是回收回来后分配给这个机器的，有可能是之前的机器造成的锁，先确定下在B机上vg varyoffvg下来，然后解锁
root@P690A7:/tmp>#pcmquerypr -ch /dev/hdisk5
root@P690A7:/tmp>#pcmquerypr -vh /dev/hdisk5
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
hdisk2 00c80dbdb4e6a8f3 oradatavg
hdisk3 0032958ebf797b5b oradatavg
hdisk4 0032958ebf17babd oradatavg
hdisk5 none None
root@P690A7:/tmp>#chdev -l hdisk5 -a pv=yes
hdisk5 changed
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
hdisk2 00c80dbdb4e6a8f3 None
hdisk3 0032958ebf797b5b None
hdisk4 0032958ebf17babd None
hdisk5 0032958e008b4789 None
问题解决了。
注：pcmquerypr -ch /dev/vpathX可以用来清除persistent reservation。注意：该命令慎用！！！
阵列IBM DS8300
主机：两台P690其中各一个分区，这里就已P690A，P690B作为区别
问题如下：
root@P690A7:/>#pcmpath query device
Total Dual Active and Active/Asymmetrc Devices : 4
DEV#: 2 DEVICE NAME: hdisk2 TYPE: 2107900 ALGORITHM: Load Balance
SERIAL: 75DNGC10B03
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 0
1 fscsi0/path1 CLOSE NORMAL 0 0
2 fscsi1/path2 CLOSE NORMAL 0 0
3 fscsi1/path3 CLOSE NORMAL 0 0
DEV#: 3 DEVICE NAME: hdisk3 TYPE: 2107900 ALGORITHM: Load Balance
SERIAL: 75DNGC11025
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 0
1 fscsi0/path1 CLOSE NORMAL 0 0
2 fscsi1/path2 CLOSE NORMAL 0 0
3 fscsi1/path3 CLOSE NORMAL 0 0
DEV#: 4 DEVICE NAME: hdisk4 TYPE: 2107900 ALGORITHM: Load Balance
SERIAL: 75DNGC1111F
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 0
1 fscsi0/path1 CLOSE NORMAL 0 0
2 fscsi1/path2 CLOSE NORMAL 0 0
3 fscsi1/path3 CLOSE NORMAL 0 0
DEV#: 5 DEVICE NAME: hdisk5 TYPE: 2107900 ALGORITHM: Load Balance
SERIAL: 75DNGC11400
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 0
1 fscsi0/path1 CLOSE NORMAL 0 0
2 fscsi1/path2 CLOSE NORMAL 0 0
3 fscsi1/path3 CLOSE NORMAL 0 0
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
hdisk2 00c80dbdb4e6a8f3 None
hdisk3 0032958ebf797b5b None
hdisk4 0032958ebf17babd None
hdisk5 none None
root@P690A7:/tmp>#chdev -l hdisk5 -a pv=yes
Method error (/etc/methods/chgdisk):
0514-047 Cannot access a device.
问题可以看到PVID加不上
最直接的方法reboot
在reboot之后使用lspv后PVID还是none老样子。
那不行，先查下ODM库了，
odmget -q "name=hdisk5 and attribute=pvid" CuAt
还是报错，报错的内容是在ODM库中找不到hdisk5,郁闷了.来点更狠的,删光纤卡了
root@P690A7:/tmp>#lsdev -p fscsi0
hdisk2 Available 3Y-08-02 IBM MPIO FC 2107
hdisk3 Available 3Y-08-02 IBM MPIO FC 2107
hdisk4 Available 3Y-08-02 IBM MPIO FC 2107
hdisk5 Available 3Y-08-02 IBM MPIO FC 2107
rmt1 Available 3Y-08-02 Other FC SCSI Tape Drive
rmt2 Available 3Y-08-02 Other FC SCSI Tape Drive
sfwcomm0 Available 3Y-08-02-FF Fiber Channel Storage Framework Comm
root@P690A7:/tmp>#rmdev -dl hdisk2
hdisk2 deleted
root@P690A7:/tmp>#rmdev -dl hdisk3
hdisk3 deleted
root@P690A7:/tmp>#rmdev -dl hdisk4
hdisk4 deleted
root@P690A7:/tmp>#rmdev -dl rmt1
rmt1 deleted
root@P690A7:/tmp>#rmdev -dl rmt2
rmt2 deleted
root@P690A7:/tmp>#rmdev -dl hdisk5
hdisk5 deleted
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
root@P690A7:/tmp>#lsdev -p fscsi0
sfwcomm0 Available 3Y-08-02-FF Fiber Channel Storage Framework Comm
root@P690A7:/tmp>#rmdev -dl sfwcomm0
sfwcomm0 deleted
root@P690A7:/tmp>#lsdev -p fcs0
fcnet0 Defined 3Y-08-01 Fibre Channel Network Protocol Device
fscsi0 Available 3Y-08-02 FC SCSI I/O Controller Protocol Device
root@P690A7:/tmp>#rmdev -dl fcnet0
fcnet0 deleted
root@P690A7:/tmp>#rmdev -dl fscsi0
fscsi0 deleted
root@P690A7:/tmp>#rmdev -dl fcs0
fcs0 deleted
root@P690A7:/tmp>#lsdev -p fscsi1
rmt3 Available 84-08-01 Other FC SCSI Tape Drive
rmt4 Available 84-08-01 Other FC SCSI Tape Drive
rmt5 Available 84-08-01 Other FC SCSI Tape Drive
rmt6 Available 84-08-01 Other FC SCSI Tape Drive
rmt7 Available 84-08-01 Other FC SCSI Tape Drive
rmt8 Available 84-08-01 Other FC SCSI Tape Drive
rmt9 Available 84-08-01 Other FC SCSI Tape Drive
rmt10 Available 84-08-01 Other FC SCSI Tape Drive
sfwcomm1 Available 84-08-01-FF Fiber Channel Storage Framework Comm
root@P690A7:/tmp>#rmdev -dl rmt3
rmt3 deleted
root@P690A7:/tmp>#rmdev -dl rmt4
rmt4 deleted
root@P690A7:/tmp>#rmdev -dl rmt5
rmt5 deleted
root@P690A7:/tmp>#rmdev -dl rmt6
rmt6 deleted
root@P690A7:/tmp>#rmdev -dl rmt7
rmt7 deleted
root@P690A7:/tmp>#rmdev -dl rmt8
rmt8 deleted
root@P690A7:/tmp>#rmdev -dl rmt9
rmt9 deleted
root@P690A7:/tmp>#rmdev -dl rmt10
rmt10 deleted
root@P690A7:/tmp>#rmdev -dl sfwcomm1
sfwcomm1 deleted
root@P690A7:/tmp>#lsdev -p fcs1
fcnet1 Defined 84-08-02 Fibre Channel Network Protocol Device
fscsi1 Available 84-08-01 FC SCSI I/O Controller Protocol Device
root@P690A7:/tmp>#rmdev -dl fcnet1
fcnet1 deleted
root@P690A7:/tmp>#rmdev -dl fscsi1
fscsi1 deleted
root@P690A7:/tmp>#rmdev -dl fcs1
fcs1 deleted
root@P690A7:/tmp>#lsdev -Cc adapter
ent0 Available 3M-08 10/100/1000 Base-TX PCI-X Adapter (14106902)
ent1 Available 6F-08 10/100/1000 Base-TX PCI-X Adapter (14106902)
ent2 Available 6k-08 10/100 Mbps Ethernet PCI Adapter II (1410ff01)
ent3 Available 7s-08 10/100/1000 Base-TX PCI-X Adapter (14106902)
sa0 Available 5J-08 IBM 8-Port EIA-232/RS-422A (PCI) Adapter
scsi0 Available 67-08 Wide/Ultra-3 SCSI I/O Controller
scsi1 Available 7b-08 Wide/Ultra-3 SCSI I/O Controller
sisscsia0 Available 3A-08 PCI-X Dual Channel Ultra320 SCSI Adapter
vsa0 Available LPAR Virtual Serial Adapter
重认设备，还不行。
root@P690A7:/tmp>#cfgmgr
cfgmgr: 0514-621 WARNING: The following device packages are required for
device support but are not currently installed.
devices.fcp.changer
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
hdisk2 00c80dbdb4e6a8f3 oradatavg
hdisk3 0032958ebf797b5b oradatavg
hdisk4 0032958ebf17babd oradatavg
hdisk5 none None
root@P690A7:/tmp>#chdev -l hdisk5 -a pv=yes
Method error (/etc/methods/chgdisk):
0514-047 Cannot access a device.
好像有点没辙了，想到会不会磁盘上存在锁的问题，先查看看
root@P690A7:/tmp>#pcmquerypr -vh /dev/hdisk5
3332393538653036
果然，这个hdisk5上存在有磁盘锁，所以磁盘不然访问。
但是在A机和B机上uname -a查看下，该不会是B机做成的锁吧
P690B7:/>#uname -a
AIX P690B7 1 6 0032958E4C00
root@P690A7:/>#uname -a
AIX P690A7 1 6 003295EE4C00
因为如果是B机做成的锁pr key应该和uname -a的值是相对应，看看又不像，因为这个hdisk5所在的LUN是回收回来后分配给这个机器的，有可能是之前的机器造成的锁，先确定下在B机上vg varyoffvg下来，然后解锁
root@P690A7:/tmp>#pcmquerypr -ch /dev/hdisk5
root@P690A7:/tmp>#pcmquerypr -vh /dev/hdisk5
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
hdisk2 00c80dbdb4e6a8f3 oradatavg
hdisk3 0032958ebf797b5b oradatavg
hdisk4 0032958ebf17babd oradatavg
hdisk5 none None
root@P690A7:/tmp>#chdev -l hdisk5 -a pv=yes
hdisk5 changed
root@P690A7:/tmp>#lspv
hdisk0 003295ee74e041d8 rootvg active
hdisk1 003295ee74fae06b None
hdisk2 00c80dbdb4e6a8f3 None
hdisk3 0032958ebf797b5b None
hdisk4 0032958ebf17babd None
hdisk5 0032958e008b4789 None
问题解决了。
注：pcmquerypr -ch /dev/vpathX可以用来清除persistent reservation。注意：该命令慎用！！！