请教一下CRC errors和enc-out问题
整体环境为全HP产品
存储为3par,SAN switch为博客系列,小型机为 rx8640
下面反应业务有中断现象，出现频率较小，大部分时间是可以正常业务的。
分别对操作系统、SAN交换、存储进行分析
1、在存储中发现异常如下：
Message Repeat Count First Time
Port 0:1:1 Degraded (Intermittent CRC Errors Detected) 1173 Dec 07 2013 04:43:09 CST
Host Port 0:1:1 experienced over 50 CRC errors (53) in 24 hours 1171 Dec 07 2013 04:43:09 CST
2、查看了SAN switch日志，在连接小型机（两台）的端口上发现以下情况：
porterrshow:
frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout
tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx
1: 1.5g 4.1g 56 14 1 0 0 13 398.4k 1 0 14 16 0 0 0 0
6: 3.7g 3.4g 22 11 6 0 0 5 1.0m 4 0 7 10 0 0 0 0
（如果看起来较乱请见谅）
端口情况：
Index Port Address Media Speed State Proto
==============================================
1 1 010100 id N4 Online FC F-Port 50:01:43:80:11:00:02:da
6 6 010600 id N4 Online FC F-Port 50:01:43:80:16:e7:ec:1c
其他端口有N4，N8不通速率混用，我查看的机房管理员所说的存储与SAN switch链接线路状态为No_Light状态，速率为N8，不知道速率是否有影响。
3、主机上MP查看硬件没发现故障，其他不知道如何排查（正在协调收集系统日志分析）
疑问是不确定这是什么问题造成的？该问题是否会导致业务异常？尝试换过光纤线，问题依旧存在，另外机房EMC设备中也出现类似问题，还请大神指教下一步该如何做，有什么需要的信息我会搜集并贴上
存储为3par,SAN switch为博客系列,小型机为 rx8640
下面反应业务有中断现象，出现频率较小，大部分时间是可以正常业务的。
分别对操作系统、SAN交换、存储进行分析
1、在存储中发现异常如下：
Message Repeat Count First Time
Port 0:1:1 Degraded (Intermittent CRC Errors Detected) 1173 Dec 07 2013 04:43:09 CST
Host Port 0:1:1 experienced over 50 CRC errors (53) in 24 hours 1171 Dec 07 2013 04:43:09 CST
2、查看了SAN switch日志，在连接小型机（两台）的端口上发现以下情况：
porterrshow:
frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout
tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx
1: 1.5g 4.1g 56 14 1 0 0 13 398.4k 1 0 14 16 0 0 0 0
6: 3.7g 3.4g 22 11 6 0 0 5 1.0m 4 0 7 10 0 0 0 0
（如果看起来较乱请见谅）
端口情况：
Index Port Address Media Speed State Proto
==============================================
1 1 010100 id N4 Online FC F-Port 50:01:43:80:11:00:02:da
6 6 010600 id N4 Online FC F-Port 50:01:43:80:16:e7:ec:1c
其他端口有N4，N8不通速率混用，我查看的机房管理员所说的存储与SAN switch链接线路状态为No_Light状态，速率为N8，不知道速率是否有影响。
3、主机上MP查看硬件没发现故障，其他不知道如何排查（正在协调收集系统日志分析）
疑问是不确定这是什么问题造成的？该问题是否会导致业务异常？尝试换过光纤线，问题依旧存在，另外机房EMC设备中也出现类似问题，还请大神指教下一步该如何做，有什么需要的信息我会搜集并贴上
14同行回答
回复 5# phanx
马上获取1，6端口sfp信息。
另外，有没有可能是主机HBA卡的问题呢？HBA卡信息如下：
Class I H/W Path Driver S/W State H/W Type Description
=======================================================================
fc 0 0/0/12/0/0/0/0 fclp CLAIMED INTERFACE HP AD299-60001 4Gb PCIe 1-port Fibre Channel Adapter
/dev/fclp0
fc 1 1/0/12/0/0/0/0 fclp CLAIMED INTERFACE HP AD299-60001 4Gb PCIe 1-port Fibre Channel Adapter
/dev/fclp1
HBA卡详细信息（fclp0）
Vendor ID is = 0x10df
Device ID is = 0xfe00
PCI Sub-system Vendor ID is = 0x103c
PCI Sub-system ID is = 0x7046
Chip version = 2
Firmware Version = 2.70X5 SLI-3 (Z3D2.70X5)
EFI Version = ZE3.21A3
EFI Boot = ENABLED
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x010100
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50014380110002db
N_Port Port World Wide Name = 0x50014380110002da
Switch Port World Wide Name = 0x20010027f893dffd
Switch Node World Wide Name = 0x10000027f893dffd
Driver state = ONLINE
Hardware Path is = 0/0/12/0/0/0/0
Maximum Frame Size = 2048
Driver Version = @(#) FCLP: PCIe Fibre Channel driver (FibrChanl-02), B.11.31.1009, May 7 2010, FCLP_IFC (3,2)收起
马上获取1，6端口sfp信息。
另外，有没有可能是主机HBA卡的问题呢？HBA卡信息如下：
Class I H/W Path Driver S/W State H/W Type Description
=======================================================================
fc 0 0/0/12/0/0/0/0 fclp CLAIMED INTERFACE HP AD299-60001 4Gb PCIe 1-port Fibre Channel Adapter
/dev/fclp0
fc 1 1/0/12/0/0/0/0 fclp CLAIMED INTERFACE HP AD299-60001 4Gb PCIe 1-port Fibre Channel Adapter
/dev/fclp1
HBA卡详细信息（fclp0）
Vendor ID is = 0x10df
Device ID is = 0xfe00
PCI Sub-system Vendor ID is = 0x103c
PCI Sub-system ID is = 0x7046
Chip version = 2
Firmware Version = 2.70X5 SLI-3 (Z3D2.70X5)
EFI Version = ZE3.21A3
EFI Boot = ENABLED
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x010100
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50014380110002db
N_Port Port World Wide Name = 0x50014380110002da
Switch Port World Wide Name = 0x20010027f893dffd
Switch Node World Wide Name = 0x10000027f893dffd
Driver state = ONLINE
Hardware Path is = 0/0/12/0/0/0/0
Maximum Frame Size = 2048
Driver Version = @(#) FCLP: PCIe Fibre Channel driver (FibrChanl-02), B.11.31.1009, May 7 2010, FCLP_IFC (3,2)收起
浏览10039
回复 2# phanx
请管理员确认了一下，证实存储与光交连接端口为2，3口，sfpshow（附件贴不上，全把文本粘上了）SAN SW1:PORT 2,3
sw300:admin> sfpshow 2
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 2,4,8_Gbps M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: HP-F BROCADE
Vendor OUI: 00:05:1e
Vendor PN: AJ716B
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF41211000046P
Date Code: 120312
DD Type: 0x68
Enh Options: 0xfa
Status/Ctrl: 0xb0
Alarm flags[0,1] = 0x5, 0x0
Warn Flags[0,1] = 0x5, 0x0
Alarm Warn
low high low high
Temperature: 45 Centigrade -10 90 -5 85
Current: 9.386 mAmps 1.000 17.000 2.000 14.000
Voltage: 3306.3 mVolts 2900.0 3700.0 3000.0 3600.0
RX Power: -5.7 dBm (272.1uW) 10.0 uW 1258.9 uW 15.8 uW 1000.0 uW
TX Power: -3.3 dBm (462.7 uW)125.9 uW 631.0 uW 158.5 uW 562.3 uW
State transitions: 2
sw300:admin> sfpshow 3
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 2,4,8_Gbps M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: HP-F BROCADE
Vendor OUI: 00:05:1e
Vendor PN: AJ716B
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF412110000392
Date Code: 120312
DD Type: 0x68
Enh Options: 0xfa
Status/Ctrl: 0xb0
Alarm flags[0,1] = 0x5, 0x0
Warn Flags[0,1] = 0x5, 0x0
Alarm Warn
low high low high
Temperature: 43 Centigrade -10 90 -5 85
Current: 9.078 mAmps 1.000 17.000 2.000 14.000
Voltage: 3309.8 mVolts 2900.0 3700.0 3000.0 3600.0
RX Power: -6.9 dBm (205.3uW) 10.0 uW 1258.9 uW 15.8 uW 1000.0 uW
TX Power: -3.3 dBm (465.2 uW)125.9 uW 631.0 uW 158.5 uW 562.3 uW
State transitions: 2
SAN SW2 PORT 2,3:
swd77:admin> sfpshow 2
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 2,4,8_Gbps M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: HP-F BROCADE
Vendor OUI: 00:05:1e
Vendor PN: AJ716B
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF41211000010W
Date Code: 120312
DD Type: 0x68
Enh Options: 0xfa
Status/Ctrl: 0xb0
Alarm flags[0,1] = 0x5, 0x0
Warn Flags[0,1] = 0x5, 0x0
Alarm Warn
low high low high
Temperature: 43 Centigrade -10 90 -5 85
Current: 7.162 mAmps 1.000 17.000 2.000 14.000
Voltage: 3296.9 mVolts 2900.0 3700.0 3000.0 3600.0
RX Power: -5.7 dBm (266.4uW) 10.0 uW 1258.9 uW 15.8 uW 1000.0 uW
TX Power: -3.3 dBm (463.5 uW)125.9 uW 631.0 uW 158.5 uW 562.3 uW
State transitions: 2
swd77:admin> sfpshow 3
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 2,4,8_Gbps M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: HP-F BROCADE
Vendor OUI: 00:05:1e
Vendor PN: AJ716B
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF412110000216
Date Code: 120312
DD Type: 0x68
Enh Options: 0xfa
Status/Ctrl: 0xb0
Alarm flags[0,1] = 0x5, 0x0
Warn Flags[0,1] = 0x5, 0x0
Alarm Warn
low high low high
Temperature: 45 Centigrade -10 90 -5 85
Current: 6.172 mAmps 1.000 17.000 2.000 14.000
Voltage: 3293.1 mVolts 2900.0 3700.0 3000.0 3600.0
RX Power: -3.4 dBm (462.1uW) 10.0 uW 1258.9 uW 15.8 uW 1000.0 uW
TX Power: -3.4 dBm (459.3 uW)125.9 uW 631.0 uW 158.5 uW 562.3 uW
State transitions: 2
swd77:admin>RX,TX数值看起来是在正常范围之内收起
请管理员确认了一下，证实存储与光交连接端口为2，3口，sfpshow（附件贴不上，全把文本粘上了）SAN SW1:PORT 2,3
sw300:admin> sfpshow 2
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 2,4,8_Gbps M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: HP-F BROCADE
Vendor OUI: 00:05:1e
Vendor PN: AJ716B
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF41211000046P
Date Code: 120312
DD Type: 0x68
Enh Options: 0xfa
Status/Ctrl: 0xb0
Alarm flags[0,1] = 0x5, 0x0
Warn Flags[0,1] = 0x5, 0x0
Alarm Warn
low high low high
Temperature: 45 Centigrade -10 90 -5 85
Current: 9.386 mAmps 1.000 17.000 2.000 14.000
Voltage: 3306.3 mVolts 2900.0 3700.0 3000.0 3600.0
RX Power: -5.7 dBm (272.1uW) 10.0 uW 1258.9 uW 15.8 uW 1000.0 uW
TX Power: -3.3 dBm (462.7 uW)125.9 uW 631.0 uW 158.5 uW 562.3 uW
State transitions: 2
sw300:admin> sfpshow 3
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 2,4,8_Gbps M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: HP-F BROCADE
Vendor OUI: 00:05:1e
Vendor PN: AJ716B
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF412110000392
Date Code: 120312
DD Type: 0x68
Enh Options: 0xfa
Status/Ctrl: 0xb0
Alarm flags[0,1] = 0x5, 0x0
Warn Flags[0,1] = 0x5, 0x0
Alarm Warn
low high low high
Temperature: 43 Centigrade -10 90 -5 85
Current: 9.078 mAmps 1.000 17.000 2.000 14.000
Voltage: 3309.8 mVolts 2900.0 3700.0 3000.0 3600.0
RX Power: -6.9 dBm (205.3uW) 10.0 uW 1258.9 uW 15.8 uW 1000.0 uW
TX Power: -3.3 dBm (465.2 uW)125.9 uW 631.0 uW 158.5 uW 562.3 uW
State transitions: 2
SAN SW2 PORT 2,3:
swd77:admin> sfpshow 2
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 2,4,8_Gbps M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: HP-F BROCADE
Vendor OUI: 00:05:1e
Vendor PN: AJ716B
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF41211000010W
Date Code: 120312
DD Type: 0x68
Enh Options: 0xfa
Status/Ctrl: 0xb0
Alarm flags[0,1] = 0x5, 0x0
Warn Flags[0,1] = 0x5, 0x0
Alarm Warn
low high low high
Temperature: 43 Centigrade -10 90 -5 85
Current: 7.162 mAmps 1.000 17.000 2.000 14.000
Voltage: 3296.9 mVolts 2900.0 3700.0 3000.0 3600.0
RX Power: -5.7 dBm (266.4uW) 10.0 uW 1258.9 uW 15.8 uW 1000.0 uW
TX Power: -3.3 dBm (463.5 uW)125.9 uW 631.0 uW 158.5 uW 562.3 uW
State transitions: 2
swd77:admin> sfpshow 3
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 2,4,8_Gbps M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: HP-F BROCADE
Vendor OUI: 00:05:1e
Vendor PN: AJ716B
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF412110000216
Date Code: 120312
DD Type: 0x68
Enh Options: 0xfa
Status/Ctrl: 0xb0
Alarm flags[0,1] = 0x5, 0x0
Warn Flags[0,1] = 0x5, 0x0
Alarm Warn
low high low high
Temperature: 45 Centigrade -10 90 -5 85
Current: 6.172 mAmps 1.000 17.000 2.000 14.000
Voltage: 3293.1 mVolts 2900.0 3700.0 3000.0 3600.0
RX Power: -3.4 dBm (462.1uW) 10.0 uW 1258.9 uW 15.8 uW 1000.0 uW
TX Power: -3.4 dBm (459.3 uW)125.9 uW 631.0 uW 158.5 uW 562.3 uW
State transitions: 2
swd77:admin>RX,TX数值看起来是在正常范围之内收起
浏览6523
回复 2# phanx
我让管理员收取一下信息，有消息贴上，另外，查看os dmesg，也发现lun出现问题
class : disk, instance 320
LUN (dev=0xd000020) capacity has shrunk from 62914559 to 4194303.
class : disk, instance 310
LUN (dev=0xd00001f) capacity has shrunk from 838860799 to 629145599.
class : lunpath, instance 95
lun path (class = lunpath, instance = 95) belonging to LUN (default minor = 0x1f) has gone offline. The lunpath hwpath is 0/0/12/0/0/0/0.0x50001fe1501c745a.0x4012000000000000
class : lunpath, instance 89
lun path (class = lunpath, instance = 89) belonging to LUN (default minor = 0x1e) has gone offline. The lunpath hwpath is 1/0/12/0/0/0/0.0x50001fe1501c745d.0x4011000000000000
class : lunpath, instance 84
lun path (class = lunpath, instance = 84) belonging to LUN (default minor = 0x1d) has gone offline. The lunpath hwpath is 1/0/12/0/0/0/0.0x50001fe1501c7459.0x4010000000000000收起
我让管理员收取一下信息，有消息贴上，另外，查看os dmesg，也发现lun出现问题
class : disk, instance 320
LUN (dev=0xd000020) capacity has shrunk from 62914559 to 4194303.
class : disk, instance 310
LUN (dev=0xd00001f) capacity has shrunk from 838860799 to 629145599.
class : lunpath, instance 95
lun path (class = lunpath, instance = 95) belonging to LUN (default minor = 0x1f) has gone offline. The lunpath hwpath is 0/0/12/0/0/0/0.0x50001fe1501c745a.0x4012000000000000
class : lunpath, instance 89
lun path (class = lunpath, instance = 89) belonging to LUN (default minor = 0x1e) has gone offline. The lunpath hwpath is 1/0/12/0/0/0/0.0x50001fe1501c745d.0x4011000000000000
class : lunpath, instance 84
lun path (class = lunpath, instance = 84) belonging to LUN (default minor = 0x1d) has gone offline. The lunpath hwpath is 1/0/12/0/0/0/0.0x50001fe1501c7459.0x4010000000000000收起
浏览6321