求救——一台 P570 突然宕机

今天 我们的 tsm 服务起不来了  是一台 P570  突然宕机  报错信息如下
# errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
A6DF45AA   0301115110 I O RMCdaemon      The daemon is started.
AA8AB241   0301115110 T O OPERATOR       OPERATOR NOTIFICATION
BC3BE5A3   0301115110 P S SRC            SOFTWARE PROGRAM ERROR
2BFA76F6   0301114910 T S SYSPROC        SYSTEM SHUTDOWN BY USER
9DBCFDEE   0301115110 T O errdemon       ERROR LOGGING TURNED ON
192AC071   0301112010 T O errdemon       ERROR LOGGING TURNED OFF
DE3B8540   0227010010 P H hdisk5         PATH HAS FAILED
DE3B8540   0225010010 P H hdisk5         PATH HAS FAILED
DE3B8540   0223010010 P H hdisk5         PATH HAS FAILED
C43F90ED   0222061210 P H hdisk3         SUBSYSTEM COMPONENT FAILURE
C43F90ED   0222031810 P H hdisk9         SUBSYSTEM COMPONENT FAILURE
DE3B8540   0215010010 P H hdisk5         PATH HAS FAILED
DE3B8540   0211010010 P H hdisk5         PATH HAS FAILED
DE3B8540   0210010010 P H hdisk5         PATH HAS FAILED
DE3B8540   0208010010 P H hdisk5         PATH HAS FAILED

# errpt -aj DE3B8540 | more
---------------------------------------------------------------------------
LABEL:          SC_DISK_ERR7
IDENTIFIER:     DE3B8540
Date/Time:       Sat Feb 27 01:00:32 GMT+08:00 2010
Sequence Number: 1238
Machine Id:      00CFCF234C00
Node Id:         czqas
Class:           H
Type:            PERM
WPAR:            Global
Resource Name:   hdisk5         
Resource Class:  disk
Resource Type:   mpioapdisk
Location:        U789D.001.DQD45HD-P1-C2-T1-W201500A0B8476EFA-L3000000000000
VPD:            
        Manufacturer................IBM     
        Machine Type and Model......1815      FAStT
        ROS Level and ID............30393134
        Serial Number...............
        Device Specific.(Z0)........0000053245004032
        Device Specific.(Z1)........
Description
PATH HAS FAILED
Probable Causes
ADAPTER HARDWARE OR CABLE
DASD DEVICE
Failure Causes
UNDETERMINED
        Recommended Actions
        PERFORM PROBLEM DETERMINATION PROCEDURES
        CHECK PATH
Detail Data
PATH ID
           0
SENSE DATA
0A00 2A00 0523 3BD0 0000 0804 0000 0000 0000 0000 0000 0000 0102 0000 0000 0000
Standard input


---------------------------------------------------------------------------
LABEL:          SC_DISK_PCM_ERR1
IDENTIFIER:     C43F90ED
Date/Time:       Mon Feb 22 06:12:03 GMT+08:00 2010
Sequence Number: 1235
Machine Id:      00CFCF234C00
Node Id:         czqas
Class:           H
Type:            PERM
WPAR:            Global
Resource Name:   hdisk3         
Resource Class:  disk
Resource Type:   mpioapdisk
Location:        U789D.001.DQD45HD-P1-C2-T1-W201500A0B8476EFA-L1000000000000
VPD:            
        Manufacturer................IBM     
        Machine Type and Model......1815      FAStT
        ROS Level and ID............30393134
        Serial Number...............
        Device Specific.(Z0)........0000053245004032
        Device Specific.(Z1)........
Description
SUBSYSTEM COMPONENT FAILURE
Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
        Recommended Actions
        PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
PATH ID
           2
SENSE DATA
0600 0308 0000 FF04 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600




errpt -aj 192AC071 | more
---------------------------------------------------------------------------
LABEL:          ERRLOG_OFF
IDENTIFIER:     192AC071
Date/Time:       Mon Mar  1 11:20:08 GMT+08:00 2010
Sequence Number: 1239
Machine Id:      00CFCF234C00
Node Id:         czqas
Class:           O
Type:            TEMP
WPAR:            Global
Resource Name:   errdemon        
Description
ERROR LOGGING TURNED OFF
Probable Causes
ERRSTOP COMMAND
User Causes
ERRSTOP COMMAND
        Recommended Actions
        RUN ERRDEAD COMMAND
        TURN ERROR LOGGING ON



# errpt -aj 9DBCFDEE | more
---------------------------------------------------------------------------
LABEL:          ERRLOG_ON
IDENTIFIER:     9DBCFDEE
Date/Time:       Mon Mar  1 11:51:07 GMT+08:00 2010
Sequence Number: 1240
Machine Id:      00CFCF234C00
Node Id:         czqas
Class:           O
Type:            TEMP
WPAR:            Global
Resource Name:   errdemon        
Description
ERROR LOGGING TURNED ON
Probable Causes
ERRDEMON STARTED AUTOMATICALLY
User Causes
/USR/LIB/ERRDEMON COMMAND
        Recommended Actions
        NONE


# errpt -aj 2BFA76F6 | more
---------------------------------------------------------------------------
LABEL:          REBOOT_ID
IDENTIFIER:     2BFA76F6
Date/Time:       Mon Mar  1 11:49:49 GMT+08:00 2010
Sequence Number: 1241
Machine Id:      00CFCF234C00
Node Id:         czqas
Class:           S
Type:            TEMP
WPAR:            Global
Resource Name:   SYSPROC         
Description
SYSTEM SHUTDOWN BY USER
Probable Causes
SYSTEM SHUTDOWN
Detail Data
USER ID
           0
0=SOFT IPL 1=HALT 2=TIME REBOOT
           1
TIME TO REBOOT (FOR TIMED REBOOT ONLY)
           0

# errpt -aj BC3BE5A3 | more
---------------------------------------------------------------------------
LABEL:          SRC_SVKO
IDENTIFIER:     BC3BE5A3
Date/Time:       Mon Mar  1 11:51:21 GMT+08:00 2010
Sequence Number: 1243
Machine Id:      00CFCF234C00
Node Id:         czqas
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SRC            
Description
SOFTWARE PROGRAM ERROR
Probable Causes
APPLICATION PROGRAM
Failure Causes
SOFTWARE PROGRAM
        Recommended Actions
        MANUALLY RESTART SUBSYSTEM IF NEEDED
Detail Data
SYMPTOM CODE
         256
SOFTWARE ERROR CODE
       -9017
ERROR CODE
           0
DETECTING MODULE
[email=]'srchevn.c'@line:'376'[/email]
FAILING MODULE
named


# errpt -aj AA8AB241 | more
---------------------------------------------------------------------------
LABEL:          OPMSG
IDENTIFIER:     AA8AB241
Date/Time:       Mon Mar  1 11:51:33 GMT+08:00 2010
Sequence Number: 1244
Machine Id:      00CFCF234C00
Node Id:         czqas
Class:           O
Type:            TEMP
WPAR:            Global
Resource Name:   OPERATOR        
Description
OPERATOR NOTIFICATION
User Causes
ERRLOGGER COMMAND
        Recommended Actions
        REVIEW DETAILED DATA
Detail Data
MESSAGE FROM ERRLOGGER COMMAND
Mon Mar 1 11:51:33 GMT+08:00 2010 SMagent started.



# errpt -aj A6DF45AA | more
---------------------------------------------------------------------------
LABEL:          RMCD_INFO_0_ST
IDENTIFIER:     A6DF45AA
Date/Time:       Mon Mar  1 11:51:35 GMT+08:00 2010
Sequence Number: 1245
Machine Id:      00CFCF234C00
Node Id:         czqas
Class:           O
Type:            INFO
WPAR:            Global
Resource Name:   RMCdaemon      
Description
The daemon is started.
Probable Causes
The Resource Monitoring and Control daemon has been started.
User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.
        Recommended Actions
        Confirm that the daemon should be started.
Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.62,213                          
ERROR ID
6eKora05bnW9/nKR/P.U/8....................
REFERENCE CODE