LABEL: INIT_RAPID
IDENTIFIER: 3A30359F
Date/Time: Fri Nov 6 17:09:47 2015
Sequence Number: 26937
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: S
Type: TEMP
WPAR: Global
Resource Name: init
Description
SOFTWARE PROGRAM ERROR
Probable Causes
SOFTWARE PROGRAM
User Causes
PERFORMANCE DEGRADED
Recommended Actions
REVIEW DETAILED DATA
Detail Data
SOFTWARE ERROR CODE
Command is respawning too rapidly. Check for possible errors.
COMMAND
id: was "/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh ser
---------------------------------------------------------------------------
LABEL: INIT_RAPID
IDENTIFIER: 3A30359F
Date/Time: Fri Nov 6 15:00:37 2015
Sequence Number: 26936
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: S
Type: TEMP
WPAR: Global
Resource Name: init
Description
SOFTWARE PROGRAM ERROR
Probable Causes
SOFTWARE PROGRAM
User Causes
PERFORMANCE DEGRADED
Recommended Actions
REVIEW DETAILED DATA
Detail Data
SOFTWARE ERROR CODE
Command is respawning too rapidly. Check for possible errors.
COMMAND
id: was "/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh ser
---------------------------------------------------------------------------
LABEL: INIT_RAPID
IDENTIFIER: 3A30359F
Date/Time: Fri Nov 6 14:40:27 2015
Sequence Number: 26935
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: S
Type: TEMP
WPAR: Global
Resource Name: init
Description
SOFTWARE PROGRAM ERROR
Probable Causes
SOFTWARE PROGRAM
User Causes
PERFORMANCE DEGRADED
Recommended Actions
REVIEW DETAILED DATA
Detail Data
SOFTWARE ERROR CODE
Command is respawning too rapidly. Check for possible errors.
COMMAND
id: was "/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh ser
---------------------------------------------------------------------------
LABEL: INIT_RAPID
IDENTIFIER: 3A30359F
Date/Time: Fri Nov 6 14:00:37 2015
Sequence Number: 26934
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: S
Type: TEMP
WPAR: Global
Resource Name: init
Description
SOFTWARE PROGRAM ERROR
Probable Causes
SOFTWARE PROGRAM
User Causes
PERFORMANCE DEGRADED
Recommended Actions
REVIEW DETAILED DATA
Detail Data
SOFTWARE ERROR CODE
Command is respawning too rapidly. Check for possible errors.
COMMAND
id: was "/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh ser
---------------------------------------------------------------------------
LABEL: INIT_RAPID
IDENTIFIER: 3A30359F
Date/Time: Fri Nov 6 13:20:37 2015
Sequence Number: 26933
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: S
Type: TEMP
WPAR: Global
Resource Name: init
Description
SOFTWARE PROGRAM ERROR
Probable Causes
SOFTWARE PROGRAM
User Causes
PERFORMANCE DEGRADED
Recommended Actions
REVIEW DETAILED DATA
Detail Data
SOFTWARE ERROR CODE
Command is respawning too rapidly. Check for possible errors.
COMMAND
id: was "/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh ser
---------------------------------------------------------------------------
LABEL: INIT_RAPID
IDENTIFIER: 3A30359F
Date/Time: Fri Nov 6 13:03:57 2015
Sequence Number: 26932
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: S
Type: TEMP
WPAR: Global
Resource Name: init
Description
SOFTWARE PROGRAM ERROR
Probable Causes
SOFTWARE PROGRAM
User Causes
PERFORMANCE DEGRADED
Recommended Actions
REVIEW DETAILED DATA
Detail Data
SOFTWARE ERROR CODE
Command is respawning too rapidly. Check for possible errors.
COMMAND
id: was "/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh ser
---------------------------------------------------------------------------
LABEL: INIT_RAPID
IDENTIFIER: 3A30359F
Date/Time: Fri Nov 6 10:13:13 2015
Sequence Number: 26931
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: S
Type: TEMP
WPAR: Global
Resource Name: init
Description
SOFTWARE PROGRAM ERROR
Probable Causes
SOFTWARE PROGRAM
User Causes
PERFORMANCE DEGRADED
Recommended Actions
REVIEW DETAILED DATA
Detail Data
SOFTWARE ERROR CODE
Command is respawning too rapidly. Check for possible errors.
COMMAND
id: was "/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh ser
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905
Date/Time: Fri Nov 6 10:10:02 2015
Sequence Number: 26930
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: O
Type: INFO
WPAR: Global
Resource Name: grpsvcs
Description
Group Services daemon started
Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user
User Causes
Daemon started manually by user
Recommended Actions
Check that Group Services daemon is running
Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.62.1.22,727
ERROR ID
63Y7ej0uj.DK/n0i1/cU08....................
REFERENCE CODE
DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_trace.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60
Date/Time: Fri Nov 6 10:10:01 2015
Sequence Number: 26929
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: O
Type: INFO
WPAR: Global
Resource Name: topsvcs
Description
Topology Services daemon started
Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user
User Causes
Daemon started manually by user
Recommended Actions
Confirm that this is desirable
Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.215.1.13,4956
ERROR ID
6UpNEL0tj.DK/UJR//cU08....................
REFERENCE CODE
Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.06.101000.DSW_cluster.C
Topology Services daemon run directory
/var/ha/run/topsvcs.DSW_cluster/
---------------------------------------------------------------------------
LABEL: RMCD_INFO_0_ST
IDENTIFIER: A6DF45AA
Date/Time: Fri Nov 6 10:08:53 2015
Sequence Number: 26928
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: O
Type: INFO
WPAR: Global
Resource Name: RMCdaemon
Description
The daemon is started.
Probable Causes
The Resource Monitoring and Control daemon has been started.
User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.
Recommended Actions
Confirm that the daemon should be started.
Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.75.1.2,229
ERROR ID
6eKora0pi.DK/H1E1/cU08....................
REFERENCE CODE
---------------------------------------------------------------------------
LABEL: HEA_PORT_DOWN
IDENTIFIER: 38A8D4BB
Date/Time: Fri Nov 6 10:07:21 2015
Sequence Number: 26926
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: H
Type: INFO
WPAR: Global
Resource Name: ent4
Resource Class:
Resource Type:
Location:
VPD:
IBM Host Ethernet Adapter:
Network Address.............5CF3FC857D00
Description
HEA PORT DOWN
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
FILE NAME
line: 2079 file: hea_prim.c
HEA LOGICAL PORT STATE:
UP
HEA PHYSICAL PORT STATE:
DOWN
SENSE DATA
0000 0002 0261 081B 0000 0000 0000 0000 0000 0000 0001 8000 0000 0300 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0006 0000 0000 0000 008E 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
DEVICE DRIVER INTERNAL STATE
3333 3333 0000 0000 0000 0001 0000 0000 0000 0000
SOURCE ADDRESS
5CF3 FC85 7D00
---------------------------------------------------------------------------
LABEL: REBOOT_ID
IDENTIFIER: 2BFA76F6
Date/Time: Fri Nov 6 10:05:33 2015
Sequence Number: 26925
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: S
Type: TEMP
WPAR: Global
Resource Name: SYSPROC
Description
SYSTEM SHUTDOWN BY USER
Probable Causes
SYSTEM SHUTDOWN
Detail Data
USER ID
0
0=SOFT IPL 1=HALT 2=TIME REBOOT
0
TIME TO REBOOT (FOR TIMED REBOOT ONLY)
0
---------------------------------------------------------------------------
LABEL: ERRLOG_ON
IDENTIFIER: 9DBCFDEE
Date/Time: Fri Nov 6 10:08:18 2015
Sequence Number: 26924
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: O
Type: TEMP
WPAR: Global
Resource Name: errdemon
Description
ERROR LOGGING TURNED ON
Probable Causes
ERRDEMON STARTED AUTOMATICALLY
User Causes
/USR/LIB/ERRDEMON COMMAND
Recommended Actions
NONE
---------------------------------------------------------------------------
LABEL: ERRLOG_OFF
IDENTIFIER: 192AC071
Date/Time: Fri Nov 6 10:02:54 2015
Sequence Number: 26923
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: O
Type: TEMP
WPAR: Global
Resource Name: errdemon
Description
ERROR LOGGING TURNED OFF
Probable Causes
ERRSTOP COMMAND
User Causes
ERRSTOP COMMAND
Recommended Actions
RUN ERRDEAD COMMAND
TURN ERROR LOGGING ON
---------------------------------------------------------------------------
LABEL: TS_NIM_ERROR_STUCK_
IDENTIFIER: 3D32B80D
Date/Time: Tue Nov 3 13:27:53 2015
Sequence Number: 26922
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: S
Type: PERM
WPAR: Global
Resource Name: topsvcs
Description
NIM thread blocked
Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
Recommended Actions
Examine I/O and memory activity on the system
Reduce load on the system
Tune virtual memory parameters
Call IBM Service if problem persists
Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O
Recommended Actions
Examine I/O and memory activity on the system
Reduce load on the system
Tune virtual memory parameters
Call IBM Service if problem persists
Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916
ERROR ID
6BUfAx.NL2CK/aPI//cU08....................
REFERENCE CODE
Thread which was blocked
send thread
Interval in seconds during which process was blocked
19
Interface name
rmndhb_lv_01.1_2
---------------------------------------------------------------------------
LABEL: LVM_GS_RLEAVE
IDENTIFIER: AB59ABFF
Date/Time: Tue Nov 3 13:27:50 2015
Sequence Number: 26921
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: U
Type: UNKN
WPAR: Global
Resource Name: LIBLVM
Resource Class: NONE
Resource Type: NONE
Location:
Description
Remote node Concurrent Volume Group failure detected
Probable Causes
Remote node Concurrent Volume Group forced offline
Failure Causes
Remote node left VGSA/VGDA groups due to failure
Recommended Actions
Examine error log on identified remote node
Detail Data
Remote Node Name
DSW02
Volume Group ID
00F6 CC49 0000 4C00 0000 0132 F7D7 0C7F
MAJOR/MINOR DEVICE NUMBER
002D 0000
SENSE DATA
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
---------------------------------------------------------------------------
LABEL: LVM_GS_RLEAVE
IDENTIFIER: AB59ABFF
Date/Time: Tue Nov 3 13:27:34 2015
Sequence Number: 26920
Machine Id: 00F6CC494C00
Node Id: SVR-DSW-01
Class: U
Type: UNKN
WPAR: Global
Resource Name: LIBLVM
Resource Class: NONE
Resource Type: NONE
Location:
Description
Remote node Concurrent Volume Group failure detected
Probable Causes
Remote node Concurrent Volume Group forced offline
Failure Causes
Remote node left VGSA/VGDA groups due to failure
Recommended Actions
Examine error log on identified remote node
Detail Data
Remote Node Name
DSW02
Volume Group ID
00F6 CC49 0000 4C00 0000 0132 F7D7 0C7F
MAJOR/MINOR DEVICE NUMBER
002D 0000
SENSE DATA
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000