AIX swap耗尽导致oracle实例down
早上发现数据库实例无法访问，ping 数据库服务器网络正常，无法telnet登陆进去。通过管理口进去重启没相应，强制重启后进入系统检查如下：# errptIDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
AFA89905 0830100915 I O grpsvcs Group Services daemon started
97419D60 0830100915 I O topsvcs Topology Services daemon started
A6DF45AA 0830100915 I O RMCdaemon The daemon is started.
BE0A03E5 0830100515 P H sysplanar0 ENVIRONMENTAL PROBLEM
9DBCFDEE 0830100715 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0830094715 P H sysplanar0 ENVIRONMENTAL PROBLEM
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
E18E984F 0830032415 P S SRC SOFTWARE PROGRAM ERROR
CB4A951F 0830032415 I S SRC SOFTWARE PROGRAM ERROR
AA8AB241 0830032215 T O clstrmgrDANGER OPERATOR NOTIFICATION
AA8AB241 0801021415 T O clstrmgrES OPERATOR NOTIFICATION
51E537B5 0729003315 P H sysplanar0 platform_dump saved to file
A6D1BD62 0729003215 I H unspecified Firmware Event
291D64C3 0729003115 I H sysplanar0 Platform dump data
3C81E43F 0724030315 P U topsvcs Late in sending heartbeat
3C81E43F 0714030315 P U topsvcs Late in sending heartbeat
3C81E43F 0627030215 P U topsvcs Late in sending heartbeat
然后# errpt -aj C5C09FFA|pg 查看错误
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 876
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
expdp
USER'S PROCESS ID:
29098094
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
2408
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 875
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
25559232
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
3668
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 874
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
27000900
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
9616
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 873
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
30408938
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
40336
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 872
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
33947854
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
1536
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 871
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
29491234
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
0
#
提示oracle 进程导致swap耗尽。。各位有谁能给予解决思路？
AFA89905 0830100915 I O grpsvcs Group Services daemon started
97419D60 0830100915 I O topsvcs Topology Services daemon started
A6DF45AA 0830100915 I O RMCdaemon The daemon is started.
BE0A03E5 0830100515 P H sysplanar0 ENVIRONMENTAL PROBLEM
9DBCFDEE 0830100715 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0830094715 P H sysplanar0 ENVIRONMENTAL PROBLEM
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA 0830033915 P S SYSVMM SOFTWARE PROGRAM ABNORMALLY TERMINATED
E18E984F 0830032415 P S SRC SOFTWARE PROGRAM ERROR
CB4A951F 0830032415 I S SRC SOFTWARE PROGRAM ERROR
AA8AB241 0830032215 T O clstrmgrDANGER OPERATOR NOTIFICATION
AA8AB241 0801021415 T O clstrmgrES OPERATOR NOTIFICATION
51E537B5 0729003315 P H sysplanar0 platform_dump saved to file
A6D1BD62 0729003215 I H unspecified Firmware Event
291D64C3 0729003115 I H sysplanar0 Platform dump data
3C81E43F 0724030315 P U topsvcs Late in sending heartbeat
3C81E43F 0714030315 P U topsvcs Late in sending heartbeat
3C81E43F 0627030215 P U topsvcs Late in sending heartbeat
然后# errpt -aj C5C09FFA|pg 查看错误
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 876
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
expdp
USER'S PROCESS ID:
29098094
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
2408
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 875
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
25559232
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
3668
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 874
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
27000900
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
9616
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 873
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
30408938
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
40336
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 872
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
33947854
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
1536
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Aug 30 03:39:33 CST 2015
Sequence Number: 871
Machine Id: 00F75D6A4C00
Node Id: calldb2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
29491234
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
0
#
提示oracle 进程导致swap耗尽。。各位有谁能给予解决思路？