oracle RAC 节点频繁重启

我一直把注意集中在了心跳上，但是看来还是有其他因素的。下面两个主机的报错请大家看下。可能是因为定时作业的 exp作业：

我1年前维护时时无规律的宕机，当时处理心跳解决了。现在刚接手，听说近俩月又开始频繁重启了。贴日志给大家：
节点1：
[rcy55a01][root][/oraexp]#errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
A6DF45AA   0616022013 I O RMCdaemon      The daemon is started.
2BFA76F6   0616021813 T S SYSPROC        SYSTEM SHUTDOWN BY USER
9DBCFDEE   0616021913 T O errdemon       ERROR LOGGING TURNED ON
EC0BCCD4   0614151113 T H ent1           ETHERNET DOWN
F3931284   0614150913 I H ent1           ETHERNET NETWORK RECOVERY MODE
F3931284   0614150713 I H ent3           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614150713 T H ent3           ETHERNET DOWN
EC0BCCD4   0614145713 T H ent1           ETHERNET DOWN
F3931284   0614145713 I H ent1           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614145713 T H ent1           ETHERNET DOWN
F3931284   0614145613 I H ent1           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614145413 T H ent1           ETHERNET DOWN
F3931284   0614145413 I H ent1           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614145413 T H ent1           ETHERNET DOWN
F3931284   0614145313 I H ent1           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614145213 T H ent1           ETHERNET DOWN
F3931284   0614145213 I H ent1           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614145213 T H ent1           ETHERNET DOWN
F3931284   0614145113 I H ent1           ETHERNET NETWORK RECOVERY MODE
F3931284   0614144913 I H ent3           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614144813 T H ent3           ETHERNET DOWN
A6DF45AA   0614022213 I O RMCdaemon      The daemon is started.
EC0BCCD4   0614022213 T H ent1           ETHERNET DOWN
2BFA76F6   0614022013 T S SYSPROC        SYSTEM SHUTDOWN BY USER
9DBCFDEE   0614022213 T O errdemon       ERROR LOGGING TURNED ON
节点2：
[rcy55a02][root][/]#errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
F3931284   0616021913 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0616021913 T H ent0           ETHERNET DOWN
F3931284   0616021713 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0616021713 T H ent0           ETHERNET DOWN
F3931284   0616021713 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0616021713 T H ent0           ETHERNET DOWN
F3931284   0616021713 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0616021713 T H ent0           ETHERNET DOWN
F3931284   0616021713 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0616021713 T H ent0           ETHERNET DOWN
F3931284   0616021713 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0616021713 T H ent0           ETHERNET DOWN
F3931284   0614150713 I H ent3           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614150713 T H ent3           ETHERNET DOWN
F3931284   0614144913 I H ent3           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614144813 T H ent3           ETHERNET DOWN
F3931284   0614022413 I H ent3           ETHERNET NETWORK RECOVERY MODE
F3931284   0614022413 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614022413 T H ent3           ETHERNET DOWN
EC0BCCD4   0614022413 T H ent0           ETHERNET DOWN
F3931284   0614022213 I H ent0           ETHERNET NETWORK RECOVERY MODE
F3931284   0614022213 I H ent3           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614022213 T H ent0           ETHERNET DOWN
F3931284   0614022213 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614022213 T H ent0           ETHERNET DOWN
F3931284   0614022213 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614022213 T H ent0           ETHERNET DOWN
F3931284   0614022213 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614022213 T H ent0           ETHERNET DOWN
F3931284   0614022213 I H ent0           ETHERNET NETWORK RECOVERY MODE
EC0BCCD4   0614022213 T H ent3           ETHERNET DOWN
EC0BCCD4   0614022213 T H ent0           ETHERNET DOWN


上面2点以后有个定时作业，是ORACLE的EXP，然后GZIP。宕机时正是在GZIP。定时作业只周一，周五。周日做。正好有宕机。

刚重新维护这系统，再观察下吧。我把定时作业的GZIP去掉了。观察下吧。希望能够解决问题。

另外实在不行，我考虑通过交换机来做心跳，不知道效果会不会好。 收起

网卡多的话，做个etherchannel，这样不会因为单个网卡失效导致节点重启。而且rac心跳的cache fusion对网络带宽和延迟要求还是挺高的，有条件尽量做下聚合。 收起

遇见同样的问题，但客户说是在NBU备份后才出现的，客户用的是AIX 6.1 10G RAC 裸设备。开始怀疑是备份太占资源了，现在看来不一定是，因为备份输出的日志并没有报错。 收起

比较明显的，应该是由于心跳引起的。 收起

alert日志中有记录CSSD Reconfiguration complete，不知道该进程Reconfiguration是不是可以理解成进程重启，如果是，这必然会引起系统的重启，因为10g的rac中进程cssd的级别为fatal，而引起心跳异常、进程reconfiguration的原因应该为私有网络连接的问题，建议改为连接交换机而非直连，到目前为止还没有见过私有网络采用直连模式的。 收起

网络心跳不稳定。

oracle官方不支持使用网线直接连，虽然测试上可以：
Is crossover cable supported as an interconnect with RAC on any platform ?
NO. CROSS OVER CABLES ARE NOT SUPPORTED. The requirement is to use a switch:

Detailed Reasons:

1) cross-cabling limits the expansion of RAC to two nodes

2) cross-cabling is unstable:

a) Some NIC cards do not work properly with it. They are not able to negotiate the DTE/DCE clocking, and will thus not function. These NICS were made cheaper by assuming that the switch was going to have the clock. Unfortunately there is no way to know which NICs do not have that clock.

b) Media sense behaviour on various OS's (most notably Windows) will bring a NIC down when a cable is disconnected. Either of these issues can lead to cluster instability and lead to ORA-29740 errors (node evictions).

Due to the benefits and stability provided by a switch, and their afforability ($200 for a simple 16 port GigE switch), and the expense and time related to dealing with issues when one does not exist, this is the only supported configuration.

From a purely technology point of view Oracle does not care if the customer uses cross over cable or router or switches to deliver a message. However, we know from experience that a lot of adapters misbehave when used in a crossover configuration and cause a lot of problems for RAC. Hence we have stated on certify that we do not support crossover cables to avoid false bugs and finger pointing amongst the various parties: Oracle, Hardware vendors, Os vendors etc...


-------------------------------------------------------------------------------- 收起

你要是两台主机一根网线直连的话，就考虑换网线，或者报错的网卡吧，看情况是网络心跳引起的 收起

接交换机吧！别直连 收起

碰到过这样问题，直连不稳定，你接交换机上看看。 收起

一根网线直连 收起