在RAC中重建EM

    今天一项目要求对数据库进行巡检，在巡检的过程中发现EM不能使用，之前虽然在这个项目上也动过几次手，但没想过要去用EM，用户一直都是用9i下的OEM工具进行数据库管理的。用户在楼下的一家单位看人家用了WEB的EM后感觉不错，希望我顺便弄一下。

    之前也在其它项目中弄过，于是大胆的开始做，但做了两次才成功，第一次是集群名录入错了，数据库是别人装的，用了默认的CRS当集群名，而我录成了数据库名，结果是很奇怪，用emctl status dbconsole检查老说EM没启动起来，但实际上又可以登录，但注意到实例那里一个都没显示，但管理页的的表空间管理、用户管理等这些好像又正常。后来怀疑到会不会是集群名录错了，于是又去查怎么查集群名，ocrconfig -showbackup就可以查到，最后的CRS就是集群名。但后面发现端口都被占用了，只好改了端口，改成1159了。

    参考资料：http://space.itpub.net/17997/viewspace-616938

    详细过程如下

$ ocrconfig -showbackup

acc02     2011/08/04 12:03:01     /oracle/product/10.2.0/crs_1/cdata/crs

acc02     2011/08/04 08:03:01     /oracle/product/10.2.0/crs_1/cdata/crs

acc02     2011/08/04 04:03:01     /oracle/product/10.2.0/crs_1/cdata/crs

acc02     2011/08/02 20:02:59     /oracle/product/10.2.0/crs_1/cdata/crs

acc02     2011/07/27 08:02:52     /oracle/product/10.2.0/crs_1/cdata/crs
$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.4.0 - Production on Thu Aug 4 13:26:27 2011

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

sys@ACC> drop user sysman cascade;

User dropped.

sys@ACC> drop user MGMT_VIEW cascade;

User dropped.

sys@ACC> drop role MGMT_USER;

Role dropped.

sys@ACC> drop public synonym MGMT_TARGET_BLACKOUTS;

Synonym dropped.

sys@ACC> drop public synonym SETEMVIEWUSERCONTEXT;

Synonym dropped.

sys@ACC> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
$ emca -config dbcontrol db -repos create -cluster

STARTED EMCA at Aug 4, 2011 1:28:55 PM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database unique name: acc
Database Control is already configured for the database acc
You have chosen to configure Database Control for managing the database acc
This will remove the existing configuration and the default settings and perform a fresh configuration
Do you wish to continue? [yes(Y)/no(N)]: y
Listener port number: 1521
Cluster name: crs
Password for SYS user: 
Password for DBSNMP user: 
Password for SYSMAN user: 
Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /oracle/product/10.2.0/db_1

Database instance hostname ................ acc01
Listener port number ................ 1521
Cluster name ................ crs
Database unique name ................ acc
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: y
Aug 4, 2011 1:29:52 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /oracle/product/10.2.0/db_1/cfgtoollogs/emca/acc/emca_2011-08-04_01-28-55-PM.log.
Aug 4, 2011 1:29:53 PM oracle.sysman.emcp.util.PortManager isPortInUse
WARNING: Specified port 5540 is already in use.
Aug 4, 2011 1:29:53 PM oracle.sysman.emcp.util.PortManager isPortInUse
WARNING: Specified port 5520 is already in use.
Aug 4, 2011 1:29:53 PM oracle.sysman.emcp.util.PortManager isPortInUse
WARNING: Specified port 1158 is already in use.
Aug 4, 2011 1:29:53 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Aug 4, 2011 1:30:04 PM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...
Aug 4, 2011 1:31:54 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully created
Aug 4, 2011 1:31:58 PM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
INFO: Propagating /oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_acc01_acc1 to remote nodes ...
Aug 4, 2011 1:31:59 PM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
INFO: Propagating /oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_acc02_acc2 to remote nodes ...
Aug 4, 2011 1:32:03 PM oracle.sysman.emcp.EMAgentConfig deployStateDirs
INFO: Propagating /oracle/product/10.2.0/db_1/acc01_acc1 to remote nodes ...
Aug 4, 2011 1:32:05 PM oracle.sysman.emcp.EMAgentConfig deployStateDirs
INFO: Propagating /oracle/product/10.2.0/db_1/acc02_acc2 to remote nodes ...
Aug 4, 2011 1:32:06 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Cannot start Database Control. The following ports are already in use: [EM agent port:3938].
Refer to the log file at /oracle/product/10.2.0/db_1/cfgtoollogs/emca/acc/emca_2011-08-04_01-28-55-PM.log for more details.
Could not complete the configuration. Refer to the log file at /oracle/product/10.2.0/db_1/cfgtoollogs/emca/acc/emca_2011-08-04_01-28-55-PM.log for more details.
$ emctl stop dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0 
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
http://acc01:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 10g Database Control ...

 Cannot determine Oracle Enterprise Manager 10g Database Control process. /oracle/product/10.2.0/db_1/acc01_acc1/emctl.pid does not exist.
$ ps -ef|grep dbconsole | awk ' { print $2} ' | xargs kill -9
kill: 193128: 0509-015 The specified process does not exist.

$ ps -ef|grep emagent (selectively remove the dbconsole's emagent process)
ksh: 0403-057 Syntax error: `(' is not expected.
$ emca -reconfig ports -cluster -DBCONTROL_HTTP_PORT 1159 -AGENT_PORT 3939 -RMI_PORT 5522 -JMS_PORT 5542

STARTED EMCA at Aug 4, 2011 1:35:21 PM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database unique name: acc

Do you wish to continue? [yes(Y)/no(N)]: y
Aug 4, 2011 1:35:29 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /oracle/product/10.2.0/db_1/cfgtoollogs/emca/acc/emca_2011-08-04_01-35-21-PM.log.
Aug 4, 2011 1:35:31 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Aug 4, 2011 1:36:17 PM oracle.sysman.emcp.EMDBCConfig performReconfiguration
INFO: Propagating /oracle/product/10.2.0/db_1/acc01_acc1/sysman/config/emoms.properties,/oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_acc01_acc1/config/http-web-site.xml,/oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_acc01_acc1/config/rmi.xml,/oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_acc01_acc1/config/jms.xml to remote nodes ...
Aug 4, 2011 1:36:18 PM oracle.sysman.emcp.EMDBCConfig performReconfiguration
INFO: Propagating /oracle/product/10.2.0/db_1/acc02_acc2/sysman/config/emoms.properties,/oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_acc02_acc2/config/http-web-site.xml,/oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_acc02_acc2/config/rmi.xml,/oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_acc02_acc2/config/jms.xml to remote nodes ...
Aug 4, 2011 1:36:20 PM oracle.sysman.emcp.EMAgentConfig performReconfiguration
INFO: Propagating /oracle/product/10.2.0/db_1/acc01_acc1/sysman/config/emd.properties to remote nodes ...
Aug 4, 2011 1:36:20 PM oracle.sysman.emcp.EMAgentConfig performReconfiguration
INFO: Propagating /oracle/product/10.2.0/db_1/acc02_acc2/sysman/config/emd.properties to remote nodes ...
Aug 4, 2011 1:36:21 PM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
Aug 4, 2011 1:38:32 PM oracle.sysman.emcp.EMDBPostConfig performReconfiguration
INFO: Database Control started successfully
Aug 4, 2011 1:38:32 PM oracle.sysman.emcp.EMDBPostConfig performReconfiguration
INFO: >>>>>>>>>>> The Database Control URL is http://acc01:1159/em <<<<<<<<<<<
Aug 4, 2011 1:38:32 PM oracle.sysman.emcp.EMDBPostConfig showClusterDBCAgentMessage
INFO:
****************  Current Configuration  ****************
 INSTANCE            NODE           DBCONTROL_UPLOAD_HOST
----------        ----------        ---------------------

acc1              acc01             acc01
acc2              acc02             acc01

Enterprise Manager configuration completed successfully
FINISHED EMCA at Aug 4, 2011 1:38:32 PM
$ emctl status dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0 
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
http://acc01:1159/em/console/aboutApplication
Oracle Enterprise Manager 10g is running.
------------------------------------------------------------------
Logs are generated in directory /oracle/product/10.2.0/db_1/acc01_acc1/sysman/log