Patching CRS to 10.2.0.5

A small blog post about patching CRS 10.2.0.1 to 10.2.0.5 instead of applying the latest PSU on top of 10.2.0.4.

So it finally happened! Oracle has released the terminal patchset for 10g Release 2. Long overdue! So the task at hand today is to upgrade CRS to 10.2.0.5.

Patch

Download patch 8202632 to a convenient location and unzip it. The patch is quite large, again, about 1.2G in size for Linux 64bit RHEL5.5, my platform.

[oracle@crs102node1 10205]$ ls -lh p8202632_10205_Linux-x86-64.zip -rw-r--r-- 1 1000 users 1.2G May 25 16:48 p8202632_10205_Linux-x86-64.zip

The first thing that caught my attention was the built-in support for SLES 11 and RHEL 6. This is interesting as RHEL 6 is still beta, where SLES 11 is out for production use. Means no more hacking the oraparam.ini file then, good (if you are on one of these platforms).

[oracle@crs102node1 10205]$ ./Disk1/runInstaller Starting Oracle Universal Installer... Checking installer requirements... Checking operating system version: must be redhat-3, SuSE-9, SuSE-10, redhat-4, redhat-5, redhat-6, UnitedLinux-1.0, asianux-1, asianux-2, asianux-3, enterprise-4, enterprise-5 or SuSE-11 Passed All installer requirements met. Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-05-26_10-14-13AM. Please wait ...[oracle@crs102node1 10205]$ Oracle Universal Installer, Version 10.2.0.5.0 Production Copyright (C) 1999, 2010, Oracle. All rights reserved.

My current Clusterware installation is the base release, as this is a fresh install. There shouldn’t be a difference to 10.2.0.4 though. Clusterware will have an active version and software version. The active version in my case is the base release, and until the patch hasn’t completed (i.e. root.sh executed on last node) on all nodes that will remain the original. When root.sh completes on the last node, the active version will match the software version. Example prior to the migration:

[oracle@crs102node1 10205]$ /u01/crs/oracle/product/10.2.0/crs/bin/crsctl query crs softwareversion CRS software version on node [crs102node1] is [10.2.0.1.0] [oracle@crs102node1 10205]$ /u01/crs/oracle/product/10.2.0/crs/bin/crsctl query crs activeversion CRS active version on the cluster is [10.2.0.1.0]

Start OUI in the Disk1 directory of the unzipped patch. The steps are pretty much the same as for previous patch sets, with the exception that the prerequisite checks croaked at me. I have installed the oracle validated RPM which should set all kernel parameters and other settings, but the ip port range was wrong

Checking for ip_local_port_range=9000 - 65500; found ip_local_port_range=1024 - 65000.    Failed <<<< Checking for rmem_default=1048576; found rmem_default=262144.    Failed <<<<

These are my settings:

[oracle@crs102node1 10205]$ cat /etc/sysctl.conf | egrep "(port_range|rmem_default)" net.ipv4.ip_local_port_range = 1024 65000

Simple to fix-but needs to be done on all nodes. Become root, and change /etc/sysctl.conf keys net.core.rmem_default and net.ipv4.ip_local_port_range. Save the document and execute “sysctl -p”. Finally, check if the new settings are active:

[root@crs102node2 ~]# sysctl -a | egrep "(port_range|rmem_default)" net.ipv4.ip_local_port_range = 9000     65000 net.core.rmem_default = 1048576

The recheck actually picked the new values up, a nice yet long overdue improvement. I then went for a coffee while the installer copied all the files to the local and remote nodes.

A few moments later, OUI reported success, and this is where the “rolling” upgrade becomes a nice feature. We do patch each node individually, and when root102.sh finishes on the local node all its resources are going to be restarted. Then over to the next one! Here’s the output for crs102node1:

root@crs102node1 ~]# /u01/crs/oracle/product/10.2.0/crs/bin/crsctl stop crs Stopping resources. Successfully stopped CRS resources Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@crs102node1 ~]# /u01/crs/oracle/product/10.2.0/crs/install/root102.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /u01/crs/oracle/product/10.2.0/crs Relinking some shared libraries. Relinking of patched files is complete. WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root WARNING: directory '/u01/crs/oracle/product' is not owned by root WARNING: directory '/u01/crs/oracle' is not owned by root WARNING: directory '/u01/crs' is not owned by root WARNING: directory '/u01' is not owned by root Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. 10205 patch successfully applied. clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully deleted 1 values from OCR. Successfully deleted 1 keys from OCR. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: crs102node1 crs102node1-priv crs102node1 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/u01/crs/oracle/product/10.2.0/crs/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /u01/crs/oracle/product/10.2.0/crs/install/paramfile.crs

This is the output from the second node

[root@crs102node2 ~]# /u01/crs/oracle/product/10.2.0/crs/install/root102.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /u01/crs/oracle/product/10.2.0/crs Relinking some shared libraries. Relinking of patched files is complete. WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root WARNING: directory '/u01/crs/oracle/product' is not owned by root WARNING: directory '/u01/crs/oracle' is not owned by root WARNING: directory '/u01/crs' is not owned by root WARNING: directory '/u01' is not owned by root Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . 10205 patch successfully applied. clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully deleted 1 values from OCR. Successfully deleted 1 keys from OCR. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 2: crs102node2 crs102node2-priv crs102node2 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/u01/crs/oracle/product/10.2.0/crs/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /u01/crs/oracle/product/10.2.0/crs/install/paramfile.crs [root@crs102node2 ~]#

Success!

Now does the cluster also know which version it is using?

[oracle@crs102node1 ~]$ /u01/crs/oracle/product/10.2.0/crs/bin/crsctl query crs activeversion CRS active version on the cluster is [10.2.0.5.0] [oracle@crs102node1 ~]$ fc -e vi /u01/crs/oracle/product/10.2.0/crs/bin/crsctl query crs softwareversion CRS software version on node [crs102node1] is [10.2.0.5.0]

Great, update succeeeded. Now over to do some more testing…