Search

Top 60 Oracle Blogs

Recent comments

February 2011

11.2.0.2 Bundled Patch 3 for Linux x86-64bit Take 2

Yesterday I wrote about the application of Bundle Patch 3 to my 2 node RAC cluster. Unfortunately I have run into problems when applying the patches for the GRID_HOME. I promised a fix to the situation, and here it is.

First of all I was unsure if I could apply the missing patches manually, but then decided against it. The opatch output for interim patches lists the patch together with a unique patch as shown here:

Interim patches (3) :

Patch  10626132     : applied on Wed Feb 02 16:08:43 GMT 2011
Unique Patch ID:  13350217
 Created on 31 Dec 2010, 00:18:12 hrs PST8PDT
 Bugs fixed:
 10626132

The formatting unfortunately is lost when pasting this here.

I was not sure if that patch/unique patch combination would appear if I patched manually so decided to not be brave and roll the bundle patch back altogether before applying it again.

Patch Rollback

This was actually very simple: I have opted to rollback all the applied patches from the GRID_HOME. The documentation states that you have to simply append the “-rollback” flag to the opatch command. I tried it on the node where the application of 2 patches failed:

[root@node1 stage]# opatch auto /u01/app/oracle/product/stage/10387939 -oh /u01/app/oragrid/product/11.2.0.2 -rollback
Executing /usr/bin/perl /u01/app/oragrid/product/11.2.0.2/OPatch/crs/patch112.pl -patchdir /u01/app/oracle/product/stage -patchn 10387939 -oh /u01/app/oragrid/product/11.2.0.2 -rollback -paramfile /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
opatch auto log file location is /u01/app/oragrid/product/11.2.0.2/OPatch/crs/../../cfgtoollogs/opatchauto2011-02-03_09-04-13.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/app/oracle/product/stage/ocm.rsp
Successfully unlock /u01/app/oragrid/product/11.2.0.2
patch 10387939  rollback successful for home /u01/app/oragrid/product/11.2.0.2
The patch  10157622 does not exist in /u01/app/oragrid/product/11.2.0.2
The patch  10626132 does not exist in /u01/app/oragrid/product/11.2.0.2
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.

So that was simple enough. Again – you won’t see anything in your opatch session and you might think that the command somehow stalled. I usually start a “screen” on my terminal and open a new window to tail the opatchauto file in $GRID_HOME/cfgtoollogs/.

Re-Applying the Patch

The next step is to re-apply the patch. The initial failure was a lack of disk space as it has been evident from the log file.

2011-02-02 15:57:45: The apply patch output is Invoking OPatch 11.2.0.1.4

 Oracle Interim Patch Installer version 11.2.0.1.4
 Copyright (c) 2010, Oracle Corporation.  All rights reserved.

 UTIL session

 Oracle Home       : /u01/app/oragrid/product/11.2.0.2
 Central Inventory : /u01/app/oracle/product/oraInventory
 from           : /etc/oraInst.loc
 OPatch version    : 11.2.0.1.4
 OUI version       : 11.2.0.2.0
 OUI location      : /u01/app/oragrid/product/11.2.0.2/oui
 Log file location : /u01/app/oragrid/product/11.2.0.2/cfgtoollogs/opatch/opatch2011-02-02_15-57-35PM.log

 Patch history file: /u01/app/oragrid/product/11.2.0.2/cfgtoollogs/opatch/opatch_history.txt

 Invoking utility "napply"
 Checking conflict among patches...
 Checking if Oracle Home has components required by patches...
 Checking conflicts against Oracle Home...
 OPatch continues with these patches:   10157622

 Do you want to proceed? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y

 Running prerequisite checks...
 Prerequisite check "CheckSystemSpace" failed.
 The details are:
 Required amount of space(2086171834) is not available.
 UtilSession failed: Prerequisite check "CheckSystemSpace" failed.

 OPatch failed with error code 73

2011-02-02 15:57:45: patch /u01/app/oracle/product/stage/10387939/10157622  apply  failed  for home  /u01/app/oragrid/product/11.2.0.2
2011-02-02 15:57:45: Performing Post patch actions

So this time around I ensured that I had enough free space (2.5G recommended minimum) available in my GRID_HOME. The procedure is the inverse to the rollback:

[root@node1 stage]# opatch auto /u01/app/oracle/product/stage/10387939 -oh /u01/app/oragrid/product/11.2.0.2
Executing /usr/bin/perl /u01/app/oragrid/product/11.2.0.2/OPatch/crs/patch112.pl -patchdir /u01/app/oracle/product/stage -patchn 10387939 -oh /u01/app/oragrid/product/11.2.0.2 -paramfile /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
opatch auto log file location is /u01/app/oragrid/product/11.2.0.2/OPatch/crs/../../cfgtoollogs/opatchauto2011-02-03_09-27-39.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/app/oracle/product/stage/ocm.rsp
Successfully unlock /u01/app/oragrid/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10387939  apply successful for home  /u01/app/oragrid/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10157622  apply successful for home  /u01/app/oragrid/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10626132  apply successful for home  /u01/app/oragrid/product/11.2.0.2
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@node1 stage]#

And this time it worked-all 3 patches were applied. However my free space diminished quite drastically from 2.5G to around 780M. And that was after purging lots of logs from $GRID_HOME/log/`hostname -s`/ . Nevertheless, this concludes the patch application.

Summary

In summary I am quite impressed with this patch. It looks as if it had been designed to be deployable by OEM and it’s silent, doesn’t require input (except for the ocm.rsp file) and is a rolling patch. However, the user has to manually check the installed patches vs the list of installable targets manually or through a script to ensure that all patches have been applied.You also have to ensure you have enough free space on your $GRID_HOME mount point.

As an enhancement request I’d like to request feedback from the opatch session that it started doing things-initially I hit CTRL-C thinking the command had stalled while it was actually busy in the background, shutting down my CRS stack. The “workaround” is to tail the logfile with the “-f” option.

Flush a Single SQL Statement in 10g – Take 3

Here’s a better script to flush a single SQL statement in 10g. Flushing is not technically correct. Actually what it does is invalidate the current children, forcing a parse the next time the statement is executed. Which is what we want. By the way, this should work in all versions of 10 (since it’s based on a side affect, there are no promises though). I called the script flush_sql10p.sql. Here’s a little demo of it’s use:

> sqlplus "/ as sysdba"
 
SQL*Plus: Release 10.1.0.3.0 - Production on Sat Feb 20 17:16:16 2010
 
Copyright (c) 1982, 2004, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 10g Enterprise Edition Release 10.1.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options
 
SYS@LAB101> @flush_pool
 
System altered.
 
SYS@LAB101> @avgskewi
 
AVG(PK_COL)
-----------
   15636133
 
SYS@LAB101> @fs
Enter value for sql_text: %skew%
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
84q0zxfzn5u6s      0 3330598925          1           .02          186 select avg(pk_col) from kso.skew where col1 = 136133
 
1 row selected.
 
SYS@LAB101> @avgskewi
 
AVG(PK_COL)
-----------
   15636133
 
1 row selected.
 
SYS@LAB101> @fs
Enter value for sql_text: %skew%
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
84q0zxfzn5u6s      0 3330598925          2           .01          111 select avg(pk_col) from kso.skew where col1 = 136133
 
1 row selected.
 
SYS@LAB101> @flush_sql10
Enter value for sql_id: 84q0zxfzn5u6s
 
sql_id: 84q0zxfzn5u6s flushed.
 
SYS@LAB101> @fs
Enter value for sql_text: 
Enter value for sql_id: 84q0zxfzn5u6s
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
84q0zxfzn5u6s      0          0          0           .00            0 select avg(pk_col) from kso.skew where col1 = 136133
 
1 row selected.
 
SYS@LAB101> l4
  4* sql_text
SYS@LAB101> del
SYS@LAB101> l3
  3* buffer_gets/decode(nvl(executions,0),0,1,executions) avg_lio,
SYS@LAB101> i
  4i last_load_time, invalidations
  5i /
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO LAST_LOAD_TIME      INVALIDATIONS
------------- ------ ---------- ---------- ------------- ------------ ------------------- -------------
84q0zxfzn5u6s      0          0          0           .00            0                                 1
 
1 row selected.
 
SYS@LAB101> !date   
Sat Feb 20 17:17:49 CST 2010
 
SYS@LAB101> @avgskewi
 
AVG(PK_COL)
-----------
   15636133
 
1 row selected.
 
SYS@LAB101> @fs
Enter value for sql_text: %skew%
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
84q0zxfzn5u6s      0 3330598925          1           .01           36 select avg(pk_col) from kso.skew where col1 = 136133
 
1 row selected.
 
SYS@LAB101> l4
  4* sql_text
SYS@LAB101> del
SYS@LAB101> l3
  3* buffer_gets/decode(nvl(executions,0),0,1,executions) avg_lio,
SYS@LAB101> i
  4i last_load_time, invalidations
  5i 
SYS@LAB101> /
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO LAST_LOAD_TIME      INVALIDATIONS
------------- ------ ---------- ---------- ------------- ------------ ------------------- -------------
84q0zxfzn5u6s      0 3330598925          1           .01           36 2010-02-20/17:17:52             1
 
1 row selected.
 
SYS@LAB101> @flush_sql10
Enter value for sql_id: 84q0zxfzn5u6s
 
sql_id: 84q0zxfzn5u6s flushed.
 
SYS@LAB101> @fs
Enter value for sql_text: %skew%
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
84q0zxfzn5u6s      0          0          0           .00            0 select avg(pk_col) from kso.skew where col1 = 136133
 
1 row selected.
 
SYS@LAB101> @avgskewi
 
AVG(PK_COL)
-----------
   15636133
 
1 row selected.
 
SYS@LAB101> @avgskewi
 
AVG(PK_COL)
-----------
   15636133
 
1 row selected.
 
SYS@LAB101> @fs
Enter value for sql_text: %skew%
 
Enter value for sql_id: 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
84q0zxfzn5u6s      0 3330598925          2           .00           36 select avg(pk_col) from kso.skew where col1 = 136133
 
1 row selected.

11.2.0.2 Bundled Patch 3 for Linux x86-64bit

It has become a habit recently to issue explicit warnings when presenting information that-if it doesn’t work as described-might cause you harm. So to follow suit, here’s mine:

Warning! This is a Patch Set Update installation war story-it worked for me. That does by no means imply that the same works for you! So always apply the PSU or bundle patch on a non-critical test system first. And always take a backup of your ORACLE_HOME and inventory first before applying any patches-you have been warned. I like to be rather safe than sorry.

A good one I think …  First of all thanks to Coskan for the pointer to the Oracle blog entry: http://blogs.oracle.com/UPGRADE/2011/01/11202_bundled_patch_3_for_linu.html which actually prompted me to apply the patch in the first place.

My environment is a 2 node 11.2.0.2 cluster on OEL 5.5 64bit. Judging by the readme file I need the latest OPatch, 11.2.0.1.4, (patch p6880880)  and the bundle patch itself (patch 10387939). Unfortunately Oracle leaves it as an exercise to the reader to make sense of the patch readme and note 1274453.1 which is applicable if ACFS is involved on the cluster. Although I am using ACFS all my ACFS volumes were disabled before patching (i.e. they are not shared ORACLE_HOMEs). If you have ACFS in use then follow note 1274453.1 and shut all of them down before proceeding. Note 1274453.1 is divided into a number of sections. First it’s important to find the right case-I wanted to try the opatch auto feature this time, but patch GRID_HOME independently of the RDBMS_HOME. I knew from other users that opatch auto has gained a bad reputation in the past but was curious whether or not Oracle fixed it. Regardless of your configuration, you have to stop dbconsole instances on all nodes if there are any.

If applicable, the next step is to stop all database instances and their corresponding ACFS volumes. The ACFS mounts also have to be dismounted. If you are unsure you can use /sbin/acfsutil registry to list the ACFS volumes. Those registered with Clusterware show up in the output of crsctl status resource -t.

Important! If not already done so, update OPatch on all nodes for GI and RDBMS home. It’s a lot of manual work, I wish there was a cluster wide OPatch location…it would make my life so much easier. Just patching OPatch on 4 nodes is a pain, I don’t want to imagine a 8 or more nodes cluster right now.

Now proceed with the patch installation. I’ll only let it patch the GRID_HOME at this stage. The patch should be rolling which is good. Note that you’ll be root, which can cause its own problems in larger organisations. My patch is staged in /u01/app/oracle/product/stage, and the patch readme suggests I should use the opatch executable in the $GRID_HOME.

[root@node1 ~]# cd /u01/app/oracle/product/stage
[root@node1 stage]# cd 10387939
[root@node1 10387939]# export PATH=/u01/app/oragrid/product/11.2.0.2/OPatch:$PATH

Oracle Configuration Manager response file

Unlike with previous patchsets which prompted the user each time you ran opatch, this time the OCM configuration is not part of the patch set installation. Instead, you have to create an OCM configuration “response” file before you apply the patch. The file is created in the current directory. As the GRID_OWNER, execute $ORACLE_HOME/OPatch/ocm/bin/emocmrsp as shown here:

[root@node1 stage]# su - oragrid
[oragrid@node1 ~]$ . oraenv
ORACLE_SID = [oragrid] ? grid
The Oracle base for ORACLE_HOME=/u01/app/oragrid/product/11.2.0.2 is /u01/app/oragrid/product/admin
[oragrid@node1 ~]$ /u01/app/oragrid/product/11.2.0.2/OPatch/ocm/bin/emocmrsp
OCM Installation Response Generator 10.3.1.2.0 - Production
Copyright (c) 2005, 2009, Oracle and/or its affiliates.  All rights reserved.

Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y
The OCM configuration response file (ocm.rsp) was successfully created.
[oragrid@node1 ~]$ pwd
/home/oragrid

I don’t want automatic email notifications in this case so I left the email address blank. I also prefer to be uninformed of future patch sets. I then moved the file into /u01/app/oracle/product/stage/ocm.rsp as root for the next steps. It is good practise to save the detailed oracle inventory information for later. Again as the grid owner I ran this command:

[root@node1 stage]# su - oragrid
[oragrid@node1 ~]$ . oraenv
ORACLE_SID = [oragrid] ? grid
The Oracle base for ORACLE_HOME=/u01/app/oragrid/product/11.2.0.2 is /u01/app/oragrid/product/admin
[oragrid@node1 ~]$ /u01/app/oragrid/product/11.2.0.2/OPatch/opatch lsinv -detail > /tmp/gridhomedetail

What can (will) be patched?

The bundle.xml file lists the components that can be patched with a bundle patch. In my case, the bundle is applicable for these:


 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

So in other words, the bundle exists of 3 patches – 10387939, 10157622 and 10626132. For each of these a target type is shown. SIHA is short for Single Instance High Availability, which has been renamed “Oracle Restart”. I assume SIDB is single instance DB, but don’t know for sure.

Patching the GRID_HOME

I’m now ready to apply the patch to the GRID home. The command to do so is simple, the below example worked for me:

[root@node1 stage]# opatch auto /u01/app/oracle/product/stage/10387939 -oh /u01/app/oragrid/product/11.2.0.2
Executing /usr/bin/perl /u01/app/oragrid/product/11.2.0.2/OPatch/crs/patch112.pl -patchdir /u01/app/oracle/product/stage -patchn 10387939 -oh

/u01/app/oragrid/product/11.2.0.2 -paramfile /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
opatch auto log file location is /u01/app/oragrid/product/11.2.0.2/OPatch/crs/../../cfgtoollogs/opatchauto2011-02-02_15-50-04.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:

Before entering the path I opened another terminal to look at the log file, which is in $ORACLE_HOME/cfgtoollogs/opatchauto plus a timestamp. After entering the full path to my ocm.rsp file nothing happened in my opatch session-Oracle could have told us that it was about to apply the patch. That is because it actually is doing that. Go back to your other terminal and see the log messages fly by! The opatch auto command automatically does everything that the DBA had to do previously, including unlocking the CRS stack and calling opatch napply for the relevant bits and pieces. This is indeed a nice step forward. I can vividly remember having to apply portions of a PSU to the GRID_HOME and others to the RDBMS home, 6 or 7 steps in total before a patch was applied. That was indeed hard work.

Only after CRS has been shut down (after quite a while after entering the path to the ocm.rsp file!) will you be shown this line:

Successfully unlock /u01/app/oragrid/product/11.2.0.2

Even further down the line you see those, after the patches have been applied:

patch /u01/app/oracle/product/stage/10387939/10387939  apply successful for home  /u01/app/oragrid/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10157622  apply  failed  for home  /u01/app/oragrid/product/11.2.0.2
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.

And with that I got my prompt back. One patch failed to apply-the log file indicated a lack of space (2G are required to be free).Tomorrow I’ll post an update and remove/reapply the patch manually.

I checked the other node and found that the patch has indeed been applied on the local node only. I hasn’t propagated across which is good as the readme wasn’t really clear if the patch was rolling. From what I’ve seen I’d call it a local patch, similar to the “opatch napply -local” we did manually before the opatch auto option has been introduced. And even better, it worked. Querying opatch lsinventory I got this result:

Lsinventory Output file location : /u01/app/oragrid/product/11.2.0.2/cfgtoollogs/opatch/lsinv/lsinventory2011-02-02_16-01-27PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Grid Infrastructure                                           11.2.0.2.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :

Patch  10387939     : applied on Wed Feb 02 15:56:50 GMT 2011
Unique Patch ID:  13350217
 Created on 30 Dec 2010, 22:55:01 hrs PST8PDT
 Bugs fixed:
 10158965, 9940990, 10190642, 10031806, 10228635, 10018789, 9744252
 10010252, 9956713, 10204358, 9715581, 9770451, 10094635, 10121589
 10170431, 9824198, 10071193, 10145612, 10035737, 9845644, 10086980
 10052141, 10039731, 10035521, 10219576, 10184634, 10207092, 10138589
 10209232, 8752691, 9965655, 9819413, 9500046, 10106828, 10220118, 9881076
 9869287, 10040531, 10122077, 10218814, 10261389, 10033603, 9788588
 9735237, 10126219, 10043801, 10073205, 10205715, 9709292, 10105926
 10079168, 10098253, 10005127, 10013431, 10228151, 10092153, 10142909
 10238786, 10260808, 10033071, 9791810, 10052956, 9309735, 10026972
 10080579, 10073683, 10004943, 10019218, 9539440, 10022980, 10061490
 10006008, 6523037, 9724970, 10142776, 10208386, 10113803, 10261680
 9671271, 10084145, 10051966, 10355493, 10227133, 10229719, 10046912
 10228393, 10353054, 10142788, 10221016, 9414040, 10127360, 10310299
 10094201, 9591812, 10129643, 10332589, 10026193, 10195991, 10260870
 10248523, 9951423, 10261072, 10299224, 10230571, 10222719, 10233732
 10113633, 10102506, 10094949, 10077191, 10329146, 8685446, 10048701
 10314582, 10149223, 10245259, 10151017, 9924349, 10245086, 11074393

Rac system comprising of multiple nodes
 Local node = node1
 Remote node = node2

--------------------------------------------------------------------------------

OPatch succeeded.

Patching the RDBMS home

Now with that I’ll try applying it to the RDBMS home on the same node:

[root@node1 stage]# opatch auto /u01/app/oracle/product/stage/10387939 -oh /u01/app/oracle/product/11.2.0.2
Executing /usr/bin/perl /u01/app/oragrid/product/11.2.0.2/OPatch/crs/patch112.pl -patchdir /u01/app/oracle/product/stage -patchn 10387939 -oh

/u01/app/oracle/product/11.2.0.2 -paramfile /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
opatch auto log file location is /u01/app/oragrid/product/11.2.0.2/OPatch/crs/../../cfgtoollogs/opatchauto2011-02-02_16-05-29.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/app/oracle/product/stage/ocm.rsp
patch /u01/app/oracle/product/stage/10387939/10387939  apply successful for home  /u01/app/oracle/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10157622/custom/server/10157622  apply successful for home  /u01/app/oracle/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10626132  apply successful for home  /u01/app/oracle/product/11.2.0.2
[root@node1 stage]#

Cool! That was indeed easy. Did it work?

$ORACLE_HOME/OPatch/opatch lsinv
[...]
--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.2.0
There are 1 products installed in this Oracle Home.

Interim patches (3) :

Patch  10626132     : applied on Wed Feb 02 16:08:43 GMT 2011
Unique Patch ID:  13350217
 Created on 31 Dec 2010, 00:18:12 hrs PST8PDT
 Bugs fixed:
 10626132

Patch  10157622     : applied on Wed Feb 02 16:08:27 GMT 2011
Unique Patch ID:  13350217
 Created on 19 Nov 2010, 01:41:19 hrs PST8PDT
 Bugs fixed:
 9979706, 9959110, 10016083, 10015460, 10014392, 9918485, 10157622
 10089120, 10057296, 9971646, 10053985, 10040647, 9978765, 9864003
 10069541, 10110969, 10107380, 9915329, 10044622, 10029119, 9812970
 10083009, 9812956, 10048027, 10036193, 10008467, 10040109, 10015210
 10083789, 10033106, 10073372, 9876201, 10042143, 9963327, 9679401
 10062301, 10018215, 10075643, 10007185, 10071992, 10057680, 10038791
 10124517, 10048487, 10078086, 9926027, 10052721, 9944948, 10028235
 10146768, 10011084, 10027079, 10028343, 10045436, 9907089, 10073075
 10175855, 10178670, 10072474, 10036834, 9975837, 10028637, 10029900, 9949676

Patch  10157622     : applied on Wed Feb 02 16:08:27 GMT 2011
Unique Patch ID:  13350217
 Created on 19 Nov 2010, 01:41:19 hrs PST8PDT
 Bugs fixed:
 9979706, 9959110, 10016083, 10015460, 10014392, 9918485, 10157622
 10089120, 10057296, 9971646, 10053985, 10040647, 9978765, 9864003
 10069541, 10110969, 10107380, 9915329, 10044622, 10029119, 9812970
 10083009, 9812956, 10048027, 10036193, 10008467, 10040109, 10015210
 10083789, 10033106, 10073372, 9876201, 10042143, 9963327, 9679401
 10062301, 10018215, 10075643, 10007185, 10071992, 10057680, 10038791
 10124517, 10048487, 10078086, 9926027, 10052721, 9944948, 10028235
 10146768, 10011084, 10027079, 10028343, 10045436, 9907089, 10073075
 10175855, 10178670, 10072474, 10036834, 9975837, 10028637, 10029900, 9949676

Patch  10387939     : applied on Wed Feb 02 16:07:18 GMT 2011
Unique Patch ID:  13350217
 Created on 30 Dec 2010, 22:55:01 hrs PST8PDT
 Bugs fixed:
 10158965, 9940990, 10190642, 10031806, 10228635, 10018789, 9744252
 10010252, 9956713, 10204358, 9715581, 9770451, 10094635, 10121589
 10170431, 9824198, 10071193, 10145612, 10035737, 9845644, 10086980
 10052141, 10039731, 10035521, 10219576, 10184634, 10207092, 10138589
 10209232, 8752691, 9965655, 9819413, 9500046, 10106828, 10220118, 9881076
 9869287, 10040531, 10122077, 10218814, 10261389, 10033603, 9788588
 9735237, 10126219, 10043801, 10073205, 10205715, 9709292, 10105926
 10079168, 10098253, 10005127, 10013431, 10228151, 10092153, 10142909
 10238786, 10260808, 10033071, 9791810, 10052956, 9309735, 10026972
 10080579, 10073683, 10004943, 10019218, 9539440, 10022980, 10061490
 10006008, 6523037, 9724970, 10142776, 10208386, 10113803, 10261680
 9671271, 10084145, 10051966, 10355493, 10227133, 10229719, 10046912
 10228393, 10353054, 10142788, 10221016, 9414040, 10127360, 10310299
 10094201, 9591812, 10129643, 10332589, 10026193, 10195991, 10260870
 10248523, 9951423, 10261072, 10299224, 10230571, 10222719, 10233732
 10113633, 10102506, 10094949, 10077191, 10329146, 8685446, 10048701
 10314582, 10149223, 10245259, 10151017, 9924349, 10245086, 11074393

Rac system comprising of multiple nodes
 Local node = node1
 Remote node = node2

--------------------------------------------------------------------------------

OPatch succeeded.

It did indeed. As a nice side effect I didn’t even have to worry about the srvctl start/stop home commands, they were automatically done for me. From the log:

2011-02-02 16:08:45: /u01/app/oracle/product/11.2.0.2/bin/srvctl start home -o /u01/app/oracle/product/11.2.0.2 -s

/u01/app/oracle/product/11.2.0.2/srvm/admin/stophome.txt -n node1 output is
2011-02-02 16:08:45: Started resources from datbase home /u01/app/oracle/product/11.2.0.2

Now I’ll simply repeat this on the remaining nodes and should be done. To test the stability of the process I didn’t limit the opatch command to a specific home but had it patch them all.

Patching all homes in one go

This takes a little longer but otherwise is just the same. I have added the output here for the sake of completeness:

[root@node2 stage]# which opatch
/u01/app/oragrid/product/11.2.0.2/OPatch/opatch
[root@node2 stage]# opatch auto /u01/app/oracle/product/stage/10387939
Executing /usr/bin/perl /u01/app/oragrid/product/11.2.0.2/OPatch/crs/patch112.pl -patchdir /u01/app/oracle/product/stage -patchn 10387939 -paramfile /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
opatch auto log file location is /u01/app/oragrid/product/11.2.0.2/OPatch/crs/../../cfgtoollogs/opatchauto2011-02-02_16-36-09.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/oragrid/product/11.2.0.2/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/app/oracle/product/stage/ocm.rsp
patch /u01/app/oracle/product/stage/10387939/10387939  apply successful for home  /u01/app/oracle/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10157622/custom/server/10157622  apply successful for home  /u01/app/oracle/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10626132  apply successful for home  /u01/app/oracle/product/11.2.0.2
Successfully unlock /u01/app/oragrid/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10387939  apply successful for home  /u01/app/oragrid/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10157622  apply successful for home  /u01/app/oragrid/product/11.2.0.2
patch /u01/app/oracle/product/stage/10387939/10626132  apply successful for home  /u01/app/oragrid/product/11.2.0.2
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@node2 stage]#

Well that’s for the technical part. When I compared the number of patches applied in the $GRID_HOME on node 2 (which simply used “opatch auto”) then I found 3 interim patches applied, as compared to only 1 on node 1 when I only patched the GRID_HOME. I’ll have to raise this with Oracle…

[oragrid@node2 ~]$ /u01/app/oragrid/product/11.2.0.2/OPatch/opatch lsinv
Invoking OPatch 11.2.0.1.4

Oracle Interim Patch Installer version 11.2.0.1.4
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oragrid/product/11.2.0.2
Central Inventory : /u01/app/oracle/product/oraInventory
 from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.4
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/oragrid/product/11.2.0.2/oui
Log file location : /u01/app/oragrid/product/11.2.0.2/cfgtoollogs/opatch/opatch2011-02-02_17-13-40PM.log

Patch history file: /u01/app/oragrid/product/11.2.0.2/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oragrid/product/11.2.0.2/cfgtoollogs/opatch/lsinv/lsinventory2011-02-02_17-13-40PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Grid Infrastructure                                           11.2.0.2.0
There are 1 products installed in this Oracle Home.

Interim patches (3) :

Patch  10626132     : applied on Wed Feb 02 16:48:59 GMT 2011
Unique Patch ID:  13350217
 Created on 31 Dec 2010, 00:18:12 hrs PST8PDT
 Bugs fixed:
 10626132

Patch  10157622     : applied on Wed Feb 02 16:48:34 GMT 2011
Unique Patch ID:  13350217
 Created on 19 Nov 2010, 01:41:33 hrs PST8PDT
 Bugs fixed:
 9979706, 9959110, 10016083, 10015460, 10014392, 9918485, 10157622
 10089120, 10057296, 9971646, 10053985, 10040647, 9978765, 9864003
 10069541, 10110969, 10107380, 9915329, 10044622, 10029119, 9812970
 10083009, 9812956, 10048027, 10036193, 10008467, 10040109, 10015210
 10083789, 10033106, 10073372, 9876201, 10042143, 9963327, 9679401
 10062301, 10018215, 10075643, 10007185, 10071992, 10057680, 10038791
 10124517, 10048487, 10078086, 9926027, 10052721, 9944948, 10028235
 10146768, 10011084, 10027079, 10028343, 10045436, 9907089, 10073075
 10175855, 10072474, 10036834, 9975837, 10028637, 10029900, 9949676
 9974223, 10260251

Patch  10387939     : applied on Wed Feb 02 16:46:29 GMT 2011
Unique Patch ID:  13350217
 Created on 30 Dec 2010, 22:55:01 hrs PST8PDT
 Bugs fixed:
 10158965, 9940990, 10190642, 10031806, 10228635, 10018789, 9744252
 10010252, 9956713, 10204358, 9715581, 9770451, 10094635, 10121589
 10170431, 9824198, 10071193, 10145612, 10035737, 9845644, 10086980
 10052141, 10039731, 10035521, 10219576, 10184634, 10207092, 10138589
 10209232, 8752691, 9965655, 9819413, 9500046, 10106828, 10220118, 9881076
 9869287, 10040531, 10122077, 10218814, 10261389, 10033603, 9788588
 9735237, 10126219, 10043801, 10073205, 10205715, 9709292, 10105926
 10079168, 10098253, 10005127, 10013431, 10228151, 10092153, 10142909
 10238786, 10260808, 10033071, 9791810, 10052956, 9309735, 10026972
 10080579, 10073683, 10004943, 10019218, 9539440, 10022980, 10061490
 10006008, 6523037, 9724970, 10142776, 10208386, 10113803, 10261680
 9671271, 10084145, 10051966, 10355493, 10227133, 10229719, 10046912
 10228393, 10353054, 10142788, 10221016, 9414040, 10127360, 10310299
 10094201, 9591812, 10129643, 10332589, 10026193, 10195991, 10260870
 10248523, 9951423, 10261072, 10299224, 10230571, 10222719, 10233732
 10113633, 10102506, 10094949, 10077191, 10329146, 8685446, 10048701
 10314582, 10149223, 10245259, 10151017, 9924349, 10245086, 11074393

Rac system comprising of multiple nodes
 Local node = node2
 Remote node = node1

--------------------------------------------------------------------------------

OPatch succeeded.

Congrats to Fahd Mirza on becoming an Oracle ACE

Last week brought great news to Pythian — one of our DBAs in Pakistan, Fahd Mirza, has become an Oracle ACE. Fahd joined Pythian in September 2010 as the very first Pythian employee in Pakistan and thanks to his skills and ambitions ended up on the team supporting Exadata environments. Fahd is a long standing active community member, frequent blogger and passionate Oracle technologist evangelizing for Oracle technology in Pakistan. No wonder he got nominated as an Oracle ACE and was accepted.

I should also mention that another Oracle ACE DBA joined us recently Jared Still. Jared is a well respected member of Oracle community, member of the OakTable Network and a veteran of the Oracle-L mailing list. Jared is a top notch Oracle DBA and huge fan of the most popular programming language at Pythian — Perl — and event wrote a book “Perl for Oracle DBAs“.

With all this, we have 5 Oracle ACEs & ACE Directors at Pythian now including Gwen Shapira, Christo Kutrovsky and myself. But that’s not all, Pythian is known as an incubator for Oracle ACEs (I think we were called the Oracle ACE Factory in one of the Oracle ACE newsletters) and it’s been a pleasure to have worked side by side with other Oracle ACEs and ACE Directors — Riyaj Shamsudeen, Doug Burns, Sheeri Cabral and Dmitri Volkov. Some of them became Oracle ACEs at Pythian, some before or after that and even though they are not working at Pythian now, they are still our good friends and help us out on many occasions with training or collaborating on exciting projects.

It’s a great initiative by Oracle through the Oracle ACE program to recognize active community contributors and passionate Oracle professionals around the globe! Well done Oracle!

Bug

There’s a note on the OTN database forum that’s worth reading if you’re puzzled by some of the Statspack or AWR results in 11g. The author is talking about 11.2.0.2 and walks through a couple of inconsistent reports from Statspack – but this prompted me to take a look at some underlying details using version 11.1.0.6 – and I think there’s a bug (or change) in the way that enqueues (locks) are handled that can make both statspack and AWR reports “lose time”.

Essentially the following sequence of events is possible:

    Session A starts waiting on a TM lock
    after a few seconds v$system_event records a wait for “enq: TM – contention”.
    AWR (or statspack) takes a snapshot (which includes v$system_event).
    Some time later session A acquires the TM lock – the total_waits for this event does not get incremented in v$system_event.
    AWR (or statspack) takes the next snapshot

When you run the AWR/Statspack report across the interval, the “Top N Timed Events” and the other sections on wait events will show time waited only if the total_waits for the event has changed in the interval. But we incremented the count before the first snapshot, and haven’t incremented it since – so the time we spent waiting in this interval disappears. (The time we spent waiting in the previous interval will be in the report for the previous interval, of course.)

Note – this is not the same as the inconsistencies that used to appear in statspack with statement-level errors, where the total resource usage across multiple periods would be reported against the snapshot where the statement completed. In those cases we finally saw the total resource usage even though it was summed against one interval; in this case we lose track of any time spent waiting after the first snapshot.

Of course the problem is not a total whitewash. The time may appear in other parts of the report – for example it will show up in the “DB Time” total – and the results will only be seriously deceptive for waits that are (a) long and (b) are so unluckily synchronised that an interval with a lot of wait time in it doesn’t have a wait event start in the interval and increment the total_waits. The problem may, therefore, only appear as a slight inconsistency in most systems.

Footnote: I haven’t spent much time looking at this, so I don’t know which versions are affected, nor which enqueues the new mechanism applies to (it may only be “TM” enqueue, as other waits tend to time out and restart every few seconds). This note is just an early warning to help you recognise the situation if you ever come across it.

Pr0n Attack…

It seems my forum has become subject to an onslaught of pr0n spam. Sorry if anyone has got an eyeful. I’m trying to delete it as soon as it arrives, but I can’t be on the forum every minute of the day.

If anyone has any tips regarding anti-spam on phpBB I would be happy to hear them. I’m currently using reCAPTCHA, but am looking at other mods to see what else I can do to prevent it.

Cheers

Tim…

Patch 10270073 – 11.1.0.1.2 Patch Set Update for Oracle Management Service

Today is patch day – on my current site we are having quite a few problems with our management server (OMS from now on). It occasionally simply “hangs” and doesn’t respond when connecting to the SSH port. A few minutes later it’s back to normal-but this behaviour is not reproducible.

So in an effort to please Oracle support who couldn’t find a reason for this I decided to apply Patch Set Update 2 to the OMS to get it to 11.1.0.1.2. I would also like to filter the corresponding agent patch for our 11.1 agents through as well. The PSU has been released 2 weeks ago so it’s reasonably fresh.The patches for OMS and agent are generic, which is nice as it implies they are available for every platform. Our OMS had had problems before, and one-off patches have been applied. So the first step as always with PSUs is to check if there are conflicts. The readme file has the required instructions. I unzipped p10270073_111010_Generic.zip in /tmp/ and then executed the prerequisite checker in /tmp as shown in this example:

[oms]oracle@oms $ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./10270073
Invoking OPatch 11.1.0.8.0

Oracle Interim Patch Installer version 11.1.0.8.0
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/oracle/product/middleware/oms11g
Central Inventory : /u01/app/oracle/product/oraInventory
 from           : /var/opt/oracle/oraInst.loc
OPatch version    : 11.1.0.8.0
OUI version       : 11.1.0.8.0
OUI location      : /u01/app/oracle/product/middleware/oms11g/oui
Log file location : /u01/app/oracle/product/middleware/oms11g/cfgtoollogs/opatch/opatch2011-02-01_08-48-14AM.log

Patch history file: /u01/app/oracle/product/middleware/oms11g/cfgtoollogs/opatch/opatch_history.txt

OPatch detects the Middleware Home as "/u01/app/oracle/product/middleware"

Invoking prereq "checkconflictagainstohwithdetail"

ZOP-40: The patch(es) has conflicts/supersets with other patches installed in the Oracle Home (or) among themselves.

Prereq "checkConflictAgainstOHWithDetail" failed.

Summary of Conflict Analysis:

Patches that can be applied now without any conflicts are :
10270073

Following patches are not required, as they are subset of the patches in Oracle Home or subset of the patches in the given list :
9563902, 9537948, 9489355

Following patches will be rolled back from Oracle Home on application of the patches in the given list :
9563902, 9537948, 9489355

Conflicts/Supersets for each patch are:

Patch : 10270073

 Bug Superset of 9563902
 Super set bugs are:
 9491872,  9476313,  9544428

 Bug Superset of 9537948
 Super set bugs are:
 9537948

 Bug Superset of 9489355
 Super set bugs are:
 9489355

OPatch succeeded.
[oms]oracle@oms $

OK, so a few one-offs will be rolled back. Let’s get started. First of all we have to stop the OMS as shown here:

[oms]oracle@oms $ emctl stop oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

The next step is to apply the patch. Here’s the sample session:

[oms]oracle@oms $ cd 10270073

[oms]oracle@oms $ $ORACLE_HOME/OPatch/opatch apply
Invoking OPatch 11.1.0.8.0

Oracle Interim Patch Installer version 11.1.0.8.0
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/middleware/oms11g
Central Inventory : /u01/app/oracle/product/oraInventory
 from           : /var/opt/oracle/oraInst.loc
OPatch version    : 11.1.0.8.0
OUI version       : 11.1.0.8.0
OUI location      : /u01/app/oracle/product/middleware/oms11g/oui
Log file location : /u01/app/oracle/product/middleware/oms11g/cfgtoollogs/opatch/opatch2011-02-01_09-00-00AM.log

Patch history file: /u01/app/oracle/product/middleware/oms11g/cfgtoollogs/opatch/opatch_history.txt

OPatch detects the Middleware Home as "/u01/app/oracle/product/middleware"

ApplySession applying interim patch '10270073' to OH '/u01/app/oracle/product/middleware/oms11g'
Interim patch 10270073 is a superset of the patch(es) [  9563902 9537948 9489355 ] in the Oracle Home
OPatch will rollback the subset patches and apply the given patch.
Execution of 'sh /tmp/10270073/custom/scripts/init -apply 10270073 ':

Return Code = 0

Running prerequisite checks...

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '10270073' for restore. This might take a while...
Backing up files affected by the patch '9563902' for restore. This might take a while...
Backing up files affected by the patch '9537948' for restore. This might take a while...
Backing up files affected by the patch '9489355' for restore. This might take a while...
ApplySession rolling back interim patch '9563902' from OH '/u01/app/oracle/product/middleware/oms11g'

Patching component oracle.sysman.oms.core, 11.1.0.1.0...
Copying file to "/u01/app/oracle/product/middleware/oms11g/sysman/omsca/scripts/wls/create_domain.py"
RollbackSession removing interim patch '9563902' from inventory
ApplySession rolling back interim patch '9537948' from OH '/u01/app/oracle/product/middleware/oms11g'

Patching component oracle.sysman.oms.core, 11.1.0.1.0...
Updating jar file "/u01/app/oracle/product/middleware/oms11g/sysman/jlib/emInstall.jar" with "/u01/app/oracle/product/middleware/oms11g/.patch_storage/9537948_Apr_12_2010_03_37_52/files//sysman/jlib/emInstall.jar/oracle/sysman/configassistant/addon/AddOnConfigAssistantDriver.class"
Updating jar file "/u01/app/oracle/product/middleware/oms11g/sysman/jlib/emInstall.jar" with "/u01/app/oracle/product/middleware/oms11g/.patch_storage/9537948_Apr_12_2010_03_37_52/files//sysman/jlib/emInstall.jar/oracle/sysman/configassistant/addon/AddOnConfigAssistantDriver$1.class"
Updating jar file "/u01/app/oracle/product/middleware/oms11g/sysman/jlib/emInstall.jar" with "/u01/app/oracle/product/middleware/oms11g/.patch_storage/9537948_Apr_12_2010_03_37_52/files//sysman/jlib/emInstall.jar/oracle/sysman/configassistant/addon/AddOnConfigAssistantDriver$AddOnFileFilter.class"
RollbackSession removing interim patch '9537948' from inventory
ApplySession rolling back interim patch '9489355' from OH '/u01/app/oracle/product/middleware/oms11g'

Patching component oracle.sysman.oms.core, 11.1.0.1.0...
Copying file to "/u01/app/oracle/product/middleware/oms11g/bin/HAConfigCmds.pm"
RollbackSession removing interim patch '9489355' from inventory

OPatch back to application of the patch '10270073' after auto-rollback.

Backing up files affected by the patch '10270073' for rollback. This might take a while...

Patching component oracle.sysman.oms.core, 11.1.0.1.0...
Updating jar file "/u01/app/oracle/product/middleware/oms11g/sysman/jlib/emCORE.jar" with "/sysman/jlib/emCORE.jar/oracle/sysman/eml/ecm/policy/PolicyViolationsController.class"
[...]
Copying file to "/u01/app/oracle/product/middleware/oms11g/bin/SecureOMSCmds.pm"
Copying file to "/u01/app/oracle/product/middleware/oms11g/sysman/emdrep/scripts/SecureAgent_oms.pl"
ApplySession adding interim patch '10270073' to inventory

Verifying the update...
Inventory check OK: Patch ID 10270073 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 10270073 are present in Oracle Home.
Execution of 'sh /tmp/10270073/custom/scripts/post -apply 10270073 ':

Return Code = 0
--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1) OUI-67620:Interim patch 10270073 is a superset of the patch(es) [  9563902 9537948 9489355 ] in the Oracle Home
--------------------------------------------------------------------------------
OPatch Session completed with warnings.

OPatch completed with warnings.

This process took 1 hour on my SPARC zone-not too impressed actually. But I’m blaming it on the overloaded box instead of the patch process.The outcome of the patching looks ok to me-I already knew I applied PSU 2 as a superset of 3 other patches. The next step is to apply the post install script. This is done by calling a JDBC application as shown here:

[oms]oracle@oms $ $ORACLE_HOME/bin/rcuJDBCEngine sys/secretpassword@repositoryHost:1821:REPOSDB JDBC_SCRIPT post_install_script.sql
Extracting Statement from File Name: 'post_install_script.sql' Line Number: 1
Extracted SQL Statement: [alter session set current_schema=SYSMAN]
 Statement Type: 'DDL Statement'
Executing SQL statement: alter session set current_schema=SYSMAN
Extracting Statement from File Name: 'post_install_script.sql' Line Number: 1
Extracting Statement from File Name: 'post_install_script.sql' Line Number: 1
Extracted SQL Statement: [SET serveroutput on size 1000000]
Skipping Unsupported Statement
 Statement Type: 'Oracle RCU NotSupported SQLPlus Statement'
Extracting Statement from File Name: 'post_install_script.sql' Line Number: 2
Extracted SQL Statement: [BEGIN
 EXECUTE IMMEDIATE 'drop table bundle_component_files';
 EXCEPTION WHEN OTHERS THEN
 IF sqlcode = -942 THEN
 NULL;
 ELSE
 RAISE;
 END IF;
END;
]
....
 END;
 END IF;
 END IF;
 RAISE;
END;
]
 Statement Type: 'BEGIN/END Anonymous Block'
Completed SQL script execution normally.
1 scripts were processed
[oms]oracle@oms $

Now finally it’s time to start the OMS and cross our fingers to see if it worked:

[oms]oracle@oms $ emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Successfully Started
Oracle Management Server is Up
[oms]oracle@oms $

I’d call this success-I managed to log on to the OMS and performed some basic testing which implied the patch was successfully applied.