Patch and Provision in EM12c: #6 Patch 12.1.0.1.0 GI + RDBMS Home with the April 2014 CPU

While setting up Database as a Service, I came across this particular issue on DB12c while creating a Pluggable database. Now, I am not certain whether this has anything whatsoever to do with the fact that it is on a Direct NFS (dNFS), but the creation fails with the error message below.

[oracle@dc01db2a ~]$ tail -100f /u01/app/oracle/cfgtoollogs/dbca/raccdba/pdb1/plugDatabase.log
database_running
/u01/app/oracle/diag/rdbms/raccdba/raccdba1/trace
alter pluggable database pdb1 open instances=all
*
ERROR at line 1:
ORA-12805: parallel query server died unexpectedly

Upon further investigation of the database alert log, I found this error message below:

Sun May 11 20:08:13 2014
Errors in file /u01/app/oracle/diag/rdbms/raccdba/raccdba1/trace/raccdba1_ora_32406.trc (incident=60690):
ORA-00600: internal error code, arguments: [OCIKDBLinkConn-8], [24327], [ORA-24327: need explicit attach before authenticating a user
], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/raccdba/raccdba1/incident/incdir_60690/raccdba1_ora_32406_i60690.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

A quick search on MOS yields that this is a known bug in the 12.1.0.1.0 release and the fix is the application of a patch.

Both GI and RDBMS installations are at the Base 12.1.0.1.0 release, and I figured it was a good time to patch, i.e. April 2014 Critical Patch Update. Now, I’m sure there are plenty of posts out there on how to apply it, but I wanted to document the process via Enterprise Manager, and thought it would make a good post to share with you.

With that being said, let’s begin.

Download Latest OPatch into the Software Library:

An important pre-requisite to patching is that the latest OPatch for the specific platform to patch is uploaded into the Software Library.

1. Make your way to the “Patches & Updates” home page.

NewImage

2. Enter the patch number of OPatch, i.e. 6880880 and search.

NewImage

3. Click on the relevant release/platform and click on download to Software Library.

NewImage

4. Under the “Saved Patches” section, you can view the now download and ready OPatch update.

NewImage

Patch OMS

Please note, this is only applicable if you are on OMS 12.1.0.3.0 you will need to apply the patch 17846072 or else you will get an error message (mentioned in the MOS Note Name below) if you patch multiple DB12c Oracle Homes.

EM 12c: Patching multiple Software Only Oracle Homes from Enterprise manager 12.1.0.3 Cloud Control Fails with Message: Error evaluating expression: ${doTargetChecks(8)} : Unexpected Error Occured! (Doc ID 1608524.1)

Patching!

1. Make your way to the Patches & Updates home page.

NewImage

2. At the Patch & Updates home page, enter the patch number and click search.

Please note: The patch number can be found within the annals of the actual CPU Note on MOS. In this case, its RDBMS 18031528 which has since become part of GI 18139660.

NewImage

3. Pick the correct platform

3.1. First of all download the patch to the Software Library.

NewImage

3.2. Second, add the patch to a new Patch Plan.

NewImage

3.3. Enter an appropriate Plan Name, select all “Oracle Home” targets, and finally select the Cluster where the software needs to be patched. You will be prompted for whether all targets within the cluster need to be patched, please confirm and continue.

NewImage

4. Let’s take a look at whether patch download has completed. To find the list of downloaded patches, we need to go to the “Saved Patches” section.

NewImage

4.1. Bingo, its there!

NewImage

5. How about the Patch Plan?

5.1. Head back to the “Patches & Updates” section.

NewImage

5.2. There it is. Click on the plan name to proceed.

NewImage

6. On to patching!

6.1. On the patch plan home page, there are a few sections (seen on the left) which have to be clicked through before patching can begin.

6.2. First off, change the patch name if you want, and/or grant access to the various roles. Click “Next” to continue.

NewImage

6.3. The two Oracle Home targets I added in step 3.3 are already present. Click “Next”.

NewImage

6.4. Since this is a CPU, we can patch “In Place”.

NewImage

6.5. Override the Cluster Oracle Home Credentials, RAC Oracle Home Credentials, and Privilege Credentials, and set them accordingly. In my case they were the grid, oracle, and root users.

Please note, if you do not have root user access, SUDO/PowerBroker will be necessary to continue. To configure SUDO within OEM see the documentation here for Privilege Delegation.

NewImage

6.6. This section is used to validate the patch on the respective targets. Click on “Analyze”. This will likely take a while and assuming all goes well, you will be shown the section where it says “Ready for Deployment”. Click “Next”.

NewImage

6.7. At this point, we are ready to patch! Click “Deploy” when ready.

NewImage

7. Given enough time, the Deployment Procedure will complete successfully. Check each the Grid Infrastructure and Database Homes for the latest applied patches:

[grid@dc01db2a ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@dc01db2a ~]$ /u01/app/12.1.0.1/grid/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/12.1.0.1/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/12.1.0.1/grid/oraInst.loc
OPatch version    : 12.1.0.1.2
OUI version       : 12.1.0.1.0
Log file location : /u01/app/12.1.0.1/grid/cfgtoollogs/opatch/opatch2014-05-12_17-43-23PM_1.log

Lsinventory Output file location : /u01/app/12.1.0.1/grid/cfgtoollogs/opatch/lsinv/lsinventory2014-05-12_17-43-23PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Grid Infrastructure 12c                                       12.1.0.1.0
There are 1 products installed in this Oracle Home.


Interim patches (3) :

Patch  18031528     : applied on Mon May 12 13:23:13 EDT 2014
Unique Patch ID:  17262469
Patch description:  "Database Patch Set Update : 12.1.0.1.3 (18031528)"
   Created on 18 Mar 2014, 14:50:40 hrs PST8PDT
   Bugs fixed:
     17716305, 17034172, 16694728, 16855202, 17439871, 16320173, 17462687
     17082983, 16313881, 16715647, 17362796, 17777061, 16450169, 16392068
     17761775, 16977973, 14197853, 16712618, 17552800, 17922172, 17441661
     16524071, 16856570, 18362376, 16410570, 13866822, 16372203, 16849982
     16837842, 16459685, 16101465, 18393024, 16802693, 16978185, 16845022
     16195633, 17042658, 14536110, 17983206, 17579911, 16787973, 16850996
     17311728, 16838328, 16178562, 17391312, 17244462, 16503473, 16935643
     18126350, 17039360, 14355775, 17721717, 17489214, 16362358, 16994576
     16928832, 16864359, 17080436, 16679874, 18031528, 16788832, 16585173
     15986012, 18099539, 17467075, 14852021, 16191248, 17174391, 17830435
     17249820, 16946990, 16589507, 16924879, 16173738, 16874123, 17797453
     16784143, 15987992, 17343514, 17346196, 17324822, 16495802, 16859937
     16590848, 17316776, 17068536, 16986421, 16910001, 16527374, 16730813
     16663303, 17490498, 17897716, 16186165, 16675739, 16170787, 16457621
     16524968, 17032726, 16543323, 18355572, 17981677, 17005047, 17442449
     16795944, 16668226, 16070351, 16698577, 17088068, 16621274, 17330580
     16888264, 16448848, 16863422, 16465158, 16634384, 17443671, 16816103
     16910734, 16517900, 16911800, 16825779, 16936008, 17019974, 16707927
     16347904, 17273253, 17263661, 17325413, 16902138, 17179261, 17810688
     16465149, 17468141, 17226980, 17184677, 16689109, 16705020, 16682595
     16864864, 16972213, 16721594, 16946613, 16964279, 16855292, 17205719
     17026503, 16964686, 16674842, 16757934, 16864562, 15996344, 16842274
     17000176, 16769019, 17572525, 17659488, 16485876, 16709437, 17289787
     16919176, 17217040, 16613964, 16796277, 16462834, 17421502, 18092561
     16617325, 17308691, 16921340, 16483559, 16057129, 16733884, 16784167
     16286774, 16822629, 16660558, 17596344, 17954431, 17570606, 16674666
     16697600, 16993424, 17797837, 17716565, 16772060, 16991789, 16790307
     16275522, 17491753, 16603924, 16427054, 16227068, 16784901, 16479182
     13521413, 17478811, 16836849, 16007562, 16663465, 16551086, 17027533
     16675710, 16406802, 17437634, 16433745, 17610418, 17465741, 17171530
     16523150, 16212405, 16741246, 16930325, 16443657

Patch  17303297     : applied on Mon May 12 13:15:10 EDT 2014
Unique Patch ID:  16881795
Patch description:  "ACFS Patch Set Update 12.1.0.1.1"
   Created on 14 Oct 2013, 07:25:50 hrs US/Central
   Bugs fixed:
     14487556, 16398970, 16552813, 16930184, 16420645, 16170117, 16436434
     16476044, 16458315, 16463033, 16095100, 16545876, 16429953, 14826673
     16001893, 16482869, 16371746, 16435343, 14476443, 16294308, 16671486
     16386110, 15978267, 16085530, 16347837, 16814544, 16022372, 16167084
     14510092, 16450287, 16399406

Patch  17077442     : applied on Mon May 12 13:00:50 EDT 2014
Unique Patch ID:  16881794
Patch description:  "Oracle Clusterware Patch Set Update 12.1.0.1.1"
   Created on 12 Oct 2013, 06:33:53 hrs US/Central
   Bugs fixed:
     16505840, 16505255, 16390989, 16399322, 16505617, 16505717, 17486244
     16168869, 16444109, 16505361, 13866165, 16505763, 16208257, 16904822
     17299876, 16246222, 16505214, 16505540, 15936039, 16580269, 16838292
     16505449, 16801843, 16309853, 16505395, 17507349, 17475155, 16493242
     17039197, 16196609, 17463260, 16505667, 15970176, 16488665, 16670327



Patch level status of Cluster nodes :

 Patching Level     		 Nodes
 --------------     		 -----
 2305600176         		 dc01db2b,dc01db2a

--------------------------------------------------------------------------------

OPatch succeeded.

[oracle@dc01db2a ~]$ . oraenv
ORACLE_SID = [AGENT] ? raccdba1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@dc01db2a ~]$ /u01/app/oracle/product/12.1.0.1/db_1/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/12.1.0.1/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0.1/db_1/oraInst.loc
OPatch version    : 12.1.0.1.2
OUI version       : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0.1/db_1/cfgtoollogs/opatch/opatch2014-05-12_17-46-34PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/12.1.0.1/db_1/cfgtoollogs/opatch/lsinv/lsinventory2014-05-12_17-46-34PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 12c                                                  12.1.0.1.0
There are 1 products installed in this Oracle Home.


Interim patches (2) :

Patch  18031528     : applied on Mon May 12 12:38:15 EDT 2014
Unique Patch ID:  17262469
Patch description:  "Database Patch Set Update : 12.1.0.1.3 (18031528)"
   Created on 18 Mar 2014, 14:50:40 hrs PST8PDT
   Bugs fixed:
     17716305, 17034172, 16694728, 16855202, 17439871, 16320173, 17462687
     17082983, 16313881, 16715647, 17362796, 17777061, 16450169, 16392068
     17761775, 16977973, 14197853, 16712618, 17552800, 17922172, 17441661
     16524071, 16856570, 18362376, 16410570, 13866822, 16372203, 16849982
     16837842, 16459685, 16101465, 18393024, 16802693, 16978185, 16845022
     16195633, 17042658, 14536110, 17983206, 17579911, 16787973, 16850996
     17311728, 16838328, 16178562, 17391312, 17244462, 16503473, 16935643
     18126350, 17039360, 14355775, 17721717, 17489214, 16362358, 16994576
     16928832, 16864359, 17080436, 16679874, 18031528, 16788832, 16585173
     15986012, 18099539, 17467075, 14852021, 16191248, 17174391, 17830435
     17249820, 16946990, 16589507, 16924879, 16173738, 16874123, 17797453
     16784143, 15987992, 17343514, 17346196, 17324822, 16495802, 16859937
     16590848, 17316776, 17068536, 16986421, 16910001, 16527374, 16730813
     16663303, 17490498, 17897716, 16186165, 16675739, 16170787, 16457621
     16524968, 17032726, 16543323, 18355572, 17981677, 17005047, 17442449
     16795944, 16668226, 16070351, 16698577, 17088068, 16621274, 17330580
     16888264, 16448848, 16863422, 16465158, 16634384, 17443671, 16816103
     16910734, 16517900, 16911800, 16825779, 16936008, 17019974, 16707927
     16347904, 17273253, 17263661, 17325413, 16902138, 17179261, 17810688
     16465149, 17468141, 17226980, 17184677, 16689109, 16705020, 16682595
     16864864, 16972213, 16721594, 16946613, 16964279, 16855292, 17205719
     17026503, 16964686, 16674842, 16757934, 16864562, 15996344, 16842274
     17000176, 16769019, 17572525, 17659488, 16485876, 16709437, 17289787
     16919176, 17217040, 16613964, 16796277, 16462834, 17421502, 18092561
     16617325, 17308691, 16921340, 16483559, 16057129, 16733884, 16784167
     16286774, 16822629, 16660558, 17596344, 17954431, 17570606, 16674666
     16697600, 16993424, 17797837, 17716565, 16772060, 16991789, 16790307
     16275522, 17491753, 16603924, 16427054, 16227068, 16784901, 16479182
     13521413, 17478811, 16836849, 16007562, 16663465, 16551086, 17027533
     16675710, 16406802, 17437634, 16433745, 17610418, 17465741, 17171530
     16523150, 16212405, 16741246, 16930325, 16443657

Patch  17077442     : applied on Mon May 12 12:32:50 EDT 2014
Unique Patch ID:  16881794
Patch description:  "Oracle Clusterware Patch Set Update 12.1.0.1.1"
   Created on 12 Oct 2013, 06:33:53 hrs US/Central
   Bugs fixed:
     16505840, 16505255, 16390989, 16399322, 16505617, 16505717, 17486244
     16168869, 16444109, 16505361, 13866165, 16505763, 16208257, 16904822
     17299876, 16246222, 16505214, 16505540, 15936039, 16580269, 16838292
     16505449, 16801843, 16309853, 16505395, 17507349, 17475155, 16493242
     17039197, 16196609, 17463260, 16505667, 15970176, 16488665, 16670327



Rac system comprising of multiple nodes
  Local node = dc01db2a
  Remote node = dc01db2b

--------------------------------------------------------------------------------

OPatch succeeded.

With that being said, the application of the patch succeeded and is validated. Unfortunately I lost the screenshots I took of the patch deployment procedure activity. I will update this post with another patching activity once I have those.

Safe to say, I was able to successfully create a pluggable database once the patch was applied.

Lots more to come where this one came from – stay tuned for more posts on patching via #em12c.

Cheers.