This is the third part of the ODA patch series, from 18.3 to 19.8. I separate in multiple parts and you can use this part as a direct guide to patch ODA from 19.6 to 19.7. Each part can be used alone since they cover all the needed steps. Some steps of this post are similar to the upgrade from 18.8 to 19.6 that I described in the previous post.
The process of patch ODA is not complicated but requires attention over some steps. The 19.6 version was the first that was possible to patch from 18.8 version, and the version that allows upgrades to newer. If you want to go directly to 19.5 you need to reimage of the appliance. In this post, I will cover the process that I made recently to patch from 18.3 to 19.8 version.
Patch Matrix
The matrix of what can be done can be found at this post from ODA blog, and you can check below:
Another important detail is to check the MOS note ODA: Quick Reference Matrix for Linux Release and Kernel by ODA Hardware Type and Version (Doc ID 2680219.1) and verify if your hardware is still compatible.
Remember that in this process, the ODA will reboot several times, so, you need to inform your teams that databases will be unavailable during the process.
Actual environment
The environment is ODA X5-2, running the version 19.6. This same environment I reimaged last year, and you can see the steps that I made in a previous post. And it running bare metal (BM).
The first thing to do is to check the actual hardware about errors. If you have, please fix it before the start. Another important detail is to verify if your disks have smartctl warnings. To do that, execute smartctl -q errorsonly -H -l xerror /dev/DISK; If you have “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]” report (or similar warning), please open an SR to replace the disks. Even the warning can lead to an error during the patch.
I recommend you to save outside of ODA some config files from BOTH nodes (to have quick access in case you need):
[root@odat1 ~]# mkdir /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/hosts /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/fstab /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/resolv.conf /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/sysconfig/network-scripts/* /nfs/ODA_PATCH_18/bkp_files_odat1/ cp: omitting directory `/etc/sysconfig/network-scripts/backupifcfgFiles' cp: omitting directory `/etc/sysconfig/network-scripts/bkupIfcfgOrig' [root@odat1 ~]# [root@odat1 ~]# cp /u01/app/18.0.0.0/grid/network/admin/listener* /nfs/ODA_PATCH_18/bkp_files_odat1/ [root@odat1 ~]#
Another detail before the start is to check the actual components and running versions. To do that we execute the odacli describe-component. If we need we just need to update the components with odacli update-storage. This will (if some firmware is applied) reboot the ODA.
Before starting the patch I recommend stopping all databases running at ODA. This will speed up the process. So, left just GI and ASM running.
If you got any error in the next steps related to lack of space. You can check my previous posts of this series because there I describe how to increase the filesystem if needed. And I also show how to decrease the filesystem as well (like /u01) to have raw space that can be allocated to other mountpoints.
19.7
The upgrade from 19.6 to 19.7 is a simple process and follows a well know procedure. In this process, we do not upgrade Linux (like 18.8 to 19.6), just some new rpms and we upgrade the GI to 19.6. The patch to download is 31220620.
Update repository
The first step is to upload the unzipped patch files at the internal ODA repository. This process is done using the odacli update-repository and you pass the full path to unzip patch files (here my files are in NFS folder):
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-repository -f /nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server1of2.zip,/nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server2of2.zip
{
"jobId" : "ad0389f3-2e2b-4523-b054-de8e84a84949",
"status" : "Created",
"message" : "/nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server1of2.zip,/nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server2of2.zip",
"reports" : [ ],
"createTimestamp" : "July 26, 2020 11:21:27 AM CEST",
"resourceList" : [ ],
"description" : "Repository Update",
"updatedTime" : "July 26, 2020 11:21:27 AM CEST"
}
You have new mail in /var/spool/mail/root
[root@odat1 ~]#
After that we can follow the job with odacli describe-job:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "ad0389f3-2e2b-4523-b054-de8e84a84949"
Job details
----------------------------------------------------------------
ID: ad0389f3-2e2b-4523-b054-de8e84a84949
Description: Repository Update
Status: Success
Created: July 26, 2020 11:21:27 AM CEST
Message: /nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server1of2.zip,/nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server2of2.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace July 26, 2020 11:21:28 AM CEST July 26, 2020 11:21:28 AM CEST Success
Setting up ssh equivalance July 26, 2020 11:21:29 AM CEST July 26, 2020 11:21:29 AM CEST Success
Copy BundleFile July 26, 2020 11:21:30 AM CEST July 26, 2020 11:25:50 AM CEST Success
Validating CopiedFile July 26, 2020 11:25:50 AM CEST July 26, 2020 11:26:21 AM CEST Success
Unzip bundle July 26, 2020 11:26:22 AM CEST July 26, 2020 11:43:55 AM CEST Success
Unzip bundle July 26, 2020 11:43:55 AM CEST July 26, 2020 11:46:16 AM CEST Success
Delete PatchBundles July 26, 2020 11:46:22 AM CEST July 26, 2020 11:46:45 AM CEST Success
Removing ssh keys July 26, 2020 11:46:45 AM CEST July 26, 2020 11:46:47 AM CEST Success
[root@odat1 ~]#
DCSAgent
The next step is to update the internal DCS agent for ODA 19.7. Is a simple step executed using the odacli update-dcsagent. The parameter is the version 19.7.0.0:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.7.0.0.0
{
"jobId" : "1af29269-73cd-4eb6-8208-e5885376b274",
"status" : "Created",
"message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
"reports" : [ ],
"createTimestamp" : "July 26, 2020 11:51:01 AM CEST",
"resourceList" : [ ],
"description" : "DcsAgent patching",
"updatedTime" : "July 26, 2020 11:51:01 AM CEST"
}
[root@odat1 ~]#
And as usual, we need to wait for the job to finish before continuing. We use the odacli describe-job to check this:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "1af29269-73cd-4eb6-8208-e5885376b274"
Job details
----------------------------------------------------------------
ID: 1af29269-73cd-4eb6-8208-e5885376b274
Description: DcsAgent patching
Status: Success
Created: July 26, 2020 11:51:01 AM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
dcs-agent upgrade to version 19.7.0.0.0 July 26, 2020 11:51:02 AM CEST July 26, 2020 11:52:49 AM CEST Success
dcs-agent upgrade to version 19.7.0.0.0 July 26, 2020 11:52:49 AM CEST July 26, 2020 11:54:28 AM CEST Success
Update System version July 26, 2020 11:54:35 AM CEST July 26, 2020 11:54:35 AM CEST Success
Update System version July 26, 2020 11:54:35 AM CEST July 26, 2020 11:54:35 AM CEST Success
[root@odat1 ~]#
You can receive some errors if you try to describe the jobID. This can occur because the process restarts the DCS. If you got this, just wait some minutes and try again (some common errors are: DCS-10001:Internal error encountered: Fail to get credential or DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070).
DCSAdmin
The next step will be to update the dcsadmin using the command odacli update-dcsadmin:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.7.0.0.0
{
"jobId" : "edf88f79-529c-438e-8a4a-a412749c8a70",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "July 26, 2020 11:56:45 AM CEST",
"resourceList" : [ ],
"description" : "DcsAdmin patching",
"updatedTime" : "July 26, 2020 11:56:45 AM CEST"
}
[root@odat1 ~]#
And as usual, we can check the end of the job before continue:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "edf88f79-529c-438e-8a4a-a412749c8a70"
Job details
----------------------------------------------------------------
ID: edf88f79-529c-438e-8a4a-a412749c8a70
Description: DcsAdmin patching
Status: Success
Created: July 26, 2020 11:56:45 AM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation July 26, 2020 11:56:47 AM CEST July 26, 2020 11:56:47 AM CEST Success
Patch location validation July 26, 2020 11:56:47 AM CEST July 26, 2020 11:56:47 AM CEST Success
dcsadmin upgrade July 26, 2020 11:56:48 AM CEST July 26, 2020 11:56:54 AM CEST Success
dcsadmin upgrade July 26, 2020 11:56:54 AM CEST July 26, 2020 11:57:01 AM CEST Success
Update System version July 26, 2020 11:57:02 AM CEST July 26, 2020 11:57:02 AM CEST Success
Update System version July 26, 2020 11:57:02 AM CEST July 26, 2020 11:57:02 AM CEST Success
[root@odat1 ~]#
DCSComponents
The next is to update the dcscomponents with command odacli update-dcscomponents. As occurred with other versions, the commands will generate one jobID, but will be invalid and you need to use the command “odacli list-jobs” to see the jobs (will be just updated SSH keys – two jobs):
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.7.0.0.0
{
"jobId" : "e4f87d5e-9b79-4bcd-a74f-daa9ff672474",
"status" : "Success",
"message" : null,
"reports" : null,
"createTimestamp" : "July 26, 2020 11:57:28 AM CEST",
"description" : "Job completed and is not part of Agent job list",
"updatedTime" : "July 26, 2020 11:57:28 AM CEST"
}
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "e4f87d5e-9b79-4bcd-a74f-daa9ff672474"
DCS-10000:Resource Job with ID e4f87d5e-9b79-4bcd-a74f-daa9ff672474 is not found.
[root@odat1 ~]#
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli list-jobs
ID Description Created Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
…
…
ad0389f3-2e2b-4523-b054-de8e84a84949 Repository Update July 26, 2020 11:21:27 AM CEST Success
1af29269-73cd-4eb6-8208-e5885376b274 DcsAgent patching July 26, 2020 11:51:01 AM CEST Success
edf88f79-529c-438e-8a4a-a412749c8a70 DcsAdmin patching July 26, 2020 11:56:45 AM CEST Success
c8d8bcde-4015-468d-8332-d2569e5ffa6d SSH keys update July 26, 2020 11:57:51 AM CEST Success
a1c4cdc8-1b9e-4d25-8fac-cf85ae561e5e SSH key delete July 26, 2020 11:57:56 AM CEST Success
7a9aff35-a04e-43ab-bc3b-7c3aee9a72d9 SSH keys update July 26, 2020 11:58:25 AM CEST Success
d0817c75-3fa3-4974-ade1-470a0c7993d8 SSH key delete July 26, 2020 12:00:00 PM CEST Success
[root@odat1 ~]#
Create-prepatchreport
The next step is to verify if you can update it to 19.7. The odacli will do a lot of tests for the operational system, GI, and ILOM. To do that, we just need to call odacli create-prepatchreport, and after that use the odacli describe-prepatchreport to check the report.
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.7.0.0.0
Job details
----------------------------------------------------------------
ID: d7d1c155-1312-41bf-afe6-c85570073d9b
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
Status: Created
Created: July 26, 2020 1:59:04 PM CEST
Message: Use 'odacli describe-prepatchreport -i d7d1c155-1312-41bf-afe6-c85570073d9b' to check details of results
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@odat1 ~]#
And after some time we got the full result (30 minutes more or less):
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i d7d1c155-1312-41bf-afe6-c85570073d9b
Patch pre-check report
------------------------------------------------------------------------
Job ID: d7d1c155-1312-41bf-afe6-c85570073d9b
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
Status: FAILED
Created: July 26, 2020 1:59:04 PM CEST
Result: One or more pre-checks failed for [ORACHK]
Node Name
---------------
odat1
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.7.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
__ILOM__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.7.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Patch already applied
Patch location validation Success Successfully validated location
__GI__
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.7.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate GI patch metadata Success Validated patching tag: 19.7.0.0.0.
Validate clones location exist Success Validated clones location
Is patch location available Success Patch location is available.
Patch location validation Success Successfully validated location
Patch verification Success Patches 30869156 not applied on GI
home /u01/app/19.0.0.0/grid on node
odat1
Validate Opatch update Success Successfully updated the opatch in
GiHome /u01/app/19.0.0.0/grid on node
odat1
Patch conflict check Success No patch conflicts found on GiHome
/u01/app/19.0.0.0/grid on node odat1
__ORACHK__
Running orachk Failed Orachk validation failed: .
Software home Failed Software home check failed
Node Name
---------------
odat2
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.7.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
__ILOM__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.7.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Patch already applied
Patch location validation Success Successfully validated location
__GI__
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.7.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate GI patch metadata Success Validated patching tag: 19.7.0.0.0.
Validate clones location exist Success Validated clones location
Is patch location available Success Patch location is available.
Patch location validation Success Successfully validated location
Patch verification Success Patches 30869156 not applied on GI
home /u01/app/19.0.0.0/grid on node
odat2
Validate Opatch update Success Successfully updated the opatch in
GiHome /u01/app/19.0.0.0/grid on node
odat2
Patch conflict check Success No patch conflicts found on GiHome
/u01/app/19.0.0.0/grid on node odat2
__ORACHK__
Running orachk Failed Orachk validation failed: .
Software home Failed Software home check failed
[root@odat1 ~]#
As you can see FAILED. Some known issues can be checked before. One that I passed was the existence of /u01/app/oraInventory/locks. I needed to remove it from both nodes to have success in the report.
The ORACHK is more tricky to check the source. To check we need go to folder /opt/oracle/dcs/oracle.ahf/orachk/SERVER/<jobID#>/orachk_<hostname><random#>/ and check the HTML file:
[root@odat1 ~]# ls -l /opt/oracle/dcs/oracle.ahf/orachk/SERVER/d7d1c155-1312-41bf-afe6-c85570073d9b/orachk_odat1_072620_14087/ total 1024 drwxr-xr-x 2 root root 4096 Jul 26 14:12 log -rw------- 1 root root 993400 Jul 26 14:12 orachk_odat1_072620_14087.html drwxr-xr-x 4 root root 24576 Jul 26 14:12 outfiles drwxr-xr-x 2 root root 4096 Jul 26 14:12 reports drwxr-xr-x 2 root root 12288 Jul 26 14:12 scripts drwxr-xr-x 2 root root 4096 Jul 26 14:12 upload [root@odat1 ~]#
There we can see internally the ERROR. In this case that we got the “Orachk validation failed” and “Software home check failed” the error is related with a check made but CVU that try to check some files (that are listed in XML file) and was not found:
Cluster Verification Utility (CVU 19.6.0.0.0 ) result Status Type Message Status On Details FAIL OS Check Software home check failed All Database Servers View Status on odat1: FAIL => Software home check failed Error Message: File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libjavafx_font_t2k.so" could not be verified on node "odat1". OS error: "No such file or directory" Error Message: File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libkcms.so" could not be verified on node "odat1". OS error: "No such file or directory" Error Message: File "/u01/app/19.0.0.0/grid/rdbms/lib/ksms.o" could not be verified on node "odat1". OS error: "No such file or directory"
This error occurs because the CVU uses the context of file /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml to verify the files that need to be checked. But this file has more contents that need:
[root@odat1 ~]# cat /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml |grep libkcms
<File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
<File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
<File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
[root@odat1 ~]#
So, to fix, need to copy the file /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml over /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml. This is described at Bug 31208297 since the new version is aligned with 19c (while the other is aligned with other versions).
Another option is simply ignore the error and continue with the process. If you have doubts or other errors, please check the release notes from the version to check if they are listed there. Another option is open SR.
Update-Server
After the create patchreport we can patch the server calling odacli update-server:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-server -v 19.7.0.0.0
{
"jobId" : "91abf816-b171-4e2f-a206-23013f53fb8e",
"status" : "Created",
"message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
"reports" : [ ],
"createTimestamp" : "July 26, 2020 16:04:06 PM CEST",
"resourceList" : [ ],
"description" : "Server Patching",
"updatedTime" : "July 26, 2020 16:04:06 PM CEST"
}
[root@odat1 ~]#
This will create one jobIB that we can follow:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "91abf816-b171-4e2f-a206-23013f53fb8e"
Job details
----------------------------------------------------------------
ID: 91abf816-b171-4e2f-a206-23013f53fb8e
Description: Server Patching
Status: Success
Created: July 26, 2020 4:04:06 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation July 26, 2020 4:04:42 PM CEST July 26, 2020 4:04:43 PM CEST Success
Patch location validation July 26, 2020 4:04:42 PM CEST July 26, 2020 4:04:42 PM CEST Success
dcs-controller upgrade July 26, 2020 4:04:43 PM CEST July 26, 2020 4:04:50 PM CEST Success
dcs-controller upgrade July 26, 2020 4:04:50 PM CEST July 26, 2020 4:04:57 PM CEST Success
Patch location validation July 26, 2020 4:05:00 PM CEST July 26, 2020 4:05:00 PM CEST Success
Patch location validation July 26, 2020 4:05:00 PM CEST July 26, 2020 4:05:00 PM CEST Success
dcs-cli upgrade July 26, 2020 4:05:02 PM CEST July 26, 2020 4:05:02 PM CEST Success
dcs-cli upgrade July 26, 2020 4:05:02 PM CEST July 26, 2020 4:05:03 PM CEST Success
Creating repositories using yum July 26, 2020 4:05:07 PM CEST July 26, 2020 4:05:10 PM CEST Success
Updating YumPluginVersionLock rpm July 26, 2020 4:05:11 PM CEST July 26, 2020 4:05:11 PM CEST Success
Applying OS Patches July 26, 2020 4:05:11 PM CEST July 26, 2020 4:24:31 PM CEST Success
Creating repositories using yum July 26, 2020 4:24:32 PM CEST July 26, 2020 4:24:34 PM CEST Success
Applying HMP Patches July 26, 2020 4:24:34 PM CEST July 26, 2020 4:24:35 PM CEST Success
Patch location validation July 26, 2020 4:24:36 PM CEST July 26, 2020 4:24:36 PM CEST Success
Patch location validation July 26, 2020 4:24:36 PM CEST July 26, 2020 4:24:36 PM CEST Success
oda-hw-mgmt upgrade July 26, 2020 4:24:37 PM CEST July 26, 2020 4:25:11 PM CEST Success
oda-hw-mgmt upgrade July 26, 2020 4:25:11 PM CEST July 26, 2020 4:25:47 PM CEST Success
OSS Patching July 26, 2020 4:25:47 PM CEST July 26, 2020 4:25:49 PM CEST Success
Applying Firmware Disk Patches July 26, 2020 4:26:57 PM CEST July 26, 2020 4:27:59 PM CEST Success
Applying Firmware Expander Patches July 26, 2020 4:28:13 PM CEST July 26, 2020 4:28:30 PM CEST Success
Applying Firmware Controller Patches July 26, 2020 4:28:44 PM CEST July 26, 2020 4:28:59 PM CEST Success
Checking Ilom patch Version July 26, 2020 4:29:01 PM CEST July 26, 2020 4:29:03 PM CEST Success
Checking Ilom patch Version July 26, 2020 4:29:03 PM CEST July 26, 2020 4:29:06 PM CEST Success
Patch location validation July 26, 2020 4:29:06 PM CEST July 26, 2020 4:29:07 PM CEST Success
Patch location validation July 26, 2020 4:29:06 PM CEST July 26, 2020 4:29:07 PM CEST Success
Save password in Wallet July 26, 2020 4:29:09 PM CEST July 26, 2020 4:29:09 PM CEST Success
Apply Ilom patch July 26, 2020 4:29:09 PM CEST July 26, 2020 4:29:11 PM CEST Success
Apply Ilom patch July 26, 2020 4:29:11 PM CEST July 26, 2020 4:29:12 PM CEST Success
Copying Flash Bios to Temp location July 26, 2020 4:29:12 PM CEST July 26, 2020 4:29:12 PM CEST Success
Copying Flash Bios to Temp location July 26, 2020 4:29:12 PM CEST July 26, 2020 4:29:12 PM CEST Success
Starting the clusterware July 26, 2020 4:31:30 PM CEST July 26, 2020 4:33:32 PM CEST Success
clusterware patch verification July 26, 2020 4:33:38 PM CEST July 26, 2020 4:33:45 PM CEST Success
clusterware patch verification July 26, 2020 4:33:38 PM CEST July 26, 2020 4:33:45 PM CEST Success
Patch location validation July 26, 2020 4:33:45 PM CEST July 26, 2020 4:33:53 PM CEST Success
Patch location validation July 26, 2020 4:33:45 PM CEST July 26, 2020 4:33:53 PM CEST Success
Opatch update July 26, 2020 4:34:37 PM CEST July 26, 2020 4:34:41 PM CEST Success
Opatch update July 26, 2020 4:34:37 PM CEST July 26, 2020 4:34:40 PM CEST Success
Patch conflict check July 26, 2020 4:34:42 PM CEST July 26, 2020 4:36:18 PM CEST Success
Patch conflict check July 26, 2020 4:36:18 PM CEST July 26, 2020 4:37:54 PM CEST Success
clusterware upgrade July 26, 2020 4:38:09 PM CEST July 26, 2020 5:26:26 PM CEST Success
clusterware upgrade July 26, 2020 5:26:27 PM CEST July 26, 2020 6:14:40 PM CEST Success
Updating GiHome version July 26, 2020 6:14:40 PM CEST July 26, 2020 6:15:03 PM CEST Success
Updating GiHome version July 26, 2020 6:14:40 PM CEST July 26, 2020 6:15:03 PM CEST Success
Update System version July 26, 2020 6:16:31 PM CEST July 26, 2020 6:16:32 PM CEST Success
Update System version July 26, 2020 6:16:32 PM CEST July 26, 2020 6:16:32 PM CEST Success
preRebootNode Actions July 26, 2020 6:16:32 PM CEST July 26, 2020 6:17:15 PM CEST Success
preRebootNode Actions July 26, 2020 6:17:15 PM CEST July 26, 2020 6:18:00 PM CEST Success
Reboot Ilom July 26, 2020 6:18:01 PM CEST July 26, 2020 6:18:01 PM CEST Success
Reboot Ilom July 26, 2020 6:18:01 PM CEST July 26, 2020 6:18:02 PM CEST Success
[root@odat1 ~]#
In ODAX5-2 this process took around two hours. If you follow the /opt/oracle/dcs/log/dcs-agent.log you can see more detail like and steps executed. Some example:
... 2020-07-26 16:54:07,687 DEBUG [clusterware upgrade : JobId=91abf816-b171-4e2f-a206-23013f53fb8e] [] c.o.d.a.u.AgentCommonUtils: dbGiHome version:19.0.0.0: 2020-07-26 16:54:07,687 DEBUG [clusterware upgrade : JobId=91abf816-b171-4e2f-a206-23013f53fb8e] [] c.o.d.a.r.s.p.PatchingOperations: Is 18c Database:is18c 2020-07-26 16:54:07,687 DEBUG [clusterware upgrade : JobId=91abf816-b171-4e2f-a206-23013f53fb8e] [] c.o.d.c.u.CommonsUtils: run: cmd= '[/u01/app/19.0.0.0/grid/OPatch/opatchauto, apply, /u01/patching/19.7.0.0.200414/31145668, -oh, /u01/app/19.0.0.0/grid, -ocmrf, /u01/app/19.0.0.0/grid/default-ocm.rsp]' 2020-07-26 16:54:07,900 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT 2020-07-26 16:54:07,900 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output : OPatchauto session is initiated at Sun Jul 26 16:54:07 2020 2020-07-26 16:54:13,077 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT 2020-07-26 16:54:13,077 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output : System initialization log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-07-26_04-54-12PM.log. 2020-07-26 16:54:13,077 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT ...
After you finish this jobIB, and around 10 minutes later, ODA nodes will reboot to boot the new version.
Update-storage
The last step is to patch the storage with command odacli update-storage. This is needed because the previous step does not update all the firmawares.
You can see from odacli describe-component that some part are not up to date:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
19.7.0.0.0
System node Name
---------------
odat1
Local System Version
---------------
19.7.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 19.7.0.0.0 up-to-date
GI 19.7.0.0.200414 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.10.0.0.200414
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.200414
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.200414
}
DCSAGENT 19.7.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 7.8 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 13.00.00.00 16.00.08.00
}
FIRMWAREEXPANDER 001E up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
HMP 2.4.5.0.1 up-to-date
System node Name
---------------
odat2
Local System Version
---------------
19.7.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 19.7.0.0.0 up-to-date
GI 19.7.0.0.200414 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.10.0.0.200414
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.200414
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.200414
}
DCSAGENT 19.7.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 7.8 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 13.00.00.00 16.00.08.00
}
FIRMWAREEXPANDER 001E up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
HMP 2.4.5.0.1 up-to-date
[root@odat1 ~]#
Look above that diskcontrolers are not with the last version of the firmware. So, we need to call odacli update-storage to patch everything:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-storage -v 19.7.0.0.0
{
"jobId" : "095e10f4-cefc-410d-af3f-b3c8f9cd0af9",
"status" : "Created",
"message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
"reports" : [ ],
"createTimestamp" : "July 26, 2020 18:35:00 PM CEST",
"resourceList" : [ ],
"description" : "Storage Firmware Patching",
"updatedTime" : "July 26, 2020 18:35:00 PM CEST"
}
[root@odat1 ~]#
And we can follow the jobID:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "095e10f4-cefc-410d-af3f-b3c8f9cd0af9"
Job details
----------------------------------------------------------------
ID: 095e10f4-cefc-410d-af3f-b3c8f9cd0af9
Description: Storage Firmware Patching
Status: Success
Created: July 26, 2020 6:35:00 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches July 26, 2020 6:39:37 PM CEST July 26, 2020 6:42:40 PM CEST Success
Applying Firmware Controller Patches July 26, 2020 6:49:16 PM CEST July 26, 2020 6:55:47 PM CEST Success
preRebootNode Actions July 26, 2020 6:55:48 PM CEST July 26, 2020 6:55:48 PM CEST Success
preRebootNode Actions July 26, 2020 6:55:48 PM CEST July 26, 2020 6:55:48 PM CEST Success
Reboot Ilom July 26, 2020 6:55:48 PM CEST July 26, 2020 6:55:48 PM CEST Success
Reboot Ilom July 26, 2020 6:55:48 PM CEST July 26, 2020 6:55:49 PM CEST Success
[root@odat1 ~]#
After it finishes, the ODA nodes will reboot again. To reboot be effective maybe you need to wait 10 minutes or more.
And after the reboot you can see that everything is up-to-date:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
19.7.0.0.0
System node Name
---------------
odat1
Local System Version
---------------
19.7.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 19.7.0.0.0 up-to-date
GI 19.7.0.0.200414 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.10.0.0.200414
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.200414
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.200414
}
DCSAGENT 19.7.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 7.8 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 16.00.08.00 up-to-date
}
FIRMWAREEXPANDER 001E up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
HMP 2.4.5.0.1 up-to-date
System node Name
---------------
odat2
Local System Version
---------------
19.7.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 19.7.0.0.0 up-to-date
GI 19.7.0.0.200414 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.10.0.0.200414
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.200414
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.200414
}
DCSAGENT 19.7.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 7.8 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 16.00.08.00 up-to-date
}
FIRMWAREEXPANDER 001E up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
HMP 2.4.5.0.1 up-to-date
[root@odat1 ~]#
After that, you can start all databases that you stopped before and continue to use ODA.
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”
Pingback: Patch ODA from 18.3 to 19.8. Part 4 – 19.7 to 19.8 | Fernando Simon