The process of patch ODA is not complicated but requires attention over some steps. The 19.6 version was the first that was possible to patch from 18.8 version, and the version that allows upgrades to newer. If you want to go directly to 19.5 you need to reimage of the appliance. In this post, I will cover the process that I made recently to patch from 18.3 to 19.8 version.
The first part covers the upgrade from 18.3 to 18.8.
Patch Matrix
The matrix of what can be done can be found at this post from ODA blog, and you can check below:
Another important detail is to check the MOS note ODA: Quick Reference Matrix for Linux Release and Kernel by ODA Hardware Type and Version (Doc ID 2680219.1) and verify if your hardware is still compatible.
Remember that in this process, the ODA will reboot several times, so, you need to inform your teams that databases will be unavailable during the process.
Actual environment
The environment is ODA X5-2, running the version 18.3. This same environment I reimaged last year, and you can see the steps that I made in a previous post. And it running bare metal (BM).
The first thing to do is to check the actual hardware about errors. If you have, please fix it before the start. Another important detail is to verify if your disks have smartctl warnings. To do that, execute smartctl -q errorsonly -H -l xerror /dev/DISK; If you have “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]” report (or similar warning), please open an SR to replace the disks. Even the warning can lead to an error during the patch.
I recommend you to save outside of ODA some config files from BOTH nodes (to have quick access in case you need):
[root@odat1 ~]# mkdir /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/hosts /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/fstab /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/resolv.conf /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/sysconfig/network-scripts/* /nfs/ODA_PATCH_18/bkp_files_odat1/ cp: omitting directory `/etc/sysconfig/network-scripts/backupifcfgFiles' cp: omitting directory `/etc/sysconfig/network-scripts/bkupIfcfgOrig' [root@odat1 ~]# [root@odat1 ~]# cp /u01/app/18.0.0.0/grid/network/admin/listener* /nfs/ODA_PATCH_18/bkp_files_odat1/ [root@odat1 ~]#
Another detail before the start is to check the actual components and running versions. To do that we execute the odacli describe-component:
[root@odat1 ~]# odacli describe-component
System Version
---------------
18.3.0.0.0
System node Name
---------------
odat1
Local System Version
---------------
18.3.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 18.3.0.0.0 up-to-date
GI 18.3.0.0.180717 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 up-to-date
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 up-to-date
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 up-to-date
}
DCSAGENT 18.3.0.0.0 up-to-date
ILOM 4.0.2.26.b.r125868 up-to-date
BIOS 30130500 up-to-date
OS 6.10 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 13.00.00.00 up-to-date
}
FIRMWAREEXPANDER 0018 up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d15,c2d0,c2d15 ] PD51 pag1
[ c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,c1d7, PAG1 up-to-date
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,
c2d9,c2d10,c2d11,c2d12,c2d13,c2d14 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
System node Name
---------------
odat2
Local System Version
---------------
18.3.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 18.3.0.0.0 up-to-date
GI 18.3.0.0.180717 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 up-to-date
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 up-to-date
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 up-to-date
}
DCSAGENT 18.3.0.0.0 up-to-date
ILOM 4.0.2.26.b.r125868 up-to-date
BIOS 30130500 up-to-date
OS 6.10 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 13.00.00.00 up-to-date
}
FIRMWAREEXPANDER 0018 up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d15,c2d0,c2d15 ] PD51 pag1
[ c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,c1d7, PAG1 up-to-date
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,
c2d9,c2d10,c2d11,c2d12,c2d13,c2d14 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
[root@odat1 ~]#
Check above that some disks are running the firmware PD51. This occurs because they were changed after the 18.3 version and the firmware is newer than the one that exists at 18.3. If you have something that you want to update you can call the odacli update-storage to do the update. Remember that doing this the ODA nodes can reboot.
18.8
The first thing to do is download the patch ODA of 18.8 version using the patch 30518425. since you will have some reboot during the process, is required to stop the databases. I recommend stopping all databases running to speed up the process. This reduces the load over ASM and to stop/start GI (done internally during the process).
Another point is about HAIP. The HAIP needs to be disabled after you patch the 18.8. If you are coming from a reimaged 18.3 version (like me in this case), the patch process will disable HAIP automatically (since it was never configured), but if your 18.3 came because you already upgrade is from 12.x version, you need to manually disable it.
The process to disable it is not quite simple because evolve CRS and cluster definitions. I made this for Exadata/ZDLRA and you can check in this previous post. To ODA you can check this post. Or read the ODA documentation. Disable HAIP is done after patch to 18.8, and not before.
Update Repository
The first step is to upload the unzipped patch files at the internal ODA repository. This process is done using the odacli update-repository and you pass the full path to unzip patch files (here my files are in NFS folder):
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-repository -f /nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server1of2.zip,/nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server2of2.zip
{
"jobId" : "3368e5ba-f5da-4e07-af69-f9f51c29517e",
"status" : "Created",
"message" : "/nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server1of2.zip,/nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server2of2.zip",
"reports" : [ ],
"createTimestamp" : "July 25, 2020 10:10:35 AM CEST",
"resourceList" : [ ],
"description" : "Repository Update",
"updatedTime" : "July 25, 2020 10:10:35 AM CEST"
}
[root@odat1 18.8]#
This generates one jobId that you can follow until it ends using odacli describe-job:
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "3368e5ba-f5da-4e07-af69-f9f51c29517e"
Job details
----------------------------------------------------------------
ID: 3368e5ba-f5da-4e07-af69-f9f51c29517e
Description: Repository Update
Status: Success
Created: July 25, 2020 10:10:35 AM CEST
Message: /nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server1of2.zip,/nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server2of2.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace July 25, 2020 10:10:36 AM CEST July 25, 2020 10:10:36 AM CEST Success
Setting up ssh equivalance July 25, 2020 10:10:37 AM CEST July 25, 2020 10:10:37 AM CEST Success
Copy BundleFile July 25, 2020 10:10:38 AM CEST July 25, 2020 10:13:41 AM CEST Success
Validating CopiedFile July 25, 2020 10:13:41 AM CEST July 25, 2020 10:14:05 AM CEST Success
Unzip bundle July 25, 2020 10:14:05 AM CEST July 25, 2020 10:29:48 AM CEST Success
Unzip bundle July 25, 2020 10:29:48 AM CEST July 25, 2020 10:31:43 AM CEST Success
Delete PatchBundles July 25, 2020 10:31:43 AM CEST July 25, 2020 10:31:44 AM CEST Success
Removing ssh keys July 25, 2020 10:31:44 AM CEST July 25, 2020 10:53:37 AM CEST Success
[root@odat1 18.8]#
DCSAgent
The next step is to update the internal DCS agent for ODA 18.3. This is a simple step done using the odacli update-dcsagent. Check that the version 18.8.0.0 was specified as a parameter:
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 18.8.0.0.0
{
"jobId" : "38eed874-b2ab-4124-934a-e81c26376587",
"status" : "Created",
"message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
"reports" : [ ],
"createTimestamp" : "July 25, 2020 10:58:14 AM CEST",
"resourceList" : [ ],
"description" : "DcsAgent patching",
"updatedTime" : "July 25, 2020 10:58:14 AM CEST"
}
[root@odat1 18.8]#
And again, you can check the jobId and wait it finish:
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "38eed874-b2ab-4124-934a-e81c26376587"
Job details
----------------------------------------------------------------
ID: 38eed874-b2ab-4124-934a-e81c26376587
Description: DcsAgent patching
Status: Success
Created: July 25, 2020 10:58:14 AM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
dcs-agent upgrade July 25, 2020 10:58:15 AM CEST July 25, 2020 11:00:01 AM CEST Success
dcs-agent upgrade July 25, 2020 11:00:01 AM CEST July 25, 2020 11:01:35 AM CEST Success
Update System version July 25, 2020 11:01:37 AM CEST July 25, 2020 11:01:37 AM CEST Success
Update System version July 25, 2020 11:01:37 AM CEST July 25, 2020 11:01:37 AM CEST Success
[root@odat1 18.8]#
Don’t worry because this update will restart the DCS and for a moment the odacli will not work and can report errors (again, don’t worry, it returns after a little time):
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "38eed874-b2ab-4124-934a-e81c26376587" DCS-10001:Internal error encountered: Fail to get credential . [root@odat1 18.8]# [root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "38eed874-b2ab-4124-934a-e81c26376587" DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070. [root@odat1 18.8]#
DCSAdmin
The next step will be to update the dcsadmin using the command oracle update-dcsadmin:
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 18.8.0.0.0
{
"jobId" : "78ee826d-4266-4983-a7d8-269ccc959620",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "July 25, 2020 11:03:34 AM CEST",
"resourceList" : [ ],
"description" : "DcsAdmin patching",
"updatedTime" : "July 25, 2020 11:03:34 AM CEST"
}
[root@odat1 18.8]#
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "78ee826d-4266-4983-a7d8-269ccc959620"
Job details
----------------------------------------------------------------
ID: 78ee826d-4266-4983-a7d8-269ccc959620
Description: DcsAdmin patching
Status: Success
Created: July 25, 2020 11:03:34 AM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation July 25, 2020 11:03:36 AM CEST July 25, 2020 11:03:36 AM CEST Success
Patch location validation July 25, 2020 11:03:36 AM CEST July 25, 2020 11:03:36 AM CEST Success
dcs-admin upgrade July 25, 2020 11:03:37 AM CEST July 25, 2020 11:03:38 AM CEST Success
dcs-admin upgrade July 25, 2020 11:03:38 AM CEST July 25, 2020 11:03:39 AM CEST Success
Update System version July 25, 2020 11:03:40 AM CEST July 25, 2020 11:03:41 AM CEST Success
Update System version July 25, 2020 11:03:41 AM CEST July 25, 2020 11:03:41 AM CEST Success
[root@odat1 18.8]#
DCSComponents
The next is to update the dcscomponents with command odacli update-dcscomponents. And here one detail. The commands will generate one jobID, but will be invalid and you need to use the command “odacli list-jobs” to see the jobs (will be just updated SSH keys – two jobs):
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 18.8.0.0.0
{
"jobId" : "513d54f5-2f84-433d-8b60-8cdd705a29c5",
"status" : "Success",
"message" : null,
"reports" : null,
"createTimestamp" : "July 25, 2020 11:05:56 AM CEST",
"description" : "Job completed and is not part of Agent job list",
"updatedTime" : "July 25, 2020 11:05:56 AM CEST"
}
[root@odat1 18.8]#
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i 513d54f5-2f84-433d-8b60-8cdd705a29c5
DCS-10000:Resource Job with ID 513d54f5-2f84-433d-8b60-8cdd705a29c5 is not found.
[root@odat1 18.8]#
[root@odat1 18.8]# odacli list-jobs
ID Description Created Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
...
...
3368e5ba-f5da-4e07-af69-f9f51c29517e Repository Update July 25, 2020 10:10:35 AM CEST Success
38eed874-b2ab-4124-934a-e81c26376587 DcsAgent patching July 25, 2020 10:58:14 AM CEST Success
78ee826d-4266-4983-a7d8-269ccc959620 DcsAdmin patching July 25, 2020 11:03:34 AM CEST Success
bcaff693-05f5-4a47-b919-a7c40b4f5616 SSH keys update July 25, 2020 11:06:19 AM CEST Success
07e98fb0-a334-483f-a772-f838f48228bf SSH key delete July 25, 2020 11:06:24 AM CEST Success
[root@odat1 18.8]#
Create-prepatchreport
The next step is to verify if you can update it to 18.8 or no. The odacli will do a lot of tests for the operational system, GI, and ILOM. To do that, we just need to call odacli create-prepatchreport, and after that use the odacli describe-prepatchreport to check the report.
Some known issues can be checked before. One that I passed was the existence of /u01/app/oraInventory/locks. I needed to remove it from both nodes to have success at the report. Besides I receive:
__GI__
Validate supported GI versions Success Validated minimum supported versions
Validate available space Success Validated free space under /u01
Verify DB Home versions Success Verified DB Home versions
Validate patching locks Failed Internal error encountered:
/u01/app/oraInventory/locks.
[root@odat1 18.8]#
The call for patchreport:
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 18.8.0.0.0
Job details
----------------------------------------------------------------
ID: 3d7c2782-a044-4db2-a912-4e9587920438
Description: Patch pre-checks for [OS, ILOM, GI]
Status: Created
Created: July 25, 2020 11:25:39 AM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@odat1 18.8]#
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i 3d7c2782-a044-4db2-a912-4e9587920438
Patch pre-check report
------------------------------------------------------------------------
Job ID: 3d7c2782-a044-4db2-a912-4e9587920438
Description: Patch pre-checks for [OS, ILOM, GI]
Status: SUCCESS
Created: July 25, 2020 11:25:39 AM CEST
Result: All pre-checks succeeded
Node Name
---------------
odat1
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions
Validate patching tag Success Validated patching tag: 18.8.0.0.0
Is patch location available Success Patch location is available
Verify OS patch Success There are no packages available for
an update
__ILOM__
Validate supported versions Success Validated minimum supported versions
Validate patching tag Success Validated patching tag: 18.8.0.0.0
Is patch location available Success Patch location is available
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
__GI__
Validate supported GI versions Success Validated minimum supported versions
Validate available space Success Validated free space under /u01
Verify DB Home versions Success Verified DB Home versions
Validate patching locks Success Validated patching locks
Node Name
---------------
odat2
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions
Validate patching tag Success Validated patching tag: 18.8.0.0.0
Is patch location available Success Patch location is available
Verify OS patch Success There are no packages available for
an update
__ILOM__
Validate supported versions Success Validated minimum supported versions
Validate patching tag Success Validated patching tag: 18.8.0.0.0
Is patch location available Success Patch location is available
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
__GI__
Validate supported GI versions Success Validated minimum supported versions
Validate available space Success Validated free space under /u01
Verify DB Home versions Success Verified DB Home versions
Validate patching locks Success Validated patching locks
[root@odat1 18.8]#
With success results, we can continue the patch. Without that, don’t try to continue (besides if there is a known issue that can be ignored – we will talk about that later).
Stop TFA
Before continue is a requirement stop TFA in both nodes. This is needed because will be changed the version and upgraded to AHF. So, we call /etc/init.d/init.tfa stop in both nodes:
[root@odat1 18.8]# /etc/init.d/init.tfa stop Stopping TFA from init for shutdown/reboot oracle-tfa stop/waiting WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version. Nothing to do ! Killing TFA running with pid 20464 . . . Successfully stopped TFA.. [root@odat1 18.8]# ############################################################### #NODE 2 ############################################################### [root@odat2 ~]# /etc/init.d/init.tfa stop Stopping TFA from init for shutdown/reboot oracle-tfa stop/waiting WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version. Nothing to do ! Killing TFA running with pid 49871 . . . Successfully stopped TFA.. [root@odat2 ~]#
Update-server
After the success of the report, we can call the odacli update-server. And here the magic occurs, the O.S. is updated, GI, and ILOM too:
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-server -v 18.8.0.0.0
{
"jobId" : "5d27346e-1afd-4a1c-9e79-3e3b5d523244",
"status" : "Created",
"message" : "Success of Server Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
"reports" : [ ],
"createTimestamp" : "July 25, 2020 11:31:56 AM CEST",
"resourceList" : [ ],
"description" : "Server Patching",
"updatedTime" : "July 25, 2020 11:31:56 AM CEST"
}
[root@odat1 18.8]#
As you imagine, a jobIB is generated and we can follow it until it finishes:
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "5d27346e-1afd-4a1c-9e79-3e3b5d523244"
Job details
----------------------------------------------------------------
ID: 5d27346e-1afd-4a1c-9e79-3e3b5d523244
Description: Server Patching
Status: Success
Created: July 25, 2020 11:31:56 AM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation July 25, 2020 11:32:08 AM CEST July 25, 2020 11:32:08 AM CEST Success
Patch location validation July 25, 2020 11:32:08 AM CEST July 25, 2020 11:32:08 AM CEST Success
dcs-controller upgrade July 25, 2020 11:32:08 AM CEST July 25, 2020 11:32:13 AM CEST Success
dcs-controller upgrade July 25, 2020 11:32:13 AM CEST July 25, 2020 11:32:18 AM CEST Success
Patch location validation July 25, 2020 11:32:21 AM CEST July 25, 2020 11:32:21 AM CEST Success
Patch location validation July 25, 2020 11:32:21 AM CEST July 25, 2020 11:32:21 AM CEST Success
dcs-cli upgrade July 25, 2020 11:32:22 AM CEST July 25, 2020 11:32:23 AM CEST Success
dcs-cli upgrade July 25, 2020 11:32:23 AM CEST July 25, 2020 11:32:24 AM CEST Success
Creating repositories using yum July 25, 2020 11:32:33 AM CEST July 25, 2020 11:32:41 AM CEST Success
Creating repositories using yum July 25, 2020 11:32:41 AM CEST July 25, 2020 11:32:41 AM CEST Success
Creating repositories using yum July 25, 2020 11:32:42 AM CEST July 25, 2020 11:32:42 AM CEST Success
Creating repositories using yum July 25, 2020 11:32:42 AM CEST July 25, 2020 11:32:42 AM CEST Success
Creating repositories using yum July 25, 2020 11:32:43 AM CEST July 25, 2020 11:32:43 AM CEST Success
Creating repositories using yum July 25, 2020 11:32:43 AM CEST July 25, 2020 11:32:43 AM CEST Success
Creating repositories using yum July 25, 2020 11:32:43 AM CEST July 25, 2020 11:32:43 AM CEST Success
Updating YumPluginVersionLock rpm July 25, 2020 11:32:44 AM CEST July 25, 2020 11:32:44 AM CEST Success
Applying OS Patches July 25, 2020 11:32:45 AM CEST July 25, 2020 11:45:19 AM CEST Success
Creating repositories using yum July 25, 2020 11:45:20 AM CEST July 25, 2020 11:45:21 AM CEST Success
Applying HMP Patches July 25, 2020 11:45:21 AM CEST July 25, 2020 11:45:52 AM CEST Success
Patch location validation July 25, 2020 11:45:53 AM CEST July 25, 2020 11:45:53 AM CEST Success
Patch location validation July 25, 2020 11:45:53 AM CEST July 25, 2020 11:45:53 AM CEST Success
oda-hw-mgmt upgrade July 25, 2020 11:45:54 AM CEST July 25, 2020 11:46:25 AM CEST Success
oda-hw-mgmt upgrade July 25, 2020 11:46:25 AM CEST July 25, 2020 11:46:59 AM CEST Success
OSS Patching July 25, 2020 11:47:00 AM CEST July 25, 2020 11:47:00 AM CEST Success
Applying Firmware Disk Patches July 25, 2020 11:47:27 AM CEST July 25, 2020 11:47:42 AM CEST Success
Applying Firmware Expander Patches July 25, 2020 11:48:01 AM CEST July 25, 2020 11:48:13 AM CEST Success
Applying Firmware Controller Patches July 25, 2020 11:48:31 AM CEST July 25, 2020 11:48:41 AM CEST Success
Checking Ilom patch Version July 25, 2020 11:48:43 AM CEST July 25, 2020 11:48:46 AM CEST Success
Checking Ilom patch Version July 25, 2020 11:48:46 AM CEST July 25, 2020 11:48:48 AM CEST Success
Patch location validation July 25, 2020 11:48:48 AM CEST July 25, 2020 11:48:49 AM CEST Success
Patch location validation July 25, 2020 11:48:48 AM CEST July 25, 2020 11:48:49 AM CEST Success
Save password in Wallet July 25, 2020 11:48:51 AM CEST July 25, 2020 11:48:51 AM CEST Success
Apply Ilom patch July 25, 2020 11:48:52 AM CEST July 25, 2020 12:01:47 PM CEST Success
Apply Ilom patch July 25, 2020 12:01:47 PM CEST July 25, 2020 12:14:11 PM CEST Success
Copying Flash Bios to Temp location July 25, 2020 12:14:11 PM CEST July 25, 2020 12:14:12 PM CEST Success
Copying Flash Bios to Temp location July 25, 2020 12:14:12 PM CEST July 25, 2020 12:14:12 PM CEST Success
Starting the clusterware July 25, 2020 12:15:57 PM CEST July 25, 2020 12:17:24 PM CEST Success
clusterware patch verification July 25, 2020 12:24:48 PM CEST July 25, 2020 12:24:50 PM CEST Success
clusterware patch verification July 25, 2020 12:24:48 PM CEST July 25, 2020 12:24:50 PM CEST Success
Patch location validation July 25, 2020 12:24:50 PM CEST July 25, 2020 12:25:03 PM CEST Success
Patch location validation July 25, 2020 12:24:50 PM CEST July 25, 2020 12:25:03 PM CEST Success
Opatch updation July 25, 2020 12:25:48 PM CEST July 25, 2020 12:25:53 PM CEST Success
Opatch updation July 25, 2020 12:25:48 PM CEST July 25, 2020 12:25:54 PM CEST Success
Patch conflict check July 25, 2020 12:25:54 PM CEST July 25, 2020 12:27:04 PM CEST Success
Patch conflict check July 25, 2020 12:27:07 PM CEST July 25, 2020 12:28:23 PM CEST Success
clusterware upgrade July 25, 2020 12:28:24 PM CEST July 25, 2020 12:50:42 PM CEST Success
clusterware upgrade July 25, 2020 12:50:42 PM CEST July 25, 2020 1:15:15 PM CEST Success
Updating GiHome version July 25, 2020 1:15:16 PM CEST July 25, 2020 1:15:21 PM CEST Success
Updating GiHome version July 25, 2020 1:15:16 PM CEST July 25, 2020 1:15:21 PM CEST Success
Update System version July 25, 2020 1:16:21 PM CEST July 25, 2020 1:16:21 PM CEST Success
Update System version July 25, 2020 1:16:22 PM CEST July 25, 2020 1:16:22 PM CEST Success
preRebootNode Actions July 25, 2020 1:16:22 PM CEST July 25, 2020 1:17:04 PM CEST Success
preRebootNode Actions July 25, 2020 1:17:04 PM CEST July 25, 2020 1:17:45 PM CEST Success
Reboot Ilom July 25, 2020 1:17:46 PM CEST July 25, 2020 1:17:46 PM CEST Success
Reboot Ilom July 25, 2020 1:17:46 PM CEST July 25, 2020 1:17:46 PM CEST Success
[root@odat1 18.8]#
In ODAX5-2 this process took al most two hours. And passed all the updates. If you follow the /opt/oracle/dcs/log/dcs-agent.log you can see more detail like:
... ... 2020-07-25 12:20:14,275 INFO [Disable haip feature : JobId=5d27346e-1afd-4a1c-9e79-3e3b5d523244] [] c.o.d.a.z.DCSZQueue: Node not yet received for: /nodes/node_1/cmd-out-q with prefix:323_5973_ 2020-07-25 12:20:16,276 INFO [Disable haip feature : JobId=5d27346e-1afd-4a1c-9e79-3e3b5d523244] [] c.o.d.a.z.DCSZQueue: Node not yet received for: /nodes/node_1/cmd-out-q with prefix:323_5973_ 2020-07-25 12:20:18,277 INFO [Disable haip feature : JobId=5d27346e-1afd-4a1c-9e79-3e3b5d523244] [] c.o.d.a.z.DCSZQueue: Node not yet received for: /nodes/node_1/cmd-out-q with prefix:323_5973_ ... ... 2020-07-25 12:25:54,991 DEBUG [Patch conflict check : JobId=5d27346e-1afd-4a1c-9e79-3e3b5d523244] [] c.o.d.c.u.CommonsUtils: run: cmd= '[/u01/app/18.0.0.0/grid/OPatch/opatchauto, apply, /u01/patching/18.8.0.0.191015/30518456, -analyze, -oh, /u01/app/18.0.0.0/grid, -log, /tmp/opatchAutoAnalyzePatch.log, -inplace]' ... ...
After this jobID finishes, the ODA will reboot (between the end of the job and reboot can be around 10 minutes).
Update-storage
The last step is to patch the storage with command odacli update-storage. This is needed because the previous step does not updates the firmware of the disks as an example.
You can see from odacli describe-component that some part are not up to date:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
18.8.0.0.0
System node Name
---------------
odat1
Local System Version
---------------
18.8.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 18.8.0.0.0 up-to-date
GI 18.8.0.0.191015 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.8.0.0.191015
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.191015
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.191015
}
DCSAGENT 18.8.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 6.10 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 13.00.00.00 up-to-date
}
FIRMWAREEXPANDER 0018 001e
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d15,c2d0,c2d15 ] PD51 up-to-date
[ c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,c1d7, PAG1 pd51
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,
c2d9,c2d10,c2d11,c2d12,c2d13,c2d14 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
System node Name
---------------
odat2
Local System Version
---------------
18.8.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 18.8.0.0.0 up-to-date
GI 18.8.0.0.191015 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.8.0.0.191015
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.191015
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.191015
}
DCSAGENT 18.8.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 6.10 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 13.00.00.00 up-to-date
}
FIRMWAREEXPANDER 0018 001e
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d15,c2d0,c2d15 ] PD51 up-to-date
[ c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,c1d7, PAG1 pd51
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,
c2d9,c2d10,c2d11,c2d12,c2d13,c2d14 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
[root@odat1 ~]#
And we can update calling the odacli update-storage:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-storage -v 18.8.0.0.0
{
"jobId" : "06bbdc8d-d7d2-4400-82cc-a3345ceca28d",
"status" : "Created",
"message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
"reports" : [ ],
"createTimestamp" : "July 25, 2020 14:35:15 PM CEST",
"resourceList" : [ ],
"description" : "Storage Firmware Patching",
"updatedTime" : "July 25, 2020 14:35:15 PM CEST"
}
[root@odat1 ~]#
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "06bbdc8d-d7d2-4400-82cc-a3345ceca28d"
Job details
----------------------------------------------------------------
ID: 06bbdc8d-d7d2-4400-82cc-a3345ceca28d
Description: Storage Firmware Patching
Status: Success
Created: July 25, 2020 2:35:15 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches July 25, 2020 2:49:02 PM CEST July 25, 2020 2:49:25 PM CEST Success
Applying Firmware Expander Patches July 25, 2020 2:53:03 PM CEST July 25, 2020 2:56:38 PM CEST Success
Applying Firmware Controller Patches July 25, 2020 2:56:51 PM CEST July 25, 2020 2:57:08 PM CEST Success
preRebootNode Actions July 25, 2020 2:57:09 PM CEST July 25, 2020 2:57:09 PM CEST Success
preRebootNode Actions July 25, 2020 2:57:09 PM CEST July 25, 2020 2:57:09 PM CEST Success
Reboot Ilom July 25, 2020 2:57:09 PM CEST July 25, 2020 2:57:09 PM CEST Success
Reboot Ilom July 25, 2020 2:57:10 PM CEST July 25, 2020 2:57:10 PM CEST Success
[root@odat1 ~]#
After the job finish (and after around 10 minutes) the ODA will reboot automatically.
And you can see that now that everything is up to date:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
18.8.0.0.0
System node Name
---------------
odat1
Local System Version
---------------
18.8.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 18.8.0.0.0 up-to-date
GI 18.8.0.0.191015 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.8.0.0.191015
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.191015
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.191015
}
DCSAGENT 18.8.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 6.10 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 13.00.00.00 up-to-date
}
FIRMWAREEXPANDER 001E up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
System node Name
---------------
odat2
Local System Version
---------------
18.8.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 18.8.0.0.0 up-to-date
GI 18.8.0.0.191015 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.8.0.0.191015
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.191015
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.191015
}
DCSAGENT 18.8.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 6.10 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 13.00.00.00 up-to-date
}
FIRMWAREEXPANDER 001E up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
[root@odat1 ~]#
If you check the log, you can see that firmware was updated:
2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.r.s.p.PatchingOperations: Firmware Disk Patch File---->/opt/oracle/oak/pkgrepos/firmwaredisk/hgst/h7280a520sun8.0t/pd51/H7280A520.PD51.fw 2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: StorageUtils::isBRCMProduct 2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: Firmware version to be applied---->PD51 2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: Storage Component Current Firmware Version---->PAG1 2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: Known Firmware Versions------>P554,P901,P9E2,PAG1,PD51 2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: StorageUtils::isBRCMProduct 2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: Storage Component Patch Applicable---->c2d3 2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.r.s.g.GiUtils: GiUtils:isCrsRunning 2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.c.u.CommonsUtils:
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”
Pingback: Patch ODA from 18.3 to 19.8. Part 2 – 18.8 to 19.6 | Fernando Simon
Hi,
Great blog here, may i know if the steps are the same if i upgrade from 18.5 to 18.8. Can i do it directly or need to upgrade to 18.6, 18.7 incrementally? How about 19.6 to 19.8, bypassing 19.7. I think its within the matrix of oda bare metal.
Hi,
I think that it is possible, go from where you are (18.5) directly to the target version (18.8).
Same for 19.
Best regards.
Fernando Simon