On 08/March/2023 the Oracle Exadata team released version 23.1.0.0.0 and this include a significant change, OEL 8. I already explained that in my first post that you can read here. In my second post, I wrote about how to patch storage and switch. In this post, I will discuss how to patch the dom0.
What you can do
Due to the changes for OEL 8, is important to understand the upgrade paths that you can take. As I wrote in my first post: If you are running the old Exadata with InfiniBand, your dom0 will always be updated until Oracle Linux 7 with UEK5. For domU you can upgrade to the OEL 8. And you can upgrade in any order, first dom0 or domU. If you are running RoCE, your dom0 can run the latest OEL 8 UEK6. The blog post from Oracle made an excellent explanation about the upgrade paths and below you can see the images that are there (I used the image from their post).
So, since the environment that I am patching is Exadata with InfiniBand, my dom0 will be upgraded until the OEL7 running the UEK5. But the Exadata-related software will be upgraded to version 23.1. The domU will be upgraded to OEL8, with UEK6. So, basically will be this (I used the image from the Exadata Team post):
Here, I patched first the dom0 because if I patch it first, all the versions already released for domU will be compatible with him. I am upgrading, so, the dom0 running at 23.1 will be compatible with domU running at a lower version.
Patching dom0
Dom0 is the “physical” place where your domU’s run and the patch process execute two basic things: 1 – patch the local Linux server and 2 – patch the ILOM for the physical layer. So, your domU’s needs to be stopped. Besides that, you need to check if the HW layer is fine and without errors.
One additional detail for patching dom0 is that you can’t patch them from themselves. So, you need an additional machine to do that, and for this reason, you need ssh keyless/passwordless to do the patch. Here for this appliance, I have two database nodes, so, one will path the other. If you have a bigger Exadata, you can call the patch (from one machine) to patch all others (like 7 at the same time). So, here I created one file (in each dom0) that points to another (is just a file with the hostname inside). So, you will need/have downtime for domU’s while patching the dom0. For this reason, do the planning correctly (doing communication with all other teams that use the databases). Is important to remind you of that.
So, the first step is to check the dom0 that will be patched for HW errors (before and after the reset ILOM – you can do this remove if you want):
[DOM0 - root@exadom0m01 ~]$ dcli -l root -g /root/exadom0m02 dbmcli -e "list alerthistory" [DOM0 - root@exadom0m01 ~]$ [DOM0 - root@exadom0m01 ~]$ dcli -l root -g /root/exadom0m02 "ipmitool bmc reset cold" exadom0m02.simon.mynet.lu: Sent cold reset command to MC [DOM0 - root@exadom0m01 ~]$ [DOM0 - root@exadom0m01 ~]$ dcli -l root -g /root/exadom0m02 dbmcli -e "list alerthistory" [DOM0 - root@exadom0m01 ~]$
After checking that there are no HW errors, we can stop all the running domU’s at the server that will be patched. Nothing is running here (I already called the xm destroy):
[DOM0 - root@exadom0m02 ~]$ xm list Name ID Mem VCPUs State Time(s) Domain-0 0 9216 4 r----- 30938.3 [DOM0 - root@exadom0m02 ~]$
As you can see, the dom0 that I will patch already is running at 22.1.9.0:
[DOM0 - root@exadom0m01 ~]$ dcli -l root -g /root/exadom0m02 imageinfo exadom0m02: exadom0m02: Kernel version: 4.14.35-2047.518.4.2.el7uek.x86_64 #2 SMP Thu Nov 3 14:28:31 PDT 2022 x86_64 exadom0m02: Uptrack kernel version: 4.14.35-2047.522.3.el7uek.x86_64 #2 SMP Fri Jan 20 16:05:02 PST 2023 x86_64 exadom0m02: Image kernel version: 4.14.35-2047.518.4.2.el7uek exadom0m02: Image version: 22.1.9.0.0.230302 exadom0m02: Image activated: 2023-03-09 17:36:48 +0100 exadom0m02: Image status: success exadom0m02: Node type: DOM0 exadom0m02: System partition on device: /dev/mapper/VGExaDb-LVDbSys3 exadom0m02: [DOM0 - root@exadom0m01 ~]$
This meets the requirements described in MOS Note 2772585.1 and the Exadata Team Post. If you are not running the required version you can use this same post and patch your dom0, you just need to change the ISO for the patch that you are installing. Everything else is the same.
To patch we need to have two files: 1 – the patchmgr (from Patch 21634633 – Please read note 1553103.1 if you have doubts), and 2 – the ISO/patch itself (Patch 32829545). This is important because the patch for 23.1 is just an ISO from the YUM channel, and it does not come with any patchmgr inside (like the switch/storage server patch).
So, we download and unzip patch 21634633:
[DOM0 - root@exadom0m01 ~]$ [DOM0 - root@exadom0m01 ~]$ cd /EXAVMIMAGES/patches/patchmgr/ [DOM0 - root@exadom0m01 patchmgr]$ [DOM0 - root@exadom0m01 patchmgr]$ unzip -q p21634633_231000_Linux-x86-64.zip [DOM0 - root@exadom0m01 patchmgr]$ [DOM0 - root@exadom0m01 patchmgr]$ ls -l total 553988 drwxrwxr-x 3 root root 3896 Feb 26 00:56 dbserver_patch_230225 -rw-r--r-- 1 root root 562465570 Mar 9 10:11 p21634633_231000_Linux-x86-64.zip -rw-r--r-- 1 root root 3445358 Mar 9 17:46 screenlog.0 [DOM0 - root@exadom0m01 patchmgr]$
After that we can do the precheck for the server that we want to patch and the target version:
[DOM0 - root@exadom0m01 patchmgr]$ ./dbserver_patch_230225/patchmgr --dbnodes /root/exadom0m02 --precheck --iso_repo /EXAVMIMAGES/patches/23.1.0/dom0/p32829547_231000_Linux-x86-64.zip --target_version 23.1.0.0.0.230225.1 ************************************************************************************************************ NOTE patchmgr release: 23.230225 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip) NOTE WARNING Do not interrupt the patchmgr session. WARNING Do not resize the screen. It may disturb the screen layout. WARNING Do not reboot database nodes during update or rollback. WARNING Do not open logfiles in write mode and do not try to alter them. ************************************************************************************************************ 2023-03-10 09:52:29 +0100 :INFO : Checking hosts connectivity via ICMP/ping 2023-03-10 09:52:31 +0100 :INFO : Hosts Reachable: [exadom0m02] 2023-03-10 09:52:31 +0100 :INFO : All hosts are reachable via ping/ICMP 2023-03-10 09:52:31 +0100 :Working: Verify SSH equivalence for the root user to exadom0m02 2023-03-10 09:52:33 +0100 :INFO : SSH equivalency verified to host exadom0m02 2023-03-10 09:52:33 +0100 :SUCCESS: Verify SSH equivalence for the root user to exadom0m02 2023-03-10 09:52:38 +0100 :Working: Initiate precheck on 1 node(s) 2023-03-10 09:52:54 +0100 :Working: Check for enough free space on exadom0m02 to transfer and unzip files. 2023-03-10 09:52:58 +0100 :SUCCESS: Check for enough free space on exadom0m02 to transfer and unzip files. 2023-03-10 09:53:01 +0100 :INFO : Preparing nodes: exadom0m02 2023-03-10 09:53:36 +0100 :Working: dbnodeupdate.sh running a precheck for the conventional update on node(s). 2023-03-10 09:57:52 +0100 :SUCCESS: Initiate precheck on node(s). 2023-03-10 09:57:55 +0100 :SUCCESS: Completed run of command: ./dbserver_patch_230225/patchmgr -dbnodes /root/exadom0m02 --precheck --iso_repo /EXAVMIMAGES/patches/23.1.0/dom0/p32829547_231000_Linux-x86-64.zip --target_version 23.1.0.0.0.230225.1 2023-03-10 09:57:55 +0100 :INFO : Precheck performed on dbnode(s) in file /root/exadom0m02: [exadom0m02] 2023-03-10 09:57:55 +0100 :INFO : Current image version on dbnode(s) is: 2023-03-10 09:57:55 +0100 :INFO : exadom0m02: 22.1.9.0.0.230302 2023-03-10 09:57:55 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/patches/patchmgr/dbserver_patch_230225: 2023-03-10 09:57:55 +0100 :INFO : - <dbnode_name>_dbnodeupdate.log 2023-03-10 09:57:55 +0100 :INFO : - patchmgr.log 2023-03-10 09:57:55 +0100 :INFO : - patchmgr.trc 2023-03-10 09:57:58 +0100 :INFO : Collected dbnodeupdate diag in file: Diag_patchmgr_dbnode_precheck_100323095218.tbz 2023-03-10 09:57:58 +0100 :INFO : Exit status:0 2023-03-10 09:57:58 +0100 :INFO : Exiting. [DOM0 - root@exadom0m01 patchmgr]$
Look at the parameters above. The server that will be patched is defined by the parameter “–dbnodes”, the action is the “–precheck”, the ISO used to patch is defined by “–iso_repo”, and the target version by “–target_version”. The target version you can find in note 888828.1, every patch for Storage, DBNodes, dom0, and domU has a description for the version that it represents.
Above, you can see that everything is fine, and we can call the patch, not calling the patchmgr with the “–upgrade” option:
[DOM0 - root@exadom0m01 patchmgr]$ pwd /EXAVMIMAGES/patches/patchmgr [DOM0 - root@exadom0m01 patchmgr]$ [DOM0 - root@exadom0m01 patchmgr]$ screen -L -RR Patch-DOM0 [DOM0 - root@exadom0m01 patchmgr]$ [DOM0 - root@exadom0m01 patchmgr]$ ./dbserver_patch_230225/patchmgr --dbnodes /root/exadom0m02 --upgrade --iso_repo /EXAVMIMAGES/patches/23.1.0/dom0/p32829547_231000_Linux-x86-64.zip --target_version 23.1.0.0.0.230225.1 ************************************************************************************************************ NOTE patchmgr release: 23.230225 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip) NOTE NOTE Database nodes will reboot during the update process. NOTE WARNING Do not interrupt the patchmgr session. WARNING Do not resize the screen. It may disturb the screen layout. WARNING Do not reboot database nodes during update or rollback. WARNING Do not open logfiles in write mode and do not try to alter them. ************************************************************************************************************ 2023-03-10 10:02:14 +0100 :INFO : Checking hosts connectivity via ICMP/ping 2023-03-10 10:02:15 +0100 :INFO : Hosts Reachable: [exadom0m02] 2023-03-10 10:02:15 +0100 :INFO : All hosts are reachable via ping/ICMP 2023-03-10 10:02:15 +0100 :Working: Verify SSH equivalence for the root user to exadom0m02 2023-03-10 10:02:17 +0100 :INFO : SSH equivalency verified to host exadom0m02 2023-03-10 10:02:17 +0100 :SUCCESS: Verify SSH equivalence for the root user to exadom0m02 2023-03-10 10:02:23 +0100 :Working: Initiate prepare steps on node(s). 2023-03-10 10:02:26 +0100 :Working: Check for enough free space on exadom0m02 to transfer and unzip files. 2023-03-10 10:02:30 +0100 :SUCCESS: Check for enough free space on exadom0m02 to transfer and unzip files. 2023-03-10 10:02:33 +0100 :INFO : Preparing nodes: exadom0m02 2023-03-10 10:03:07 +0100 :SUCCESS: Initiate prepare steps on node(s). 2023-03-10 10:03:07 +0100 :Working: Initiate update on 1 node(s). 2023-03-10 10:03:10 +0100 :Working: dbnodeupdate.sh running a backup on 1 node(s). 2023-03-10 10:09:06 +0100 :SUCCESS: dbnodeupdate.sh running a backup on 1 node(s). 2023-03-10 10:09:06 +0100 :Working: Initiate update on node(s) 2023-03-10 10:09:06 +0100 :Working: Get information about any required OS upgrades from exadom0m02. 2023-03-10 10:09:17 +0100 :SUCCESS: Get information about any required OS upgrades from exadom0m02. 2023-03-10 10:09:18 +0100 :Working: dbnodeupdate.sh running an update step on exadom0m02. 2023-03-10 10:22:19 +0100 :INFO : exadom0m02 is ready to reboot. 2023-03-10 10:22:19 +0100 :SUCCESS: dbnodeupdate.sh running an update step on exadom0m02. 2023-03-10 10:22:47 +0100 :Working: Initiate reboot on exadom0m02. 2023-03-10 10:23:34 +0100 :SUCCESS: Initiate reboot on exadom0m02. 2023-03-10 10:23:34 +0100 :Working: Waiting to ensure exadom0m02 is down before reboot. 2023-03-10 10:23:47 +0100 :SUCCESS: Waiting to ensure exadom0m02 is down before reboot. 2023-03-10 10:23:47 +0100 :Working: Waiting to ensure exadom0m02 is up after reboot. 2023-03-10 10:31:08 +0100 :SUCCESS: Waiting to ensure exadom0m02 is up after reboot. 2023-03-10 10:31:08 +0100 :Working: Waiting to connect to exadom0m02 with SSH. During Linux upgrades this can take some time. 2023-03-10 10:31:50 +0100 :SUCCESS: Waiting to connect to exadom0m02 with SSH. During Linux upgrades this can take some time. 2023-03-10 10:31:50 +0100 :Working: Wait for exadom0m02 is ready for the completion step of update. 2023-03-10 10:31:51 +0100 :SUCCESS: Wait for exadom0m02 is ready for the completion step of update. 2023-03-10 10:31:52 +0100 :Working: Initiate completion step from dbnodeupdate.sh on exadom0m02 2023-03-10 10:35:31 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on exadom0m02. 2023-03-10 10:36:30 +0100 :SUCCESS: Initiate update on node(s). 2023-03-10 10:36:30 +0100 :SUCCESS: Initiate update on all node(s) 2023-03-10 10:36:31 +0100 :SUCCESS: Initiate update on 1 node(s). 2023-03-10 10:36:34 +0100 :SUCCESS: Completed run of command: ./dbserver_patch_230225/patchmgr --dbnodes /root/exadom0m02 --upgrade --iso_repo /EXAVMIMAGES/patches/23.1.0/dom0/p32829547_231000_Linux-x86-64.zip --target_version 23.1.0.0.0.230225.1 2023-03-10 10:36:34 +0100 :INFO : Upgrade performed on dbnode(s) in file /root/exadom0m02: [exadom0m02] 2023-03-10 10:36:34 +0100 :INFO : Current image version on dbnode(s) is: 2023-03-10 10:36:34 +0100 :INFO : exadom0m02: 23.1.0.0.0.230225.1 2023-03-10 10:36:34 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/patches/patchmgr/dbserver_patch_230225: 2023-03-10 10:36:34 +0100 :INFO : - <dbnode_name>_dbnodeupdate.log 2023-03-10 10:36:34 +0100 :INFO : - patchmgr.log 2023-03-10 10:36:34 +0100 :INFO : - patchmgr.trc 2023-03-10 10:36:55 +0100 :INFO : Collected dbnodeupdate diag in file: Diag_patchmgr_dbnode_upgrade_100323100200.tbz 2023-03-10 10:36:55 +0100 :INFO : Exit status:0 2023-03-10 10:36:55 +0100 :INFO : Exiting. [DOM0 - root@exadom0m01 patchmgr]$
As you can see above, everything was fine the target dom0 was patched. As you can see, I called it inside screen session. So, in case of network failure for my ssh session, it continues to patch.
After the patch is ok, I recommend you to start any domU inside at patched dom0 to check if everything is fine (here is OVM/Xen, so is xm command):
[DOM0 - root@exadom0m02 ~]$ xm list Name ID Mem VCPUs State Time(s) Domain-0 0 9216 4 r----- 897.2 [DOM0 - root@exadom0m02 ~]$ [DOM0 - root@exadom0m02 ~]$ cd /EXAVMIMAGES/GuestImages/exadU2m04.simon.mynet.lu/ [DOM0 - root@exadom0m02 exadU2m04.simon.mynet.lu]$ [DOM0 - root@exadom0m02 exadU2m04.simon.mynet.lu]$ xm create vm.cfg Using config file "./vm.cfg". Started domain exadU2m04.simon.mynet.lu (id=1) [DOM0 - root@exadom0m02 exadU2m04.simon.mynet.lu]$
If the startup was fine and all databases started, we can call the cleanup for the dom0 because we don’t need tempfiles/traces anymore.
[DOM0 - root@exadom0m01 patchmgr]$ ./dbserver_patch_230225/patchmgr --dbnodes /root/exadom0m02 --cleanup --iso_repo /EXAVMIMAGES/patches/23.1.0/dom0/p32829547_231000_Linux-x86-64.zip --target_version 23.1.0.0.0.230225.1 ************************************************************************************************************ NOTE patchmgr release: 23.230225 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip) NOTE WARNING Do not interrupt the patchmgr session. WARNING Do not resize the screen. It may disturb the screen layout. WARNING Do not reboot database nodes during update or rollback. WARNING Do not open logfiles in write mode and do not try to alter them. ************************************************************************************************************ 2023-03-10 10:39:38 +0100 :Working: Verify SSH equivalence for the root user to exadom0m02 2023-03-10 10:39:39 +0100 :INFO : SSH equivalency verified to host exadom0m02 2023-03-10 10:39:40 +0100 :SUCCESS: Verify SSH equivalence for the root user to exadom0m02 2023-03-10 10:39:46 +0100 :Working: Initiate cleanup on 1 node(s). 2023-03-10 10:39:46 +0100 :Working: Check for enough free space on exadom0m02 to transfer and unzip files. 2023-03-10 10:39:50 +0100 :SUCCESS: Check for enough free space on exadom0m02 to transfer and unzip files. 2023-03-10 10:41:32 +0100 :SUCCESS: Initiate cleanup on 1 node(s). 2023-03-10 10:41:36 +0100 :SUCCESS: Completed run of command: ./dbserver_patch_230225/patchmgr --dbnodes /root/exadom0m02 --cleanup --iso_repo /EXAVMIMAGES/patches/23.1.0/dom0/p32829547_231000_Linux-x86-64.zip --target_version 23.1.0.0.0.230225.1 2023-03-10 10:41:36 +0100 :INFO : Cleanup performed on dbnode(s) in file /root/exadom0m02: [exadom0m02] 2023-03-10 10:41:36 +0100 :INFO : Current image version on dbnode(s) is: 2023-03-10 10:41:36 +0100 :INFO : exadom0m02: 23.1.0.0.0.230225.1 2023-03-10 10:41:36 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/patches/patchmgr/dbserver_patch_230225: 2023-03-10 10:41:36 +0100 :INFO : - <dbnode_name>_dbnodeupdate.log 2023-03-10 10:41:36 +0100 :INFO : - patchmgr.log 2023-03-10 10:41:36 +0100 :INFO : - patchmgr.trc 2023-03-10 10:41:36 +0100 :INFO : Exit status:0 2023-03-10 10:41:36 +0100 :INFO : Exiting. [DOM0 - root@exadom0m01 patchmgr]$
Is the same patchmgr command with the “–cleanup” option.
So, we reached the goal, dom0 patched to the 23.1 version:
[DOM0 - root@exadom0m01 patchmgr]$ dcli -l root -g /root/exadom0m02 imageinfo exadom0m02: exadom0m02: Kernel version: 4.14.35-2047.518.4.2.el7uek.x86_64 #2 SMP Thu Nov 3 14:28:31 PDT 2022 x86_64 exadom0m02: Uptrack kernel version: 4.14.35-2047.522.3.el7uek.x86_64 #2 SMP Fri Jan 20 16:05:02 PST 2023 x86_64 exadom0m02: Image kernel version: 4.14.35-2047.518.4.2.el7uek exadom0m02: Image version: 23.1.0.0.0.230225.1 exadom0m02: Image activated: 2023-03-10 10:33:12 +0100 exadom0m02: Image status: success exadom0m02: Exadata software version: 22.1.9.0.0.230302 exadom0m02: Node type: DOM0 exadom0m02: System partition on device: /dev/mapper/VGExaDb-LVDbSys3 exadom0m02: [DOM0 - root@exadom0m01 patchmgr]$ [DOM0 - root@exadom0m02 ~]$ [DOM0 - root@exadom0m02 ~]$ imagehistory Version : 12.1.2.3.3.161109 Image activation date : 2016-11-25 11:26:42 +0100 Imaging mode : fresh Imaging status : success Version : 18.1.6.0.0.180529 Image activation date : 2018-07-23 17:55:58 +0200 Imaging mode : patch Imaging status : success Version : 19.2.7.0.0.191012 Image activation date : 2020-03-11 10:35:23 +0100 Imaging mode : patch Imaging status : success Version : 19.2.19.0.0.201013 Image activation date : 2020-11-26 15:43:10 +0100 Imaging mode : patch Imaging status : success Version : 20.1.8.0.0.210317 Image activation date : 2021-04-22 11:38:32 +0200 Imaging mode : patch Imaging status : success Version : 22.1.9.0.0.230302 Image activation date : 2023-03-09 17:36:48 +0100 Imaging mode : patch Imaging status : success Version : 23.1.0.0.0.230225.1 Image activation date : 2023-03-10 10:33:12 +0100 Imaging mode : patch Imaging status : success [DOM0 - root@exadom0m02 ~]
So, now you can call the patches for other dom0 that you need. You use exactly the same commands as described above, just need to change the file with the hostname list.
If during the patch (precheck or even upgrade phases) you got an error related to DSS SSH keys, please read my second post of this series. There I explain how to fix the issue.
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purposes, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”