Starting with Grid Infrastructure/ASM 19.16 Oracle changed how the REQUIRED_MIRROR_FREE_MB is calculated and the impact is more than expected. Check below examples of the changes, and how this will impact you. This is valid for all GI/ASM starting with 19.16 and only for Exadata/ExaCC.
Please read my new post about this issue.
REQUIRED_MIRROR_FREE_MB
The REQUIRED_MIRROR_FREE_MB (according to 19c documentation) is:
“amount of space that must be available in a disk group to restore full redundancy after the worst failure that can be tolerated by the disk group without adding additional storage. This requirement ensures that there are sufficient failure groups to restore redundancy”.
And (at Exadata environment until 19.16) is calculated based on the disk redundancy that you have. If you choose the HIGH, the raw size of two disks (the largest in your diskgroup) is reserved; at NORMAL, is the raw size of one disk. At Exadata, it differs from other environments because does not consider the whole failgroup failure and the way that extends are written/spread (more info below and in another post).
But for now, understand that the required size is what you need to reserve (as raw space) at your diskgroup to ensure protection in case of disk failure. And it is directly related to the USABLE_FILE_MB because the space that you can allocate at your diskgroup (USABLE_FILE_MB) comes from (FREE_MB- REQUIRED_MIRROR_FREE_MB)/redundancy factor (3 for HIGH, 2 for NORMAL). So, when you increase the REQUIRED_MIRROR_FREE_MB you reduce the USABLE_FILE_MB. I will explain more later.
Before GI/ASM 19.16
Below is one example of one Exadata running GI 19.14. In this example I have:
- 5 cells
- 12 disks per cell
- Disk size 1200GB (almost 1.2TB)
- HIGH redundancy for diskgroup DATAC8
You can see this here:
[grid@exasivm08 +ASM1]$ asmcmd
ASMCMD>
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED HIGH N 512 512 4096 4194304 73728000 73724256 2457600 23755552 0 Y DATAC8/
MOUNTED NORMAL N 512 512 4096 4194304 6144000 6093368 102400 2995484 0 N RECOC8/
ASMCMD>
ASMCMD> exit
[grid@exasivm08 +ASM1]$
[grid@exasivm08 +ASM1]$ sqlplus / as sysasm
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 3 14:12:16 2022
Version 19.14.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.14.0.0.0
SQL> set linesize 255
SQL> SELECT name, type, total_mb, free_mb, required_mirror_free_mb, usable_file_mb FROM V$ASM_DISKGROUP;
NAME TYPE TOTAL_MB FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------------------------ ------ ---------- ---------- ----------------------- --------------
DATAC8 HIGH 73728000 73724256 2457600 23755552
RECOC8 NORMAL 6144000 6093368 102400 2995484
SQL> select GROUP_NUMBER, NAME, TOTAL_MB, FREE_MB, HOT_USED_MB, COLD_USED_MB, REQUIRED_MIRROR_FREE_MB, USABLE_FILE_MB from v$asm_diskgroup;
GROUP_NUMBER NAME TOTAL_MB FREE_MB HOT_USED_MB COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------ ------------------------------ ---------- ---------- ----------- ------------ ----------------------- --------------
1 DATAC8 73728000 73724256 0 3744 2457600 23755552
2 RECOC8 6144000 6093368 0 50632 102400 2995484
SQL> select max(TOTAL_MB) from v$asm_disk_stat where GROUP_NUMBER=1;
MAX(TOTAL_MB)
-------------
1228800
SQL> select max(TOTAL_MB), max(TOTAL_MB)/1024 from v$asm_disk_stat where GROUP_NUMBER=1;
MAX(TOTAL_MB) MAX(TOTAL_MB)/1024
------------- ------------------
1228800 1200
SQL>
SQL> l
1 select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks
2 from v$asm_disk d, v$asm_diskgroup dg
3 where d.group_number = dg.group_number
4 and dg.name IN ('DATAC8')
5 group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status
6* order by 1,2,3
SQL> /
NAME FAILGROUP STATE HEADER_STATU MOUNT_S MODE_ST NUM_DISKS
------ ------------------------------ -------- ------------ ------- ------- ----------
DATAC8 EXACELBRM01 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELBRM02 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELBRM03 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELBRM04 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELBRM05 NORMAL MEMBER CACHED ONLINE 12
SQL>
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.14.0.0.0
[grid@exasivm08 +ASM1]$
So, let’s do some math to validate the numbers:
- Redundancy HIGH: means 3 copies.
- Number of failgropups/cells: 5
- Disks per cell: 12
- Disk size: 1200 GB = 1200*1024 = 1228800 MB
- RAW size per cell (#of disks per cell * disk size): 12 * 1228800 = 14745600 MB
- Total RAW size for Exadata (raw size for FG * #of FG): 5 *14745600 = 73728000 MB
- REQUIRED_MIRROR_FREE_MB (disk size * #redudancy factor): 1228800 * 2 = 2457600 MB (for HIGH need to have two extra copies, for NORMAL 1 extra copy)
- USABLE_FILE_MB ( (FREE_MB – REQUIRED_MIRROR_FREE_MB)/REDUNDANCY ) = ( (73724256 – 2457600)/3 ) = 23755552 MB (in this case is 3 for HIGH because we need 3 copies of the same block).
As you can is pretty easy to calculate and validate the numbers from ASM. The GI until 19.16 calculate the REQUIRED_MIRROR_FREE_MB just based on the number of extra copies that are needed.
After GI/ASM 19.16
But let’s check the same server after upgrading the GI to 19.16 or newer:
[grid@exasivm08 +ASM1]$ $ORACLE_HOME/OPatch/opatch lspatches
34580338;TOMCAT RELEASE UPDATE 19.0.0.0.0 (34580338)
34444834;OCW RELEASE UPDATE 19.17.0.0.0 (34444834)
34428761;ACFS RELEASE UPDATE 19.17.0.0.0 (34428761)
34419443;Database Release Update : 19.17.0.0.221018 (34419443)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)
OPatch succeeded.
[grid@exasivm08 +ASM1]$
[grid@exasivm08 +ASM1]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED HIGH N 512 512 4096 4194304 73728000 73724256 6635520 22362912 0 Y DATAC8/
MOUNTED NORMAL N 512 512 4096 4194304 6144000 6093368 552960 2770204 0 N RECOC8/
ASMCMD> exit
[grid@exasivm08 +ASM1]$
[grid@exasivm08 +ASM1]$ sqlplus / as sysasm
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 3 17:40:33 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
SQL> set linesize 255
SQL> SELECT name, type, total_mb, free_mb, required_mirror_free_mb, usable_file_mb FROM V$ASM_DISKGROUP;
NAME TYPE TOTAL_MB FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------------------------ ------ ---------- ---------- ----------------------- --------------
DATAC8 HIGH 73728000 73724256 6635520 22362912
RECOC8 NORMAL 6144000 6093368 552960 2770204
SQL> select GROUP_NUMBER, NAME, TOTAL_MB, FREE_MB, HOT_USED_MB, COLD_USED_MB, REQUIRED_MIRROR_FREE_MB, USABLE_FILE_MB from v$asm_diskgroup;
GROUP_NUMBER NAME TOTAL_MB FREE_MB HOT_USED_MB COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------ ------------------------------ ---------- ---------- ----------- ------------ ----------------------- --------------
1 DATAC8 73728000 73724256 0 3744 6635520 22362912
2 RECOC8 6144000 6093368 0 50632 552960 2770204
SQL>
SQL> l
1 select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks
2 from v$asm_disk d, v$asm_diskgroup dg
3 where d.group_number = dg.group_number
4 and dg.name IN ('DATAC8')
5 group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status
6* order by 1,2,3
SQL> /
NAME FAILGROUP STATE HEADER_STATU MOUNT_S MODE_ST NUM_DISKS
------ ------------------------------ -------- ------------ ------- ------- ----------
DATAC8 EXACELBRM01 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELBRM02 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELBRM03 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELBRM04 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELBRM05 NORMAL MEMBER CACHED ONLINE 12
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
[grid@exasivm08 +ASM1]$
Again, let’s do some math to validate the numbers:
- Redundancy HIGH: means 3 copies.
- Number of failgropups/cells: 5
- Disks per cell: 12
- Disk size: 1200 GB = 1200*1024 = 1228800 MB
- RAW size per cell (#of disks per cell * disk size): 12 * 1228800 = 14745600 MB
- Total RAW size for Exadata (raw size for FG * #of FG): 5 *14745600 = 73728000 MB
Until now, everything is exactly the same. But you can see that now the REQUIRED_MIRROR_FREE_MB jumped from 2457600 MB (almost 2.3TB) to 6635520 MB (almost 6.3 TB). But why?
The reason is quite complex but I will try to summarize it. The note Understanding ASM Capacity and Reservation of Free Space in Exadata (Doc ID 1551288.1) and Oracle Exadata Database Machine Disk Swap Service Process (Doc ID 1544637.1) clearly explain that that exists Disk Failure Coverage (DFC) and Cell failure Coverage (CFC) and you can see above that until 19.16 the REQUIRED_MIRROR_FREE_MB just leverage DFC, and if you want CFC you need to use a script to calculate the correct size.
Imagine that you have a complete cell failure, you will need a lot of space inside of the diskgroup (more than what is declared in REQUIRED_MIRROR_FREE_MB) to support that. In fact, when (for Exadata) ASM writes a block, the extend is placed in several disks (not in all disks) in other cells, so, blocks from disk 1 from cell 1 (by example) can be protected by different disks that the disk 2 (from the same cell 1). This is the reason that you can (sometimes) have more than 2 outages (for a high redundancy) and your diskgroup continues to run. And for this reason (maybe) you (in reality) need more than just 2 disks as reserved space.
So, taking that into consideration, Oracle changed the way that REQUIRED_MIRROR_FREE_MB is calculated, According to note Understanding ASM Capacity and Reservation of Free Space in Exadata (Doc ID 1551288.1) this change started with GI 19.16 and was because of the bug 32166950. Now, it follows the simple rule:
- Equal or more than 5 FG: 9% of the total size of diskgroup.
- Less than 5 FG: 15% of the total size of diskgroup.
So, now we have (from my example above):
- REQUIRED_MIRROR_FREE_MB (RAX size of diskgroup * 9%): 73728000 * 9% = 6635520 MB
- USABLE_FILE_MB ( (FREE_MB – REQUIRED_MIRROR_FREE_MB)/REDUNDANCY ) = ( (73724256 – 6635520)/3 ) = 22362912 MB).
As you can the drop was huge, and the max space for my diskgroup drop from 23.7 TB to 22.3TB (almost 1.3TB).
Good? Bad? Ugly?
You can think that the value above is not too much, but now look at another example from a full Exadata:
####################################
#
# At 19.14
#
####################################
SQL> select max(TOTAL_MB) from v$asm_disk_stat where GROUP_NUMBER=1;
MAX(TOTAL_MB)
-------------
3145728
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.14.0.0.0
[grid@exasivm03 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED HIGH N 512 512 4096 4194304 452984832 74558648 6291456 22755730 0 Y DATAC3/
MOUNTED HIGH N 512 512 4096 4194304 113246208 100866828 1572864 33097988 0 N RECOC3/
ASMCMD>
####################################
#
# At 19.16
#
####################################
SQL> select max(TOTAL_MB) from v$asm_disk_stat where GROUP_NUMBER=1;
MAX(TOTAL_MB)
-------------
3145728
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0
[grid@exasivm03 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED HIGH N 512 512 4096 4194304 452984832 86527396 40768634 15252920 0 Y DATAC3/
MOUNTED HIGH N 512 512 4096 4194304 113246208 102025608 10192158 30611150 0 N RECOC3/
ASMCMD>
As you can (after upgrading to 19.16) the REQ_MIR_FREE_MB jumped from 6291456 MB (6 TB) to 40768634 MB (almost 38TB). And the available space (USABLE_FILE_MB) dropped from almost 22TB to 5TB (just check above that between the took shots the FREE_MB changed – this means that almost 10TB of datafiles was removed and cleaned from this DG as well, so I deducted this of account). And if you check, the REQ_MIR_FREE_MB is more that the RAW size of one entire cell/FG. REQ_MIR_FREE_MB = 40768634 MB and the size of the entire cell is 12 * 3145728 = 37748736 MB.
The point is that REQ_MIR_FREE_MB became exponential because is based on the total size of your diskgroup and is not per cell or per disk. Again, is based on the total size of your diskgroup. So, the more cells you have, more the bar is raised.
Let’s discussed raising the bar and its consequences. You can argue that just the REQ_MIR_FREE_MB was changed, and this does not mean that something was allocated at disk (it was not), but changing the REQ_MIR_FREE_MB affects the USABLE_FILE_MB. And (working with Exadata since 2010) I guarantee that A LOT of customers based their lives at the USABLE_FILE_MB when they need to check if they need to allocate more space to a diskgroup (if is bare metal, no way).
In the end, this change does not physically reserve space at all but reduced the USABLE_FILE_MB metric. And from the customer’s point of view, is the same thing. I don’t think that any customer will want to have USABLE_FILE_MB in negative value.
You can still argue that is possible to work with the negative value of USABLE_FILE_MB, but what will be the limit? How much do I go further in the negative valley? What will be your protection threshold? Being in negative value will protect you from failures/outages. Is it a calculated risk?
Is up to you to decide if the change was Good, Bad, or Ugly. Unfortunately, you will need to be prepared when you upgrade to GI 19.16 at Exadata/ExaCC, your available space can become negative without any explained reason. And until this moment, there is no way to bypass that (or return to the old metric calculation), not even a parameter to allow the customer to decide it. The change occurred, and no one (cleared) was told about that.
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purposes, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”