During shutdown, CA MIA takes all locally managed unallocated Tape Devices OFFLINE. Locally managed, allocated Tape Devices are marked Pending OFFLINE, and transition to an OFFLINE state when the local allocation of the device completes.
CA MIA takes this action to preserve the integrity of data residing on devices that are currently allocated on other systems in the MIAplex.
If Tape Device Allocation is not QUIESCED prior to CA MIA SHUTDOWN on a system, TASKs attempting to allocate Tape Devices on the system will enter Recovery Allocation. During Recovery Allocation, WTORs are issued listing the set of options available for completing the current allocation request.
While a Recovery Allocation WTOR is outstanding, the TASK for which it is issued holds important system resources that serialize Recovery Allocation processing with other z/OS processes.
For example, while a Recovery Allocation WTOR is outstanding on a system:
CA MIA extends z/OS Tape Device Allocation Serialization across all systems in an MIAplex. As a result, z/OS Recovery Allocation related serialization on one system can affect VARY processing and Tape Device Allocation throughput within an MIAplex in the following ways:
EXAMPLE 1: VARY ONLINE processing delayed by Recovery Allocation WTORs
This example shows the effect that an outstanding Recovery Allocation WTOR on system XE03 has on VARY ONLINE processing on an external system system in the MIAplex, XE13. The delay would occur on ALL MIAplex systems.
TASK TAPE1 is started on system XE03. It is unable to allocate a suitable, ONLINE UNALLOCATED device, and enters Recovery Allocation. The MIM2060 Recovery Allocation WTOR is issued and remains outstanding on XE03.
XE03 TASK TAPE1 is in recovery for UNIT=3490 Tape Devices
#MIM2120 TAPE1 TAPE1 UNABLE TO ALLOCATE #MIM2042 TAPE1 TAPE1 TAPEDD DEVICES OFFLINE 068 ==> 0740,0741,0742,0743,0744,0745,0746,0747,0760,0761,0762,0763 ==> 0764,0765,0766,0767,0768,0769,076A,076B,076C,076D,076E,076F ==> 0E50,0E51,0E52,0E53,0E54,0E55,0E56,0E57,0E58,0E59,0E5A,0E5B ==> 0E5C,0E5D,0E5E,0E5F,0E80,0E81,0E82,0E83,0E84,0E85,0E86,0E87 ==> 0E88,0E89,0E8A,0E8B,0E8C,0E8D,0E8E,0E8F,0E90,0E91,0E92,0E93 ==> 0E94,0E95,0E96,0E97,0E98,0E99,0E9A,0E9B,0E9C,0E9D,0E9E,0E9F ==> 0E20,0E21,0E22,0E23,0E24,0E25,0E26,0E27,0E28,0E29,0E2A,0E2B ==> 0E2C,0E2D,0E2E,0E2F,0E70,0E71,0E72,0E73,0E74,0E75,0E76,0E77 ==> 0E78,0E79,0E7A,0E7B,0E7C,0E7D,0E7E,0E7F *0117 #MIM2060 TAPE1 - REPLY DEVICE NAME OR 'CANCEL'.
While the MIM2060 WTOR (or the z/OS equivalent) is outstanding, TASK TAPE1 holds the Tape Device Group Locks for each of the devices listed on the MIM2042 (or z/OS equivalent). Until a valid response to the Recovery Allocation WTOR is entered, these resources are unavailable on any other system in anactive MIAplex. This condition will delay VARY ONLINE command processing and Tape allocation throughput on all MIAplex systems.
MIA on EXTERNAL system XE13 is started and synchronizes. The Recovery Allocation WTOR remains outstanding on XE03. VARY ONLINE commands for the devices involved in Recovery Allocation on XE03 (UNIT=3490) are issued during MIA synchronization on XE13. These commands will be delayed from completing because of the Recovery Allocation condition on XE03, as identified by a DIAGNOSE ALL command issued on XE13.
XE13
*#MIM0022I system XE13 in file 00 synchronization UNDERWAY #MIM0023I system XE13 in file 00 synchronization COMPLETE #VARY 740-747,ONLINE #VARY 760-76F,ONLINE #VARY E50-E5F,ONLINE #VARY E80-E8F,ONLINE #VARY E90-E9F,ONLINE #VARY E20-E2F,ONLINE #VARY E70-E7F,ONLINE #MIM0067I Command VARY 310 #MIM2021I VARY process pending #MIM0067I Command VARY 311 #MIM2021I VARY process pending #MIM0067I Command VARY 312 #MIM2021I VARY process pending #MIM0067I Command VARY 313 #MIM2021I VARY process pending #MIM0067I Command VARY 314 #MIM2021I VARY process pending #MIM0067I Command VARY 315 #MIM2021I VARY process pending #MIM0067I Command VARY 316 #MIM2021I VARY process pending
The SYSTEMS DISPLAY portion of the MIM2150I command response to a DIAGNOSE ALL command issued on XE13 identifies that Tape Device Group Locks are in use on system XE03. The Tape Device Group Locks are held by TASK TAPE1, which is in Recovery Allocation on XE03.
#DIAGNOSE ALL
#MIM0067I Command DIAGNOSE 320
#MIM2150I DIAGNOSE ALLOCATION DISPLAY
BEGIN SYSTEMS DISPLAY =========>
System XE03 has 'AB' locks for devices
740 741 742 743 744 745 746 747 760 761 762 763 764 765 766 767 768 769 76A 76B 76C 76D 76E 76F E20 E21 E22 E23 E24 E25 E26 E27 E28 E29 E2A E2B E2C E2D E2E E2F E50 E51 E52 E53 E54 E55 E56 E57 E58 E59 E5A E5B E5C E5D E5E E5F E70 E71 E72 E73 E74 E75 E76 E77 E78 E79 E7A E7B E7C E7D E7E E7F E80 E81 E82 E83 E84 E85 E86 E87 E88 E89 E8A E8B E8C E8D E8E E8F E90 E91 E92 E93 E94 E95 E96 E97 E98 E99 E9A E9B E9C E9D E9E E9F
END OF SYSTEMS DISPLAY
The JOBSTATUS portion of the MIM2150I command response shows that the GTAF component of CA MIA, which is responsible for Global Serialization of Tape Device Allocation, is delaying requests by the MIA started Task on XE13 for +A+ Tape device Group Locks for 8 of the devices being VARY’d ONLINE, (740-747).
MIA is attempting to serialize these VARY ONLINE requests with z/OS Tape Device Allocation within the MIAplex. MIA cannot achieve the serialization because the Tape Device Group Lock resources required by MIA to do so are held by TASK TAPE1 on system XE03.
The +A+ designation of the Tape Device Group Lock request indicates that the request is for a VARY command.
BEGIN JOBSTATUS DISPLAY =======> JOBSTATUS: DELAYED
GTAF is delaying MIAB713's request for +A+ group locks for devices 740 GTAF is delaying MIAB713's request for +A+ group locks for devices 742 GTAF is delaying MIAB713's request for +A+ group locks for devices 743 GTAF is delaying MIAB713's request for +A+ group locks for devices 745 GTAF is delaying MIAB713's request for +A+ group locks for devices 746 GTAF is delaying MIAB713's request for +A+ group locks for devices 747 GTAF is delaying MIAB713's request for +A+ group locks for devices 741 GTAF is delaying MIAB713's request for +A+ group locks for devices 744
JOBSTATUS: RELEASED
NO ENTRIES FOR JOBSTATUS.RELEASED
JOBSTATUS: WAITING
NO ENTRIES FOR JOBSTATUS.WAITING
JOBSTATUS: GIVEN
NO ENTRIES FOR JOBSTATUS.GIVEN
JOBSTATUS: WAITING FOR DEVICES
NO ENTRIES FOR JOBSTATUS.WAITING FOR DEVICES
END OF JOBSTATUS DISPLAY
Lastly, the MANAGED VARY portion of the MIM2150I command response shows the status of all VARY commands for MIA managed devices on the system where the DIAGNOSE ALL command was issued.
It identifies that VARY ONLINE requests for devices 0740-0747 are ACTIVE,but are being delayed by MIA from processing due to Tape Device Group Lock contention (Wait-Rsn: Grp Lock).
MIA on XE13 will requeue these 8 VARY commands, and attempt to process the next 8 SCHDEDULED VARY ONLINE requests. If the Recovery Allocation WTOR remains outstanding on XE03, NO VARY ONLINE commands for the Group of Tape Devices (ESOTERIC or GENERIC) involved in Recovery Allocation on XE03 will complete until the Recovery Allocation WTOR is replied to.
Begin managed VARY display ====>
Active VARY device queue Dev State Option Source Requeue# Queue-Tm Wait-Rsn 0740 ONLINE *INTVARY 1 7 Grp Lock 0741 ONLINE *INTVARY 1 7 Grp Lock 0742 ONLINE *INTVARY 1 7 Grp Lock 0743 ONLINE *INTVARY 1 7 Grp Lock 0744 ONLINE *INTVARY 1 7 Grp Lock 0745 ONLINE *INTVARY 1 7 Grp Lock 0746 ONLINE *INTVARY 1 7 Grp Lock 0747 ONLINE *INTVARY 1 7 Grp Lock
Scheduled VARY device queue
Dev State Option Source Requeue# Queue-Tm Wait-Rsn
0760 ONLINE *INTVARY 0 7
0761 ONLINE *INTVARY 0 7
0762 ONLINE *INTVARY 0 7
0763 ONLINE *INTVARY 0 7
0764 ONLINE *INTVARY 0 7
0765 ONLINE *INTVARY 0 7
0766 ONLINE *INTVARY 0 7
0767 ONLINE *INTVARY 0 7
•
•
•
0E7C ONLINE *INTVARY 0 8
0E7D ONLINE *INTVARY 0 8
0E7E ONLINE *INTVARY 0 8
0E7F ONLINE *INTVARY 0 8
End managed VARY display
EXAMPLE 2: LOCAL VARY OFFLINE processing delayed by Recovery Allocation WTORs
Outstanding Recovery Allocation WTORs will delay the completion of VARY OFFLINE command processing on the system where the WTOR is outstanding.
The delay is caused by z/OS ENQ Serialization of VARY OFFLINE processing with Allocation, NOT by MIA Global Serialization of the request.
While a TASK is in RECOVERY ALLOCATION, it holds a SHR ENQ for z/OS Allocation resource SYSIEFSD/Q4. z/OS VARY OFFLINE processing requires a SYSIEFSD/Q4 EXCL ENQ to serialize VARY OFFLINE with z/OS Allocation.
When processing a VARY OFFLINE request, z/OS will WAIT for 5 seconds to acquire the EXCL ENQ for SYSIEFSD/Q4. If the 5 second timer expires before the SYSIEFSD/Q4 EXCL ENQ can be obtained, z/OS will drop then attempt to reacquire the EXCL ENQ.
The effect is that during periods that a SYSIEFSD/Q4 ENQ is held for extended periods of time, as is the case with outstanding Recovery Allocation WTORs, VARY OFFLINE processing is delayed, and z/OS Allocation requests, which are serialized by z/OS with a SHR SYSIEFSD/Q4 ENQ, will be serviced at 5 second intervals.
Recovery Allocation related SYSIEFSD/Q4 ENQ contention delays z/OS VARY OFFLINE processing from completing for ALL device types while a Recovery Allocation WTOR is outstanding.
This example illustrates how Recovery Allocation related ENQ contention can impact LOCAL system VARY OFFLINE processing.
XE03 JOB in recovery for UNIT=3480
#MIM2120 TAPE1 TAPE1 UNABLE TO ALLOCATE #MIM2042 TAPE1 TAPE1 TAPEDD DEVICES OFFLINE 068 ==> 0740,0741,0742,0743,0744,0745,0746,0747,0760,0761,0762,0763 ==> 0764,0765,0766,0767,0768,0769,076A,076B,076C,076D,076E,076F ==> 0E50,0E51,0E52,0E53,0E54,0E55,0E56,0E57,0E58,0E59,0E5A,0E5B ==> 0E5C,0E5D,0E5E,0E5F,0E80,0E81,0E82,0E83,0E84,0E85,0E86,0E87 ==> 0E88,0E89,0E8A,0E8B,0E8C,0E8D,0E8E,0E8F,0E90,0E91,0E92,0E93 ==> 0E94,0E95,0E96,0E97,0E98,0E99,0E9A,0E9B,0E9C,0E9D,0E9E,0E9F ==> 0E20,0E21,0E22,0E23,0E24,0E25,0E26,0E27,0E28,0E29,0E2A,0E2B ==> 0E2C,0E2D,0E2E,0E2F,0E70,0E71,0E72,0E73,0E74,0E75,0E76,0E77 ==> 0E78,0E79,0E7A,0E7B,0E7C,0E7D,0E7E,0E7F *0117 #MIM2060 TAPE1 - REPLY DEVICE NAME OR 'CANCEL'
While the MIM2060 Recovery Allocation WTOR is outstanding on XE03, VARY OFFLINE commands are issued for Tape Devices 078C-078F, but the commands do not complete.
Note: Devices 078C-078F are NOT involved in the outstanding Recovery Allocation on the system.
The Managed VARY display within the CA MIA DIAGNOSE ALL command response, issued on system XE03, provides insight into the cause of the delay in VARY OFFLINE processing.
Begin managed VARY display ====>
Active VARY device queue Dev State Option Source Requeue# Queue-Tm Wait-Rsn 078C OFFLINE *INTVARY 1 5 VARY Dev 078D OFFLINE *INTVARY 1 5 VARY Dev 078E OFFLINE *INTVARY 1 5 VARY Dev 078F OFFLINE *INTVARY 2 5 VARY Dev End managed VARY display
END OF DIAGNOSE COMMAND
The Wait-Rsn of VARY Dev, appearing for each of the VARY OFFLINE commands shown in the example above, indicates that MIA has passed the VARY OFFLINE requests to the z/OS VARY Service, IEEVARYD, for processing. MIA is waiting for the z/OS VARY Service to complete the OFFLINE requests and return control. MIA is not delaying the VARY OFFLINE commands.
A GRS display for the SYSIEFSD resource identifies SYSIEFSD resource utilization on system XE03. While TASK TAPE1 is in Recovery Allocation, it OWNS SHR ENQs on SYSIEFSD resources CHNGDEVS, DDRDA, DDRTPUR and Q4.
The MIA started task on XE03, MIAB703, WAITS EXCL for SYSIEFSD/Q4. This EXCL SYSIEFSD/Q4 ENQ, as well as the EXCL SYSIEFSD/VARYDEV ENQs below it, are raised by the z/OS VARY Service, IEEVARYD, on behalf of the caller, MIAB703. Until Recovery Allocation on XE03 is relieved, VARY OFFLINE completion and z/OS Allocation throughput for DASD or TAPE will be impacted.
D GRS,RES=(SYSIEFSD,*)
ISG343I 10.55.09 GRS STATUS 229 S=SYSTEM SYSIEFSD CHNGDEVS SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS XE03 TAPE1 0068 007FF890 SHARE OWN S=SYSTEM SYSIEFSD DDRDA SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS XE03 TAPE1 0068 007FF890 SHARE OWN S=SYSTEM SYSIEFSD DDRTPUR SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS XE03 TAPE1 0068 007FF890 SHARE OWN S=SYSTEM SYSIEFSD Q4 SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS XE03 TAPE1 0068 007FF890 SHARE OWN XE03 MIAB703 0066 006E4108 EXCLUSIVE WAIT S=SYSTEM SYSIEFSD VARYDEV SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS XE03 MIAB703 0066 006E4108 EXCLUSIVE OWN XE03 MIAB703 0066 006E7900 EXCLUSIVE WAIT XE03 MIAB703 0066 006E44D8 EXCLUSIVE WAIT XE03 MIAB703 0066 006E4340 EXCLUSIVE WAIT
|
Copyright © 2014 CA.
All rights reserved.
|
|