How You Diagnose Allocation Problems

Troubleshooting › Tape Device Allocation Problems › How You Diagnose Allocation Problems

How You Diagnose Allocation Problems

You can diagnose possible allocation delays identified through GTALOCAL enqueue conflicts by performing the following procedure. The following flowchart chart connects a series of steps, instructing you on what action to take in the event of a delay or a hang. Alternate instructions for recurrent problems follow this procedure.

Identify the system on which the troubleshooting commands will be issued by specifying the following command on any system in the MIMplex:
```
F MIM,DIAGNOSE SYSTEMS
```
Is there a system listed as holding locks?
YES

Go to Step 2.

NO
Have MIM2162I and MIM2165I or MIM22075 messages been issued?

YES

To see which job is holding the resource mentioned in message MIM2165I, issue one of the following z/OS commands (depending on which resource is listed in the MIM216x messages):
```
    D GRS,RES=(SYSIEFSD,*)
```
```
    D GRS,RES=(SYSZIOS,*)
```
Note: If the resource in the message is SYSIEFSD DDRTPUR, allocation is most likely being delayed by an outstanding SWAP. In addition to issuing the D GRS command, issue a z/OS D R, L, CN=(ALL) command to look for outstanding SWAP WTORs. Also, look for an outstanding MIM2207I message indicating a SWAP has started.

If a SWAP is outstanding, take whatever action is necessary to allow the SWAP to continue. This may include replying to jobs with outstanding allocation recovery messages or replying to outstanding SWAP WTORs. If you cannot determine why the job holding the SYSIEFSD DDRTPUR resource is causing the allocation delay, go to Step 6.

NO

Problem circumstances may have changed. If you come to this point repeatedly and still believe there is a problem, contact CA Technical Support.

Return to Step 1.
On the system identified in Step 1, issue the following CA MIA command:
```
F MIM,DIAGNOSE JOB
```
Has any job been given ownership of device group locks?
YES

Go to Step 3.

NO
Is the CA MIA task waiting for device group locks?

YES

Go to Step 3.

NO
1. Issue the following command on each system in the complex.
```
    F MIM,SYSDUMP JOBNAMES=(ALLOCAS,IOSAS,GRS) TITLE="ALLOCATION HANG"
```
2. Issue the following command to CA MIA:
```
    SHUTDOWN LOCAL
```
  If the problem is not resolved, call CA Technical Support and ask for help with CA MIA.
Issue the following z/OS command:
```
D R,L,CN=(ALL).
```
Are there any outstanding SWAP messages or any outstanding allocation recovery messages?

YES

Reply to the outstanding WTOR.

NO

Go to Step 4.
Issue the following CA MIA command:
```
F MIM,DIAGNOSE VARY
```
Are there any active VARY requests that show a Wait-RSN of “VARY Dev?”

YES
1. Check to see if the devices involved in outstanding VARY processing are functional. If the problem is not resolved, issue the following command:
```
     F MIM,SYSDUMP JOBNAMES=(ALLOCAS,IOSAS,GRS) TITLE="VARY DELAY"
```
2. Issue a the following command to CA MIA:
```
    SHUTDOWN LOCAL
```
If the problem is not resolved, contact CA Technical Support and ask for help with CA MIA.

NO

Go to Step 5.
Do the following:
1. Look in the system log for IOS003A INTERVENTION REQUIRED messages for tape devices. “Ready” any tape device mentioned in these messages. Allocation hangs can occur when jobs perform AVR processing on devices that are experiencing I/O problems.
2. Review the system log for I/O error messages for tape devices. If any have been issued for devices that do not appear to be allocated, VARY that device OFFLINE. Allocation hangs can occur when jobs perform AVR processing on devices that are experiencing I/O problems.
3. If I/O errors are not the cause of the allocation delay, go to Step 6.
Issue the following z/OS command to obtain related z/OS allocation enqueue information:
```
D GRS,RES=(SYSIEFSD,*)
```
Retain the information in the resulting display for diagnostic purposes.

Go to Step 7.
Get an SVC dump of the job identified in Step 1 or Step 2 by issuing the following command:
```
F MIM,SYSDUMP JOB=(jobname,ALLOCAS.ISOAS,GRS),TITLE='ALLOCATION DELAY', SYSTEM=ALL
```
Note: When the DIAGNOSE display shows a job ID of MASTER as being 'delayed for' or 'given' a single device group lock, this most likely represents a job that is in the middle of SWAP processing. Look for the MIM2207I message to determine the actual job associated with the SWAP. Then, issue the preceding dump command for two job names - the job name for the MASTER address space and the swapping job name (JOB=(*MASTER*,jobname)).

Can this job be canceled?

YES

Cancel the job.

NO

Issue a SHUTDOWN LOCAL command to CA MIA. If the problem is not resolved, call CA Technical Support and ask for CA MIA.

Go to Step 8.
Gather any dumps taken, SYSLOGs for the problem time period, along with traces and job logs of any cancelled jobs, and contact CA Technical Support during normal business hours to receive an incident number for this problem.
Important: If the problem is recurring, then adequate documentation is essential in solving the problem. Technical Support needs GTAF trace information before they can analyze the problem. Follow these steps:
1. Define the MIMTRACE DD in the CA MIM PROC to go to a SYSOUT hold class.
2. Issue the following command
```
F MIM,SET TRACE=ON
```
3. To set the internal GTAF trace, issue the following command:
```
F MIM,SET GTAF SETT=(LOCKS,MASKS,SWAP,UNITALOC),SETP=ALL
```
4. Issue the following command to spin off current trace data and reopen the SYSOUT file
```
F MIM,SET TRACE=(SPIN,CLASS=c)
```
  This allows you to purge data from time periods when no problem was experienced.
5. Call CA Technical Support for assistance.
If the problem is not recurrent, then global dumps of CA MIA and SYSLOGs are the minimum documentation required. If a job appears in the DIAGNOSE JOB command as having the device locks, then a dump of this job may also be essential for problem diagnosis.

Tell Technical Publications how we can improve this information