Previous Topic: 4.2.3 Monitoring CA MICS Operations

Next Topic: 4.2.5 System Restart and Recovery

4.2.4 Diagnosing Operational Problems


Use Operational Status and Tracking to analyze problems in
DAILY, WEEKLY, MONTHLY, or YEARLY CA MICS database update
processing.  The primary Operational Status and Tracking
display lists the current status for each unit database.
Overdue processing and Database update failures are noted.

This section presents guidelines and samples for CA MICS
operational problem diagnosis.  Specific steps for CA MICS
operational job restart are covered in the section 4.2.5 and
the Reference section.

Note:  This section does not directly address CA MICS
       incremental update processing.  While the analysis
       concepts are the same, the actual techniques may
       differ.  For example CA MICS Operational Status and
       Tracking does not support incremental update and you
       cannot use the STATUS command to research problems in
       INCRccc job execution.  You can, however, get the same
       information from the INCRccc job MICSLOG output and
       the incremental update checkpoint data set.


DIAGNOSING OVERDUE PROCESSES

Operational Status and Tracking shows that a CA MICS
production process (e.g., DAILY) is OVERDUE.

OVERDUE means that the processing due today has not yet been
submitted via Operational Status and Tracking or has not yet
executed.

Determine why the process has not run.

  o Has the CA MICS process been submitted for processing?
    If not, check your scheduling procedures to determine why
    scheduled processing has not begun.

    - If input data is not yet available, determine the cause
      and reschedule daily processing.

    - If input data is available, submit scheduled
      processing.

  o Is scheduled processing still in the JES input queue? If
    so, determine why the job has not yet executed and take
    corrective action.

  o Was scheduled processing submitted but not executed? If
    so, examine batch job outputs, especially the JES log and
    messages data set.

    - Did the operator cancel scheduled processing before
      execution? If so, determine why and resubmit scheduled
      processing.

    - Did scheduled processing fail with a JCL error prior to
      executing the first batch job step? If so, correct the
      problem and resubmit scheduled processing.

    - Did a system crash or JES failure purge scheduled
      processing from the system prior to executing the first
      batch job step? If so, resubmit scheduled processing.


DIAGNOSING INCOMPLETE PROCESSES

Operational Status and Tracking shows that a CA MICS
production process (e.g., DAILY) is INCOMPLETE.

INCOMPLETE means that today's processing began execution.
The operational job may still be executing.

Determine why the process is incomplete.

  o Is the operational job still executing? If so, allow it
    to complete.

  o If the operational job is not executing, the batch job
    may have been flushed by a system failure (e.g., an IPL)
    or may have failed with a JCL error so that the RSR (run
    status) step did not execute.

    - In this situation, Operational Status and Tracking will
      not report the correct job status and there is no Run
      Status Report for problem diagnosis.

      Enter the RSTATUS command.  RSTATUS will update
      Operational Status and Tracking with the current status
      of CA MICS processing.

  o If, after running RSTATUS, CA MICS processing status is
    OVERDUE or DUE TODAY, resubmit scheduled processing.  If
    status is FAILED, then refer to the discussion below.


DIAGNOSING FAILED PROCESSES

Operational Status and Tracking shows that a CA MICS
production process (e.g., DAILY) has FAILED.

FAILED means that the today's processing terminated due to an
error.  You must correct the error and restart scheduled
processing.

Determine why the process failed.

  o The Operational Status and Tracking display lists the
    operational job and step where the failure occurred.

  o Issue the STATUS command or review the Run Status Report
    to gather additional information.

    - The STATUS display lists the scheduled CA MICS
      operational process, last completed job step, the
      failing job step, and may also list the abnormal
      termination code (abend code).

    - If the STATUS display does NOT list the abend code,
      refer to the batch job printed outputs, particularly
      the JES log and JES messages.

  o Look up the abend code in the Reference section of this
    guide.  If the abend is not covered in the Reference
    section, refer to IBM documentation for information on
    causes and corrective action.

  o Examine the MICSLOG from the failing job step for error
    messages documenting the cause of job termination.

  o Examine the JES log and messages output from the failing
    job for additional information concerning an abend or
    other failure.

  o If the failure occurred during CA MICS Database aging,
    call CA MICS Product Support for assistance.

  o If the failure occurred in step DAY199, refer to the
    CA MICS Accounting and Chargeback User Guide for
    additional information and guidance.

The research done here determines what to do next.  For the
most part, you will correct the error and restart the failing
job.  In some cases, you may be instructed to restore the
Database from a backup and rerun scheduled processing from
the beginning (along with any other processing executed since
the backup was taken).

Restart and recovery are discussed in section 4.2.5.

Contact CA MICS Product Support if you need assistance in
problem diagnosis.


DIAGNOSIS EXAMPLE

The Operational Status and Tracking display (see sample
panels on next page) shows that MONTHLY processing for the R
(REMOTE) unit failed.  Last completed job step was DAYALL.

The Operational Status and Tracking STATUS command shows that
the DAILY job failed in step DAYSMF with an S001 abend.

IBM documentation defines an S001 abend as an I/O error.

The JES log includes a message which indicates that the I/O
error occurred on the INPUTSMF DD statement.

The Reference section in this guide shows that the INPUTSMF
DD statement is used in the DAYSMF step to read the input
data.

Thus, the CA MICS DAILY job failed because of an I/O error on
the input data set.  Restart and recovery are covered in
section 4.2.5, but some notes on S001 abend recovery follow
to complete this example.

  o If the problem is due to a hardware failure (e.g., a
    dirty tape drive), you can restart the DAILY job at step
    DAYSMF.

  o If the problem is a media problem (e.g., a bad tape) and
    a backup copy is available, you can restart the DAILY job
    at step DAYSMF pointing CA MICS to the backup input tape.

  o If the backup copy of the input data is on a consolidated
    tape (e.g., a week-to-date raw SMF tape), you can use the
    Select facility to select just the current data from the
    consolidated input tape.

  o If a backup copy of the input is not available, you will
    need to try to salvage the tape using system utilities to
    copy the tape, skipping unreadable records.  The data is
    incomplete, but this may be more tolerable than not
    having data at all.

---------------------- Operational Status and Tracking --------- ROW 1 OF 6Command ===> STATUS R Scroll ===> CSR

Commands: Schedule, Daily, Weekly, Monthly, Yearly, Backup, Restore, Restart, Status/History/Checkpt/Joblog, Suspend/Resume, Force Database Current Operation Last Completed Edit Suspend Cmd ID Label Type as of 10 OCT 2001 Job/Step Date JCL Updates -------- -- -------- - ------------------- ------------------ --- --- ________ C CICS U MONTHLY Completed MONTHLY 10OCT2001 NO NO ________ D DASD U MONTHLY DUE TODAY WEEKLY 09OCT2001 NO NO ________ I IMS U MONTHLY Completed MONTHLY 10OCT2001 NO NO ________ P PRIMARY P MONTHLY FAILED DAY020 10OCT2001 NO NO ________ R REMOTE U MONTHLY FAILED DAYALL 10OCT2001 NO NO ________ T TEST T DAILY OVERDUE DAILY 19SEP2001 NO NO ****************************** BOTTOM OF DATA ********************************



. . . . . . . . . . . . . . . . . . . . . . . . . .



--------------------------- Unit Database Status -------------------------


Command ===>



Database: R (REMOTE) - REMOTE DATA CENTER The status information was recorded at: 10OCT01 08:45 Status of this unit Database: NON-UPDATABLE Status of the cycle aging process: Completed CA MICS Last Completed Job Step & Date Status of Current Operation: MONTHLY -------- --- ------- ----------------------------------------------------- DAILY ALL 10OCT01 FAILED DAYSMF S001 MONTHLY 900 05SEP01 HELD Prior job failed BACKUP 900 09OCT01 HELD Prior job failed Status of Other Jobs ----------------------------------------------------- WEEKLY 900 09OCT01 Completed YEARLY 900 10JAN01 Completed RESTORE 900 09OCT01 Completed



. . . . . . . . . . . . . . . . . . . . . . . . . .