System Description › Overlapping SMF Data

Overlapping SMF Data

Overlapping SMF data is detected at SMF dump processing time and a warning message is issued. The index record is written to the appropriate system configuration (SID) or logstream configuration, and the dump process will not abend or stop processing when an overlap or duplicate record condition occurs. At this time you can use the LISTH commands to display data around the time frame of the overlapping data, and use ADDX and DELETEX to "fix" the SCDS index entries.

If no action is taken to correct the SCDS entries after Dump processing, when an EXTRACT or PRINT command is executed that encompasses the overlapped index entries, an error message will be issued and processing will stop. At this point there are three options. Option one is to specify the SMFD PARM "PERMIT", which will cause the EXTRACT output to contain all the data from the overlapping indexes. Use this option with caution and only after you have determined there is no duplicate data in the overlapping index entries. Option two is to specify the SMFD PARM "NODUPS", which will cause the EXTRACT or PRINT process to execute and produce in the resultant output, data that does not overlap. This method produces the desired results in the output but leaves the overlapping index entries in the SCDS file. The major benefit with option two is that no manual intervention against the SCDS file is required. Option three is to use the LISTH command to view index entries around the time frame in question, and manually update the SCDS index entries using ADDX and DELETEX commands.

In addition to the above mentioned remedies for dealing with overlapped data conditions, there is a new complex level configuration option OVERLAP(n). The proper specification of this option can eliminate false detections of overlap conditions by this product. This option specifies a time from zero (no overlap grace time period) to five seconds. If a number other than zero is specified, it is used as the number of seconds of overlap allowed to occur between index entries before an overlap condition is detected. This option was introduced due to the irregular way some applications write out SMF records, which, depending on the time the SMF MANx files are dumped, may trigger false overlap detection without this option specified. See Chapter 4 in the User Guide for information on the OVERLAP(n) option.

The following examples illustrate how this product deals with various settings of the OVERLAP complex level options, and the PERMIT and NODUPS execution time parameters.

In the first example, the complex level overlap option has been specified as OVERLAP(0). This means that any data overlaps will be flagged as a warning during the dump process, and flagged as an error during EXTRACT/PRINT processing, unless the PERMIT or NODUPS execution parameter has been specified. Consider that three SMF dumps have occurred for the XAT1 system, with the following FROM and TO date/time ranges:

-------- FROM --------        -------- To --------
03/16/2008    14:00:00        03/16/2008  14:00:59 ------>  Dump 1
03/16/2008    14:00:56        03/16/2008  14:01:59 ------>  Dump 2
03/16/2008    14:01:56        03/16/2008  14:02:59 ------>  Dump 3

Because the overlap option has been set to zero, the overlaps detected during the dump process will be flagged with warning messages. In this particular instance, the dump process for dump(1) will not produce a warning message at the time the dump occurs because it does not overlap any existing index entries for SID XAT1. Dump(2) and dump(3) will issue the following warning messages because the FROM date/time of their dumps is less than the TO date/time of the previous dump.

CAF2417W SMF DATA FOR XAT1 03/16/2008 at 14:00:56 OVERLAPS DATA ALREADY IN INDEX

CAF2417W SMF DATA FOR XAT1 03/16/2008 at 14:01:56 OVERLAPS DATA ALREADY IN INDEX

The dumped data is added to the SCDS file as an index entry for the SID being processed (XAT1). Using the LISTH and CHECKIT commands, and upon examination of the data, if it is determined that there is duplicated data, the DELETEX and ADDX commands can be used to update the SCDS to properly reflect the correct index entries. For example, if an index entry is completely duplicated, you can use the DELETEX command to remove the duplicated entry. If there is an overlap, you can DELETEX the overlapped index entry and then manually dump SMF data using the SOURCE DUMP command after eliminating overlapped or duplicated data. Finally, if there is a small overlap and it is determined that there was no duplicate data in the SMF files, you can DELETEX and ADDX the entries back into the SCDS without the FROM and TO date/times being overlapped. If an EXTRACT is run at a later point in time, no error messages will be issued.

A second, and recommended approach, is to leave the index entries as they exist in the SCDS and use the PERMIT or NODUPS options, depending on the outcome of your research into the overlap condition. This approach requires less manual intervention and lets this product create the desired output from EXTRACT processing with very little intervention from the user. Consider an EXTRACT request that encompasses all three of the above dumps:

EXTRACT SID(XAT1) FROM(031608 140000) TO (031608 140300).

If this EXTRACT command is issued with OVERLAP(0) and without the execution parameter PERMIT or NODUPS specified, all of the data from dump(1) is written to the extract file. Two error messages, one for the dump(2) index entry, and one for the dump(3) index entry are issued.

CAF2411E SMF DATA FOR XAT1 03/16/2008 AT 14:00:56 OVERLAPS DATA ALREADY IN INDEX

CAF2411E SMF DATA FOR XAT1 03/16/2008 AT 14:01:56 OVERLAPS DATA ALREADY IN INDEX

Data from the dump(2) and dump(3) index entries are not extracted.

If this EXTRACT command is issued with OVERLAP(0), and the execution parm PERMIT specified, warning messages are issued, and all data from all three index entries are written to the extract file. The PERMIT option should only be used when the data has been examined and it has been determined that no duplicate data exists. The warning messages are as follows:

CAF2417W SMF DATA FOR XAT1 03/16/2008 AT 14:00:56 OVERLAPS DATA ALREADY IN INDEX

CAF2417W SMF DATA FOR XAT1 03/16/2008 AT 14:01:56 OVERLAPS DATA ALREADY IN INDEX

If this EXTRACT command is issued with OVERLAP(0), and the execution parm NODUPS specified, warning messages are issued, and all data in the three dump files not considered to be in an overlap condition are written to the output file. In more precise terms, SMF data in dump(2) from 03/16/2008 at 14:00:56 up until the first SMF record in dump(2), with a start date/time greater than 14:00:59, is discarded. All SMF data in dump(3) from 03/16/2008 at 14:01:56 up until the first SMF record with a start date/time greater than 14:01:59 is discarded. This PARM should be used when it has been determined that the overlapped portion of files does indeed contain duplicate data, and you want to skip data in the overlap condition. The following warning messages are issued in this case:

CAF2417W SMF DATA FOR XAT1 03/16/2008 AT 14:00:56 OVERLAPS DATA ALREADY IN INDEX

CAF2417W SMF DATA FOR XAT1 03/16/2008 AT 14:01:56 OVERLAPS DATA ALREADY IN INDEX

Another possible condition is an SMF data set that somehow is dumped twice, the second dump creating a complete duplicate index entry. For example, consider that dump(3) data was dumped twice, and the following index entries for SID XAT1 now exist:

-------- FROM --------        -------- To --------
03/16/2008    14:00:00        03/16/2008  14:00:59 ------>  Dump 1
03/16/2008    14:00:56        03/16/2008  14:01:59 ------>  Dump 2
03/16/2008    14:01:56        03/16/2008  14:02:59 ------>  Dump 3
03/16/2008    14:01:56        03/16/2008  14:02:59 ------>  Dump 4

At the time of the dump processing, a warning message appears indicating that overlapping data exists, but the index entry is added to the SCDS file. At extract time, when a duplicate index entry is detected, the following message is issued:

CAF3902W DUPLICATE SMF DATA DETECTED FOR XAT1 03/16/2008 14:02:59 DATA SKIPPED

This message is issued regardless of the execution parameters PERMIT or NODUPS. PERMIT and NODUPS have no effect on processing of complete duplicate index entries. No data from a duplicate index entry is written to the output file, and the index entry is skipped.

In the previous examples, the dump(2) and dump(3) index entries overlapped their preceding index entries by three seconds. If the complex level option overlap had been specified as OVERLAP(5), indicating that index entries that overlap other index entries by less than the OVERLAP(N) value should not be flagged as overlaps, no warning or error messages would be received during dump or extract processing, and the PERMIT or NODUPS options would not be needed. If the overlap condition exceeds the overlap option specification, then processing occurs exactally as previously explained. For example, using the same dump files, if the complex level option is specified as OVERLAP(2), all of the above warning and error messages will appear as if OVERLAP(0) has been specified, because the index entries overlapped by three seconds, which triggers the overlap condition.