REORG Processing Details

Utility Statements › REORG › Usage › REORG Processing Details

REORG Processing Details

How the unload phase works

During the unload phase, most record types are unloaded as part of an area sweep. Certain database structures necessitate exceptions to this general processing flow.

A record that is stored VIA a System Index in which sequence order cannot be determined from the member record (such as an unsorted index or one in which duplicates are FIRST or LAST), is offloaded by walking its index.
Members of a VIA set are unloaded at the time their owner is unloaded by walking the VIA set. If the VIA member in turn owns VIA members, they too are unloaded in a similar fashion.
When unloading a CALC record for which duplicates are allowed, the CALC set is walked in the prior direction to determine the relative position of the record with respect to its duplicates.

Records that are unloaded are assigned a target page number during the unload phase. Target page numbers are assigned as follows:

CALC records are assigned a target page based on their CALC key and target page range
VIA records that are members of their VIA set are assigned a target page based on their owner's assigned or target page. If a VIA record is in the same target page range as its owner, it is assigned the same target page plus any VIA displacement. If in a different page range, it is assigned a target page in its range proportional to its owner's page in its assigned or target page range plus any VIA displacement.
VIA records that are not members of their VIA set are assigned a target page that is proportional to their position in their source page range.
DIRECT records are assigned a target page that is the same as their source page if the page is in both the source and target page ranges; otherwise, DIRECT records are assigned a target page that is proportional to their position in their source page range.

REORG slicing

A database is divided into slices based upon the value coded in the DIVIDE PROCESSING n WAYS parameter. Generally, each area is divided symmetrically, but if an area contains a subarea and no outside records span that subarea, REORG divides the subarea independently of the rest of the area.

If dividing an area or subarea produces a piece that is too small, smaller than 100 pages, that area or subarea is divided again into fewer, but larger pieces. If an area or subarea is less than 200 pages, it is not divided at all.

After all areas and subareas have been divided into pieces, each piece is assigned to a slice. Each slice typically ends up with approximately the same number of pages.

Usually, the number of slices equals the n-WAYS value, but if the total number of pages being processed is small, there may be fewer slices. If the total size of all areas is less than 200 pages, all pages are assigned to one slice, regardless of the n-WAYS value.

Dependent areas are areas with physical linkages to the areas being processed. They are also divided and assigned to slices.

Areas and subareas that contain only system indexes are not divided and are not assigned to a slice.

The source and target database are sliced independently of one another and consequently may not be divided into the same number of slices.

Index groups

A user-owned index occurrence is processed with the slice to which its owner record is assigned. System-owned indexes are assigned to groups and processed independently of slices. System indexes that have the same or overlapping target page ranges are assigned to the same group; otherwise, they are assigned to different groups. If an index has a distinct (non-overlapping) target page range, it is assigned its own index group. Index groups are processed independently of each other and therefore can be processed in parallel. Once assigned to a group, an index is in that group for the duration of REORG processing.

REORG tasks and phases

REORG processing is divided into tasks and grouped into phases. Each phase processes a type of work needed to unload or reload a database or rebuild an index. Each task processes a slice or index group within a phase.

For example, if a database were divided into two unload slices and had three index groups, the unload phase could have up to five tasks: one for each slice and possibly one for each index group. All unload tasks must successfully complete before REORG can begin to process tasks in the RELOAD or REBUILD phases.

RELOAD contains six phases numbered 1 through 6. These phases reload slices, rebuild user indexes, and reconnect pointers. Each reload task in a given phase must successfully complete before processing can move to the next RELOAD phase.

REBUILD can contain up to three phases numbered 1 through 3. Each index group has one task per phase, but each index group can run independently of the others and independently of RELOAD phases 3 through 5.

If a system index related page range conflict exists between tasks in RELOAD and REBUILD, or between REBUILD tasks, updates of the page range are serialized.

The phases are:

Phase	Description
SETUP	Analyzes syntax, subschemas, and DMCLs to initialize the REORG process. It decides how to divide the source and target databases for parallel processing. It creates tasks and decides what work files are needed to support those tasks. It builds the RORGCTL file.
UNLOAD	Consists of Unload Slice and Unload Index tasks. Slices are offloaded through area sweeps and by walking VIA sets. If an index needs its sequence preserved, the index is walked to determine the relative sequence number of each member. The sequence data is merged with information extracted from the loaded record in REBUILD2. If VIA System Indexes exist and their sequence needs to be preserved, the member records are offloaded by walking the index. VIA members are offloaded when their owners are offloaded regardless of how the owners are offloaded.
REBUILD1	This phase exists only when there is a VIA SYSTEM INDEX that spans multiple reload slices. If needed, it runs prior to RELOAD1. It sorts VIA INDEX member records into index order and assigns the records to target slices.
RELOAD1	Loads data into the target database. Set connection and index key data is generated and processed by later phases. If a task in this phase fails, when restarted, it formats only the page ranges assigned to its slice and restarts processing from the beginning.
RELOAD2	This is an overflow phase. If a reload slice fills up, the data is saved and processed by this phase. There is only one task in this phase, and it updates across slices. It generates the same set connection and index key data as RELOAD1 for records it stores. If an area fills up during this phase, processing stops. If the task in this phase fails, when restarted, REORG starts over by reprocessing all RELOAD1 tasks. Each formats its slice and restarts.
RELOAD3	Merges user-owned index data generated by UNLOAD, RELOAD1, and RELOAD2. Member keys and/or sequence numbers are merged in owner sequence and generated records are passed to RELOAD4.
RELOAD4	Merges data produced by RELOAD1 and RELOAD2 about old and new db-keys for chained sets and passes it to RELOAD6. If data is passed from RELOAD3, it rebuilds user indexes using that data. INDEX and OWNER pointer data is generated and passed to RELOAD6. If a task in this phase fails, when restarted, it reprocesses data from the beginning. It erases and rebuilds all user index occurrences in its slice. If a target slice fills before all data is processed, unconnected data is passed to RELOAD5. Tasks in this phase serialize on system index related page ranges.
RELOAD5	This is an overflow phase for RELOAD4. Data that would not fit into a slice is processed here. There is only one task in this phase, and it updates pages across all slices. If an area fills up during this phase, processing stops. Index key connection begins with the key that failed in RELOAD4. If the task in this phase fails, when restarted, REORG reprocesses all tasks in RELOAD4. User indexes that were already built are rebuilt. Tasks in this phase serialize on system index related page ranges.
RELOAD6	Updates prefix pointers using data generated by RELOAD phases 4 and 5; and REBUILD phase 3.
REBUILD2	This phase merges sequence data from an UNLOAD INDEX task with member data from RELOAD1 and RELOAD2. It is only generated for groups that have indexes for which sequence must be preserved. It can run in parallel with all REBUILD phases 2 through 4 for other index groups and in parallel with RELOAD phases 3 through 6.
REBUILD3	This phase uses member data from RELOAD1, RELOAD2, and REBUILD2 to rebuild system indexes. Tasks in one group can run in parallel with reload tasks and index tasks from other groups as long as there are no page range conflicts. Tasks in this phase serialize on system index related page ranges.
CLEANUP	There are no tasks in this phase. When all other processing has successfully completed, this phase optionally unlocks the reload areas and deletes work files created by the REORG utility.

Classes of work files

Each phase of REORG processing can produce different classes of work files. A task can produce one or more occurrences of a given class of work file. Each work file is read by a task in a later phase along with similar work files from other tasks.

The following table describes each class of work file. When discussing how many work files are generated, sometimes a task's phase name is referenced and sometimes the number of reload or unload slices is referenced. For each unload slice, there is one UNLOAD SLICE task and one RELOAD3 task. For each reload slice, there is one RELOAD1, RELOAD4, and RELOAD6 task. The number of reload slices is the same as the number of RELOAD1 tasks. Sometimes these terms are interchanged when it is thought to clarify a description.

The classes of work files are:

Work File Classes	Description
SYS002	Contains offloaded record and set information. Output: UNLOAD SLICE and UNLOAD INDEX Input: RELOAD1 One is created for each unload slice times the number of reload slices. One is created for each index group that contains a VIA system index where all members target one reload slice.
SYSX02	Contains offloaded record and set information for members of a VIA index. Output: UNLOAD SLICE and UNLOAD INDEX Input: REBUILD1 One is created for each unload slice times the number of index groups that contain VIA system indexes that target multiple reload slices. One is created for each index group that contains VIA system indexes that target multiple reload slices
SYSIX2	Contains sorted record and set information for members of a VIA index. Output: REBUILD1 Input: RELOAD1 One is created for each index group that contains VIA system indexes that target multiple reload slices times the number of reload slices.
SYSOF2	Contains record and set information for records that overflowed their target slices. Output: RELOAD1 Input: RELOAD2 One is created for each reload slice or RELOAD1 task.
SYS003	Contains offloaded index sequence information for user indexes. Output: UNLOAD SLICE Input: RELOAD3 One is created for each unload slice.
SYSX03	Contains offloaded index sequence information for system indexes. Output: UNLOAD INDEX Input: REBUILD2 One is created for each index group that contains system indexes that are not VIA and where member sequence is not being preserved.
SYS004	Work file containing sorted record and set information. Not created when SORTEXIT is specified. One is created for each RELOAD1 and RELOAD2 task.
SYS005	Contains old and new db-key values for chained sets. Output: RELOAD1 and RELOAD2 Input: RELOAD4 One is created for each (RELOAD1 and RELOAD2 task) times the number of RELOAD4 tasks.
SYS006	Contains user index owner and member db-key and symbolic key information. Output: RELOAD1 and RELOAD2 Input: RELOAD3 One is created for each reload slice times the number of unload slices.
SYSX06	Contains system index member db-key and symbolic key information. Output: RELOAD1 and RELOAD6 Input: REBUILD2 One is created for each reload slice times the number of index groups that contained a non-VIA system index where sequence was being preserved.
SYS007	Work file containing sorted user index owner and member work record information. Not created when SORTEXIT is specified. One is created for each RELOAD3 task.
SYSX07	Work file containing sorted user index owner and member work record information. Not created when SORTEXIT is specified. One is created for each REBUILD2 task.
SYS008	Contains user index information that is used to rebuild indexes. Output: RELOAD3 Input: RELOAD4 One is created for each unload slice times the number of reload slices.
SYSX08	Contains system index information that is used to rebuild indexes. Output: RELOAD1, RELOAD2, and REBUILD2 Input: REBUILD3 One is created for each REBUILD2 task that is generated. Also, one is created for each index group (that contains a VIA system index or a non-VIA system index where sequence was not being preserved) times the number of RELOAD1 and RELOAD2 tasks.
SYS009	Work file containing sorted user index work records and sorted old/new db-keys for chained sets. Not created when SORTEXIT is specified. One is created for each RELOAD4 and RELOAD5 task.
SYSX09	Work file containing sorted system index work records. Not created when SORTEXIT is specified. One is created for each REBUILD3 task.
SYS010	Contains new db-key information for updating record prefixes. Output: RELOAD4, RELOAD5, and REBUILD3. Input: RELOAD6 One is created for each (RELOAD4 and RELOAD5) task times the number of RELOAD6 tasks, and one is created for each index group (that contains indexes where the member contains UP or OWNER pointers) times the number of RELOAD6 tasks.
SYS011	Work file containing sorted prefix pointer information for all records. Not created when SORTEXIT is specified. One is created for each RELOAD6 task.
DBKEYS	A file created when the DBKEYS option is specified. Output: RELOAD1 and RELOAD2 One is created for each RELOAD1 and RELOAD2 task. All should be concatenated together when processed.