How Targets of an Operator are Processed › Operators Auto Recovery Example

Operators Auto Recovery Example

If Operators Auto Recovery is selected (the default), you do not have to initiate the recovery. Each Orchestrator in the environment detects that the agent or Orchestrator is active and runs the recovery. Each Orchestrator runs recovery on any enabled touchpoint, host group, or Orchestrator that is mapped to the agent or Orchestrator in the environment.

When an agent is mapped to many touchpoints, the Orchestrators run the recovery on all enabled touchpoints that are mapped to the agent. When an agent or Orchestrator becomes active, the Domain Orchestrator broadcasts the changes to the other Orchestrators. In a clustered setup, only the master node runs the recovery.

Example: Automatic Recovery

Consider the following scenario, in which an agent is mapped to the following touchpoints and host groups in an environment:

• TP_user01-w500,
• TP_user01-w500_2
• TP_user01-w500_3
• TP_manyAgents
• HG_user01-w500

Assume that all touchpoints and host groups are enabled and that only the following touchpoints and host groups are set to Operators Auto Recovery:

• TP_user01-w500
• TP_user01-w500_3
• HG_user01-w500

When the agent becomes active, each Orchestrator in the environment tries to run the automatic recovery on the three touchpoints.

Assume there are two Orchestrators in the environment:

Orchestrator1 log file contains:

• Message to signal the agent that became active again:

  2010-06-28 19:22:59,984 DEBUG 
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Checking whether to initiate Touchpoint/Orchestrator Recovery of 'System_Error' operators in 'Waiting' or 'Running' or 'Blocked' Processes (with auto recovery flag set), that ran against Agent/Server ID: f3492322-5517-4a21-8a19-92838ccb3f65
  

• Orchestrator mapped the agent to HG_user01-w500:

  2011-03-28 19:22:59,984 INFO 
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Initiating Touchpoint/Orchestrator: HG_user01-w500 Recovery... Recovering 'System_Error' Operators that ran against: HG_user01-w500 and belong to 'Waiting' or 'Running' or 'Blocked' Processes (with auto recovery flag set).
  2011-03-28 19:22:59,984 DEBUG 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieve the processes (with enabled auto recovery flag) that are in 'Waiting' or ‘Running’ or 'Blocked' states and contain operators in 'System Error' when ran against: HG_user01-w500
  

• Orchestrator signals 0 processes to be recovered for HG_user01-w500:

  2011-03-28 19:23:00,000 WARN  
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieved - 0 processes to recover for HG_user01-w500 recovery.
  2011-03-28 19:23:00,000 INFO  
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Found no Processes to recover for Touchpoint/Orchestrator: HG_user01-w500 Recovery.
  

• Orchestrator mapped the agent to TP_user01-w500:

  2011-03-28 19:23:00,000 INFO  
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Initiating Touchpoint/Orchestrator: TP_user01-w500 Recovery... Recovering 'System_Error' Operators that ran against: TP_user01-w500 and belong to 'Waiting' or 'Running' or 'Blocked' Processes (with auto recovery flag set).
  2011-03-28 19:23:00,000 DEBUG 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieve the processes (with enabled auto recovery flag) that are in 'Waiting' or ‘Running’ or 'Blocked' states and contain operators in 'System Error' when ran against: TP_user01-w500.
  

• Orchestrator signals 1 process to be recovered for TP_user01-w500:

  2011-03-28 19:23:00,015 WARN 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieved - 1 processes to recover for TP_user01-w500 recovery.
  

• Orchestrator queues recovery request for the 1 process (ROID: 2889) to be recovered for TP_user01-w500:

  2011-03-28 19:23:00,015 WARN 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Queueing recovery of process with ROID - 2889 for TP_user01-w500 recovery.
  2011-03-28 19:23:00,015 DEBUG 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Special Response for process recovery. ROID is 2889 UUID is 3fe95f08-a347-4d6d-a0a3-b3639836e130
  2011-03-28 19:23:00,015 DEBUG 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Special Response for process recovery was posted. ROID is 2889 UUID is 3fe95f08-a347-4d6d-a0a3-b3639836e130
  2011-03-28 19:23:00,015 INFO  
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Queued requests for Touchpoint: TP_user01-w500 Recovery... Recovering Processes with the following ROIDs: 2889
  

• Orchestrator signals that Touchpoint TP_user01-w500_2 is mapped to the agent but it is not set to automatic recovery:

  2011-03-28 19:23:00,015 DEBUG 
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Touchpoint: TP_user01-w500_2 mapped to Agent: f3492322-5517-4a21-8a19-92838ccb3f65 is not set to Automatic Recovery of 'System_Error' operators in 'Waiting' or ‘Running’ or 'Blocked' Processes.
  

• Orchestrator mapped the agent to TP_user01-w500_3:

  2011-03-28 19:23:00,015 INFO  
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Initiating Touchpoint/Orchestrator: TP_user01-w500_3 Recovery. Recovering 'System_Error' Operators that ran against: TP_user01-w500_3 and belong to 'Waiting' or 'Running' or 'Blocked' Processes (with auto recovery flag set).
  2011-03-28 19:23:00,015 DEBUG 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieve the processes (with enabled auto recovery flag) that are in 'Waiting' or ‘Running’ or 'Blocked' states and contain operators in 'System Error' when ran against:: TP_user01-w500_3
  

• Orchestrator signals 0 processes to be recovered for TP_user01-w500_3:

  2011-03-28 19:23:00,015 WARN  
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieved - 0 processes to recover for TP_user01-w500_3 recovery.
  2011-03-28 19:23:00,031 INFO  
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Found no Processes to recover for Touchpoint/Orchestrator: TP_user01-w500_3 Recovery.
  

• Orchestrator signals that Touchpoint TP_manyAgents is mapped to the agent but it is not set to automatic recovery:

  2011-03-28 19:23:00,031 DEBUG 
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Touchpoint: TP_manyAgents mapped to Agent: f3492322-5517-4a21-8a19-92838ccb3f65 is not set to Automatic Recovery  of 'System_Error' operators in 'Waiting' or ‘Running’ or 'Blocked' Processes 
  

• Orchestrator picks up the recovery request from the queue for process 2889:

  2011-03-28 19:23:30,015 INFO  
  [com.optinuity.c2o.workflowengine.FlowManagerCache] [onPool Worker-3] Registering flow manager: 2889
  2011-03-28 19:23:30,015 INFO  
  [com.optinuity.c2o.workflowengine.WorkflowManager] [onPool Worker-3] Handling Response: [$$TouchpointRecovery&&]. flow id is 2889
  2011-03-28 19:23:30,171 INFO  
  [com.optinuity.c2o.workflowengine.WorkflowManager] [onPool Worker-3] Start Recovery of Process with ROID:2889, for TP_user01-w500 Recovery.
  

• Orchestrator resets the operator(s) with System_Error when ran against TP_user01-w500 within the process 2889:

  2011-03-28 19:23:30,171 INFO  
  [com.optinuity.c2o.workflowengine.WorkflowManager] [onPool Worker-3] Resetting System_Error Operators in Process with ROID: 2889 for TP_user01-w500 Recovery.
  2011-03-28 19:23:30,203 INFO  
  [com.optinuity.c2o.workflowengine.WorkflowManager] [onPool Worker-3] Resetting Operator with ROID: 2912 in Process with ROID: 2889 for TP_user01-w500 Recovery.
  

• Orchestrator resumes the entire process 2889 after resetting the appropriate operators:

  2011-03-28 19:23:30,343 INFO  
  [com.optinuity.c2o.workflowengine.WorkflowManager] [onPool Worker-3] Resuming Process with ROID: 2889 for TP_user01-w500 Recovery.
  2011-03-28 19:23:30,343 INFO  
  [com.optinuity.c2o.workflowengine.WorkflowManager] [onPool Worker-3] Resuming WorkFlow - ROID is 2889
  2011-03-28 19:23:30,703 INFO  
  [com.optinuity.c2o.workflowengine.WorkflowManager] [onPool Worker-3] Done resuming Process with ROID: 2889 for TP_user01-w500 Recovery.
  

  Process 2889 completes successfully. Because the recovery process started automatically, the process logs indicate that the engine initiated the recovery. For manual recovery, logs include the name of the user who started the recovery.

Orchestrator2 (OrchestratorD610b) log file contains:

• The following for OrchestratorD610b is similar to Orchestrator:

  2011-03-28 19:23:41,444 INFO  
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Initiating Touchpoint/Orchestrator: HG_user01-w500 Recovery. Recovering 'System_Error' Operators that ran against: HG_user01-w500 and belong to 'Waiting' or 'Running' or 'Blocked' Processes (with auto recovery flag set).
  2011-03-28 19:23:41,444 WARN 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieved - 0 processes to recover for HG_user01-w500 recovery.
  2011-03-28 19:23:41,444 INFO  
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Found no Processes to recover for Touchpoint/Orchestrator: HG_user01-w500 Recovery...
  2011-03-28 19:23:41,444 INFO  
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Initiating Touchpoint/Orchestrator: TP_user01-w500 Recovery... Recovering 'System_Error' Operators that ran against: TP_user01-w500 and belong to 'Waiting' or 'Running' or 'Blocked' Processes (with auto recovery flag set).
  2011-03-28 19:23:41,444 WARN  
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieved - 0 processes to recover for TP_user01-w500 recovery.
  2011-03-28 19:23:41,444 INFO 
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Found no Processes to recover for Touchpoint/Orchestrator: TP_user01-w500 Recovery...
  2011-03-28 19:23:41,444 INFO 
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Initiating Touchpoint/Orchestrator: TP_user01-w500_3 Recovery... Recovering 'System_Error' Operators that ran against: TP_user01-w500_3 and belong to 'Waiting' or 'Running' or 'Blocked' Processes (with auto recovery flag set).
  

• OrchestratorD610b finds 1 process to recover for TP_user01-W500_3 (process ROID: 541) and queues its recovery request:

  2011-03-28 19:23:41,444 WARN 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieved - 1 processes to recover for TP_user01-w500_3 recovery.
  2011-03-28 19:23:41,444 WARN 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Queueing recovery of process with ROID - 541 for TP_user01-w500_3 recovery.
  2011-03-28 19:23:41,444 INFO  
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Queued requests for Touchpoint: TP_user01-w500_3 Recovery... Recovering Processes with the following ROIDs: 541
  2011-03-28 19:23:41,444 WARN 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Retrieved - 1 processes to recover for TP_user01-w500_3 recovery.
  2011-03-28 19:23:41,444 WARN 
  [com.optinuity.c2o.workflowengine.C2oWorkFlowEngine] [aListenerThread] recoverSystemErrorProcesses: Queueing recovery of process with ROID - 541 for TP_user01-w500_3 recovery.
  2011-03-28 19:23:41,444 INFO 
  [com.optinuity.c2o.c2oserver.ServerManager] [aListenerThread] Queued requests for Touchpoint: TP_user01-w500_3 Recovery... Recovering Processes with the following ROIDs: 541
  

• OrchestratorD610b picks up the recovery request, resets the Operators with System_Error, which ran against TP_user01-w500_3, in the process and resumes the process:

  2011-03-28 19:24:11,461 INFO 
  [com.optinuity.c2o.workflowengine.WorkflowManager] [nPool Worker-63] Handling Response: [$$TouchpointRecovery&&]. flow id is 541
  2011-03-28 19:24:11,804 INFO 
  [com.optinuity.c2o.workflowengine.WorkflowManager] [nPool Worker-63] Start Recovery of Process with ROID:541, for TP_user01-w500_3 Recovery.
  2011-03-28 19:24:11,820 INFO 
  [com.optinuity.c2o.workflowengine.WorkflowManager] [nPool Worker-63] Resetting System_Error Operators in Process with ROID: 541 for TP_user01-w500_3 Recovery.
  2011-03-28 19:24:11,883 INFO 
  [com.optinuity.c2o.workflowengine.WorkflowManager] [nPool Worker-63] Resetting Operator with ROID: 564 in Process with ROID: 541 for TP_user01-w500_3 Recovery.
  2011-03-28 19:24:12,039 INFO 
  [com.optinuity.c2o.workflowengine.C2OSvcIconInstanceRefObject] [nPool Worker-63] Inside Update ConnectorTraversed, Current/destination is Start_System_Process_1 Source is Start_1
  2011-03-28 19:24:12,117 INFO 
  [com.optinuity.c2o.workflowengine.WorkflowManager] [nPool Worker-63] Resuming Process with ROID: 541 for TP_user01-w500_3 Recovery.
  2011-03-28 19:24:12,117 INFO 
  [com.optinuity.c2o.workflowengine.WorkflowManager] [nPool Worker-63] Resuming WorkFlow - ROID is 541
  2011-03-28 19:24:12,133 INFO 
  [com.optinuity.c2o.workflowengine.C2OSvcIconInstanceRefObject] [nPool Worker-63] Inside Update ConnectorTraversed, Current/destination is Start_System_Process_1 Source is Start_1
  2011-03-28 19:24:12,648 INFO  [com.optinuity.c2o.workflowengine.WorkflowManager] [nPool Worker-63] Done resuming Process with ROID: 541 for TP_user01-w500_3 Recovery.