Event and Alert Management Deployment Scenarios › Event Management Scenarios › Event Management Example 3: Create a New Event to Indicate a Crashing Service

Event Management Example 3: Create a New Event to Indicate a Crashing Service

This scenario illustrates how you can correlate events that occur together to indicate a different or more severe condition than when the events occur separately. You create an event to indicate the correlated condition. Several conditions are detectable only with the correlation of separate event occurrences or the same event. The follow events are such situations:

• A persistent CPU or memory deficiency (which can be more severe than an occasional spike)
• Servers in a cluster going down at the same time
• Any situation where the root cause of a condition may not be evident through service or CI hierarchy and relationships

Correlating events lets you represent the true condition in a new event that you can use to trigger escalation policy to resolve the problem.

This scenario assumes that you have connectors monitoring running services, and you have had problems in the past with services that shut down immediately after they are started. It does the following:

• Detects when service startup and shutdown occur within a short amount of time from one another
• Creates a new event that increases the severity and modifies the event summary to indicate the problem
• Discards the original events
• Creates escalation policy that triggers based on the new event summary

Follow these steps:

1. Select the Mid-tier connector in the Data Source list and enter the following in the Event Pattern fields in the Event Search tab:

   AlertedMdrElementID=? and matches (Summary,'service has started')
   

   AlertedMdrElementID=? and matches (Summary,'service has stopped')
   

   This search criteria returns events from the same connector and CI, where the first event summary contains the text 'service has started', and the second event summary contains the text 'service has stopped'.

2. Select ALL events occur within 45 seconds in the Additional Criterion pane, and select the Sequence enforced check box.

   This selection specifies that the events must occur within 45 seconds of each other and that the 'service has started' event must occur before the 'service has stopped' event.

3. Click Search.

   The search results appear.

4. Click Create Policy.

   The Create Event Policy wizard opens and displays the New Policy page.

5. Enter ServiceCrash in the Policy Name field, select Create New Event, and click Next.

   The Create New Event page opens.

6. Edit the properties of the new event as follows and click Next:
   • Set the Severity to Critical.
   • Set the Summary to 'Service crashing immediately after startup'.
   • Retain the defaults for other properties, which inherits the properties of the event from the first pattern.
   • Set the mdrElementID to fx:uniqueidentifier().

   This change increases the severity to critical and changes the summary to a specific indication of the correlated problem.

   The Select Data Sources page opens.

7. Select Save and Deploy policy, move the Mid-tier connector to the Selected Data Sources pane, and click Next.

   Note: If only certain connectors, such as the CA NSM connector, are monitoring services, you can assign to specific connectors instead.

   The Confirm page opens.

8. Confirm the policy information and click Finish.

   The policy is deployed.

9. Select the deployed policy in the Events tab, and click Edit Policy.

   The Create Event Policy wizard opens and displays the New Policy page.

10. Enter FilterCorrelatedEvents in the Policy Name field, select Filter Events and then Exclude, and click Next.

    The Select Data Sources page opens.

11. Select Save and Deploy policy, retain the Mid-tier connector in the Selected Data Sources pane, and click Finish.

    The filter event policy is deployed. This policy discards the original service startup and shutdown events, so that only the created event becomes an alert in the Operations Console.

12. Select Tools, Escalation Policies and Actions.

    The Escalation Policies and Actions dialog opens.

13. Click Add.

    The Alert Escalation Policy Editor dialog opens.

14. Enter Service Crash Policy in the Name field and click the Attributes tab.

    A pane opens for specifying alert attribute-specific criteria.

15. Select Summary in the Attribute drop-down list, Equal To in the Comparison Type drop-down list, and enter 'Service crashing immediately after startup' in the Attribute Value field. Click Add.

    The policy triggers when an alert occurs with the summary you specified for the new event.

16. Select the Policy Actions tab and click New.

    The Escalation Action Editor dialog opens.

17. Enter Create Service Crash Ticket in the Action Name field and select Create Ticket in the Action Type drop-down list.

    Tabs appear for specifying ticket properties.

18. Select Summary in the Property Name drop-down list, enter 'Service is crashing immediately after startup' in the Property Value field, and click Add.

    The ticket summary matches the alert summary.

19. Click OK.

    The action is saved.

20. Click OK on the Alert Escalation Policy Editor dialog.

    CA SOI saves the escalation policy. When the deployed event policy detects the correlated event condition, the following actions occur:

    • A new Critical event is created with a descriptive summary
    • The original events are discarded
    • When the event appears in the Operations Console as an alert, it triggers an escalation policy that creates a help desk ticket