Reference Information › Appliance Catalog Reference Guide › System Catalog › Monitoring Appliances › MON Application Monitoring Appliance

MON Application Monitoring Appliance

Latest version: 4.1.3-1

Functional Overview

MON is an application monitoring appliance that collects performance and resource usage statistics, such as counters, from other appliances in an application. Appliances report their counters to MON through the mon terminal.

MON provides a visual interface using a web browser to access and view the appliance counters. Through the visual interface, you create views containing one or more graphs. Each graph displays one or more appliance counters.

In addition, MON supports background monitoring of appliance counters. All counters in a single view, as specified by the alarm_view property, are monitored in the background by MON. Whenever one of these counters falls below or exceeds a specified threshold, MON generates an alarm to the grid controller, which posts a message to the dashboard. A browser to the MON visual interface does not have to be open for MON to monitor the appliance counters in the background.

Collection is done using a custom REST-based protocol, running over HTTP. For additional information on the monitoring API functionality, refer to the Monitoring API Reference.

Resources

Terminals

The default interface is enabled. The default interface is used by the appliance to report to AppLogic that it has booted successfully. It can also be used to log in over secure shell to the appliance from the controller, primarily for diagnostic and troubleshooting purposes. The default interface is used by the GUI to access the visual interface of a MON appliance.

User Volumes

Properties

Error Messages

The following messages may appear in either the appliance log file or the system log of the grid controller when the appliance fails to start:

• failed to mount data volume
• httpd did not start

Counter Groups and Counters

The following counters groups are defined for all appliances:

• CPU Summary
• CPU n (only if more than 1 CPU -- otherwise no such group(s))
• Scheduler
• Memory statistics
• Filesystem
• Interface name (one group for each interface: Terminals or raw)
• Volume name (one group per volume, including class and placeholders)

See Creating Custom Counters on how to create and monitor custom counters in an appliance.

Monitoring Applications Through the CA AppLogic GUI

The following diagram shows a typical usage of MON for a simple application. In this application, the monitor can be accessed through the AppLogic GUI only.

Appliances in use:

• in - input gateway, class IN
• srv - web server, class WEB6
• db - database server, class MYSQL5
• mon - monitoring appliance, class MON

The mon terminal of all appliances are connected to the mon terminal of mon.

Note: To display graphs using Windows Internet Explorer, you must set display all websites in compatibility mode. To set this mode, access the menu bar, click Tools and Compatibility View Settings, then select the Display all websites in Compatibility View check box and click close.

Monitoring Application Through Both the GUI and an IP Address

The following diagram shows how MON can be used to access the application monitor through the GUI or through an IP address on port 80.

Appliances in use:

• usr - input gateway, class IN
• admin - input gateway, class IN
• srv - web server, class WEB6
• db - database server, class MYSQL5
• mon - monitoring appliance, class MON

The mon terminal of all appliances are connected to the mon terminal of mon.

The out terminal of admin is connected to the web terminal of mon so users can access the monitor through the IP address specified by the ip_addr property on admin.

Monitoring Applications Through Both the CA AppLogic GUI and an Application IP Address on Port 8080

The following diagram shows how MON can be used to access the application monitor through the GUI or through an application IP address on port 8080.

Appliances in use:

• INSSLR - input gateway, class INSSLR
• WEBx8 - web servers assembly, class WEBx8
• MYSQL - database server, class MYSQL5
• NAS - storage appliance, class NAS
• PS8 - port switch, class PS8
• mail - mail server, class MAIL
• NET - output gateway, class NET
• mon - monitoring appliance, class MON

The mon terminal of all appliances are connected to the mon terminal of mon.

INSSLR is configured to forward https traffic to the WEBx8 and all other TCP traffic to the aux terminal. PS8 is configured to forward all traffic on port 8080 through out8 to the web terminal of mon. This allows users to monitor the application through the application IP address specified by the ip_addr property on INSSLR on port 8080.

Note: The MAIL appliance is not distributed with CA AppLogic.

Background Monitoring

This use case describes how to use MON to monitor appliance counters in the background and have it generate alerts when an appliance counter is out of range.

1. Set the alarm_view property on MON to the name of a view that contains the counters to be monitored.
2. (Re)start the application.
3. Login to the Application Monitor.
4. Click on the New button to create a new view and give it the same name as the alarm_view property.
5. Click on the Add Graph button to create a new graph and add it to the view.
6. Select a counter to be monitored and add it to the graph.
7. Set the alarm threshold by filling in the two boxes next to Alarm.
   To set a minimum threshold, enter a value in the left text box. There is no minimum threshold if this text box is left empty.
   To set a maximum threshold, enter a value in the right text box. There is no maximum threshold if this text box is left empty.
8. Repeat steps 5-7 to monitor additional appliance counters.

Whenever an appliance counter falls below or exceeds the specified threshold, MON generates an alert to the CA AppLogic Controller. A message is then logged to the dashboard specifying which counter of which appliance is out of range. For each appliance, the CA AppLogic Controller only logs the last alert to the dashboard, however it will specify in the dashboard message if multiple alerts have been received for the appliance. All alerts are logged to /var/log/messages on the CA AppLogic Controller.

Note: A browser to the Application Monitor does not have to be open for MON to monitor the appliance counters in the background.

To stop monitoring a specific appliance counter:

• If the counter is the only counter on the graph, delete the graph.
• If the counter is on a graph with multiple counters, remove the counter from the graph.

To disable background monitoring, execute one of the following:

• Rename the view to a name that is different than the alarm_view property on MON.
• Delete all graphs in the view.
• Delete the view.
• Stop the MON appliance in the application (for example, comp stop myapp:main.mon).
• Clear the alarm_view property on MON and restart the MON instance (or restart the application).

Data Collection Interface

Since version 1.1.3, the MON appliance provides a REST-service-like data collection API, it exports counter configuration and values in XML, JSON and UDL formats. For more details on the API, visit MonDataCollectionInterface.

Notes

Known Limitations

To monitor an application through the GUI, the name of the MON appliance in the application should be mon.

Open source and 3rd party software used inside of the appliance

Appliance Counters

Overview

This document defines the list of all standard counters exported by CCA. CCA is the counter collection agent that runs in all AppLogic appliances. CCA is used to report counters to the monitoring GUI exposed by the MON appliance. The availability of these counters might be limited on some of the supported appliance platforms. When the appliance OS supports a given counter it should be visible in the MON GUI when the appliance's mon output terminal is connected to the MON appliance.

CPU Summary

File System

Memory Statistics

Scheduler

CPU#X

Volume X

Terminal X

Overview

Each appliance running the ccad (counter collection agent) daemon allows for custom counters to be defined and collected by 3rd party utilities. This ability is named extension interface and provides appliance creators with option to monitor appliance-specific counter data through the standard MON user interface.

Creating Custom Counters

To monitor custom counters you need to add the counters definitions to the ccad configuration and feed the actual counter values to ccad

Adding counter definitions

The custom counters have to be defined in the optional /etc/ccad.conf configuration file in UDL format. When you change the configuration, the ccad daemon must be restart for the changes to take effect.

Here is a simple example configuration:

counters Apache
   {
   pace = 1000
   pipe = /tmp/cca
   counter Total_Accesses
      {
      name        = "Total hits"
      desc        = "Total number of hits"
      units       = "#"
      type        = "MAX"
      }
   counter Total_kBytes
      {
      name        = "Total bytes"
      desc        = "Total number of bytes"
      units       = "bytes"
      type        = "MAX"
      }
   counter BusyWorkers
      {
      name        = "Active requests"
      desc        = "Number of active requests"
      units       = "#"
      type        = "MAX"
      }
   counter IdleWorkers
      {
      name        = "Idle servers"
      desc        = "Number of idle servers"
      units       = "#"
      type        = "MAX"
      }
   }

This example defines 4 counters. Each counter entity supports the following attributes:

• name - Human-readable name of the counter (set to counter entity name by default)
• desc - Human-readable description of the counter (empty by default)
• type - Type of the counter (SUM, AVG, MIN or MAX, the default is AVG)
• units - Units in which the counter values should be interpreted (empty by default)
• range_lower - The lowest possible value of that counter (smaller values will be cut)
• range_upper - The highest possible value of that counter (larger values will be cut)
• alarm_below - The lowest non-alarming value of the counter (smaller values will trigger alarm)
• alarm_above - The highest non-alarming value of the counter (larger values will trigger alarm)

The root-level counters entity's name is used for Entity name when the counters are later shown in MON. Additionally, the following attributes are meaningful for this entity:

• pace - Pace (in milliseconds) of the custom counters collection (1000 if not specified)
• pipe - Path to the named pipe used for collection (/tmp/cca if not specified)

In addition to counter definition method described above, ccad.conf allows counters to be grouped together. This is achieved by placing them one level deeper in group entities as in the following example:

counters Plants
   {
   pace = 2000
   pipe = /my-pipe
   counter tree: desc = "Any tree" , units = "#"
   group Fruits
      {
      desc = "Delicious Fruits!!!"
      counter pears : desc = "Number of pears"   , units = "pears"
      counter apples: desc = "Number of apples"  , units = "apples"
      }
   group Vegetables
      {
      counter potato: desc = "Number of potatoes", units = "potatoes"
      counter tomato: desc = "Number of tomatoes", units = "tomatoes"
      }
   }

This configuration would produce 3 custom counter entities in MON's GUI - Plants, Delicious Fruits!!! and Vegetables with the 5 defined counters distributed among them. Groups support a desc attribute (just like counter entities) that can be used to define a richer group name as opposed to the more constrained group entity name. Counter names should be unique across the configuration file otherwise the counter collection would be made impossible for the duplicate counter names.

Feeding counter data

Once you have the ccad running with the proper counter definitions, you need to feed the actual counter values. To do so, write the counter values to the named pipe create by ccad (specified by the pipe attribute in /etc/ccad.conf, default /tmp/cca).
The proper format is one of the following:

counter = value (name and value are separated with =)

counter : value (name and value are separated with :)
A counter set should be flushed (to MON) by printing . or form feed character (\f) in the named pipe on a line by itself. A counter collection script (in Bash) could do the following to feed counter to ccad:

echo "Total_Accesses = 31" > /tmp/cca
echo "Total_kBytes = 22241" > /tmp/cca
echo "BusyWorkers = 6" > /tmp/cca
echo "IdleWorkers = 34" > /tmp/cca
echo "." > /tmp/cca

This should be done every pace milliseconds as specified in /etc/ccad.conf

Here is a simple bash loop that would collect the needed data and feed it to ccad. We feed more data than we need as it faster to give all the collected data to ccad (which will ignore the lines that are not counter values) than to parse the data before feeding it to ccad:

while true; do
   curl -s http://in/server-status?auto | sed "s/^Total\ /Total_/g" > /tmp/cca # this parses the data so it matches out counter definitions
   sleep 1 || exit 4        
   echo "." > /tmp/cca
done

Examples

Windows Example

The following Windows custom counter example uses Perl.  To install the Perl Win32 package on a managed Windows appliance if it is not already present:

• Copy the Cygwin legacy command line installer to the appliance.  The installer can be downloaded from http://www.cygwin.com/setup-legacy.exe
• Copy the version 0.27 Perl Win32 package to the appliance (for example, to c:\local_cygwin_pkg).  This package can be downloaded from several sources including http://cygwin.basemirror.de/release-legacy/perl/perl-libwin32/perl-libwin32-0.27-1.tar.bz2
• Install the Perl Win32 package.  In a command shell in the graphic console of the appliance execute:  setup-legacy.exe -q -n -N -d -R c:\cygwin -l c:\local_cygwin_pkg -L

The following is the code example:

#!/bin/perl
use Win32::Pipe;

my $defaultPace=1000;
my $defaultPipe="\\\\.\\pipe\\cca";
my $ccadConf="/etc/ccad.conf";

sub get_value_from_config
   {
   open(F, $ccadConf) || die "can't open $ccadConf: $!\n";
   while(<F>)
      {
      #($name, $value) = split('=');
      my $row=$_;
      $row =~ m/^\s*(\w+)\s*=\s*(.*)\s*$/;
      if ($_[0] eq $1)
         {
         close F;
         return $2;
         }
      }
   close F;
   return "";
   }

my $cfgPace = &get_value_from_config("pace");
my $cfgPipe = &get_value_from_config("pipe");
if ($cfgPace eq "") { $cfgPace = $defaultPace; }
if ($cfgPipe eq "") { $cfgPipe = $defaultPipe; }
else { $cfgPipe = "\\\\.\\pipe\\".$cfgPipe; }

my $pipe;
$pipe = new Win32::Pipe($cfgPipe) || die "can't connect to pipe\n";
$pipe->ResizeBuffer(4096);

# this loop feeds a counter named Test with a value which increments on each iteration
my $val = 0;
while (1)
  {
   $val ++;
   my $str = "Test = $val\n.";
   $pipe->Write( $str );
   sleep( $cfgPace / 1000 );
   }

Mysql

#!/bin/bash

DEFAULT_PACE=1000
DEFAULT_PIPE=/tmp/cca
CCAD_CONF=/etc/ccad.conf
MYSQL='/usr/bin/mysql'
TR=/usr/bin/tr
SLEEP=/bin/sleep
function get_value_from_config {
   CONF_VALUE=`grep -oE "$1[[:space:]]*=[[:space:]]*[^[:space:]]+" $CCAD_CONF |cut -d"=" -f 2|tr -d " "`
}
get_value_from_config "pace"
PACE=${CONF_VALUE:-$DEFAULT_PACE}
PACE=$(( $PACE / 1000 ))
get_value_from_config "pipe"
PIPE=${CONF_VALUE:-$DEFAULT_PIPE}
test -p $PIPE || exit 3
while true; do
   $MYSQL -e 'show status'|$TR '[[:blank:]]' '=' > $PIPE
   $SLEEP $PACE || exit 4
   echo "." > $PIPE
done

Linux Examples

Apache

Note: The server must have 'extended status' enabled

#!/bin/bash

DEFAULT_PACE=1000
DEFAULT_PIPE=/tmp/cca
CCAD_CONF=/etc/ccad.conf
STATUS_URL='http://in/server-status?auto'
CURL=/usr/bin/curl
SED=/bin/sed
SLEEP=/bin/sleep
function get_value_from_config {
   CONF_VALUE=`grep -oE "$1[[:space:]]*=[[:space:]]*[^[:space:]]+" $CCAD_CONF |cut -d"=" -f 2|tr -d " "`
}
get_value_from_config "pace"
PACE=${CONF_VALUE:-$DEFAULT_PACE}
PACE=$(( $PACE / 1000 ))
get_value_from_config "pipe"
PIPE=${CONF_VALUE:-$DEFAULT_PIPE}
test -p $PIPE || exit 3
while true; do
   $CURL -s "$STATUS_URL" | $SED "s/^Total\ /Total_/g" > $PIPE
   sleep $PACE || exit 4
   echo "." > $PIPE
done