System Catalog › Monitoring Appliances › MON: Application Monitoring Appliance

MON: Application Monitoring Appliance

Latest version: 3.0.3-1

Functional Overview

MON is an application monitoring appliance which collects performance and resource usage statistics (that is, counters) from other appliances within an application. Appliances report their counters to MON through the mon terminal.

MON provides a visual interface using a web browser to access and view the appliance counters. Through the visual interface, the user creates views containing one or more graphs. Each graph displays one or more appliance counters. The visual interface is accessible through the following:

• The CA 3Tera AppLogic GUI
  • a Monitor button at the bottom of the Applications tab of the Dashboard
  • a monitor icon in the toolbar of the infrastructure Editor
• through a gateway (that is, IN or INSSLR gateway) connected to the web terminal of MON

See the User Interface Reference Guide for additional details on accessing and using the visual interface.

MON also supports background monitoring of appliance counters. All counters in a single view, as specified by the alarm_view property, are monitored in the background by MON. Whenever one of these counters falls below or exceeds a specified threshold, MON generates an alarm to the CA 3Tera AppLogic Grid Controller, which posts a message to the dashboard. A browser to MON's visual interface does not have to be open for MON to monitor the appliance counters in the background.

Boundary

Resources

Terminals

The default interface is enabled. The default interface is used by the appliance to report to CA 3Tera AppLogic that it has booted successfully. It can also be used to log in over secure shell to the appliance from the CA 3Tera AppLogic controller, primarily for diagnostic and troubleshooting purposes. The default interface is used by the CA 3Tera AppLogic GUI to access the visual interface of a MON appliance.

User Volumes

Properties

Error Messages

The following messages may appear in either the appliance log file or the system log of the grid controller when the appliance fails to start:

• failed to mount data volume

httpd did not start

Counter Groups and Counters

The following counters groups are defined for all appliances:

• CPU Summary
• CPU n (only if more than 1 CPU -- otherwise no such group(s))
• Scheduler
• Memory statistics
• Filesystem
• Interface name (one group for each interface: Terminals or raw)
• Volume name (one group per volume, including class and placeholders)

See Creating Custom Counters on how to create and monitor custom counters in an appliance.

Typical Usage

Monitoring Applications Through the CA 3Tera AppLogic GUI

The following diagram shows a typical usage of MON for a simple application. In this application, the monitor can be accessed through the CA 3Tera AppLogic GUI only.

Appliances in use:

• in - input gateway, class IN
• srv - web server, class WEB5
• db - database server, class MYSQL5
• mon - monitoring appliance, class MON

The mon terminal of all appliances are connected to the mon terminal of mon.

Monitoring Application Through Both the CA 3Tera AppLogic GUI and an IP Address on Port 80

The following diagram shows how MON can be used to access the application monitor through the CA 3Tera AppLogic GUI or through an IP address.

Appliances in use:

• usr - input gateway, class IN
• admin - input gateway, class IN
• srv - web server, class WEB5
• db - database server, class MYSQL5
• mon - monitoring appliance, class MON

The mon terminal of all appliances are connected to the mon terminal of mon.

The out terminal of admin is connected to the web terminal of mon so users can access the monitor through the IP address specified by the ip_addr property on admin.

Monitoring Applications Through Both the CA 3Tera AppLogic GUI and an Application IP Address on Port 8080

The following diagram shows how MON can be used to access the application monitor through the CA 3Tera AppLogic GUI or through an application IP address on port 8080.

Appliances in use:

• INSSLR - input gateway, class INSSLR
• WEBx8 - web servers assembly, class WEBx8
• MYSQL - database server, class MYSQL5
• NAS - storage appliance, class NAS
• PS8 - port switch, class PS8
• mail - mail server, class MAIL
• NET - output gateway, class NET
• mon - monitoring appliance, class MON

The mon terminal of all appliances are connected to the mon terminal of mon.

INSSLR is configured to forward https traffic to the WEBx8 and all other TCP traffic to the aux terminal. PS8 is configured to forward all traffic on port 8080 through out8 to the web terminal of mon. This allows users to monitor the application through the application IP address specified by the ip_addr property on INSSLR on port 8080.

Note: The MAIL appliance is not distributed with CA 3Tera AppLogic.

Background Monitoring

This use case describes how to use MON to monitor appliance counters in the background and have it generate alerts when an appliance counter is out of range.

1. Set the alarm_view property on MON to the name of a view that contains the counters to be monitored.
2. (Re)start the application.
3. Login to the Application Monitor.
4. Click on the New button to create a new view and give it the same name as the alarm_view property.
5. Click on the Add Graph button to create a new graph and add it to the view.
6. Select a counter to be monitored and add it to the graph.
7. Set the alarm threshold by filling in the two boxes next to Alarm.
   To set a minimum threshold, enter a value in the left text box. There is no minimum threshold if this text box is left empty.
   To set a maximum threshold, enter a value in the right text box. There is no maximum threshold if this text box is left empty.
8. Repeat steps 5-7 to monitor additional appliance counters.

Whenever an appliance counter falls below or exceeds the specified threshold, MON generates an alert to the CA 3Tera AppLogic Controller. A message is then logged to the dashboard specifying which counter of which appliance is out of range. For each appliance, the CA 3Tera AppLogic Controller only logs the last alert to the dashboard, however it will specify in the dashboard message if multiple alerts have been received for the appliance. All alerts are logged to /var/log/messages on the CA 3Tera AppLogic Controller.

Note: A browser to the Application Monitor does not have to be open for MON to monitor the appliance counters in the background.

To stop monitoring a specific appliance counter:

• If the counter is the only counter on the graph, delete the graph.
• If the counter is on a graph with multiple counters, remove the counter from the graph.

To disable background monitoring, execute one of the following:

• Rename the view to a name that is different than the alarm_view property on MON.
• Delete all graphs in the view.
• Delete the view.
• Stop the MON appliance in the application (for example, comp stop myapp:main.mon).

Clear the alarm_view property on MON and restart the MON instance (or restart the application).

Data Collection Interface

Since version 1.1.3, the MON appliance provides a REST-service-like data collection API, it exports counter configuration and values in XML, JSON and UDL formats. For more details on the API, visit MonDataCollectionInterface.

Notes

Known Limitations

To monitor an application through the CA 3Tera AppLogic GUI, the name of the MON appliance in the application should be mon.

Open source and 3rd party software used inside of the appliance

Appliance Counters

List of Standard Appliance Counters

Overview

This document defines the list of all standard counters exported by CCA. CCA is the counter collection agent that runs in all CA 3Tera AppLogic appliances. CCA is used to report counters to the monitoring GUI exposed by the MON appliance. The availability of these counters might be limited on some of the supported appliance platforms (as stated in the counters table below). When the appliance OS supports a given counter it should be visible in the MON GUI when the appliance's mon output terminal is connected to the MON appliance.

CPU Summary

File System

Memory Statistics

Scheduler

CPU#X

Volume X

Terminal X

Custom Counters

Overview

Each appliance running the ccad (counter collection agent) daemon allows for custom counters to be defined and collected by 3rd party utilities. This ability is named extension interface and provides appliance creators with option to monitor appliance-specific counter data through the standard MON user interface.

Creating Custom Counters

To monitor custom counters you need to add the counters definitions to the ccad configuration and feed the actual counter values to ccad

Adding counter definitions

The custom counters have to be defined in the optional /etc/ccad.conf configuration file (in UDL format). Whenever you make changes to the configuration, the ccad daemon must be restart for the changes to take effect.

Here is a simple example configuration:

counters Apache
   {
   pace = 1000
   pipe = /tmp/cca
   counter Total_Accesses
      {
      name        = "Total hits"
      desc        = "Total number of hits"
      units       = "#"
      type        = "MAX"
      }
   counter Total_kBytes
      {
      name        = "Total bytes"
      desc        = "Total number of bytes"
      units       = "bytes"
      type        = "MAX"
      }
   counter BusyWorkers
      {
      name        = "Active requests"
      desc        = "Number of active requests"
      units       = "#"
      type        = "MAX"
      }
   counter IdleWorkers
      {
      name        = "Idle servers"
      desc        = "Number of idle servers"
      units       = "#"
      type        = "MAX"
      }
   }

This example defines 4 counters. Each counter entity supports the following attributes:

• name - Human-readable name of the counter (set to counter entity name by default)
• desc - Human-readable description of the counter (empty by default)
• type - Type of the counter (SUM, AVG, MIN or MAX, the default is AVG)
• units - Units in which the counter values should be interpreted (empty by default)
• range_lower - The lowest possible value of that counter (smaller values will be cut)
• range_upper - The highest possible value of that counter (larger values will be cut)
• alarm_below - The lowest non-alarming value of the counter (smaller values will trigger alarm)
• alarm_above - The highest non-alarming value of the counter (larger values will trigger alarm)

The root-level counters entity's name is used for Entity name when the counters are later shown in MON. Additionally, the following attributes are meaningful for this entity:

• pace - Pace (in milliseconds) of the custom counters collection (1000 if not specified)
• pipe - Path to the named pipe used for collection (/tmp/cca if not specified)

In addition to counter definition method described above, ccad.conf allows counters to be grouped together. This is achieved by placing them one level deeper in group entities as in the following example:

counters Plants
   {
   pace = 2000
   pipe = /my-pipe
   counter tree: desc = "Any tree" , units = "#"
   group Fruits
      {
      desc = "Delicious Fruits!!!"
      counter pears : desc = "Number of pears"   , units = "pears"
      counter apples: desc = "Number of apples"  , units = "apples"
      }
   group Vegetables
      {
      counter potato: desc = "Number of potatoes", units = "potatoes"
      counter tomato: desc = "Number of tomatoes", units = "tomatoes"
      }
   }

This configuration would produce 3 custom counter entities in MON's GUI - Plants, Delicious Fruits!!! and Vegetables with the 5 defined counters distributed among them. Groups support a desc attribute (just like counter entities) that can be used to define a richer group name as opposed to the more constrained group entity name. Counter names should be unique across the configuration file otherwise the counter collection would be made impossible for the duplicate counter names.

Feeding counter data

Once you have the ccad running with the proper counter definitions, you need to feed the actual counter values. To do so, write the counter values to the named pipe create by ccad (specified by the pipe attribute in /etc/ccad.conf, default /tmp/cca).
The proper format is one of the following:

counter = value (name and value are separated with =)

counter : value (name and value are separated with :)
A counter set should be flushed (to MON) by printing . or form feed character (\f) in the named pipe on a line by itself. A counter collection script (in Bash) could do the following to feed counter to ccad:

echo "Total_Accesses = 31" > /tmp/cca
echo "Total_kBytes = 22241" > /tmp/cca
echo "BusyWorkers = 6" > /tmp/cca
echo "IdleWorkers = 34" > /tmp/cca
echo "." > /tmp/cca

This should be done every pace milliseconds as specified in /etc/ccad.conf

Here is a simple bash loop that would collect the needed data and feed it to ccad. We feed more data than we need as it faster to give all the collected data to ccad (which will ignore the lines that are not counter values) than to parse the data before feeding it to ccad:

while true; do
   curl -s http://in/server-status?auto | sed "s/^Total\ /Total_/g" > /tmp/cca # this parses the data so it matches out counter definitions
   sleep 1 || exit 4        
   echo "." > /tmp/cca

done

Examples

Windows Example

The following Windows custom counter example uses Perl. To install the Perl Win32 package on a managed Windows appliance if it is not already present:

• Copy the Cygwin legacy command line installer to the appliance. The installer can be downloaded from http://www.cygwin.com/setup-legacy.exe
• Copy the version 0.27 Perl Win32 package to the appliance (for example, to c:\local_cygwin_pkg). This package can be downloaded from several sources including http://cygwin.basemirror.de/release-legacy/perl/perl-libwin32/perl-libwin32-0.27-1.tar.bz2
• Install the Perl Win32 package. In a command shell in the graphic console of the appliance execute: setup-legacy.exe -q -n -N -d -R c:\cygwin -l c:\local_cygwin_pkg -L

The following is the code example:

#!/bin/perl
use Win32::Pipe;

my $defaultPace=1000;
my $defaultPipe="\\\\.\\pipe\\cca";
my $ccadConf="/etc/ccad.conf";

sub get_value_from_config
   {
   open(F, $ccadConf) || die "can't open $ccadConf: $!\n";
   while(<F>)
      {
      #($name, $value) = split('=');
      my $row=$_;
      $row =~ m/^\s*(\w+)\s*=\s*(.*)\s*$/;
      if ($_[0] eq $1)
         {
         close F;
         return $2;
         }
      }
   close F;
   return "";
   }

my $cfgPace = &get_value_from_config("pace");
my $cfgPipe = &get_value_from_config("pipe");
if ($cfgPace eq "") { $cfgPace = $defaultPace; }
if ($cfgPipe eq "") { $cfgPipe = $defaultPipe; }
else { $cfgPipe = "\\\\.\\pipe\\".$cfgPipe; }

my $pipe;
$pipe = new Win32::Pipe($cfgPipe) || die "can't connect to pipe\n";
$pipe->ResizeBuffer(4096);

# this loop feeds a counter named Test with a value which increments on each iteration
my $val = 0;
while (1)
  {
   $val ++;
   my $str = "Test = $val\n.";
   $pipe->Write( $str );
   sleep( $cfgPace / 1000 );
   }

Linux Examples

Apache

Note: The server must have 'extended status' enabled.

#!/bin/bash

DEFAULT_PACE=1000
DEFAULT_PIPE=/tmp/cca
CCAD_CONF=/etc/ccad.conf
STATUS_URL='http://in/server-status?auto'
CURL=/usr/bin/curl
SED=/bin/sed
SLEEP=/bin/sleep
function get_value_from_config {
   CONF_VALUE=`grep -oE "$1[[:space:]]*=[[:space:]]*[^[:space:]]+" $CCAD_CONF |cut -d"=" -f 2|tr -d " "`
}
get_value_from_config "pace"
PACE=${CONF_VALUE:-$DEFAULT_PACE}
PACE=$(( $PACE / 1000 ))
get_value_from_config "pipe"
PIPE=${CONF_VALUE:-$DEFAULT_PIPE}
test -p $PIPE || exit 3
while true; do
   $CURL -s "$STATUS_URL" | $SED "s/^Total\ /Total_/g" > $PIPE
   sleep $PACE || exit 4
   echo "." > $PIPE
done

Mysql

#!/bin/bash

DEFAULT_PACE=1000
DEFAULT_PIPE=/tmp/cca
CCAD_CONF=/etc/ccad.conf
MYSQL='/usr/bin/mysql'
TR=/usr/bin/tr
SLEEP=/bin/sleep
function get_value_from_config {
   CONF_VALUE=`grep -oE "$1[[:space:]]*=[[:space:]]*[^[:space:]]+" $CCAD_CONF |cut -d"=" -f 2|tr -d " "`
}
get_value_from_config "pace"
PACE=${CONF_VALUE:-$DEFAULT_PACE}
PACE=$(( $PACE / 1000 ))
get_value_from_config "pipe"
PIPE=${CONF_VALUE:-$DEFAULT_PIPE}
test -p $PIPE || exit 3
while true; do
   $MYSQL -e 'show status'|$TR '[[:blank:]]' '=' > $PIPE
   $SLEEP $PACE || exit 4
   echo "." > $PIPE
done

Counter Collection Web Service API

Monitoring API Overview

The MON appliance exports counter data and configuration collected from connected appliances on its aux output terminal. This is done via custom REST-based protocol, running over HTTP. This protocol defines 3 types of resources:

• Appliances
• Entities
• Counters

Each of these types is embedded into the scope of the previous. That is, entities have only meaning in the scope of appliances and counters are only meaningful in the scope of a particular entity. The data collection API allows for retrieving all counter configurations and values known to MON.

MON exports requested data configuration and values in any of the following structured text formats:

• XML (primary format used if no specific format is requested through the fmt argument)
• UDL (meant for CA 3Tera AppLogic in-house usage)

JSON (meant for JavaScript-enabled browser clients)

Monitoring API Protocol

The protocol only uses the GET HTTP method, as it only provides reading functionality. Thus, every supported type of protocol request can be defined by means of its URI and the output structure. Characters, that are considered special for the URI should be escaped using the standard %-encoding. We define the output by its XML schema (primary output format is XML). Translation into other output formats (UDL, JSON) is achieved by direct usage of the respective output format's means of expressing structure and attributes (for example UDL entities v.s. XML entities, UDL attributes v.s. XML attributes, JSON objects v.s. XML entities, JSON object properties v.s. XML attributes).

The root node of all outputs contains a timestamp attribute which provides the server time at the moment when the response was created.

The following is a description of all supported URLs (calls).

Counter Configuration Calls

/monapi/desc[?fmt=<format>] 

Returns a list of all available appliances in a specified output format (<format> is one of the known output formats, e.g. xml).

Output:

<response timestamp="1194967464">
   <appliance name="main.web" />
   <appliance name="main.in" />
</response>
/monapi/desc/<apps>[?fmt=<format>] 

Returns a list of all available entities within the requested appliances (<apps>). <apps> might be a single appliance name, a comma-separated list of appliance names or a wildcard character (*) meaning all available appliances.

Output:

<response timestamp="1194967464">
   <appliance name="main.web">
      <entity name="Terminal in" />
      <entity name="Terminal out" />
      <entity name="Volume data" />
      <entity name="CPU" />
      <entity name="Memory" />
      <entity name="Network" />
   </appliance>
</response>

/monapi/desc/<apps>/<ents>[?fmt=<format>] 

Returns a list of all available counters within the requested appliance entities (<ents>). <ents> might be a single entity name, a comma-separated list of entities names or a wildcard character (*) meaning all available entities. Same for <apps>.

Output:

<response timestamp="1194967464">
   <appliance name="main.web">
      <entity name="Terminal in">
         <counter name="Sent bytes" />
         <counter name="Received bytes" />
      </entity>
   </appliance>
</response>

/monapi/desc/<apps>/<ents>/<cnts>[?fmt=<format>] 

Returns the descriptions of requested counters (<cnts>). <ents>, <ents> and <ents> might be a single appliance/entity/counter, a comma-separated list of such or wildcard.

Output:

<response timestamp="1194967464">
   <appliance name="main.web">
      <entity name="Terminal in">
         <counter name="Sent bytes" description=""  units=""  alarm_below="" alarm_above="" range_lower="" range_upper="" pace="" />
         <counter name="Received bytes" description=""  units=""  alarm_below="" alarm_above="" range_lower="" range_upper="" pace="" />
      </entity>
   </appliance>
</response>

Counter Value Calls

/monapi/val/<apps>/<ents>/<cnts>[?fmt=<format>] 

Returns the values of the specified counters. If any of the app, ent or cnt parameters are omitted, wildcard value is assumed. Thus, for example, /monapi/val/main.web would mean "the values of all counters in all entities of appliance main.web".

Output:

<response timestamp="1194967464">
   <appliance name="main.web">
      <entity name="Terminal in">
         <counter name="Sent bytes" value="23234" />
         <counter name="Received bytes" value="690432" />
      </entity>
   </appliance>

</response>