Reference Information › CA AppLogic Support Knowledge Base › Overview of Support Knowledge Base › AppLogic Knowledge Library - Backend Functions

AppLogic Knowledge Library - Backend Functions

BFC

Grid node

• srvmon

  function:

  • check if server's arp cache contains an invalid entry for the controller server - SCR 3240(minutesly). If deteced, send below alert and execute "ip neigh flush all" to reflush the arp cache
  • An invalid ARP entry was found in the server's ARP cache. The server's ARP cache has been flushed to resolve the issue. Please contact technical support and reference bug 3240
  • check if ntpd is running and restart it if it is not, then update state file(minutesly)
  • check bond and eth state, then update state file(minutesly) (only on primary server) check if controller is running and start it up if it is not.(minutesly)

  cron job:

  /etc/cron.d/srvmon

  related file:

  /usr/local/apl-srv/bin/srvmon main script

  /var/spool/apl-srv/srvmon controller latest failure time

  /var/run/srvmon/bondX bond and NIC state kept by srvmon

  /var/run/srvmon/srvmd.pid srvmon pid when it's launched

  /var/run/ctl_up it indicates the controller shall run on current server

• applogic-rsc-check

  function:

  • check if controller CPU/Memory and dom0 Memory is too little. If it occurs, send below alter to dashbroad.(daily)

    Grid resources are not configured correctly. This may lead to degradation in grid performance or grid instability. Please update the following grid resource$MSG_S on your grid or contact technical support

    minimum resource checked by applolgic-rs-check

    controller minimum cpu procentage: 10%

    controller minimum memory:768 MB

    dom0 minimum memory:512 MB

  cron job:

  /etc/cron.daily/applogic-rsc-check

  related file:

  N/A

• ctl-vm-coredump

  function:

  • delete the applogic core dump file older than 30 days from /var/applogic/cores(weekly)

  cron job:

  /etc/cron.weekly/ctl-vm-coredump

  related file:

  N/A

• log rotate

  function:

  rotate log according to setting /etc/logrotate.conf(daily). all applogic log are rotated when over 100K and kept with latest 5 versions

  cron job:

  /etc/cron.daily/logrotate

• vmad-bin

  function:

  it listen to port 11000, vm agent in appliance send their status to this prort

  process:

  /usr/local/applogic/bin/vmad-bin name=srv1 rem_ip=127.0.0.1 rem_port=11000

Controller

• 3tbmc

  functionality

  • regularly collect mertering data(hoursly)
  • transfer to meterting server grm.3tera.net.(daily)

  cron job:

  /etc/cron.d/3tbmc

  related file:

  /usr/local/bin/3tbmc main script

  /var/applogic/license/<yyyy.mm.dd>.log metering data file(not compressed yet)

  /var/applogic/license/<yyyy.mm.dd>.zip metering data file(compressed format)

• 3tctlmon

  functionality

  • check for the OOM killer in the system log(/var/log/messages*), if see keyword of "oom-killer", log below message(hourly)

    Controller is out of memory. Controller restart not initiated because %s failures have been detected within the previous %s hours.

  • check that servers' time is in sync with controller time, if not synced, log below message(hourly)

    detected clock on server $server is out of sync with controller's clock, difference is: $diff seconds

    detected server $server has not synchronized with a time server in over $SRV_CLOCK_UNSYNC_THRESHOLD seconds

  • validate the grid uptime is correct(hourly)
  • network checking(not funcitonal yet)
  • check if any system volumes(boot, meta and impex) are read only(by touch a file on related file system). if detected, it will printed below messages in controller log or dashboard. If boot or meta volume is readonly, contorller VM may be automatically restarted.(hourly)

    detected read-only _impex volume

    Controller _impex volume is read-only

    detected read-only meta volume - shutdown prevented by $num historical failures

    Controller meta volume is read-only. Controller restart not initiated because %s failures have been detected within the previous %s hours

    detected read-only meta volume - shutting down controller VM

    detected read-only boot volume - shutdown prevented by $num historical failures

    Controller boot volume is read-only. Controller restart not initiated because %s failures have been detected within the previous %s hours

    detected read-only boot volume - shutting down controller VM

  • determine if the impex volume(/dev/hdc) is mounted; if not, remount it and printed below messages(hourly)

    remount failure

    detected _impex volume not mounted - corrective action failed

    Controller _impex volume is not mounted.

    remont success

    detected _impex volume not mounted - corrective action succeeded

    Controller _impex volume was unexpectedly unmounted and successfully re-mounted.

  • check for low disk space on the boot and meta volumes. If free disk space is less than 10240KB, log below alter in log and dashboard(daily)

    detected low disk space on $vol volume - $free Kb free

    Controller $vol volume is nearly full. Please take immediate action or contact technical support.

    cron job:

    /etc/cron.d/3tctlmon

    related file:

    /usr/local/applogic/bin/3tctlmon main script

    /var/spool/applogic/locks/3tctlmon/3tctlmon.pid lock file

    /var/applogic/state/3tctlmon state file

    /var/spool/applogic/3tctlmon

    /var/applogic/state/3tctlmon.oom state file having most recent oom-killer check

    /var/run/applogic/state/srvs/<srv name>.3tctlmon state file having most recent dom0 sync time

    /var/applogic/scripts/3tnetha network checking script(not funtional yet)

• 3tgridmaint

  function

  • Post a dashboard message if no metadata backup was done for more than one day(every 6 hours)
  • re-generate the resource cache(every 4 hour)
  • update dashboard messages about quota limits(every 4 hour)
  • clear old CTL messages older than 1 day(every 4 hour)

  cron job:

  /etc/cron.d/3tgridmaint

  related file:

  /var/applogic/state/last_metadata_backup_time state file updated whenever metadata backup is done.It's modify time is presented as last meta data backup time

• 3tha

  function:

  • if any secondary server state is "unknonw" or "down", set it to none role(every 2 hours)
  • if grid has less than 2 functional secondary server, set a grid node which none role to secondary node(every 2 hours)

  cron job:

  /etc/cron.d/3tha

  related file:

  /usr/local/applogic/scripts/set_secondary.sh main script

  /var/run/set_secondary.pid main script pid file, also the lock file

• 3tmsg

  function:

  • 3tmsg is a controller dashboard message utility. It's launched with time_post paramter to send dashboard message to the specific mailbox(very 5 minutes)

  cron job:

  /etc/cron.d/3tmsg

  related file

  /usr/local/bin/3tmsg main script

  /etc/applogic/messages.d/* all dashboard message, one message, one file

• vmnd

  function:

  • wrap of volclean. It's launched regularly to detect unused and broken volume and streams(every 6 hours)

    Volume check completed on %t. Volume maintenance is required. Found %s inaccessible volumes,%s broken volume links,%s unused volumes.

  cron job:

  /etc/cron.d/vmnd

  related file

  /usr/local/applogic/bin/vmnd main script

  /var/spool/applogic/locks/vmnd/vmnd.lock lock file when main script is running

• 3tgrid

  function:

  • Send "grid info" outptu to system log(daily)

  cron job:

  /etc/cron.daily/3tgrid

  related file

  /usr/local/applogic/bin/3tgrid mail script

• ctld-coredump

  function:

  delete the applogic core dump file older than 30 days from /var/applogic/cores(dialy)

  cron job:

  /etc/cron.daily/ctld-coredump

  related file:

  N/A

• 3taudit

  function:

  ???

  cron job:

  /etc/cron.hourly/3taudit

  related file:

  /usr/local/applogic/bin/3taudit main script