Installing and Maintaining CA AppLogic® › Advanced Maintenance Guide › Administration and Advanced Options › How to Maintain a Grid

How to Maintain a Grid

As a Backbone Administrator, you want to perform administrative tasks and advanced operations on a grid. These operations include retrieving server information, checking the network health, and reviewing grid failures. BFC Maintainers can perform these tasks regularly.

The following diagram shows how to maintain a grid in a typical environment:

1. Retrieve and modify the server information.
2. If you have too many servers, remove a server from the grid.

   For example, you want to change the minimum number of required servers from 5 to 4 because of underutilized resources.

3. If you want to detect problems in the grid, check the network health.
4. If the grid fails, complete the following steps:
   1. Review the grid failure logs.
   2. Restore grid controller operation.

Retrieve and Modify the Server Information

The 3tsrv utility resides in dom0 of each server. Only grid administrators have access to this utility. Use this utility to retrieve and modify server information and perform functions such as log collection for failure analysis.

Follow these steps:

1. Execute the following command:

   3tsrv command [ prop[=val] ]* [ --batch ][ --force ] 
   

   command

   Specifies the command that you want to execute.

   prop=val

   Specifies additional command-specific arguments. If you do not specify val, the product assumes a Boolean property with a value of TRUE.

   --batch

   Specifies that the utility executes from a script and should not display lengthy error messages. This variable also displays output in UDL format.

   --force

   Forces the specified operation.

2. Consider the following supported commands:

   info

   Retrieves detailed server information.

   set

   Sets server information.

   reboot

   Reboots the server.

   shutdown

   Shuts down the server.

   applogic activate

   Activates the product on the server.

   applogic deactivate

   Deactivates the product on the server.

   applogic start

   Starts the product on the server.

   applogic stop

   Stops the product on the server.

   diskchk enable

   Enables disk failure detection on the server.

   diskchk disable

   Disables disk failure detection on the server.

   bd list

   Lists the active block devices used by the product on the server

   sd get

   Displays the contents of the server data file

   sd set

   Updates the server data file

   logs collect

   Collects server logs and information

   help

   Displays help for the utility.

   For example, you want to view details information about the server.

   Execute the following command:

   3tsrv info [ --batch ]
   

Remove a Server from the Grid

You can remove a server from a grid using Server Actions in the Grid Properties page.

You may need to decrement the minimum server value if you indicated an equal value for the minimum number and target number. For example, your grid has five servers with the values set to 5/5/5 for minimum/target/maximum. You want to set the minimum value to 4 because svr5 is underutilized.

Note: If the minimum server value is less than the target value, you do not have to adjust the minimum value first. For example, if you indicated 5/7/7 for minimum/target/maximum, you can remove two servers without adjusting the minimum value.

Follow these steps:

1. From the BFC UI, open the Xen or VMware tab of the Grid Properties page.
2. Enter 4 as the minimum value.
3. Open the grid shell.
4. Execute the following command:

   3t srv disable srv<n>
   

   n

   The server number that you want to remove from the grid.

   Note: This step is optional if you select the Force Removal check box in Step 5d.

5. Complete the following steps:
   1. From the BFC UI, select the Servers tab on the Grid Properties page.
   2. Select the server that you want to remove, then click Remove from the Server Actions drop-down list.
   3. If you do not want another grid to select the server, enable the Quarantine option.
   4. To remove servers that are not disabled in CA AppLogic®, select the Force Removal option.

Check the Network Health

The 3tnetha utility lets you script various network and switch-related checks. The products invoke the 3tnetha script periodically as part of its periodic health checks.

Follow these steps:

1. Locate the script within the following directory on the controller:

   /var/applogic/scripts
   

   By default, the script does nothing except for exit 0.

2. Verify the output the script:
   • If the environment does not have problems, the script completes with a 0 exit code.
   • If the script detects a problem, the script fails with an exit code of 1. The output displays an error message to STDOUT that describes the problem. When 3tnetha fails with an exit code of 1, the message displays on the grid dashboard.

     The following example shows a sample format of the dashboard message:

     A problem has been detected with the grid's network: STDOUT from 3tnetha. Contact technical support for assistance. 
     

   • If the script fails with an exit code greater than 1, a failure occurred to determine the network state. In this case, a generic dashboard message appears which indicates the problem.

Review the Grid Failure Logs

You can troubleshoot the failures based on the behavior of the grid. Report any unexpected grid failures to CA Support. However, before you submit a bug report, review the Release Notes to verify that your problem is unknown.

Follow these steps:

1. Verify if you experience any of the following grid behavior:
   • The grid reboots for no apparent reason, and the operator did not issue a grid reboot command (this would be reported on the grid's dashboard)
   • One of the grid servers crashes or becomes unresponsive.

     Note: The product can report this behavior on the dashboard of the grid.

   • The product GUI locks up and becomes unresponsive.
   • One of the default product standard appliances fails for any of the following reasons:
     • Fails to boot
     • Crashes
     • Becomes unresponsive

     Note: The product can report this behavior on the dashboard of the grid.

   • A CLI command hangs or fails for no apparent reason.

   In this example, you cannot verify your problem as a known issue. You decide to file a bug report.

   You want to collect all the logs from the grid, including the backups. For example, xxxx.1, xxxx.2, and so on.

   The grid and servers logs require administrator access. You send these logs to CA Support.

   Note: You can use the 3tsrv utility on each server to collect the server specific logs and information.

2. Collect the Grid controller logs from the following directories:
   • /var/log/messages
   • /var/log/applogic
   • /var/log/3tshell
   • /var/log/secure

• Collect the Servers logs from the following directories:
  • /var/log/messages/
  • /var/log/applogic
  • /var/log/xen/
  • /var/log/secure
• Collect the Appliance logs (only for appliance failures) from the following directories:
  • /var/log/messages/
  • /var/log/applogic
  • /var/log/secure
• Collect the following container and BFC Logs:

  ContainerX_python.log.Y

  Contains the output of any grid commands that the BFC issues. This log also contains output from the user-identifiable actions that the BFC takes. We recommend that you consider this log as the most useful for troubleshooting.

  BFCLog.X

  Contains useful data when diagnosing issues about discovery. CA Technical Support uses this log primarily.

  bfc_setup.log

  Contains the output from the BFC installation.

  /var/log/messages folder

  Contains data about DHCP requests. You can watch this log when you discover servers. If a server powers on, but you do not see its dhcp in this log, then it is probably not configured correctly with respect to PXE.

  /var/bfcadmin/inventory

  Contains the inventory files for servers and can help in diagnosing discovery/inventory issues.

  /var/bfcadmin/deployer

  Contains the logs created upon server deployment in to a grid. If a server fails when being added to a grid, these files may help.

1. Collect the following information about each server within the grid (dom0):
   • lspci
   • lspci -n
   • lspci -v
   • lsmod
   • cat /proc/cpuinfo
   • cat /proc/meminfo
   • dmidecode
   • smartctl -a for all physical disks on the server

Restore Grid Controller Operation

If the grid controller server fails, the product detects grid controller recovery issues. These issues could potentially cause the grid controller to become inaccessible.

Follow these steps:

1. If the grid does not have a controller HA due to one or more down grid controller servers, consider the following information:
   • The dashboard alerts you that the grid does not have a controller HA. X of Y controller servers are down. To restore the controller HA, you bring Z of the following controller servers back online: list of servers
   • Bring back the servers online or add new servers to the grid. Contact your service provider for assistance.
2. If you configured the HA grid controller improperly, consider the following information:
   • The dashboard alerts you that the grid is not configured for the controller HA. You assign a secondary controller server or the grid cannot recover from grid controller server failures. To enable the controller HA on the grid, assign one of the running servers as a secondary controller server.
   • No servers are assigned as a secondary grid controller (backup grid controller). Contact your service provider immediately.
   • The product detects the following improper grid configuration that could potentially cause grid failures or application downtime.
3. If single server grids do not have HA features, consider the following information:
   • The dashboard alerts you that HA is unavailable due to the grid being a single server grid.
   • Most of the HA features require at least two servers. To take advantage of the HA features described in this document, contact your service provider to add at least one more server to your grid.
4. If you did not configure the grid with the appropriate amount of controller memory, controller cpu, or server memory, consider the following information:
   • The dashboard alerts you that grid resources are not configured correctly. This issue can lead to degradation in performance of the grid or grid instability. Update the following grid resources on your grid or contact technical support: controller memory | controller CPU | server memory.
   • Contact your service provider immediately. To use the correct amount of resources, you reconfigure the grid, or the grid can become unstable which can affect the uptime of running applications.

You have successfully performed grid maintenance.