How to Identify Performance Problems in CA SDM

How to Identify Performance Problems in CA SDM

As a system administrator, you gather information about your installation environment, resource usage, configuration details, and other available system resources. For example, understand the computer configuration to determine the Operating System type, version, and available system resources. This data helps you to identify and diagnose a performance or memory-related issue in CA SDM. The CA Diagnostic Report tool and the Interval Logging utility helps you collect diagnostic information about your CA SDM environment.

The following diagram describes how to gather diagnostic information and identify performance problems in CA SDM:

Follow these steps:

1. Define the Performance Problem.
2. Verify the Prerequisites
3. Execute the CA SDM Diagnostic Report tool.
   • Verify Windows report
   • Verify UNIX report
   • Verify Collected diagnostic report
4. Gather database server environment details.
5. Collect Performance Data from CA SDM Servers
6. Review general tuning recommendations.

Verify the Prerequisites

1. For Windows operating system, install the pslist.exe tool and add its directory path variable on each CA SDM servers.
2. Install and configure the CA SDM servers before running the diagnostic tool.

   Note:

   We recommend that you contact CA Support Online before you use the diagnostic tool.

Install Ps Tools for Windows

For a Windows installation, install the pslist.exe tool and add its directory path variable to the system path variable .

Follow these steps:

1. Download PS Tools Suite from Microsoft.
2. Extract pstools.zip to any directory of your choice and add the directory to the Windows %PATH% environment variable.
3. Execute pslist.exe once manually from the command prompt as the Local System user and accept the license agreement. To execute pslist.exe as the Local System user, run the following command:

   psexec.exe -s -i pslist.exe
   

   Note: You can start the CA SDM Windows service under a different user account other than the Local System account. For that service, execute pslist.exe under that account and accept the license agreement. If pslist.exe is added after installing & configuring CA SDM, restart the CA SDM services after accepting the license agreement of pslist.

Define the Performance Problem

You can define your performance problem by collecting the system information initially. Review the following list of example questions to determine if they are relevant for your issue:

• What did users experience to disrupt their tasks?
• When did users first discover the problem?
• Has your environment changed recently, such as hardware upgrades?
• What functionality of the product experiences issues?
• How many users do the issue impact, and what type of users?
• What users are not impacted?
• What is the geographic location of the users that see the problem?
• What is the Access Level of the impacted users?
• Do you host CA SDM on a VMWare ESX server or other virtualized environment?
• What other software are you running on the ESX server or host machine?
• What are the specifications of this environment?
• How many CPUs do you have on each computer?
• How much memory did you configure for each computer?

Review General Tuning Recommendations

As a best practice, we recommend you to monitor the key performance indicators and the resource consumption regularly. Also, ensure that routine maintenance is applied to identify small problems before they become serious.

If you suspect an issue, or users complain about slow performance, open a CA Support Online issue and send the Interval Logging data, CA SDM Diagnostic Tool Report, CA SDM standard logs, and information about the problem. Send the information to CA Support from each CA SDM server.

The following list describes common signs of performance problems:

• The long messages in the CA SDM standard logs (stdlog.x files)
• Search for the messages in the stdlogs that state "The following query took #### milliseconds...".
• Queuing on database agents in pdm_vdbinfo output
• Search for Queued Requests(#) in the pdm_vdbinfo output. You can execute this command at the OS prompt manually. The Interval Logging script also executes this command.
• A large number of connected users to each web engine. For example, more than 200 users.

  Note: Execute pdm_webstat to display the number of concurrent users per web engine. You can execute this command at the OS prompt manually. The Interval Logging script also executes this command.

• User complaints

Collect Database Server Environment Details

You can gather details about your database server to help identify performance problems in CA SDM.

Follow these steps:

1. Determine the location of your database server, such as local or remote.
2. Determine the DBMS version, Operating System version, and patch level.
3. Complete the appropriate steps for your database type:
   • Execute the following queries for SQL Server and note the results:

     select @@version
     SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel')
     

   • Execute the following query for Oracle:

     select * from v$version where banner like 'Oracle%';
     

4. Confirm the version of the database client that you installed on the application server.
5. If available, gather information about Environmental Data, such as the Operating System and other databases.
6. Gather network topology and topology information or other products that you integrate with CA SDM.

   For example, locate information about the products from available PDF files or diagrams.

Collect Information from CA Diagnostic Report Tool

The CA SDM installation media includes the Diagnostic Report Tool to help collecting information for diagnosing performance problems. You can use the diagnostic tool to collect information regarding the type of operating system, and to determine what commands to use for the data collection from the CA SDM installation.

Configure the CA SDM server before running the diagnostic tool.

Follow these steps:

Windows

1. Set $NX_ROOT as your CA SDM installation root directory.

   The default for $NX_ROOT is the C:\Program Files\CA\Service Desk\ on Windows. You can change the default when you want to change the default directory during the installation process.

2. Verify that the system path variable includes $NX_ROOT\bin.

UNIX/Linux

1. Set $NX_ROOT as your CA SDM installation root directory.

   The default for $NX_ROOT is — /opt/CAisd/ on a Unix or Linux operating system. You can change the default when you want to change the default directory during the installation.

2. Verify that your $PATH includes $NX_ROOT/bin.

Execute the CA Diagnostic Tool

The CA Diagnostic tool creates a .CAZ file on the Windows OS and .tar.gz on UNIX in the $NX_ROOT\diag\rpt directory that you upload with your issue at http://support.ca.com.

Follow these steps:

1. Execute supp_diag.cmd on Windows, or execute supp_diag.sh on UNIX.

   The diagnostic tool can take five to 10 minutes to complete.

2. If the data collection process does not complete, view the $NX_ROOT\diag\<host_name>_supp_diag.log log file to determine the errors when collecting diagnostic information.

   Note: If you want to cancel the background batch job, use CTRL-C to cancel the batch file. Some processes still run in the background, such as MSINFO32.exe. If you have any questions about using this diagnostic tool, contact the CA Support Online.

3. The script directory structure displays the location of the script files, diagnostic zip files, and log files:
   • The $NX_ROOT\diag\bin directory contains script files.
   • The $NX_ROOT\diag\rpt directory contains the diagnostics zip file (in .caz format on Windows and .tar.gz format on UNIX systems).
   • The $NX_ROOT\diag\misc_logs directory contains the log files that can be automatically included in the zip file.
4. Complete the appropriate steps to unzip the gathered files that are based on your operating system:

   Windows

   • Open a command prompt.
   • Cd to $NX_ROOT\diag\rpt or any directory where the .CAZ file is located.
   • Execute the following command.

     $NX_ROOT\diag\bin\cazipxp -u <package_name>.CAZ
     

   UNIX

   • Open a command prompt.
   • Cd to $NX_ROOT/diag/rpt or any directory where the .tar or .tar.gz file is located.
   • Uncompress and untar the file:

     gunzip -d <package_name>.tar.gz 
     tar -xvf <package_name>.tar
     

Verify Windows Report

The following list describes the Windows report files that are created and included in the CAZ\tar file package.

Ca.olf

Specifies the CA licensing information from ca_lic directory.

Lic98.log

Specifies the log file that is related to CA licensing from ca_lic directory.

Lic98version.log

Specifies the log file that is related to CA licensing from ca_lic directory.

Licdebug.log log

Specifies the file that is related to CA licensing from ca_lic directory.

Drwatsoninfo.txt

Specifies the Dr Watson configuration of the computer.

<host name>_env.txt

Specifies the environment variables that are set on the computer.

<host name>_slstat.txt

Specifies the output of slstat command.

<host name>_pdm_status.txt

Specifies the output of slstat command.

<host name>_dir_listing.txt

Specifies the Service Desk install directory listing.

<host name>_pslist.txt

Specifies the process listing when the pslist Microsoft tool is installed.

<host name>_MSINFO32.NFO

Specifies the MSINFO output gathering system information.

<host name>_SYSTEMINFO.TXT

Specifies the system information.

<host name>_appevents.csv

Specifies the application event logs created in the past seven days.

<host name>_sysevents.csv

Specifies the application event logs created in the past seven days.

<host name>_hostinfo.txt

Specifies the computer information.

<host name>_prodinstallinfo.txt

Specifies CA products installation information.

<host name>_caprod_registry.txt

Specifies the Registry information of installed CA products.

<host name>_softfeatures.txt

Specifies the list of software features that are installed for Service Desk.

<host name>_ipconfig.txt

Specifies the IP configuration information.

<host name>_supp_diag.log

Specifies the log created for running the supp_diag tool.

Verify UNIX Report

The following list describes the UNIX report files that are created and included in the CAZ\tar file.

Ca.olf

Specifies the CA licensing information from ca_lic directory.

Lic98.log

Specifies the log file that is related to the CA licensing from ca_lic directory.

<host name>_env.txt

Specifies the environment variables that are set on the computer.

<host name>_slstat.txt

Specifies the output of slstat command.

<host name>_pdm_status.txt

Specifies the output of slstat command.

<host name>_dir_listing.txt

Specifies the Service Desk install directory listing.

<host name>_pslist.txt

Specifies the process listing when the pslist microsoft tool is installed.

<host name>_uname.txt

Specifies the output of uname operating system command.

<host name>_diskinfo.txt

Specifies the output of df operating system command.

<host name>_freemem.txt

Specifies the output of memory information.

<host name>_supp_diag.log

Specifies the log created for running the supp_diag tool.

<host name>_prtconf.txt

Specifies the output of prtconf operating system command on Solaris and the AIX computers.

<host name>_solrev.txt

Specifies the OS version and patches Information on Solaris computers.

<host name>_netconf.txt

Specifies the IP configuration Information on AIX computers.

Verify Collected Diagnostic Report

The default CA SDM documentation file\directory list includes the following reports in the CAZ\tar file. Add files or directories that you include in the CAZ\tar file by placing it in the $NX_ROOT\diag\misc_logs directory.

• $NX_ROOT/GENLEVEL or $NX_ROOT/.GENLEVEL
• $NX_ROOT/<COMPUTERNAME>.his
• $NX_ROOT/NX.env
• $NX_ROOT/NX.env.last
• $NX_ROOT/log\
• $NX_ROOT/pdmconf\
• $NX_ROOT/pdmconf\version
• $NX_ROOT/bopcfg\www\*.cfg
• $NX_ROOT/site\mods\
• $NX_ROOT/site\ddict.sch
• $NX_ROOT/site\eh\
• $NX_ROOT/bopcfg\www\CATALINA_BASE\logs\
• $NX_ROOT/bopcfg\www\CATALINA_BASE\webapps\CAisd\WEB-INF\web.xml
• $NX_ROOT/bopcfg\www\CATALINA_BASE\webapps\*.xml

Collect Performance Data from CA SDM Servers

You can collect resource usage data on the different CA SDM servers in a multiple server configuration to determine the problems that are experienced on each server. The collected data sets are used to analyze and troubleshoot any performance or memory-related issues in the CA SDM servers. The interval logging utility allows you to start or stop the diagnostic data collection using the CA SDM web interface. You can share the collected data set with CA Support Online to help identify and resolve the CA SDM server problems.

Interval logging utility runs on pdm_intrvlog_nxd daemon that collects log data from a server. The daemon automatically starts when a Service Desk server start.

• To stop the interval logging daemon manually, remove the entry of interval logging utility from pdm_startup
• To start the interval logging daemon manually, double-click the pdm_intrvlog_nxd.bat file in the $NX_ROOT/bin folder

Collect Usage Data using Interval Logging

To collect the resource usage data on the CA SDM servers, configure the server for interval logging. You can select the type of data you want to collect from each server, and also change the logging options for the servers at any point.

For example, if you select only the CPU usage option for a particular server, the utility will only collect CPU usage data on the server.

Follow these steps:

1. Log in to CA SDM.
2. Select Systems, Interval Logging under the Administration tab.

   The Interval Logging Configurations List opens.

3. Click Create New to add an interval log configuration.

   The Create New Interval Log Configuration page opens.

4. Complete the following fields:

   Server Name

   Specifies the server for which you want to collect the log data. Use the Search icon to view the list of servers you can configure for the interval logging.

   Recurrence Interval

   Specifies the time interval during which the log data is collected for the server. For example, if the recurrence interval is set to 3 minutes, the log is collected for every 3 minutes.

   Default: 3 minutes

   Minimum: 2 minutes

   Enabled

   Indicates whether the interval logging is enabled or disabled for the server. If enabled, interval log is collected for the server.

   Note: To stop the interval logging before the scheduled end date, change the Enabled status to NO.

   Record Status

   Indicates whether the interval log configuration is active or inactive.

   Scheduled Start Date

   Specifies the start date and time for collecting the log data. If you do not enter a start date, interval logging starts immediately until the configuration is active and enabled or until the scheduled end date.

   Scheduled End Date

   Specifies the end date and time for collecting the log data. If you do not enter an end date, interval logging will continue to run until the configuration is active and enabled.

5. Select the appropriate log option that you want to collect from the server.

   For example, select CPU Usage if you want to capture the log for CPU usage data.

   CPU Usage

   Collect CPU usage statistics for the server by executing pslist –x on Windows, or “ps” on Unix. Depending on your configuration type, you can collect the diagnostic data on the following servers:

   • Conventional: Primary server.
   • Advanced Availability: All servers.

   Memory Usage

   Collects memory usage data for the server by executing pslist –m on Windows, or “ps” on Unix. Depending on your configuration type, you can collect the diagnostic data on the following servers:

   • Conventional: Primary server.
   • Advanced Availability: All servers.

   Network Status

   Collects information of all active connections and network statistics by executing netstat /b, or /a from the operating system. Depending on your configuration type, you can collect the diagnostic data on the following servers:

   • Conventional: Primary server.
   • Advanced Availability: All servers.

   Task List

   Collects application and services information for all tasks running on the server. Depending on your configuration type, you can collect the diagnostic data on the following servers:

   • Conventional: Primary server.
   • Advanced Availability: All servers.

   Web Session Counts

   Collects CA SDM sessions and user statistics for the web engine processes by executing the pdm_webstat command. You can collect the data for any CA SDM servers.

   Server Status

   Collect information about all the CA SDM daemons or processes on the server by executing the pdm_status command. You can collect the data for any CA SDM servers.

   DB Report

   Collects information of database record types by executing the db_report command. You can collect the data for any CA SDM servers.

   Virtual DB Info

   Collects information related to the queued database requests by executing the pdm_vdbinfo command. You can collect the data for any CA SDM servers.

   List Server Connections

   Collects information on active connections for the server by executing the pdm_listconn command. You can collect the data for any CA SDM servers.

   List Slump Processes

   Collects information about slump connections and processes by executing the slstat command. You can collect the data for any CA SDM servers.

6. Click Save.

   The server is configured for the interval logging.

7. (Optional) Repeat Steps 1-5 to create more interval log configurations.
8. Go to the $NX_ROOT\log directory and view the generated log files. Depending on the Interval Logging options you select, the following output files are generated:
   • cpu_usage_hostname
   • db_report_hostname
   • memory_usage_hostname
   • netstat_hostname
   • pdm_listconn_hostname
   • pdm_vdbinfo_hostname
   • server_status_hostname
   • session_counts_hostname
   • tasklist_hostname

   Important! The maximum limit of the output file size defaults to 30 KB. You can modify the file size by changing the @NX_LOGFILE_LIMIT value in NX_env file. If the generated output file exceeds the maximum file size limit, a new file is created with the suffix of 1. Files that are generated subsequently have the suffix of the number incrementally.

   You can share the collected log data with CA Support to help identify performance problems in your SDM installation.