Policy Server Guides › Policy Server Administration Guide › General SiteMinder Troubleshooting

General SiteMinder Troubleshooting

Policy Server Exits with LDAP Admin Limit Exceeded Error

Symptom:

The Policy Server gracefully exits when LDAP search to Policy Store/Key Store fails with the following error:

LDAP_ADMINLIMIT_EXCEEDED (error code 11)

Solution:

Enable the following optional registry key:

EnableRetryOnAdminLimitExceededFailure

Allows the Policy Server to retry the search once before giving up.

Values: 0 (disabled) or 1 (enabled).

Default: 0

Windows

Follow these steps:

1. From the Windows Start menu, select Run.
2. Enter regedit in the Run dialog box and click OK.
3. In the Registry Editor, navigate to the following location:

   HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Netegrity\SiteMinder\CurrentVersion\ObjectStore
   

4. Modify the value of the following registry key:

   EnableRetryOnAdminLimitExceededFailure
   

5. Restart the Policy Server.

UNIX

Follow these steps:

1. Navigate to the following location:

   install_directory/siteminder/registry

2. Open sm.registry in a text editor.
3. Locate the following text in the file:

   HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Netegrity\SiteMinder\CurrentVersion\ObjectStore
   

4. Modify the value of the following registry key:

   EnableRetryOnAdminLimitExceededFailure
   

5. Restart the Policy Server.

Administrative UI Becomes Unresponsive

Symptom:

On a stand–alone Administrative UI installation (with an embedded JBoss application server), the Administrative UI becomes unresponsive. That is, the Administrative UI service does not start or you cannot log in after a period of normal operation.

Solution:

1. Stop the Administrative UI application server.
2. When the application server is down, rename or delete the data directory in the following location:

   \adminui\server\default
   

3. Restart the application server.
4. Run the following command on the Policy Server:

   XPSRegClient client_name[:passphrase] -adminui -t timeout -r retries -c comment -cp -l log_path -e error_path
   -vT -vI -vW -vE -vF
   

   Note: Inserting a space between client_name and [:passphrase] results in an error.

   client_name

   Identifies the Administrative UI being registered.

   Limit: This value must be unique. For example, if you have previously used smui1 to register an Administrative UI, enter smui2.

   Note: Record this value. This value is to complete the registration process from the Administrative UI.

   passphrase

   Specifies the password that is required to complete the registration of the Administrative UI.

   Limits:

   • The passphrase must contain at least six (6) characters.
   • The passphrase cannot include an ampersand (&) or an asterisk (*).
   • If the passphrase contains a space, it must be enclosed in quotation marks.
   • If you are registering the Administrative UI as part of an upgrade, you can reuse a previous passphrase.

   Note: If you do not specify the passphrase in this step, XPSRegClient prompts you to enter and confirm one.

   Important! Record the passphrase, so that you can refer to it later.

   -adminui

   Specifies that an Administrative UI is being registered.

   -t timeout

   (Optional) Specifies how long you have to complete the registration process from the Administrative UI. The Policy Server denies the registration request when the timeout value is reached.

   Unit of measurement: minutes

   Default: 240 (four hours)

   Minimum Limit: 1

   Maximum Limit: 1440 (one day)

   -r retries

   (Optional) Specifies how many failed attempts are allowed when you complete the registration process from the Administrative UI. A failed attempt can result from an incorrect client name or passphrase submitted to the Policy Server during the registration process.

   Default: 1

   Maximum Limit: 5

   -c comment

   (Optional) Inserts the specified comments into the registration log file for informational purposes.

   Note: Surround comments with quotes.

   -cp

   (Optional) Specifies that registration log file can contain multiple lines of comments. The registration tool prompts for multiple lines of comments and inserts the specified comments into the registration log file for informational purposes.

   Note: Surround comments with quotes.

   -l log_path

   (Optional) Specifies where to export the registration log file.

   Default: siteminder_home\log

   siteminder_home

   Specifies the Policy Server installation path.

   -e error_path

   (Optional) Sends exceptions to the specified path.

   Default: stderr

   -vT

   (Optional) Sets the verbosity level to TRACE.

   -vI

   (Optional) Sets the verbosity level to INFO.

   -vW

   (Optional) Sets the verbosity level to WARNING.

   -vE

   (Optional) Sets the verbosity level to ERROR.

   -vF

   (Optional) Sets the verbosity level to FATAL.

   The registration tool lists the name of the registration log file and prompts for a passphrase.

5. Press Enter.

   The registration tool creates the client name and passphrase pairing.

6. To register the Administrative UI with a Policy Server, complete one of the following steps on the Administrative UI host:
   • Windows:
     • (Recommended) Use the Administrative UI shortcut to open the Administrative UI. Using the shortcut registers the Administrative UI over SSL. If you do not have access to the shortcut, open a web browser and go to the following location:

       https://host:8443/iam/siteminder/adminui

       Note: A self–signed certificate that is valid for ten years is created and used for the connection. The certificate is created with an RSA 2048 key strength.

     • Open a web browser and go to the following location:

       http://host:8080/iam/siteminder/adminui

   • UNIX:
     • (Recommended) Open a web browser and go to the following location to register the Administrative UI over SSL:

       https://host:8443/iam/siteminder/adminui

     • Open a browser and go to the following location:

       http://host:8080/iam/siteminder/adminui

       Note: If the host system does not have a web browser, you can remotely access the login screen.

     host

     Specifies the fully qualified Administrative UI host system name.

     The SiteMinder Administrative UI login screen appears.

7. Enter the following value in the User Name field:

   siteminder
   

8. Type the SiteMinder superuser account password in the Password field.

Command Line Troubleshooting of the Policy Server

You can run the Policy Server process interactively in a separate window with debugging options turned on to troubleshoot problems. The following server executable may be run from the command line:

install_dir/siteminder/bin/smpolicysrv

Note: On Windows systems, do not run the smpolicysrv commands from a remote desktop or Terminal Services window. The smpolicysrv command depends on inter-process communications that do not work if you run the smpolicysrv process from a remote desktop or Terminal Services window.

Use the following options with the smpolicysrv command:

-tport_number

This option is used to modify the TCP port that the server binds to for Agent connections. If this switch is not used, the server defaults to the TCP port specified through the Policy Server Management Console.

-uport_number

This option is used to modify the UDP port that the server binds to for RADIUS connections. If this switch is not used, the server defaults to the UDP port specified through the Policy Server Management Console. This switch is applicable to the authentication and accounting servers only.

-stop

This switch stops the server in the most graceful manner possible. All database and network connections are closed properly using this method.

-abort

This switch stops the server immediately, without first closing database and network connections.

-stats

This switch produces current server runtime statistics such as thread pool limit, thread pool message, and the number of connections.

-resetstats

This switch resets the current server runtime statistics without restarting the Policy Server. This switch resets the following counters:

• Max Threads is reset to the Current Threads value.
• Max Depth of the message queue is reset to the Current Depth of the message queue.
• Max Connections is reset to Current Connections.
• Msgs, Waits, Misses, and Exceeded limit are reset to zero.

This switch does not reset the following counters:

• Thread pool limit
• Current Threads
• Current Depth of the message queue
• Current Connections
• Connections Limit

-publish

Publishes information about the Policy Server.

-tadmport_number

Sets the TCP port for the administration service.

-uacport_number

Sets the UDP port for Radius accounting.

-uadmport_number

Sets the UDP port for the administration service.

-uauthport_number

Sets the UDP port for Radius authentication.

-ac

Enables the servicing of Agent API requests.

-noac

Disables the servicing of Agent API requests.

-adm

Enables the servicing of administration requests.

-noadm

Disables the servicing of administration requests.

-radius

Enables the servicing of RADIUS requests.

-noradius

Disables the servicing of RADIUS requests.

-onlyadm

Combines the following options into a single option:

• -adm
• -noac
• -noradius

–starttrace

The command:

• starts logging to a trace file and does not affect trace logging to the console.
• issues an error if the Policy Server is not running.

If the Policy Server is already logging trace data, running the –starttrace command causes the Policy server to:

• rename the current trace file with a time stamp appended to the name in the form: file_name.YYYYMMDD_HHmmss.extension
• create a new trace file with the original name

  For example, if the trace file name in Policy Server Management Console’s Profiler tab is C:\temp\smtrace.log, the Policy Server generates a new file and saves the old one as c:\temp\smtrace.20051007_121807.log. The time stamp indicates that the Policy Server created the file on October 7, 2005 at 12:18 pm. If you have not enabled the tracing of a file feature using the Policy Server Management Console’s Profiler tab, running this command does not do anything.

-stoptrace

The command:

• stops logging to a file and does not affect trace logging to the console.
• issues an error if the Policy Server is not running.

You can use two smpolicysrv command line options, -dumprequests and -flushrequests, to troubleshoot and recover more quickly from an overfull Policy Server message queue. Only use these options in the following case:

1. Agent requests waiting in the Policy Server message queue time out.
2. One or more Agents resend the timed-out requests, overfilling the message queue.

!Important Do not use -dumprequests and -flushrequests in normal operating conditions.

-dumprequests

Outputs a summary of each request in the Policy Server message queue to the audit log.

-flushrequests

Flushes the entire Policy Server message queue, so that no requests remain.

Start or Stop Debugging Dynamically

You can start or stop the debugging function of certain components at any time without restarting the Policy Server.

Note: We recommend using this feature only when directed to do so by CA Technologies technical support personnel.

Follow these steps:

1. Open a command window on the machine hosting the Policy Server.
2. Type the following command:

   SmCommand -i SiteMinder
   

   A list of options appears.

3. Select one of the following debugging options according to the instructions that your CA support representative provides.

   CA.EPM::EPMObjects_Debug

   Toggles the debugging state of the SiteMinder EPM component.

   CA.XPS::Debug

   Toggles the debugging state of the SiteMinder XPS component.

   CA.XPS::XPSEval_Debug

   Toggles the debugging state of the SiteMinder XPSEvaluate component.

Start or Stop Tracing Dynamically

You can start or stop the tracing functions of certain components at any time without restarting the Policy Server.

Follow these steps:

1. Open a command window on the machine hosting the Policy Server.
2. Type the following command:

   SmCommand -i SiteMinder
   

3. A list of options appears. The tracing options display the opposite of their current states. For example, if tracing for CA XPS is currently disabled, the option to turn it on appears as follows:

   item_number - CA.XPS::TraceOn
   

4. Select one of the following options by typing the number of the option you want:

   CA.EPM::EPMObjects_TraceState

   Toggles tracing for the EPM Objects components on or off.

   CA.XPS::TraceState

   Toggles tracing for the XPS components on or off.

   CA.XPS::XPSEval_TraceState

   Toggles tracing for the XPS Expression Evaluator components on or off.

   A confirmation message appears. The list of options is redisplayed with your changes.

5. (Optional) Repeat Step 4 to start or stop tracing on another component.
6. Type Q to quit.

   Tracing has been changed dynamically.

Policy Server Hangs after Web Agent Communication Failure

Symptom:

If a Web Agent goes offline during a Policy Server request, for example, during a network outage, and does not notify the Policy Server of the communication failure, the Policy Server continues to wait for the Web Agent data. The Policy Server continues to wait, even after the Web Agent regains network functionality and closes the connection to the Policy Server.

If many requests from one or more Web Agents are lost in this manner, the Policy Server can become unresponsive because the worker threads handling the requests are not released.

Solution:

Creating and enabling the SiteMinder Enable TCP Keep Alive (SM_ENABLE_TCP_KEEPALIVE) environment variable configures the Policy Server to send KeepAlive packets to idle Web Agent connections. The interval at which the Policy Server sends the packets is based on OS–specific TCP/IP parameters.

You can also set this variable at the locations for the Web Agent, an Application Server Agent (ASA), the Administrative UI, or a custom agent created by the SDK.

Consider the following when configuring the parameters:

• When the Policy Server must start to send the packets.
• The interval at which the Policy Server sends the packets.
• The number of times the Policy Server sends the packets before determining that the Web Agent connection is lost.

  Note: For more information about configuring TCP/IP parameters, see your OS–specific documentation.

To configure the Policy Server to send KeepAlive packets to idle Web Agent connections

1. Log into the Policy Server host system.
2. Do one of the following:
   • (Windows) Create the following system environment variable with a value of 1:

     SM_ENABLE_TCP_KEEPALIVE

   • (UNIX)
     1. Create the following system environment variable:

        SM_ENABLE_TCP_KEEPALIVE=1

     2. Export the environment variable.

   Note: The value must be 0 (disabled) or 1 (enabled). If a value other than 0 or 1 is configured, the environment variable is disabled. If the environment variable is disabled, the Policy Server does not send KeepAlive packets to idle Web Agent connections.

Check the Installed JDK Version

If a Policy Server fails to start, check that the correct version of the JDK is installed.

Override the Local Time Setting for the Policy Server Log

The Policy Server log file, install_dir/siteminder/log/smps.log, displays time in local timezone as identified by the operating system of the machine on which the Policy Server is installed.

To display the time in this log file in GMT time:

1. Locate the following registry setting:

   HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Netegrity\SiteMinder\
   CurrentVersion\LogConfig\LogLocalTime
   

2. Change the value from 1 (which is the default) to 0.

Review System Application Logs

If the Policy Server fails to start, review the event log (on Windows) or the syslog (on UNIX) for information about the Policy Server.

• On Windows, view the event log using the Event Viewer. From the Log menu of the Event Viewer, select Application.
• On UNIX, view the syslog using a text editor.

LDAP Referrals Handled by the LDAP SDK Layer

Enhancements have been made to SiteMinder’s LDAP referral handling to improve performance and redundancy. Previous versions of SiteMinder supported automatic LDAP referral handling through the LDAP SDK layer. When an LDAP referral occurred, the LDAP SDK layer handled the execution of the request on the referred server without any interaction with the Policy Server.

SiteMinder now includes support for non-automatic (enhanced) LDAP referral handling. With non-automatic referral handling, an LDAP referral is returned to the Policy Server rather than the LDAP SDK layer. The referral contains all of the information necessary to process the referral. The Policy Server can detect whether the LDAP directory specified in the referral is operational, and can terminate a request if the appropriate LDAP directory is not functioning. This feature addresses performance issues that arise when an LDAP referral to an offline system causes a constant increase in request latency. Such an increase can cause SiteMinder to become saturated with requests.

Disable LDAP Referrals

If LDAP referrals are causing errors, you can disable all LDAP referrals. Note that disabling LDAP referrals will cause any referrals in your directory to return errors.

To disable LDAP referral handling for Policy Servers on Windows

1. From the Windows Start menu, select Run.
2. Enter regedit in the Run dialog box and click OK.
3. In the Registry Editor, navigate to:

   HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Netegrity\SiteMinder\
   CurrentVersion\Ds\LDAPProvider
   

4. Modify the following registry value:

   Note: The value is shown in hexadecimal notation.

   "EnableReferrals"=dword:00000001
   

   Determines if any LDAP referrals are handled by the Policy Server. If set to 0, no LDAP referrals will be accepted by the Policy Server. If set to 1, the Policy Server accepts LDAP referrals.

   LDAP referrals are enabled by default. This setting may only be modified by editing the Registry.

5. Restart the Policy Server.

To disable LDAP referral handling for a Policy Server on Solaris

1. Navigate to:

   install_dir/siteminder/registry
   

2. Open sm.registry in a text editor.
3. Locate the following text in the file:

   HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Netegrity\SiteMinder\
   CurrentVersion\Ds\LDAPProvider
   

4. Locate the line that follows the line from step 3 and begins with:

   EnableReferrals
   

5. Modify the value that comes just before the semicolon as follows.

   Note: The value must be converted to hexadecimal notation.

   Determines if any LDAP referrals are handled by the Policy Server. If set to 0, no LDAP referrals will be accepted by the Policy Server. If set to 1, the Policy Server accepts LDAP referrals.

6. Restart the Policy Server.

Handle LDAP Referrals on Bind Operations

To configure LDAP referrals on bind operations for Policy Servers on Windows

1. From the Windows Start menu, select Run.
2. Enter regedit in the Run dialog box and click OK.
3. In the Registry Editor, navigate to:

   HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Netegrity\SiteMinder\
   CurrentVersion\Ds\LDAPProvider
   

4. Modify the following registry value:

   Note: The value is shown in hexadecimal notation.

   "ChaseReferralsOnBind"=dword:00000001
   

   Determines if LDAP referrals on a bind operation should be chased. Most LDAP directory servers handle LDAP referrals on binds. If your directory server handles referrals on binds, ChaseReferralsOnBind has no effect. However, if your directory does not, this setting allows the Policy Server to handle bind referrals.

   If your server does handle referrals on bind operations you can change this setting to 0, disabling the Policy Server’s ability to handle bind referrals.

   Referral chasing on binds is enabled by default. This setting may only be modified by editing the Registry.

5. Restart the Policy Server.

To configure LDAP referrals on bind operations for a Policy Server on Solaris

1. Navigate to:

   install_dir/siteminder/registry
   

2. Open sm.registry in a text editor.
3. Locate the following text in the file:

   HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Netegrity\SiteMinder\
   CurrentVersion\Ds\LDAPProvider
   

4. Locate the line that follows the line from step 3 and begins with:

   ChaseReferralsOnBind
   

5. Modify the value that comes just before the semicolon as follows.

   Note: The value must be converted to hexadecimal notation.

   Determines if LDAP referrals on a bind operation should be chased. Most LDAP directory servers handle LDAP referrals on binds. If your directory server handles referrals on binds, ChaseReferralsOnBind has no effect. However, if your directory does not, this setting allows the Policy Server to handle bind referrals.

   If your server does handle referrals on bind operations you can change this setting to 0, disabling the Policy Server’s ability to handle bind referrals.

6. Restart the Policy Server.

Idle Timeouts and Stateful Inspection Devices

Stateful inspection devices, such as firewalls, generally have an idle timeout setting. SiteMinder connections from Policy Servers to Agents also have idle timeout settings.

The Policy Server polls the services at a regular interval. The polling interval has a 5-minute cap. This means the idle connections will time out within 5 minutes of the configured value. For example, if the value 55 minutes is specified as the timeout, then the connections will time out between 55 and 60 minutes.

By default, connections created between a Policy Server and a Web Agent expire after 10 minutes of inactivity. If a firewall or other stateful network device exists between a Policy Server and a Web Agent and connections are idle for longer that the device’s idle timeout, then the device ends those connections without notifying either the Policy Server or the Web Agent.

When the Web Agent attempts to use a connection that has been terminated by a network device, it receives a network error, resets the connection, and reports a 500 error (20-0003) to the browser. The Agent also closes all other connections in the connection pool that are the same age or older than the one that received the error. On the Policy Server side, however, the sockets for those connections remain established. Depending on the load patterns for the site, connection growth can occur to a point that it interferes with the proper operation of the Policy Server.

To prevent a firewall or other stateful network device from terminating Policy Server – Web Agent connections, you must configure an idle timeout for Policy Server. When the Policy Server closes a TCP/IP connection, it will wait for a specified period of inactivity and then send RESET, closing the server and client ends of the connection cleanly. The period of inactivity is specified in the Idle Timeout (minutes) field on the Settings tab of the Policy Server Management Console.

Note: The Idle Timeout (minutes) field can also be used to limit the amount of time an administrator may be connected.

At installation, the Idle Timeout value is set to 10 minutes. To work with a stateful network device, set the value to a shorter time period than the TCP/IP idle timeout of the device that is located between the web agent and the policy server. It is recommended that the TCP Idle Session Timeout be set to 60% of the idle timeout of any stateful device(s) to ensure that the Policy Server’s timeout occurs first.

Error -- Optional Feature Not Implemented

When the Policy Server attempts to use an ODBC data source, but cannot connect to the database, the following error message may appear:

Optional feature not implemented.. Error code -1

Often this message indicates a component mismatch, a misconfiguration or invalid credentials.

Note: CA's configuration of the Intersolv or Merant drivers differs from the default configuration.

If you receive the above message, and you are using an ODBC data source as your policy store, or for logging, see the sections that describe the configuration of ODBC data sources in the Policy Server Installation Guide.

Errors or Performance Issues When Logging Administrator Activity

On the Audit tab of the Policy Server Management Console, if you have set Administrator Changes to Policy Store Objects to Log All Events, and you are logging to an ODBC data source, you may encounter one of the following:

• Substantial delays when saving objects in the Administrative UI
• The error message:

  Exception occurred while executing audit log insert.
  

If either of these conditions occur, log to a text file instead.

Policy Servers Sharing Policy Store Not Updated Consistently

Symptom:

If multiple Policy Servers share a single policy store, the data inside the policy store could possibly be out of synchronization. Synchronization issues can occur under the following conditions:

• The system times on the Policy Servers differ.
• Network latency.

For example, suppose the system time on Policy Server A is 10:00, and the system time on Policy Server B is 10:05. Policy Server A sends its data to the policy store at 10:00. Policy Server B does not record any changes in the data timestamped before 10:05 because those events appear to have occurred earlier.

Solution:

To accommodate different system times or network latency issues:

1. Create the following DWORD registry setting:

   SiteMinder\CurrentVersion\ObjectStore 
   Key: ServerCommandTimeDelay
   

2. Set the value of the key to the number of seconds that corresponds to the time difference. For example, for a five-minute time difference, set the value of the key to 300.

Cache Failure Timeout

The Policy Server can sometimes fail to process events after deleting the following objects:

• Policies
• Rules
• Realms
• Policy domains

Cache failure timeout functionality addresses this issue.

When the secondary cache buildup is not successful, the policy server aborts after a timeout period. You specify the timeout period using the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Netegrity\SiteMinder\CurrentVersion\ObjectStore\CacheFailureTimeout

The value of this key is in seconds. The default is 0, which implies no timeout.

After the policy server shuts down, the smexec brings up the next process event request immediately.

Key Rollover Log Messages

When the Policy Server issues key rollover commands to Web agents, they can process the commands successfully some of the time, but other times, the commands fail. To facilitate troubleshooting in this situation, the Policy Server logs three types of messages to SMPS.log.

[INFO] Key Rollover Request has been initiated manually

This message is logged when an administrator manually initiates a key rollover.

[INFO] Key Rollover Request has been initiated automatically by Policy Server

This message is logged when the Policy Server initiates a key rollover automatically.

[INFO] Key distribution has been initiated by Policy Server

This message is logged when a key rollover request has been initiated, either automatically or manually.

Cache Update Log Messages

You can enable and disable cache flushing or updates through the Administrative UI or the Command Line Interface. To facilitate troubleshooting, the Policy Server logs two types of messages to SMPS.log.

[INFO] Server 'enablecacheupdates' command received.

This message is logged when cache flushing is enabled, either through the Administrative UI or the Command Line Interface.

[INFO] Server 'disablecacheupdates' command received.

This message is logged when cache flushing is disabled, either through the Administrative UI or the Command Line Interface.

Event Handlers List Settings Warning when Opening Policy Server Management Console

Symptom:

When I log into the Policy Server Management Console for the first time after upgrading to SiteMinder r12.5, a warning message appears saying that the event handlers list should be set to XPSAudit.

Solution:

For SiteMinder r12.5, you can no longer add custom event handler libraries using the Policy Server Management Console. Use the XPSConfig command-line tool to add any custom event-handler libraries.

More information:

Add Event Handler Libraries

SiteMinder Policy Server Startup Event Log

Symptom:

My Policy Server crashed while it was starting up. I want to know what SiteMinder startup events occurred before the Policy Server crashed.

Solution:

If the Policy Server crashes on startup, a log of the startup events is stored in the following file:

policy_server_home/audit/SmStartupEvents.audit