Content Provider Development Guide › Implementing the Indexer Function › Indexer Instance

Indexer Instance

State Transitions Diagram

A terminated indexer cannot be re-started, and that a terminating indexer cannot be stopped terminating.

While active, an Indexer may also throttle, if supported by the Content Provider. Throttling is a temporary state from which The Indexer either recovers (back to “running”), or which the Indexer leaves by terminating.

Initializing

Initialization is implemented by the ICISContentIndexer::Init() method. The method supplies the ICISStatusCallback interface pointer that the Indexer uses. See ICISContentProvider.idl for more behavior details.

Terminating

A client can actively terminate an Indexer by calling either ICISContentIndexer::Abort() or Terminate(), or by the client releasing the last reference to the ICISContentIndexer interface.

Alternatively, the Content Provider can terminate the Indexer. Such a self-termination may occur for many reasons. For example, if the Content Provider cannot recover from an error situation; if the content database is in a bad state that cannot be recovered from; or if “throttling” is going on for too long and has hit a timeout limit.

Throttling

A Content Provider must be in control of its content database system. This includes monitoring its health, managing its connections, managing the data that is pushed to it.

• Throttling is a state controlled by a Content Provider [Indexer]. Throttling is triggered when the Content Provider detects that the content database is not keeping up with document indexing.
• When a Content Provider [Indexer] decides to throttle, it delays the execution of the ICISContentIndexer::Index()calls. The only effect from a client’s point of view is that calls to Index() take longer to return, both in asynchronous and synchronous behaviors. The length of the delay for each call can be fixed, variable, or fully dynamic; that is entirely up to the Content Provider and the sophistication of its throttling control and management.
• The throttling continues while the Content Provider takes remedial action. For example, this may involve doing nothing but waiting for queues to shrink.
• Upon entering the throttling state, the Indexer must send a status report. This may make a client pause calling ICISContentIndexer::Index(), independently of the delaying mechanism.
• The Indexer should send status reports at regular intervals if throttling continues.
• Finally the Indexer must send a status report when it recovers. If recovery is not possible, it must start to terminate.

Indexing

A client queues a document for indexing by calling ICISContentIndexer::Index().Index() queues the document and returns CIS_S_PENDING.

By default, indexing is an asynchronous operation, with Index() returning when the document is queued or if the queuing fails.

If queued, the document is indexed in the background. When the document indexing is complete, the Indexer sends a status report through the callback interface. See also Asynchronous Behavior below.

1. Queuing a document should succeed, and Index() should return CIS_S_PENDING.
2. A Content Provider should fail to queue a document only if the Indexer is not “active” or if the parameters of the Index() call are structurally invalid (for example, an invalid NULL pointer). Apart from these conditions, the queuing of a document can only fail due to exceptional circumstances, such as failing to allocate memory or failing to notify the client of the queuing.

   Note: Notifying the client of the queuing by sending the “queued” status report through the callback interface is an integral part of the queuing process. If calling ICISStatusCallback::ReportStatus() fails, then the Indexer should fail the queuing. This way, it is not possible to have a situation where the Indexer has a document queued, but the client has not received the “queued” status report for it.

3. If Index() fails to queue the document, it returns an error code. No status report is generated for the document.
4. If Index() succeeds in queuing the document, it returns CIS_I_PENDING. The Indexer sends a “queued” status report for the document through the callback interface, typically before Index() returns. When the document’s indexing completes, the Indexer sends a “completed” status report for the document, regardless of whether the indexing was successful. The completed report can contain a failure code.

This mechanism helps ensure that besides incorrect API usage (an issue at the client) and a minimum of unexpected failures (for example, running out of memory), the start and end of the document indexing process are always logged. In other words, the acceptance of a document for indexing is the expected behavior, even if the indexing fails almost immediately because of trivia such as an invalid parameter value.

Note: Report any failure reason (such as invalid XML syntax, invalid configuration parameter, connection failure to the content database service, inconsistent or invalid document description, and others) through the "completed" status report. This approach maximizes information provided through logging.

The Index() call supplies four separate pieces of data: the document to index (i.e. its content), its description, its metadata, and some user reference data.

The user reference data is data that the Indexer must pass back in status reports about the document. That data is for the sole use of the client. It cannot be interpreted by a Content Provider.

A client may pass the document content directly, using an IStream interface, or through a file, by indicating contentInFile="true" in the document description.

• Passing document content using files is an optional feature offered by a Content Provider. A Content Provider may support it, but it is not mandatory.
• If the Content Provider supports it, this capability must be indicated in the Indexer function information element (see the XML schema), which alerts the client that the capability is available.
• If a client erroneously calls Index() with contentInFile="true", and the Content Provider does not support the capability, then the Indexer should report that the document’s indexing failed (for example, with CIS_E_NOT_SUPPORTED). As per the rules, this means that the Index() call succeeds with CIS_S_PENDING, the document is reported as “queued”, and then reported as “completed” with a failure code.

An Indexer may delay the execution of the method if it is currently throttling.

See ICISContentProvider.idl for more behavior details.

Asynchronous Behavior

A Content Provider must support indexing document asynchronously. This is the default behavior.

This means that calls to ICISContentIndexer::Index() return quickly (unless the Indexer is throttling) with CIS_S_PENDING.

A Content Provider can also support synchronous document indexing. If so, this capability must be indicated in the Indexer function information element (see the XML schema).

The ICISContentIndexer::Init() method implementation must verify if a client is wanting to configure the Indexer for synchronous document indexing, and fail if appropriate (for example, with CIS_E_NOT_SUPPORTED).

See ICISContentProvider.idl for more detail about the expected behavior of synchronous indexing and Index() in particular. Put simply, a synchronous version of Index() wraps up the asynchronous behavior implementation and synchronizes the “queued” and “completed” status reports before returning.

Note: The CA DataMinder 14.0 and 14.1 Content Indexer Server (client) does not currently use synchronous indexing. This may change in future releases.

Outstanding Indexing Operations on Termination

If an Indexer starts to terminate through a client request or through self-termination, it goes into the “terminating” state. Two things must happen before the Indexer moves on to the “terminated” state:

1. Any Index() call that was made before termination started and is currently in progress, but with its document not yet queued, must fail with CIS_E_SHUTDOWN.
2. Any document queued, but not completed, is abandoned immediately if the termination is traumatic, for example, an Abort() call. If the termination is normal, for example, a Terminate() call, then the document is allowed to complete or to time out.

Any call to Index() after termination starts must fail with CIS_E_SHUTDOWN.

Status Reports

The Indexer sends status reports by calling the ReportStatus() method on the ICISStatusCallback interface pointer that it received when the client called ICISContentIndexer::Init().

The Indexer must send status reports when:

• Initialization starts.
• Initialization completes (independent of whether it was successful).
• Throttling starts.
• Throttling ends and normal running resumes.
• Termination starts.
• Termination completes.
• The client asks for a flushing operation (see ICISContentIndexer::Flush()).
• A document is queued for indexing.
• A document’s indexing completes (independent of whether it was successful).

The Indexer may send optional status reports when, for example:

• The indexer does not timely move on from a temporary state (initializing, throttling, terminating).
• A document’s indexing is taking too long.
• Periodically, for instance to report the current value of monitored counters.

Multithreading

The implementation of the ICISContentIndexer interface must be multithreaded.

A client may call methods on an ICISContentIndexer interface pointer from multiple threads (for example, worker threads).