Information Gathering › Run the Logging Script › Debug the Web Crawler in the Knowledge Center

Debug the Web Crawler in the Knowledge Center

The Web Crawler in the Knowledge Center provides the capability to index the websites that are running on intranet or Internet. Note the following points:

• You can invoke the Web Crawler through the Knowledge Center Setting window. Click the Index URLs tab, enter the URL of the site that you want to index, and click Index. The web crawler initiates the crawling process in a separate JVM.
• Indexing of intranet URLs does not require any proxy information.
• If the crawl server is behind a proxy, then to index any internet URL user must provide proxy information. That is, proxy host, proxy port, domain name, user name, and password.
• If the Web Crawler does not start due to low memory space, the required JVM parameter to run Web Crawler is -Xms128m --Xmx768m.
• The UI displays the following indexing-statuses:
  • In-Progress—Indexing is running.
  • Failed—Indexing is failed due to some error.
  • Canceling—Canceling the indexing is in progress.
  • Canceled—User canceled the indexing.
  • Indexed—Indexing is completed.
  • Re-Indexed—User has reindexed the existing URL.

Index Location

The Web Crawler generates the indexes in the following location:

<CHORUS_HOME>/userdoc/mfui/webcrawls/crawlresults/<DIRECTORY WITH URL NAME>

URL Log

The URL indexing logs reside in the following location:

<CHORUS_HOME>/userdoc/mfui/webcrawls/crawlresources/<DIRECTORY WITH URL NAME>

The View Log screen shows only the files (along with timestamp and status) that are being sent for indexing.

Uploading documents and indexing are two separate actions. If any indexed file appears in the Index View log window, the file that you uploaded does not get indexed automatically. Index the folder (<CHORUS_HOME>/userdoc/<folder with user name>) explicitly by using the Index Documents tab.

Server Log

The server log in the following location can indicate what documents have been uploaded to <CHORUS_HOME>/userdoc folder:

$CHORUS_HOME/logs

order.xml

The order.xml file stores Web Crawler configuration information including the following:

• Website download limits.
• Download filters.
• Download depth.
• Page parsing information.
• Crawl output type and location.
• Proxy information in case of the Internet URL.

This file stores the proxy server name and port also. You can find this file in the following location:

<CHORUS_HOME>/userdoc/mfui/webcrawls/crawlresources / <DIRECTORY WITH URL NAME>/order.xml

Error Status FilecrawlErrorDetails.txt

The Error Status FilecrawlErrorDetails.txt file stores error information for which error has occurred and stores it in crawlErrorDetails.txt. You can find this file in the following location:

<CHORUS_HOME>/userdoc/mfui/webcrawls/crawlresources / <DIRECTORY WITH URL NAME>/crawlErrorDetails.txt