Dart KnowledgeWorker™ Advanced Search Installation Guide (3.1.5) – Dart

1. INTRODUCTION

This document describes the steps required to install the KnowledgeWorker® Advanced Search Engine (ASE). The ASE is an Add-On to an existing KnowledgeWorker® installation and provides advanced search capabilities over and above the intrinsic Item, Category and Content Search features.

The Advanced Search Engine provides a wide range of facilities for managing unstructured information, including:

document indexing
concept-based searching and ranking of results
dynamic document summarisation (aka “concepts in context”)
identification of related topics and query expansion
finding related documents

The pre-requisites for installing the ASE are:

A previously installed and tested implementation of KnowledgeWorker®. Note - the minimum KnowledgeWorker® version that will support the ASE is version 3.1.

The ASE requires the following Microsoft platform tools to be installed:

SOAP – version 3.0 or higher

NET – version 1.1 or higher

This document assumes that the reader has knowledge of installing and configuring KnowledgeWorker® and that both SOAP and .NET are installed.

2. OVERVIEW OF ASE

The illustration below provides an overview of the information flow from the user’s actions in the KnowledgeWorker® interface components into the ASE.

The main ASE components are:

QueryServer

Collector

ConceptIndexer

SQL Database

ConceptDatabase

2.1 QueryServer

All interactions between the KnowledgeWorker® application and ASE are via the QueryServer and consist of high-level XML transactions over a Web Services interface.

The QueryServer receives action requests from the KnowledgeWorker® application relating to documents. These action requests are presented to the QueryServer that then stores the requests in the SQL Database for subsequent processing by the document handling modules (Collector and ConceptIndexer).

The action requests from KnowledgeWorker® to the QueryServer are typically related to the addition of documents, versions or URL’s to the repository and a smaller number of deletion actions.

The search retrieval requests to the QueryServer use the proprietary probabilistic index (aka ConceptDatabase) to identify relevant results. The QueryServer delivers the results in XML packets to KnowledgeWorker® which then processes the results to ensure that the permission model is respected.

The QueryServer manages ALL access to the ConceptDatabase and these two components must reside on the same physical server.

2.2 Collector

The Collector is responsible for importing new documents. The Collector regularly scans the SQL Database for new document references and metaphorically “collects” those documents in readiness for processing by the Indexer. In effect the Collector simply ensures the document exists and that the content is going to be accepted by the Indexer.

When an action requests a web site to be indexed the Collector will fetch all pages linked, directly or indirectly, from the specified home page.

In addition to collecting documents the Collector also performs the following:

Conversion of documents to text format

Automatic language detection (unless the language is specified)

The following document types are currently supported:

All HTML formats

All XML formats

All text formats (e.g. “.csv” files)

Microsoft Word

Microsoft PowerPoint

Microsoft Rich Text Format

Adobe PDF

Corel WordPerfect

The following languages are automatically detected and processed:

English

Danish

Dutch

Finnish

French

German

Italian

Norwegian

Portuguese

Spanish

Swedish

Welsh

The Collector manages the queue of documents to be indexed, and outputs its results, via the SQL database. It does not access the ConceptDatabase. Therefore, there is no reason to host this process on the same server as the QueryServer. It should either be hosted on the same server as the SQL Database or alternatively on its own dedicated server (see Components and Configurations).

The Collector can be deployed as an executable or a windows service. Typically the executable is used to establish the configuration and the windows service is implemented once the ASE configuration has been proved.

2.3 ConceptIndexer

The ConceptIndexer regularly scans the SQL Database and processes any documents that have been cleared for indexing by the Collector. The ConceptIndexer inserts the document index data into the ConceptDatabase and updates the document control entry in the SQL Database to show that the document has been indexed.

The indexing activity can proceed concurrently with retrieval activity. However, heavy-duty indexing activity can significantly impair retrieval performance and so, if on-going indexing is very significant, the ConceptIndexer should either be run during quiet periods (perhaps overnight) or alternatively new information should be constructed off-line with a batch process updating the live index periodically.

If the ConceptIndexer is to update the live index as a background task then it is vital that this process runs on the same server where the SQL Database is located.

The ConceptIndexer can be deployed as an executable or a windows service. Typically the executable is used to establish the configuration and the windows service is implemented once the ASE configuration has been proved.

2.4 SQL Database - (SQL or Oracle)

The current release of ASE supports either SQL Server 2000 (or later) or Oracle (9i or later) via OLE/DB and is generically defined as the SQL database in the documentation and interface dialogues.
The SQL Database is used to manage the queue of documents being indexed. It may also be used by the application to store any application-specific information independently from the Searching function.
As described above the SQL Database will show that a document:

Requires collecting

Is available to be indexed

Is indexed and available for search

In general, all document and database status information is held in the SQL Database. In future the SQL Database will also be used to hold additional information to help build classifications / taxonomies.

2.5 ConceptDatabase

The ConceptDatabase contains the probabilistic index data for all documents that have been processed by the ConceptIndexer.

The following eight files comprise the ConceptDatabase:

DocID.cse

Http.cse

Lexicon.cse

LexiconKeys.cse

Postings.cse

TermID.cse

Ter

mlist.cse

Text.cse

It is vital that the live copy of the ConceptDatabase is located on the same server as the QueryServer.

Note: The “text.cse” is not a supplied file this file will be created automatically when the first documents are processed.

3. COMPONENTS AND CONFIGURATIONS

The ASE overview describes clearly the function of each of the components and this, together with the installation instructions for KnowledgeWorker®, makes it clear that a degree of planning is required prior to installation of ASE. Two servers are the minimum recommended requirement when using KnowledgeWorker® with the ASE but it should be noted, for most installations it is recommended that three servers be used for the complete KnowledgeWorker® environment with ASE.

General

In all cases, apart from setting up small-scale demonstration systems, when installing the ASE, a new server should be introduced to the KnowledgeWorker® environment.

Connecting

All communication between KnowledgeWorker® and the Advanced Search Engine is via the QueryServer web services. Where the SQL Database is located on a different server to the QueryServer web services the appropriate (SQL / Oracle) client must be installed so that QueryServer communicate with the SQL Database.

Adding ASE to an existing KnowledgeWorker® single server installation

In order to implement the ASE in an existing single-server installation it will be necessary to separate the current website and KnowledgeWorker® vaults from the KnowledgeWorker® Database.

The SQL Database (incorporating the ASE SQL Database and the KnowledgeWorker® repository database(s)) and the ASE components (QueryServer, Collector and ConceptIndexer) would, we recommend, be located on the higher specification of two servers.

Examples for a 2-server and 3-server configurations with KnowledgeWorker® and ASE are shown below.

KnowledgeWorker® and ASE in a 2-Server Configuration

This illustrates the minimum recommended configuration for the deployment of KnowledgeWorker® and ASE.

This deployment requires that Server 1 has the client software to communicate with the SQL Database (SQL Server or Oracle 9i) and that the Collector has access to the KnowledgeWorker® file storage / vaults. The access to the KnowledgeWorker® file storage must take into account the additional security requirements of .Net since the Collector is a .Net application.

KnowledgeWorker® and ASE in a 3 Server Configuration

In the 3-server configuration both Server 1 and Server 3 must have the client software to communicate with the SQL Database (SQL Server or Oracle 9i).

Server 3 must have an IIS Virtual Folder to be created for the QueryServer to provide the link between the KnowledgeWorker® website and the various ASE components.

Note - with the ASE components (ConceptIndexer, Collector and QueryServer) installed on a single server it is possible to have a single “config.cse” file if the programs share a common folder and the executable versions of the Collector and ConceptIndexer are utilised. However, if you are using the Windows Services for the Collector and ConceptIndexer then there will need to be a "config.cse" in the Windows System folder and so, in this case, at least two configurations must be undertaken.

4. INSTALLATION

The basic steps involved in installing the ASE are:

Ensure that Microsoft SOAP and .NET are installed on the ASE server

Copy the supplied ASE software on to the ASE server

Create the SQL Database (SQL Server or Oracle)

Install Collector and configure to link to the SQL database

Install ConceptIndexer and configure to link to the SQL database

Make the QueryServer web service available and link to the SQL database

Configure KnowledgeWorker® to use ASE

Note that in the following examples it is assumed that the ASE software has been installed within a Kworker folder on drive C:

IMPORTANT: It is assumed that the same variety of database (SQL Server or Oracle) is used for both the ASE SQL Database and the KnowledgeWorker® repository.

4.1 Install Microsoft SOAP and .NET

Ensure that Microsoft SOAP and .NET Framework are installed on the ASE Interface server. The current requirements are:

SOAP 3.0 or higher

.NET Framework 1.1 or higher

4.2 Install the supplied ASE software

The ASE components are delivered in the ASE directory of the CD. An ASE search components consists of an ASE folder containing the following:

ASE\ConceptCollector

ASE\ConceptCollectorService

ASE\Indexer

ASE\IndexerService

ASE\QS

ASE\ConceptDB

Copy the supplied ASE folder onto the server that has been designated the ASE Site. If this is located on the same server as the KnowledgeWorker® website then we suggest that the ASE folder be copied into the existing KWorker folder otherwise create a new Kworker folder on the new server.
Each of the above folders will contain a zip file. The contents of each zip file must be extracted into the containing folder.

4.3 Create the ASE SQL Database

The creation of the SQL Database for ASE involves a number of steps:

Creating / selecting a database / tablespace to use

Creating / selecting a user and granting this user access to the database / tablespace

Creating the tables within the database / tablespace

4.3.1 SQL Server

It is recommended that you create a new database, although you may elect to allow an existing user to have access to the database.

In SQL Server, using SQL Enterprise Manager it is possible to create a new database and user, and grant the newly created user access to the database as the database owner (dbo).

It is important to note the name of the database and the user credentials, user name and password, as these will be required in the ASE configuration dialogue for QueryServer, Collector and ConceptIndexer.

4.3.2 Oracle

You must create a new Tablespace and associated user for the ASE database. You may perform this task using Oracle Enterprise Manager or by running the script shown below.

Whether by using Enterprise Manager or the supplied script, the Tablespace and user credentials, user name and associated password, must be noted as these will be required in the ASE configuration dialogue for QueryServer, Collector and ConceptIndexer.

Tablespace and user creation script

Create tablespace tablespace_name DATAFILE 'dat_file_name' size initial_size reuse autoextend on next extend_size maxsize maximum_size;
Create user user_name identified by password default tablespace tablespace_name quota user_quota on tablespace_name;
grant connect,resource to user_name;
commit;

where:

tablespace_name - Name of the table space to be created within the Oracle instance;
dat_file_name - The full path and data filename for the data file to support the Tablespace being created;
initial_size - The initial size (in megabytes) of the Tablespace;
extend_size - The amount (in megabytes) the Tablespace is to be extended when more storage is required;
maximum_size - The maximum size to be allowed for the Tablespace;
user_name - The user name to be associated with the Tablespace;
password - The password to be applied to the supplied user name;
user_quota - The amount of Tablespace to be made available to the specified user.

IMPORTANT

The above information is provided for information purposes. We strongly recommend that the Oracle database administrator perform the allocation of the Tablespace and user together with the various allocation settings.

In the supplied script example and associated connection configuration dialogue the following values have been used.

tablespace_name - ASE_DEMO
dat_file_name - C:\OraDB\ASE_DEMO.
initial_size - 20 mb
extend_size - 5 mb
maximum_size - 40 mb
user_name - ORA_ASE
password - ORA_ASEPWD
user_quota 10 mb

The earlier script with these values would be:

Create tablespace ASE_DEMO DATAFILE 'C:\OraDB\ASE_DEMO.dat' size 20M reuse autoextend on next 5M maxsize 40 M;

Create user ORA_ASE identified by ORA_ASEPWD default tablespace ASE_DEMO quota 10M on ASE_DEMO;

grant connect,resource to ORA_ASE;

commit;

4.4 Connect to the SQL database

For both SQL Server and Oracle the next stage in the installation is to connect to the database and to create the structure in the database for ASE.

Executing the ConceptConfig program that is found in the directory C:\KWorker\ASE\ConceptQS\bin\conceptConfig creates the connection for either database type. The dialogue contains a tab for each database and the user is advised to complete only the details for the relevant database (SQL Server or Oracle).

4.4.1 SQL

Execute the program C:\KWorker\ASE\ConceptQS\bin\conceptConfig to bring up the dialogue and click the SQLServer tab as shown below:

NOTES

For ease of system administration we suggest that the user Id. and password be those used when creating the main KnowledgeWorker® repository.

The SQLServer Database Name can be set to “.” to indicate a local host installation or, in practice, the alias name used in SQL client to reference to the SQL Server.

The ConceptSearching Database Path refers to the location of the ConceptDatabase and this will already have the 8 blank probabilistic index data files for the ConceptDatabase that were unpacked from ASE\ConceptDB. When using the Browse button to populate the ConceptSearching Database Path navigate to ConceptDatabase directory and select one of the existing files with the “.cse” extension and the path will be populated appropriately.

4.4.2 Oracle

Execute the program C:\KWorker\ASE\ConceptQS\bin\conceptConfig to bring up the dialogue and click the Oracle tab as shown below.

Assume that the sample script described in the previous section was used then the following fields would complete the above dialogue:

Oracle Data Source - SID or TNSName
Oracle Login Name - ORA_ASE
Oracle Password - ORA_ASEPWD

The ConceptSearching Database Path refers to the location of the ConceptDatabase and this will already have the 8 blank probabilistic index data files for the ConceptDatabase that were unpacked from ASE\ConceptDB. When using the Browse button to populate the ConceptSearching Database Path navigate to ConceptDatabase directory and select one of the existing files with the “.cse” extension and the path will be populated appropriately.

4.4.3 SQL Server and Oracle

When all the information has been entered into the configuration dialogue simply click the Install SQL Database button to create the required schema. The Install SQL Database button will connect to the new empty database and run a script to create the ASE schema (SQL tables). This script will take a period of time to run and the user must wait for the message below to be certain that the script has completed.

Note- The Install SQL Database button is disabled unless the “config.cse” file is deleted.

If the database has previously been created or configured, but the “config.cse” file is not available to the ASE Component (Collector or ConceptIndexer) then the user should connect to the existing database using the Save button and upon successful connection the relevant “config.cse” file will be created.

If you click Save when configuring a new installation or the connection to the database cannot be established you will receive the following message.

4.5 Create the Query Web Service

The QueryServer Web Service provides the means of communication for the Collector, the ConceptIndexer and for search queries requested by the KnowledgeWorker®.

To create the Query Web Service, create a virtual directory and set its properties.
Create a Virtual Directory in IIS for the QueryServer folder that you previously installed in the Kworker\ASE\ConceptQS folder.

Right click the hosting Web Site, normally Default Web Site and select New->Virtual Directory.

This will initiate the Virtual Directory Creation Wizard.

Enter the Alias name as shown above or an Alias name of your choice and click Next.

Enter the location of the QueryServer and click Next. Click Next through to the end of the dialogues to complete the setup of the Virtual Directory.

In IIS locate the newly created Virtual Directory, in this example QueryServer. Right click the entry and select Properties from the pop-up menu.

Go to the Documents tab of the properties dialog that appears.

Remove any existing documents by selecting each and pressing the Remove button as these are not required. Add a single document stats.aspx as shown below.

Click OK to save the setting. The setup of the Virtual Directory for QueryServer is now complete.

Note - the Alias name of the QueryServer as this will be required to complete the installation of the ASE in KnowledgeWorker®.

4.6 Installing the Collector

The Collector may be installed as either a Windows Application or as a Windows Service. The following subsections describe the installation procedure for each option.

4.6.1 Installing the Collector as a Windows Application

In the conceptcollector folder (C:\KWorker\ASE\ConceptCollector) launch the program: conceptCollector.exe to begin the collection process.

The first time this program is run you will need to complete the configuration information as described in the previous section (Connect to the SQL Database). Then the main form will appear and collection will begin.

Minimize this Window to leave the process running.

Note - The conceptCollector must always be stopped using the Stop button so that it shuts down cleanly after saving all pending data. The shutdown normally takes a few seconds.

4.6.2 Installing the Collector as a Windows Service

The installation of the Collector as a windows service requires:

The completion of the configuration information as described in the previous section (Connect to the SQL Database) but using the “conceptConfig.exe” program from the conceptCollectorService folder (C:\KWorker\ASE\conceptCollectorService)

The conceptCollectorService folder (C:\KWorker\ASE\conceptCollectorService) content is then copied to the windows system directory (C:\Windows\system32)

The conceptCollector service to be started, as it is installed with a manual start-up type

Note - The conceptConfig.exe and config.cse may already exist in the windows system directory if the conceptIndexerService has previously been configured on the machine and may be overwritten.

Install the new Service using InstallUtil.exe. The program InstallUtil.exe is supplied with the .NET framework runtime and typically exists in the C:\WINNT\Microsoft.NET\Framework\v1.1.4322\ folder.

Run InstallUtil.exe from the Command Prompt e.g.

Be careful not to use a version of InstallUtil.exe from a previous .NET framework release, as it may not work.

The service can now be started and stopped like any other Windows Service.

Note - The service is installed with the Manual startup type but should be changed to automatic for production environments.

Use the QueryServer status page at: //localhost/conceptQS/stats.aspx to monitor the progress of the service in collecting documents.

The ConceptCollector service reports into the windows event viewer log Conceptsearching. Please review the information in the event viewer if the service fails to start.

4.6.3 Uninstalling the Collector Service

To uninstall the Collector Service again use InstallUtil.exe. The program InstallUtil.exe is supplied with the .NET framework runtime and typically exists in the C:\WINNT\Microsoft.NET\Framework\v1.1.4322\ folder.

Run InstallUtil.exe from the Command Prompt and use the /u option e.g.

4.7 Installing the ConceptIndexer

The ConceptIndexer may be installed as either a Windows Application or as a Windows Service. The following subsections describe the installation procedure for each option.

4.7.1 Installing the ConceptIndexer as a Windows Application

In the ConceptIndexer folder (C:\KWorker\ASE\ConceptIndexer) launch the conceptIndexer.exe program to begin the indexing process.

The first time this program is run you will need to complete the configuration information as described in the previous section (Connect to the SQL Database). Then the main form will appear and indexing will begin.

Minimize this Window to leave the process running.

Note - the ConceptIndexer must always be stopped using the Stop button so that it shuts down cleanly after saving all pending data. The shutdown normally takes a few seconds.

4.7.2 Installing the ConceptIndexer as a Windows Service

Installation of the ConceptIndexer as a windows service requires:

The completion of the configuration information as described in the previous section (Connect to the SQL Database) but using the “conceptConfig.exe” program from the conceptCollectorService folder (C:\KWorker\ASE\conceptIndexerService)

The conceptIndexerService folder (C:\KWorker\ASE\conceptIndexerService) content is then copied to the windows system directory (C:\Windows\system32)

The ConceptIndexer service to be started, as it is installed with a manual start-up type.

The conceptConfig.exe and config.cse may already exist in the windows system directory if the conceptCollectorService has previously been configured on the machine and may be overwritten.

Install the new Service using InstallUtil.exe. InstallUtil.exe is supplied with the .NET framework runtime and typically exists in the C:\WINNT\Microsoft.NET\Framework\v1.1.4322\ folder.

Run InstallUtil.exe from the Command Prompt. e.g.

Be careful not to use a version of InstallUtil.exe from a previous .NET framework release, as it may not work.

The service can now be started and stopped like any other Windows Service.

Note - The service is installed with the Manual startup type but should be changed to automatic for production environments.

Use the QueryServer status page at: //localhost/conceptQS/stats.aspx to monitor the progress of the service in collecting documents.

The ConceptIndexer service reports into the windows event viewer log Conceptsearching. Please review the information in the event viewer if the service fails to start.

4.7.3 Uninstalling the ConceptIndexer Service

To uninstall the ConceptIndexer Service, again use InstallUtil.exe. The program InstallUtil.exe is supplied with the .NET framework runtime and typically exists in the C:\WINNT\Microsoft.NET\Framework\v1.1.4322\ folder.

Run InstallUtil.exe from the Command Prompt and use the /u option.

It may be necessary to reboot to complete the uninstall process.

4.8 Configure ASE within KnowledgeWorker®

The previous installation sections have involved installing and configuring the ASE and its components. This section identifies the changes that must now be made to KnowledgeWorker® and the repository in order that connection is made to the ASE.

4.8.1 Update the KnowledgeWorker® website

Whilst the vast majority of KnowledgeWorker® pages are ready to handle ASE, the following folders/files are provided for updating your existing KnowledgeWorker® website.

As always, ensure you have taken a backup of the existing website before copying over the new files to the relevant locations.

Scripts

FolderInfo.asp

NewFolder.asp

4.8.2 Enable / Disable ASE

This section describes how an administrative account in the repository can enable / disable the use of the ASE.

IMPORTANT: The ASE should not be enabled in the repository unless the ASE components have been previously installed and configured.

Login to repository as an administrative user, such as system, and go to
Admin->System Administration->Configure System Parameters

Change the status of the Advanced Search to Activate, as illustrated in the follow screenshot.

Click the Submit button.

Note - When activating the Advanced Search facility you MUST exit the repository and login again in order for the Configure Advanced Search item to appear in the System Administration page.

4.8.3 Configure ASE Connection & Defaults

To complete the ASE connection and default search parameters Login to the repository as an administrative user, such as system, and go to Admin->System Administration->Configure Advanced Search.

The Advanced Search Configuration page.

The Advanced Search Configuration parameters can be broken into three key elements:

The location of the QueryServer Web Server

The connection details for the SQL Database

The default parameters for the Advanced Search query pages

4.8.3.1 Configure connection to QueryServer

ASE Qs Web Site URL - URL pointing to the QueryServer Web Service.

Note: - Typically this will be the FQDN that is either held in DNS or at least the machine name and virtual directory name for the QueryServer.

4.8.3.2 Configure ASE Database Parameters

These are the connection details for the SQL Database that was created for ASE. Most of these will have already been used in configuring the ASE dialogues and therefore familiar to the administrator.

ASE Server name - The name of the machine running the database and has the database executing on it
ASE Database Name - The name used to connect to the SQL / Oracle server using the client software for the database. This will be the Alias in the SQL Server client software or TNSName in the Oracle client software
ASE Database User Id. - User Id for accessing the SQL Database
ASE Database Password - Password for accessing the SQL Database

4.8.3.3 Configure ASE Search Parameters

These items are the default values that appear in the search query page and define the criteria used in the search and the results that are returned. These values should be tailored to suit the use of the repository.

Highlight Query Terms - Show Query Terms highlighted in the result text
Min. doc relevance - Minimum document relevance to show in the results
Min. extract relevance - Minimum extract relevance to show in the results
Max Extracts - The maximum number of extracts for a result
Doc summary length - Length of the summary text for a result
Hits per page - Number of hits to show in a single page
Max related topics - Maximum number of related topics to list
Number of results to display - Defines the maximum number of results that are to be returned based upon the relevance

Click the Submit button to save your new settings.

Installation of the KnowledgeWorker® Advanced Search Engine is now complete.

5. MONITORING CONCEPT INDEX

Although actions are delivered to the ASE and queued almost immediately by the QueryServer, it is the Collector and ConceptIndexer that perform the actual processing as background tasks.

In some cases a single index request (e.g. to index a whole web site) can cause thousands of pages to be collected and indexed. This can sometimes appear to slow down the indexing of other content.

For a quick summary of indexing progress you can visit the QueryServer stats.aspx page (see below). This page is not intended to be delivered to the user community in general but to provide those managing and maintaining the system with a monitoring tool for the progress of indexing into the ConceptDatabase.

Those maintaining the ASE are also at liberty to analyse the progress of indexing in more detail with a query to the Pages table of the SQL Database where the following status codes are used:

6. CONFIGURING ASE FOR MULTIPLE REPOSITORIES

If you have more than one repository in use and you wish to use the ASE for each repository you must configure each one individually. The illustration shows the relationship between the configuration dialogue and the various ASE components.

Each time the ASE configuration is run a config.cse file is created in the ASE site folder, specifically in the QueryServer\Bin folder. The creation of and the location of the config.cse file is important as this is the vital ingredient to making the ConceptIndexer and Collector service multiple repositories.

6.1 Using separate ASE ConceptDatabase for each repository

If required, you may create an ASE Site and associated ConceptDatabase for each repository. Such a configuration would allow the flexibility of relocating an entire repository including the associated ASE without having to update and re-index the ASE.

The procedure to create such a configuration involves simply repeating the install procedure for each repository.

Each installation can be stored within a parent ASE folder so that the ASE Site and associated ConceptDatabase will be as shown below:

KWorker\ASE\ASE_Site_01\QueryServer
\Collector
\etc
KWorker\ASE\ASE_Site_01\

IMPORTANT - We recommend that when using multiple ConceptDatabases only one instance of the Collector and ConceptIndexer is employed. In order to achieve this, complete each of the install and configurations as described above and then copy one of the created QueryServer folders into the parent ASE folder.

Copy each config.cse file from the QueryServer\Bin sub-folders used in the configuration process and rename the files to:

Config1.cse
Config2.cse etc

The Indexer and Collector programs will scan the folder from which they are executed and process each confign.cse file. The ConceptIndexer and Collector programs will support a maximum of 25 repositories.

7. ADDINS

7.1 ASE Synchroniser

At various times it may be necessary to synchronise the repository with the corresponding ConceptDatabase. For example:

After adding the ASE to an existing repository

System problems may have caused the ASE server to be unavailable for a short period

For whatever reason, the ASE may have been set Inactive in the repository System Parameters

In order to ensure that the ASE ConceptDatabase is up to date and synchronised with the repository it is necessary to run the program ASE_Synch.exe. The function of the ASE_Sync.exe is to compare the content of the repository with the content of the ConceptDatabase and introduce objects from the repository as add actions to the QueryServer. The effect of the add action will be to queue content for the Collector and ConceptIndexer.

If the repository is large, it should be noted that the synchronisation will take a considerable amount of time and system resources to complete and may need to be undertaken as a background process.

The install disc contains the following files and folders:

Addins\ASE_SYNC\ASE_SYNC.EXE - Synchroniser program

Addins\ASE_SYNC\Templates - XML config files

Addins\ASE_SYNC\Log - Log file created by ASE_SYNC.EXE

The templates folder will contain the following files:

ASE_DOC_ADD_TEMPLATE.XML

ASE_URL_ADD_TEMPLATE.XML

ASE_CONFIG.XML

The first two files are used by the system and should not be amended.

The ASE_CONFIG.XML file will be supplied with dummy database and site identifiers and must be amended in accordance with the rules described below before running the synchronizer program.

<?xml version="1.0" encoding="utf-8" ?>

<xml>

<ase_configurations>

    <config>

      <kworker_database_name>KW_DB_1</kworker_database_name>

      <ase_database_name>ASE_DB_1</ase_database_name>

    <ase_qs_web_site_url>http://Server/ASE_Site/</ase_qs_web_site_url>

      <max_rec_count>1000</max_rec_count>

    </config>

</ase_configurations>

</xml>

where

<kworker_database_name> - Repository Database Name, usually this aligns with the name used in the login dialogue

<ase_database_name> - Id. of the ASE SQL/Oracle database

<ase_qs_web_site_url> - Location of the QueryServer Web Service

<max_rec_count> - Maximum number of records to process during each run of the process.

The <config> parameter set must be repeated for each repository and ASE instance as shown below.

<?xml version="1.0" encoding="utf-8" ?>

<xml>

<ase_configurations>

    <config>

      <kworker_database_name>KW_DB_1</kworker_database_name>

      <ase_database_name>ASE_DB_1</ase_database_name>

    <ase_qs_web_site_url>http://Server/ASE_Site/</ase_qs_web_site_url>

      <max_rec_count>1000</max_rec_count>

    </config>

    <config>

      <kworker_database_name>KW_DB_2</kworker_database_name>

      <ase_database_name>ASE_DB_2</ase_database_name>

    <ase_qs_web_site_url>http://Server/ASE_Site/</ase_qs_web_site_url>

      <max_rec_count>1000</max_rec_count>

    </config>

    <config>

      <kworker_database_name>KW_DB_3</kworker_database_name>

      <ase_database_name>ASE_DB_3</ase_database_name>

    <ase_qs_web_site_url>http://Server/ASE_Site/</ase_qs_web_site_url>

      <max_rec_count>1000</max_rec_count>

    </config>

</ase_configurations>

</xml>

8. TROUBLESHOOTING ASE

This section describes solutions to some common configuration problems.

8.1 I don’t know how to unzip the installation files

The most popular decompression program is available for download from http://www.winzip.com/.

This is available to purchase and there is also a free evaluation version that can be used.

8.2 I don’t have the .NET Framework installed

This is a free download available from Microsoft at http://www.microsoft.com/downloads.

8.3 I get an Error when Browsing to QueryServer

After installation you should be able to browse to http://localhost/QueryServer.

If this returns a Configuration Error with the text “This error can be caused by a virtual directory not being configured as an application in IIS” then the problem is that the web site was not created correctly using the Internet Information Services tool. See Section 4.5 for instructions on using this tool.

If this returns a Configuration Error with the text “The page cannot be displayed” then the website does exist but the required page cannot be located. Check that the default page is set to: “Stats.aspx” using the Internet Information Services tool. See Section 4.5 for instructions on using this tool.

8.4 I get a connection not initialised error in the stats.aspx

When monitoring the indexing progress by visiting the QueryServer “stats.aspx” page if the error message “connection has not been initialised” is observed this indicates that there is a problem connecting to the SQL Database.

Check that the configuration information, as described in the previous section (Connect to the SQL Database), has been completed for all of the components (QueryServer, Collector and ConceptIndexer), including those running as windows services.

8.5 All attempts to Connect to the SQL Server Database Fail

Check that the SQL Server Authentication mode is set to: “SQL Server and Windows”. To view the SQL Server Authentication mode run SQL Server Enterprise Manager and access the Properties for your server. The setting is located on the Security tab.

ASE has been tested with all version of SQL Server 2000 but not with any previous versions.

If you can successfully configure the SQL Server location using conceptConfig.exe but QueryServer complains that the “connection has not been initialised” then the problem is likely to be with the ASP.NET security configuration. In this case you should edit the machine.Config file.

Edit the text file using any text editor (e.g. Notepad) and change the “processModel Attributes” section.

If the current configuration shows userName=”machine”
Then change this to userName=”SYSTEM”

This change gives the ASP.NET process higher privileges. Save the changes and reboot your server to see if this cures the problem.

8.6 All attempts to Connect to the Oracle Database Fail

ASE has been tested with Oracle 9i and 10g on various platforms but not with any previous versions.

1. Use Oracle’s Enterprise Manager or SQL*Plus tools to try the same connection. Make sure that you use the same information when using conceptConfig.exe.

Oracle Data Source - This is the Oracle “Service Name” or “SID”
Oracle User Name - This is the Oracle “Username”
Oracle Password - This is the Oracle “Password”

Note that the SQL Database schema will be created in the default TABLESPACE of the user you specify. This Oracle user MUST have sufficient privilege to create and drop tables, indexes and sequences.

2. If you can successfully configure the Oracle location using conceptConfig.exe but QueryServer complains that the “connection has not been initialised” then the problem is likely to be with the ASP.NET security configuration. In this case you should try the editing machine.Config as detailed in the previous subsection for problems connecting to SQL Server.

3. If you can successfully configure the Oracle location using conceptConfig.exe but QueryServer complains that the “connection has not been initialised” then the problem is likely to be with the ASP.NET security configuration. In this case you should try the following:

Add the IUSR_machine name account and the IWAM_machine name account to the Oracle home folder

Give both accounts full control

To do this, right-click on the Oracle home folder in Windows Explorer

Click Properties, and then click the Security tab. Add the appropriate accounts and then give them full control

Restart your computer

4. If it still fails then try the following:

Disable SQL*Net Authentication. To do this, edit the SQLNET.ora file
This configuration file is usually stored in the Network\Admin subfolder of the Oracle home folder
Add the following lines to this file:

SQLNET.AUTHENTICATION_SERVICES = (none)
SQLNET.AUTHENTICATION = (none)

Restart the IIS service after you make these changes.

8.7 Indexing speed has slowed

When ASE is initially installed with an empty database it has been deliberately setup to have a very fast scan rate for the Collector and ConceptIndexer in order that the user community can gain confidence that it is indexing their prized documents. When the document count exceeds 1000 it is assumed that the community now trusts the service and the indexing slows to the normal frequency.

If the queue of documents to be indexed is very high and remains consistently high then this may suggest that it is time to consider reconfiguring the ASE environment to better suit the needs of the Collector and ConceptIndexer.

8.8 Documents are not Indexing when using Services

When ASE is initially installed, the Collector and ConceptIndexer are installed as windows services and are defined by default with the manual startup type. If this startup type has not been changed and the server is rebooted then these services will not start automatically. Please check that these services have been started because new documents will not be collected or indexed if these services are not running. Documents will have still been queued in the ASE database if ASE is enabled in the repository.

If the services are running correctly but documents are still not being collected, check that any changes to the configuration data, such as connection to the SQL Database have been also applied to all the components (QueryServer, Collector and ConceptIndexer). A quick way to confirm the configuration is consistent, when the ASE components are on a single machine is:

Check the configuration information held by QueryServer by launching conceptConfig (see Connect to the SQL database section)

Stop the Collector and ConceptIndexer services

Make a backup of the config.cse file in C:\windows\system32

Copy the config.cse file from C:\KWorker\ASE\ConceptQS\bin to C:\Windows\System32

Note - Overwrite the file in C:\Windows\System32. Restart the Collector and ConceptIndexer services.