Analytics Conceptual Index (.NET)

A conceptual index is a type of index used by Relativity Analytics. It uses Latent Semantic Indexing (LSI) to discover concepts between documents. This indexing process is based solely on term co-occurrence. The language, concepts, and relationships are defined entirely by the contents of your documents and learned by the index. For general information about Analytics indexes, see Analytics indexes and on the RelativityOne Documentation site.

The Analytics Conceptual Index API supports programmatically managing conceptual indexes in Relativity Analytics. It includes the following features:

  • Supports CRUD operations on Analytics indexes.
  • Provides helper methods that simplify working with index jobs. You can use these methods to submit and cancel jobs, and to check on their statuses. These jobs can populate new data, build new indexes, and activate indexes for use.

As a sample use case, you might use this API in an application to programmatically operate on a conceptual index.

You can also use the Conceptual Index API through REST. For more information, see Analytics Conceptual Index (REST).

The Relativity.Analytics.Conceptual.SDK contains this API. For compatibility and package installation instructions, see Download the SDKs and NuGet packages.

Fundamentals for the Conceptual Index API

Review the following information to learn about the methods, classes, and enumerations used by the Conceptual Index API.

The Conceptual Index API includes the following methods available on the IConceptualIndexService interface in the Relativity.Analytics.Conceptual.<VersionNumber>.Services namespace.

Note: The <VersionNumber> variable in the namespace indicates the version number of the API. The version number uses the format uppercase V and an integer version number, such as V1 or V2 in .NET.

The Conceptual Index API uses the following classes and enumerations:

  • AnalyticsIndexJobType enumeration - defines valid index job types as follows:
    • FullPopulation (0) - runs a full index population.
    • IncrementalPopulation (1) - runs an incremental population. This process adds or removes documents from the training and searchable sets. If active, the index remains active, and you can query it during the incremental population. You can't query new documents until a build followed by an activate is completed.
    • RetryErrors (2) - runs a full population if a previous population attempt failed.
    • RemoveDocumentsInError (3) - removes documents in error.
    • BuildIndex (4) - runs a full index build. If any training documents are added, this step takes longer because the engine rebuilds the concept space. If only searchable documents are added, the engine starts this build at a later step, updating searchable items, because it only maps the new searchable documents into the pre-existing concept space.
    • Activate (5) - activates the index for querying.
    • Deactivate(6) - disables queries on the index. The index must be deactivated to perform a build.
    • TagDocuments(7) - tags documents that are a part of the training and searchable sets for an index
  • AnalyticsIndexStatus class - represents the aggregate state of an Analytics index. The GetStatusAsync() method returns a status object. Its properties include:
    • Active - indicates whether the index is active and can be queried.
    • ConsoleButtons - a list of console buttons and links with their current states. The states can be used to determine which job steps can be run for the index. See AnalyticsIndexJobType enumeration in Classes and enumerations.
    • CurrentStep - the current job step when a multi-step job is in progress.
    • DocLevelErrorsExist - indicates whether documents in the training set are in error.
    • LastErrorMessage - the last error that occurred on the index
    • OverallTimeElapsed - the total time of the current job when one exists.
    • PercentageComplete - the completed percentage of the current job step when one exists.
    • PercentDenominator - the denominator value used to generate the percentage of the current job step when one exists.
    • PercentNumerator - the numerator value used to generate the percentage of the current job step when one exists.
    • SearchableSetCount - the count of items in the searchable set when an index is new or active in its final state.
    • SecondaryPercentageComplete - the completed percentage of a stage of a job step when one exists.
    • SecondaryStatusText - additional status text.
    • StatusText - the main status text of the index. For more information, see Index states and job validation.
    • StepTimeElapsed - the elapsed time of the current job step when one exists.
    • TotalSteps - the total number of job steps when a multi-step job is in progress.
    • TrainingSetCount - the count of items in the training set when the index is new or active in its final state.
  • ArtifactRef class - represents a Relativity artifact referenced by an index property, such as repeated content filters, searchable sets, and training sets. Its properties include:
    • ArtifactID - represents a unique identifier, called an Artifact ID, for an object.
    • Guids - represents the GUIDs used to identify an object.
  • ConceptualIndex class - represents the index. Its properties include:
    • Active - indicates whether the index is active
    • AnalyticsServer - the Analytics server used to build the index. This value is a ArtifactRef object.
    • ArtifactID - the Artifact ID of the index.
    • ConceptStopWords - words to suppress from the index. You can add or remove stop words from the list. Separate each word with a hard return. The default values are derived from the [EDDSDBO].[ContentAnalystDefaultNoiseWords] table.
    • ContinueIndexStepsToCompletion - a Boolean value indicating whether to automatically complete all steps necessary to activate an Analytics index after starting a step. For example, if you set the property to true, and submit a populate job, the index is automatically built and activated. If you don't set it, only the population occurs, and building and activating must be done manually.
    • Dimensions - the number of dimensions of the concept space in which documents are mapped when the index is built. More dimensions increase the conceptual values applied to documents and refine the relationships between documents.
    • EmailNotificationRecipients - the list of email recipients notified during index population and build.
    • EnableEmailHeaderFilter - removes common header fields (such as To, From, and Date) and reply-indicator lines. It doesn't remove content from the Subject line. Use this filter to ensure that the headers in the concept space don't overshadow the authored content. The default value is false.
    • Guids - the GUIDs for this artifact.
    • IndexLastUsedOn - the time when the index was last used.
    • InUseByCategorizationSet - indicates whether the index is in use by an LSI categorization set.
    • LastErrorMessage - the last error message to occur on the index.
    • Name - the user-friendly name of the index.
    • OptimizeTrainingSet - a Boolean value indicating whether to select only conceptually relevant documents from the training set saved search. To improve index quality, this property automatically excludes documents that are very large, very small, or contain many numbers when this property is set to true. The default value is false.
    • Order - an integer representing the position of the index in drop-down menus. The default value is 0.
    • RemoveDocumentErrors - removes documents from being populated when they have errored in a previous population.
    • RemoveEngSignaturesAndFooters - a Boolean value indicating whether to remove signatures and footers in English language emails from the text stored in the index. By default, this property is set to true for new indexes and false for existing ones. Setting this property to true enables the email header filter, disables the Go Words and OCR filters, and removes documents greater than 30 MB from the searchable set. The default value is false.
    • RepeatedContentFilters - the repeated content filters associated with the index.
    • TrainingSet - the saved search used to populate the training set. This value is a ArtifactRef object.
    • SearchableSet - the saved search to populate the searchable set. This value is a ArtifactRef object.

Guidelines for using the Conceptual Index API

Review the following guidelines for working with the Conceptual Index API.

Common workflows

For Analytics indexes, common workflows include building new ones or incrementally updating existing ones.

Use the following steps to build a new index:

  1. Populate the index by adding all documents from the training set and searchable set to the ready-to-index list.
  2. Build the index by building the concept space.
  3. Activate the index by make it available to users. Add the index to the search drop-down on the Documents tab and to the right-click menu in the viewer.
  1. Populate the index incrementally by checking the data sources for the training and the searchable sets. If either search includes new documents, Analytics adds them to the index. At this stage, user can query against the index.
  2. Deactivate an index by disabling the index. It can't be queried against or used for any operations with Analytics. You must deactivate the index to perform a build.
  3. Build the index by building the concept space. If any training documents are added, this step takes longer because the engine rebuilds the concept space. If only searchable documents are added, the engine starts this build at a later step, updating searchable items, because it only maps the new searchable documents into the pre-existing concept space.
  4. Activate the index to make it available to users.

Catch exceptions

When using any methods in the Conceptual Index API, catch the following exceptions:

  • PermissionException
  • ValidationException
  • Generic .NET Exception

Execute valid operations for the index state

Execute only valid operations for the state of an index. Only certain operations are valid for the current state of your index. See the following examples:

  • A call to cancel a running index job is only valid when a job is currently running for the index.
  • A call to build an index can't run on an index that hasn't been populated.

    • Note: To retrieve a list of valid operations, get the AnalyticsIndexStatus object. Examine the ConsoleButtons property on the returned object for a list of valid operations to call. You can also get additional information about the job from the StatusText and AdditionalStatusText properties. If you attempt to execute an invalid operation, you receive validation error. See Retrieve the status of a job.

Index states and job validation

When performing operations on an index, make sure the operation is applicable to its current state.

Some states overlap with each other, and others have sub-states depending on additional index metadata. For example, many states include an error state, such as STATE – 1 or more documents in error status, when state is populating, building, and so on.

For example, an error scenario occurs when items linked to the index (saved searches, Analytics profile) can't be accessed. Regardless of any additional error state, the validator treats it as an error. It subsequently only lets the two error resolution-type jobs run: RetryErrors or RemoveDocumentsInError. The message state values are returned as the StatusText property.

The states of an index are as follows:

  • Activating
  • Activation Recommended
  • Active
  • Attempting to cancel population
  • Build Recommended
  • Building
  • Configuring Analytics Engine
  • Constructing Population Table
  • Content Analyst Server is unavailable
  • Deactivating
  • Disable queries to continue
  • Disabling Queries
  • Enabling Queries
  • Failed to activate index
  • Failed to deactivate index
  • Finalizing build
  • Index Build Failed
  • Indexing Job in Queue
  • New
  • Not enough conceptual content to build
  • Please check your training set
  • Populating
  • Populating Documents
  • Population Canceled by User
  • Population Failed
  • Preparing to build
  • Removing Documents in Error
  • Waiting

Monitor the analysis progress

After submitting an index job, monitor its progress by making calls to its status. See Retrieve the status of a job.

Create a conceptual index

Use the CreateAsync() method to add a new index to Relativity. Set the following properties on the ConceptualIndex object passed to the CreateAsync() method:

  • AnalyticsServer - an ArtifactRef object representing the Analytics server used to build the index. In RelativityOne environments, set the Artifact ID -1.
  • Name - the user-friendly name of the index.
  • Order - an integer representing the position of the index in drop-down menus. The default value is 0.
  • SearchableSet - an ArtifactRef object representing the saved search used to populate the searchable set.

When the operation succeeds, it returns the Artifact ID of the new index. Use the Artifact ID to submit the index for populating and building.

Copy 
public async Task CreateAsyncSampleCode()
{
    ServiceFactorySettings settings = new ServiceFactorySettings(
      new Uri("net.pipe://localhost/relativity.services/"),
      new Uri("http://localhost/relativity.rest/api"),
      new IntegratedAuthCredentials());
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
 
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int searchableSetSearchArtifactID = 1038051;
        int trainingSetSearchArtifactID = 1038051;
        int analyticsServerArtifactID = 1016891;
  
        ConceptualIndex indexToCreate = new ConceptualIndex
        {
            Name = "my new analytics index",
            Order = 1,
            SearchableSet = new ArtifactRef { ArtifactID = searchableSetSearchArtifactID },
            TrainingSet = new ArtifactRef { ArtifactID = trainingSetSearchArtifactID },
            AnalyticsServer = new ArtifactRef { ArtifactID = analyticsServerArtifactID },
            OptimizeTrainingSet = true,
            RemoveDocumentsThatErroredDuringPopulation = true,
            ContinueIndexStepsToCompletion = true,
            ConceptStopWords = "a\r\nable\r\nabout\r\nabove\r\naccording\r\n",
            Dimensions = 100,
            RemoveEnglishSignaturesAndFooters = true,
            EnableEmailHeaderFilter = true
        };
  
        try
        {
            int conceptualIndexID = await indexService.CreateAsync(workspaceId, indexToCreate);
            Debug.WriteLine("ID of the newly created index is: " + conceptualIndexID);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "Failed to create index because of invalid permission. workspaceID: {workspaceId} - IndexToCreate: {@index}",
            workspaceId, indexToCreate);
            throw;
        }
        catch (ValidationException ex)
        {
            _logger.LogError(ex,
            "The index you were trying to create was invalid. workspaceID: {workspaceId} - IndexToCreate: {@index}",
            workspaceId, indexToCreate);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Creating index failed. workspaceID: {workspaceId} - IndexToCreate: {@index}",
            workspaceId, indexToCreate);
            throw;
        }
    }
}

Retrieve a conceptual index

Use the ReadAsync() method to retrieve an index. Pass the Artifact IDs of the workspace and the index to this method. It returns a ConceptualIndex object. You can use this object to examine the index properties before submitting an index job, or updating or deleting the index.

Copy 
public async Task ReadSingleAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1049079;
  
        try
        {
            ConceptualIndex conceptualIndex = await indexService.ReadAsync(workspaceId, existingIndexArtifactID);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "You don't have permission to retrieve the index. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (NotFoundException ex)
        {
            _logger.LogError(ex,
            "Failed to retrieve the index because index could not be found. WorkspaceID: {workspaceID} - IndexID: {IndexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Failed to retrieve the index. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

Update a conceptual index

Use the UpdateAsync() method to modify an index. You can update multiple properties on a ConceptualIndex object. For a list of properties, see the ConceptualIndex class in Classes and enumerations.

Copy 
public async Task UpdateAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1049079;
  
        ConceptualIndex conceptualIndex= null;
  
        try
        {
            try
            {
                conceptualIndex = await indexService.ReadAsync(workspaceId, existingIndexArtifactID);
            }
            catch (PermissionException ex)
            {
                _logger.LogError(ex,
                "You don't have permission to read the index. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
                workspaceId, existingIndexArtifactID, userName);
                throw;
            }
            catch (NotFoundException ex)
            {
                _logger.LogError(ex,
                "Failed to read the index because index could not be found. WorkspaceID: {workspaceID} - IndexID: {IndexId}",
                workspaceId, existingIndexArtifactID);
                throw;
            }
            conceptualIndex.Name = "My Updated Analytics Index";
            await indexService.UpdateAsync(workspaceId, existingIndexArtifactID, conceptualIndex);
  
            conceptualIndex = await indexService.ReadAsync(workspaceId, existingIndexArtifactID);
  
            Debug.WriteLine("The updated name of the retrieved index is " + conceptualIndex.Name);
  
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "You don't have permission to update the index. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (ValidationException ex)
        {
            _logger.LogError(ex,
            "The index you were trying to update has invalid data. workspaceID: {workspaceId} - IndexToUpdate: {@index}",
            workspaceId, conceptualIndex);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Failed to update the index. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

Delete a conceptual index

Use the DeleteAsync() method to remove an index from Relativity. Pass the Artifact IDs of the workspace and the index to this method.

Copy 
public async Task DeleteAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1054085;
  
        try
        {
            await indexService.DeleteAsync(workspaceId, existingIndexArtifactID);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "Deleting Index failed because of invalid permission. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (NotFoundException ex)
        {
            _logger.LogError(ex,
            "Deleting Index failed because index could not be found. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (DeletionDependencyException ex)
        {
            _logger.LogError(ex,
            "Deleting Index failed because there is a dependecy on it. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Deleting index failed. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

Helper methods for conceptual index jobs

Use the following helper methods for managing jobs related to conceptual indexes.

Submit a conceptual index job

Use the SubmitJobAsync() method to submit a job after creating the index. Pass the Artifact IDs for the workspace and index, and a job type to this method. For a list of job types, see Classes and enumerations.

Copy 
public async Task SubmitJobAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1054087;
  
        try
        {
            await indexService.SubmitJobAsync(workspaceId, existingIndexArtifactID, AnalyticsIndexJobType.FullPopulation);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "You don't have permission to submit the index. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Failed to submit the index. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

Retrieve the status of a job

Use the GetStatusAsync() method to retrieve the status of an indexing job. Pass the Artifact IDs of the workspace and the index to this method. It returns an AnalyticsIndexStatus object.

Copy 
public async Task GetStatusAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1049072;
  
        try
        {
            AnalyticsIndexStatus status = await indexService.GetStatusAsync(workspaceId, existingIndexArtifactID);
            Debug.WriteLine("Status of the index is " + status.StatusText);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "You don't have permission to get the index states. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (NotFoundException ex)
        {
            _logger.LogError(ex,
            "Couldn't get the index status because index could not be found. WorkspaceID: {workspaceID} - IndexID: {IndexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Failed to get the index status. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

Cancel a job

Use the CancelJobAsync() method to cancel an indexing job. Pass the Artifact IDs of the workspace and the index to this method.

Copy 
public async Task CancelJobAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1054087;
  
        try
        {
            await indexService.CancelJobAsync(workspaceId, existingIndexArtifactID);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "Cancelling job failed because of invalid permission. WorkspaceID {workspaceID} - IndexID {indexID} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (NotFoundException ex)
        {
            _logger.LogError(ex,
            "Cancelling Job failed because index could not be found. WorkspaceID {workspaceID} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Cancelling Job failed. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}