Analytics Conceptual Index (.NET)

The Conceptual Index API includes the following methods available on the IConceptualIndexService interface in the Relativity.Analytics.Conceptual.<VersionNumber>.Services namespace.

Note: The <VersionNumber> variable in the namespace indicates the version number of the API. The version number uses the format uppercase V and an integer version number, such as V1 or V2 in .NET.

• CreateAsync() method - adds a new index to Relativity. See Create a conceptual index.
• ReadAsync() method - retrieves an index. See Retrieve a conceptual index.
• UpdateAsync() method - modifies an index. See Update a conceptual index.
• DeleteAsync() method - removes an index from Relativity. See Delete a conceptual index.
• SubmitJobAsync() method - submits a job after the index is created. See Submit a conceptual index job.
• GetStatusAsync() method - retrieves the status of an indexing job. See Retrieve the status of a job.
• CancelJobAsync() method - cancels an indexing job. See Cancel a job.

The Conceptual Index API uses the following classes and enumerations:

• AnalyticsIndexJobType enumeration - defines valid index job types as follows:
  • FullPopulation (0) - runs a full index population.
  • IncrementalPopulation (1) - runs an incremental population. This process adds or removes documents from the training and searchable sets. If active, the index remains active, and you can query it during the incremental population. You can't query new documents until a build followed by an activate is completed.
  • RetryErrors (2) - runs a full population if a previous population attempt failed.
  • RemoveDocumentsInError (3) - removes documents in error.
  • BuildIndex (4) - runs a full index build. If any training documents are added, this step takes longer because the engine rebuilds the concept space. If only searchable documents are added, the engine starts this build at a later step, updating searchable items, because it only maps the new searchable documents into the pre-existing concept space.
  • Activate (5) - activates the index for querying.
  • Deactivate(6) - disables queries on the index. The index must be deactivated to perform a build.
  • TagDocuments(7) - tags documents that are a part of the training and searchable sets for an index
• AnalyticsIndexStatus class - represents the aggregate state of an Analytics index. The GetStatusAsync() method returns a status object. Its properties include:
  • Active - indicates whether the index is active and can be queried.
  • ConsoleButtons - a list of console buttons and links with their current states. The states can be used to determine which job steps can be run for the index. See AnalyticsIndexJobType enumeration in Classes and enumerations.
  • CurrentStep - the current job step when a multi-step job is in progress.
  • DocLevelErrorsExist - indicates whether documents in the training set are in error.
  • LastErrorMessage - the last error that occurred on the index
  • OverallTimeElapsed - the total time of the current job when one exists.
  • PercentageComplete - the completed percentage of the current job step when one exists.
  • PercentDenominator - the denominator value used to generate the percentage of the current job step when one exists.
  • PercentNumerator - the numerator value used to generate the percentage of the current job step when one exists.
  • SearchableSetCount - the count of items in the searchable set when an index is new or active in its final state.
  • SecondaryPercentageComplete - the completed percentage of a stage of a job step when one exists.
  • SecondaryStatusText - additional status text.
  • StatusText - the main status text of the index. For more information, see Index states and job validation.
  • StepTimeElapsed - the elapsed time of the current job step when one exists.
  • TotalSteps - the total number of job steps when a multi-step job is in progress.
  • TrainingSetCount - the count of items in the training set when the index is new or active in its final state.
• ArtifactRef class - represents a Relativity artifact referenced by an index property, such as repeated content filters, searchable sets, and training sets. Its properties include:
  • ArtifactID - represents a unique identifier, called an Artifact ID, for an object.
  • Guids - represents the GUIDs used to identify an object.
• ConceptualIndex class - represents the index. Its properties include:
  • Active - indicates whether the index is active
  • AnalyticsServer - the Analytics server used to build the index. This value is a ArtifactRef object.
  • ArtifactID - the Artifact ID of the index.
  • ConceptStopWords - words to suppress from the index. You can add or remove stop words from the list. Separate each word with a hard return. The default values are derived from the [EDDSDBO].[ContentAnalystDefaultNoiseWords] table.
  • ContinueIndexStepsToCompletion - a Boolean value indicating whether to automatically complete all steps necessary to activate an Analytics index after starting a step. For example, if you set the property to true, and submit a populate job, the index is automatically built and activated. If you don't set it, only the population occurs, and building and activating must be done manually.
  • Dimensions - the number of dimensions of the concept space in which documents are mapped when the index is built. More dimensions increase the conceptual values applied to documents and refine the relationships between documents.
  • EmailNotificationRecipients - the list of email recipients notified during index population and build.
  • EnableEmailHeaderFilter - removes common header fields (such as To, From, and Date) and reply-indicator lines. It doesn't remove content from the Subject line. Use this filter to ensure that the headers in the concept space don't overshadow the authored content. The default value is false.
  • Guids - the GUIDs for this artifact.
  • IndexLastUsedOn - the time when the index was last used.
  • InUseByCategorizationSet - indicates whether the index is in use by an LSI categorization set.
  • LastErrorMessage - the last error message to occur on the index.
  • Name - the user-friendly name of the index.
  • OptimizeTrainingSet - a Boolean value indicating whether to select only conceptually relevant documents from the training set saved search. To improve index quality, this property automatically excludes documents that are very large, very small, or contain many numbers when this property is set to true. The default value is false.
  • Order - an integer representing the position of the index in drop-down menus. The default value is 0.
  • RemoveDocumentErrors - removes documents from being populated when they have errored in a previous population.
  • RemoveEngSignaturesAndFooters - a Boolean value indicating whether to remove signatures and footers in English language emails from the text stored in the index. By default, this property is set to true for new indexes and false for existing ones. Setting this property to true enables the email header filter, disables the Go Words and OCR filters, and removes documents greater than 30 MB from the searchable set. The default value is false.
  • RepeatedContentFilters - the repeated content filters associated with the index.
  • TrainingSet - the saved search used to populate the training set. This value is a ArtifactRef object.
  • SearchableSet - the saved search to populate the searchable set. This value is a ArtifactRef object.

Use the following steps to build a new index:

1. Populate the index by adding all documents from the training set and searchable set to the ready-to-index list.
2. Build the index by building the concept space.
3. Activate the index by make it available to users. Add the index to the search drop-down on the Documents tab and to the right-click menu in the viewer.

1. Populate the index incrementally by checking the data sources for the training and the searchable sets. If either search includes new documents, Analytics adds them to the index. At this stage, user can query against the index.
2. Deactivate an index by disabling the index. It can't be queried against or used for any operations with Analytics. You must deactivate the index to perform a build.
3. Build the index by building the concept space. If any training documents are added, this step takes longer because the engine rebuilds the concept space. If only searchable documents are added, the engine starts this build at a later step, updating searchable items, because it only maps the new searchable documents into the pre-existing concept space.
4. Activate the index to make it available to users.

Some states overlap with each other, and others have sub-states depending on additional index metadata. For example, many states include an error state, such as STATE – 1 or more documents in error status, when state is populating, building, and so on.

For example, an error scenario occurs when items linked to the index (saved searches, Analytics profile) can't be accessed. Regardless of any additional error state, the validator treats it as an error. It subsequently only lets the two error resolution-type jobs run: RetryErrors or RemoveDocumentsInError. The message state values are returned as the StatusText property.

The states of an index are as follows:

• Activating
• Activation Recommended
• Active
• Attempting to cancel population
• Build Recommended
• Building
• Configuring Analytics Engine
• Constructing Population Table
• Content Analyst Server is unavailable
• Deactivating
• Disable queries to continue
• Disabling Queries
• Enabling Queries
• Failed to activate index
• Failed to deactivate index
• Finalizing build
• Index Build Failed
• Indexing Job in Queue
• New
• Not enough conceptual content to build
• Please check your training set
• Populating
• Populating Documents
• Population Canceled by User
• Population Failed
• Preparing to build
• Removing Documents in Error
• Waiting

public async Task CreateAsyncSampleCode()
{
    ServiceFactorySettings settings = new ServiceFactorySettings(
      new Uri("net.pipe://localhost/relativity.services/"),
      new Uri("http://localhost/relativity.rest/api"),
      new IntegratedAuthCredentials());
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
 
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int searchableSetSearchArtifactID = 1038051;
        int trainingSetSearchArtifactID = 1038051;
        int analyticsServerArtifactID = 1016891;
  
        ConceptualIndex indexToCreate = new ConceptualIndex
        {
            Name = "my new analytics index",
            Order = 1,
            SearchableSet = new ArtifactRef { ArtifactID = searchableSetSearchArtifactID },
            TrainingSet = new ArtifactRef { ArtifactID = trainingSetSearchArtifactID },
            AnalyticsServer = new ArtifactRef { ArtifactID = analyticsServerArtifactID },
            OptimizeTrainingSet = true,
            RemoveDocumentsThatErroredDuringPopulation = true,
            ContinueIndexStepsToCompletion = true,
            ConceptStopWords = "a\r\nable\r\nabout\r\nabove\r\naccording\r\n",
            Dimensions = 100,
            RemoveEnglishSignaturesAndFooters = true,
            EnableEmailHeaderFilter = true
        };
  
        try
        {
            int conceptualIndexID = await indexService.CreateAsync(workspaceId, indexToCreate);
            Debug.WriteLine("ID of the newly created index is: " + conceptualIndexID);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "Failed to create index because of invalid permission. workspaceID: {workspaceId} - IndexToCreate: {@index}",
            workspaceId, indexToCreate);
            throw;
        }
        catch (ValidationException ex)
        {
            _logger.LogError(ex,
            "The index you were trying to create was invalid. workspaceID: {workspaceId} - IndexToCreate: {@index}",
            workspaceId, indexToCreate);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Creating index failed. workspaceID: {workspaceId} - IndexToCreate: {@index}",
            workspaceId, indexToCreate);
            throw;
        }
    }
}

public async Task ReadSingleAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1049079;
  
        try
        {
            ConceptualIndex conceptualIndex = await indexService.ReadAsync(workspaceId, existingIndexArtifactID);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "You don't have permission to retrieve the index. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (NotFoundException ex)
        {
            _logger.LogError(ex,
            "Failed to retrieve the index because index could not be found. WorkspaceID: {workspaceID} - IndexID: {IndexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Failed to retrieve the index. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

public async Task UpdateAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1049079;
  
        ConceptualIndex conceptualIndex= null;
  
        try
        {
            try
            {
                conceptualIndex = await indexService.ReadAsync(workspaceId, existingIndexArtifactID);
            }
            catch (PermissionException ex)
            {
                _logger.LogError(ex,
                "You don't have permission to read the index. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
                workspaceId, existingIndexArtifactID, userName);
                throw;
            }
            catch (NotFoundException ex)
            {
                _logger.LogError(ex,
                "Failed to read the index because index could not be found. WorkspaceID: {workspaceID} - IndexID: {IndexId}",
                workspaceId, existingIndexArtifactID);
                throw;
            }
            conceptualIndex.Name = "My Updated Analytics Index";
            await indexService.UpdateAsync(workspaceId, existingIndexArtifactID, conceptualIndex);
  
            conceptualIndex = await indexService.ReadAsync(workspaceId, existingIndexArtifactID);
  
            Debug.WriteLine("The updated name of the retrieved index is " + conceptualIndex.Name);
  
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "You don't have permission to update the index. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (ValidationException ex)
        {
            _logger.LogError(ex,
            "The index you were trying to update has invalid data. workspaceID: {workspaceId} - IndexToUpdate: {@index}",
            workspaceId, conceptualIndex);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Failed to update the index. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

public async Task DeleteAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1054085;
  
        try
        {
            await indexService.DeleteAsync(workspaceId, existingIndexArtifactID);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "Deleting Index failed because of invalid permission. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (NotFoundException ex)
        {
            _logger.LogError(ex,
            "Deleting Index failed because index could not be found. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (DeletionDependencyException ex)
        {
            _logger.LogError(ex,
            "Deleting Index failed because there is a dependecy on it. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Deleting index failed. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

public async Task SubmitJobAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1054087;
  
        try
        {
            await indexService.SubmitJobAsync(workspaceId, existingIndexArtifactID, AnalyticsIndexJobType.FullPopulation);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "You don't have permission to submit the index. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Failed to submit the index. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

public async Task GetStatusAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1049072;
  
        try
        {
            AnalyticsIndexStatus status = await indexService.GetStatusAsync(workspaceId, existingIndexArtifactID);
            Debug.WriteLine("Status of the index is " + status.StatusText);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "You don't have permission to get the index states. WorkspaceID {workspaceId} - IndexID {IndexId} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (NotFoundException ex)
        {
            _logger.LogError(ex,
            "Couldn't get the index status because index could not be found. WorkspaceID: {workspaceID} - IndexID: {IndexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Failed to get the index status. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}

public async Task CancelJobAsyncSampleCode()
{
    string userName = "user@test.com";
    string password = "Test1234!";
  
    string relativityServicesUri = "http://localhost/relativity.services";
    string relativityRestUri = "http://localhost/relativity.rest/api";
  
    var usernamePasswordCredentials = new UsernamePasswordCredentials(userName, password);
    ServiceFactorySettings settings = new ServiceFactorySettings(new Uri(relativityServicesUri), new Uri(relativityRestUri), usernamePasswordCredentials);
    ServiceFactory serviceFactory = new ServiceFactory(settings);
  
    using (IConceptualIndexService indexService = serviceFactory.CreateProxy<IConceptualIndexService>())
    {
        int workspaceId = 1023289;
        int existingIndexArtifactID = 1054087;
  
        try
        {
            await indexService.CancelJobAsync(workspaceId, existingIndexArtifactID);
        }
        catch (PermissionException ex)
        {
            _logger.LogError(ex,
            "Cancelling job failed because of invalid permission. WorkspaceID {workspaceID} - IndexID {indexID} - username: {username}",
            workspaceId, existingIndexArtifactID, userName);
            throw;
        }
        catch (NotFoundException ex)
        {
            _logger.LogError(ex,
            "Cancelling Job failed because index could not be found. WorkspaceID {workspaceID} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
            "Cancelling Job failed. WorkspaceID {workspaceId} - IndexID {indexId}",
            workspaceId, existingIndexArtifactID);
            throw;
        }
    }
}