Analytics Conceptual Index (REST)

A conceptual index is a type of index used by Relativity Analytics. It uses Latent Semantic Indexing (LSI) to discover concepts between documents. This indexing process is based solely on term co-occurrence. The language, concepts, and relationships are defined entirely by the contents of your documents and learned by the index. For general information, see Analytics indexes and on the RelativityOne Documentation site.

The Analytics Conceptual Index service supports programmatically managing conceptual indexes in Relativity Analytics. It includes the following features:

  • Supports CRUD operations on Analytics indexes.
  • Provides helper endpoints that simplify working with index jobs. You can use these endpoints to submit and cancel jobs, and to check on their statuses. These jobs can populate new data, build new indexes, and activate indexes for use.

As a sample use case, you might use the Conceptual Index service in an application to programmatically operate on a conceptual index.

You can also use the Conceptual Index service through .NET. For more information, see Analytics Conceptual Index (.NET).

This page contains the following information:

Guidelines for using the Conceptual Index service

Review the following guidelines for working with the Conceptual Index service. For additional information, see Analytics Conceptual Index (.NET).

URLs

The URLs for REST endpoints contain path parameters that you need to set before making a call:

  • Set the {versionNumber} placeholder to the version of the REST API that you want to use, using the format lowercase v and the version number, for example v1 or v2
  • Set other path parameters in the URLs to the Artifact ID of a given entity, for example setting {workspaceID} to the Artifact ID of a workspace.

    • Note: To indicate the admin-level context, set the {workspaceID} path parameter to -1.

For example, you can use the following URL to retrieve a conceptual index:

<host>/Relativity.REST/API/conceptual-analytics/{versionNumber}/workspaces/{workspaceID}/indexes/{indexID}

Set the path parameters as follows:

  • {versionNumber} to the version of the service, such as v1.
  • {workspaceID} to the Artifact ID of the workspace that contains the index.
  • {indexID} to the Artifact ID of a specific index.

Postman sample file

You can use the Postman sample file to become familiar with making calls to endpoints on the Agent Manager service. To download the sample file, click Conceptual Index Postman file.

To get started with Postman, complete these steps:

  1. Obtain access to a Relativity environment. You need a username and password to make calls to your environment.
  2. Install Postman.
  3. Import the Postman sample file for the service. For more information, see Working with data files on the Postman web site.
  4. Select an endpoint. Update the URL with the domain for your Relativity environment and any other variables.
  5. In the Authorization tab, set the Type to Basic Auth, enter your Relativity credentials, and click Update Request.
  6. See the following sections on this page for more information on setting required fields for a request.
  7. Click Send to make a request.

Create a conceptual index

To create an index, send a POST request with a URL in the following format:

<host>/Relativity.REST/API/conceptual-analytics/{versionNumber}/workspaces/{workspaceID}/indexes

The request payload must include a valid JSON representation of the index object.

The following properties are required:

  • AnalyticsServer - the Analytics server used to build the index.
  • Name - the user-friendly name of the index.
  • IndexTypeIdentifier - the type of index, such as Conceptual (LSI-based) or Classification (SVM-based) index.
  • Order - an integer representing the position of the index in drop-down menus. The default value is 0.
  • SearchableSet - the saved search to populate the searchable set.

The following properties are optional:

  • ConceptStopWords - words to suppress from the index. You can add or remove stop words from the list. Separate each word with a hard return. The default values are derived from the [EDDSDBO].[ContentAnalystDefaultNoiseWords] table.
  • ContinueIndexStepsToCompletion - a Boolean value indicating whether to automatically complete all steps necessary to activate an Analytics index after starting a step.
  • Dimensions - the number of dimensions of the concept space in which documents are mapped when the index is built. More dimensions increase the conceptual values applied to documents and refine the relationships between documents.
  • EmailNotificationRecipients - the list of email recipients notified during index population and build.
  • EnableEmailHeaderFilter - removes common header fields (such as To, From, and Date) and reply-indicator lines. It doesn't remove content from the Subject line. Use this filter to ensure that the headers in the concept space don't overshadow the authored content. The default value is false.
  • OptimizeTrainingSet - a Boolean value indicating whether to select only conceptually relevant documents from the training set saved search. To improve index quality, this property automatically excludes documents that are very large, very small, or contain many numbers when this property is set to true. The default value is false.
  • RemoveDocumentsThatErroredDuringPopulation - removes documents from being populated when they have errored in a previous population.
  • RemoveEngSignaturesAndFooters - a Boolean value indicating whether to remove signatures and footers in English language emails from the text stored in the Analytics index. By default, this property is set to true for new indexes and false for existing ones. Setting this property to true enables the email header filter, disables the Go Words and OCR filters, and removes documents greater than 30 MB from the searchable set. The default value is false.
  • RepeatedContentFilters - repeated content filters associated with the index.
  • TrainingSet - the saved search used to populate the training set.

    • Note: For both the TrainingSet and SearchableSet fields, the Artifact ID for the SavedSearchRef object can be set to 0 to use the built-in search for all documents in a workspace, or -1 for the default training set and default searchable. Setting this value to 0 can negatively impact on system performance.

{
   "index":{
      "AnalyticsServer":{
         "ArtifactID":1018616
      },
      "ConceptStopWords":"a able about above according accordingly across actually after afterwards again against ain't all allow allows almost alone along already also although always am among amongst an and another any anybody anyhow anyone anything anyway anyways anywhere apart appear appreciate appropriate apr are aren't around as a's aside ask asking associated at aug available away awfully b back bcc be became because become becomes becoming been before beforehand behind being believe below beside besides best better",
      "Dimensions":100,
      "EmailNotificationRecipients":[
 
      ],
      "Name":"Jane Doe Case Analytics Index",
      "OptimizeTrainingSet":true,
      "Order":1,
      "RemoveEnglishSignaturesAndFooters":true,
      "EnableEmailHeaderFilter":true,
      "RepeatedContentFilters":[
 
      ],
      "SearchableSet":{
         "ArtifactID":1038051
      },
      "TrainingSet":{
         "ArtifactID":1038051
      },
      "Active":true
   }
}

When the index is successfully created, the response returns the status code of 200.

The following exceptions may occur:

  • ValidationException – An exception representing the case where an invalid or malformed request is made. For example, this exception occurs when an invalid value is specified for an index property.
  • PermissionException – An exception representing the case where the user doesn't have permission to perform an attempted action. For example, this exception occurs when a user doesn't have access to a workspace, a searchable or training set, or to perform index operations.

Retrieve a conceptual index

To read an index, send a GET request with a URL in the following format:

<host>/Relativity.REST/API/conceptual-analytics/{versionNumber}/workspaces/{workspaceID}/indexes/{index artifact ID}

The body of the request is empty.

The response returns an index object with the following properties:

  • ArtifactID - the Artifact ID of the index.
  • Guids - the GUIDs for this index.
  • AnalyticsServer - the Analytics server used to build the index.
  • ConceptStopWords - words to suppress from the index. You can add or remove stop words from the list. Separate each word with a hard return. The default values are derived from the [EDDSDBO].[ContentAnalystDefaultNoiseWords] table.
  • Dimensions - the number of dimensions of the concept space in which documents are mapped when the index is built. More dimensions increase the conceptual values applied to documents and refine the relationships between documents.
  • EmailNotificationRecipients - the list of email recipients notified during index population and build.
  • Name - the user-friendly name of the index.
  • OptimizeTrainingSet - a Boolean value indicating whether to select only conceptually relevant documents from the training set saved search. To improve index quality, this property automatically excludes documents that are very large, very small, or contain many numbers when this property is set to true. The default value is false.
  • Order - an integer representing the position of the index in drop-down menus. The default value is 0.
  • RemoveEngSignaturesAndFooters - a Boolean value indicating whether to remove signatures and footers in English language emails from the text stored in the Analytics index. By default, this property is set to true for new indexes and false for existing ones. Setting this property to true enables the email header filter, disables the Go Words and OCR filters, and removes documents greater than 30 MB from the searchable set. The default value is false.
  • EnableEmailHeaderFilter - removes common header fields (such as To, From, and Date) and reply-indicator lines. It doesn't remove content from the Subject line. Use this filter to ensure that the headers in the concept space don't overshadow the authored content. The default value is false.
  • RepeatedContentFilters - repeated content filters associated with the index.
  • SearchableSet - the saved search to populate the searchable set. This value is a SavedSearchRef object.
  • TrainingSet - the saved search used to populate the training set. This value is a SavedSearchRef object.
  • Active - indicates whether the search provider associated with the index has been activated.
  • RemoveDocumentsThatErroredDuringPopulation - removes documents from being populated when they have errored in a previous population.
  • ContinueIndexStepsToCompletion - a Boolean value indicating whether to automatically complete all steps necessary to activate an Analytics index after starting a step.
  • LastErrorMessage - the last error message to occur on the index.
  • IndexLastUsedOn - the time when this index was last used.

This JSON sample represents an index object:

{
  "ArtifactID": 1118331,
  "Guids": [],
  "AnalyticsServer": {
    "ArtifactID": 1018616
  },
  "ConceptStopWords": "a able about above according accordingly across actually after afterwards again against ain't all allow allows almost alone along already also although always am among amongst an and another any anybody anyhow anyone anything anyway anyways anywhere apart appear appreciate appropriate apr are aren't around as a's aside ask asking associated at aug available away awfully b back bcc be became because become becomes ",
  "Dimensions": 100,
  "EmailNotificationRecipients": [],
  "Name": "John Doe Case Analytics Index",
 
  "OptimizeTrainingSet": true,
  "Order": 1,
  "RemoveEnglishSignaturesAndFooters": true,
  "EnableEmailHeaderFilter": true,
  "RepeatedContentFilters": [],
  "SearchableSet": {
    "ArtifactID": 1038051
  },
  "TrainingSet": {
    "ArtifactID": 1038051
  },
  "Active":true,
  "RemoveDocumentsThatErroredDuringPopulation": true,
  "ContinueIndexStepsToCompletion": true,
  "LastErrorMessage": "",
  "IndexLastUsedOn": "2017-05-09T19:22:17"
}

The following exceptions may occur:

  • NotFoundException – An exception representing the case where an item isn't available or doesn't exist. For example, this exception occurs when reading or deleting an index that doesn't exist.
  • PermissionException – An exception representing the case where the user doesn't have permission to perform an attempted action. For example, this exception occurs when a user doesn't have access to a workspace, a searchable or training set, or to perform index operations.

Update a conceptual index

To update an index, send a PUT request with a URL in the following format:

<host>/Relativity.REST/API/conceptual-analytics/{versionNumber}/workspaces/{workspaceID}/indexes/{index artifact ID}

The request for an update operation contains the same fields as those for a create request. See the field descriptions for the request in Create a conceptual index.

{
  "index": {
     "AnalyticsServer": {
     "ArtifactID": 1018616
    },
    "ConceptStopWords": "a able about above according accordingly across actually after afterwards again against ain't all allow allows almost alone along already also although always am among amongst an and another any anybody anyhow anyone anything anyway anyways anywhere apart appear appreciate appropriate apr are aren't around as a's aside ask asking associated at aug available away awfully b back bcc be became because become becomes ",
    "Dimensions": 120,
    "EmailNotificationRecipients": [],
    "Name": "Jane Doe Case Analytics Index",
    "OptimizeTrainingSet": true,
    "Order": 2,
    "RemoveEnglishSignaturesAndFooters": true,
    "EnableEmailHeaderFilter": true,
    "RepeatedContentFilters": [],
    "SearchableSet": {
      "ArtifactID": 1038051
    },
    "TrainingSet": {
      "ArtifactID": 1038051
    },
    "Active": false
  }
}

When the index is successfully updated, the response returns the status code of 201.

The following exceptions may occur:

  • ValidationException – An exception representing the case where an invalid or malformed request is made. For example, this exception occurs when an invalid value is specified for an index property.
  • PermissionException – An exception representing the case where the user doesn't have permission to perform an attempted action. For example, this exception occurs when a user doesn't have access to a workspace, a searchable or training set, or to perform index operations.

Delete a conceptual index

To delete an index, send a DELETE request with a URL in the following format:

<host>/Relativity.REST/API/conceptual-analytics/{versionNumber}/workspaces/{workspaceID}/indexes/{indexID}

The body of the request is empty.

When the index is successfully deleted, the response returns the status code of 200.

The following exceptions may occur:

  • DeletionDependencyException - An exception representing the case where an object deletion fails because it's referenced by dependencies. For example, this exception occurs when attempting to delete an index that's being used.
  • NotFoundException – An exception representing the case where an item isn't available or doesn't exist. For example, this exception occurs when reading or deleting an index that doesn't exist.
  • PermissionException – An exception representing the case where the user doesn't have permission to perform an attempted action. For example, this exception occurs when a user doesn't have access to a workspace, a searchable or training set, or to perform index operations.

Helper endpoints for conceptual index jobs

Use the following endpoints for managing jobs related to conceptual indexes.

Submit a conceptual index job

To submit an index job, send a POST request with a URL in the following format:

<host>/Relativity.REST/API/conceptual-analytics/{versionNumber}/workspaces/{workspaceID}/indexes/{indexID}/job

The request payload must include a valid JSON representations of the index job type. Valid values include:

  • FullPopulation - runs a full index population.
  • IncrementalPopulation - runs an incremental population.
  • RetryErrors - retries errors that occurred during population.
  • RemoveDocumentsInError - removes documents in error.
  • BuildIndex - runs a full index build.
  • Activate - activates the index for querying.
  • Deactivate - disables queries on the index.
{
    "jobType": "FullPopulation"
}

When the job is successfully submitted, the response returns the status code of 200.

The following exceptions may occur:

  • NotFoundException – An exception representing the case where an item isn't available or doesn't exist. For example, this exception occurs when reading or deleting an index that doesn't exist.
  • PermissionException – An exception representing the case where the user doesn't have permission to perform an attempted action. For example, this exception occurs when a user doesn't have access to a workspace, a searchable or training set, or to perform index operations.
  • ValidationException – An exception representing the case where an invalid or malformed request is made. For example, this exception occurs when an invalid value is specified for an index property.

Retrieve the status of a job

To retrieve the status of an index, send a GET request with a URL in the following format:

<host>/Relativity.REST/API/conceptual-analytics/{versionNumber}/workspaces/{workspaceID}/indexes/{indexID}/status

The body of the request is empty.

The response returns an index status object with the following properties:

  • Active - indicates whether the index is active and can be queried.
  • ConsoleButtons - a list of console buttons and links with their current states. The states can be used to determine which job steps can be run for the index. See AnalyticsIndexJobType enumeration in Fundamentals for the Conceptual Index API.
  • CurrentStep - the current job step when a multi-step job is in progress.
  • DocLevelErrorsExist - indicates whether documents in the training set are in error.
  • LastErrorMessage - the last error that occurred on this index.
  • OverallTimeElapsed - the total time of the current job when one exists.
  • PercentageComplete - the completed percentage of the current job step when one exists.
  • PercentDenominator - the denominator value used to generate the percentage of the current job step when one exists.
  • PercentNumerator - the numerator value used to generate the percentage of the current job step when one exists.
  • SearchableSetCount - the count of items in the searchable set when an index is new or active in its final state.
  • SecondaryPercentageComplete - the completed percentage of a stage of a job step when one exists.
  • SecondaryStatusText - additional status text.
  • StatusText - the main status text of the index.
  • StepTimeElapsed - the elapsed time of the current job step when one exists.
  • TotalSteps - the total number of job steps when a multi-step job is in progress.
  • TrainingSetCount - the count of items in the training set when the index is new or active in its final state.

This JSON sample represents an index status object:

{
  "Active": true,
  "ConsoleButtons": [
    {
      "DisplayText": "Populate Index: Full",
      "Identifier": "populateIndexFull",
      "Class": "consoleButtonEnabled",
      "HelpText": "",
      "Enabled": true
    },
    {
      "DisplayText": "Cancel Full Population",
      "Identifier": "cancelPopulateIndexFull",
      "Class": "consoleButtonHide",
      "HelpText": "Cannot cancel full population if index is not populating.",
      "Enabled": false
    },
    {
      "DisplayText": "Populate Index: Incremental",
      "Identifier": "populateIndexIncremental",
      "Class": "consoleButtonEnabled",
      "HelpText": "",
      "Enabled": true
    },
    {
      "DisplayText": "Cancel Incremental Population",
      "Identifier": "cancelPopulateIndexIncremental",
      "Class": "consoleButtonHide",
      "HelpText": "Cannot cancel incremental population if index is not populating.",
      "Enabled": false
    },
    {
      "DisplayText": "Retry Errors",
      "Identifier": "retryErrors",
      "Class": "consoleButtonDisabled",
      "HelpText": "There are no errors to retry.",
      "Enabled": false
    },
    {
      "DisplayText": "Cancel Retry Errors",
      "Identifier": "cancelRetryErrors",
      "Class": "consoleButtonHide",
      "HelpText": "Cannot cancel retry errors if index is not populating.",
      "Enabled": false
    },
    {
      "DisplayText": "Remove Documents in Error",
      "Identifier": "removeDocumentsInError",
      "Class": "consoleButtonDisabled",
      "HelpText": "There are no errored documents to remove.",
      "Enabled": false
    },
    {
      "DisplayText": "Build Index",
      "Identifier": "buildIndex",
      "Class": "consoleButtonDisabled",
      "HelpText": "Index cannot be built while the index is active.",
      "Enabled": false
    },
    {
      "DisplayText": "Activate",
      "Identifier": "activate",
      "Class": "consoleButtonHide",
      "HelpText": "Index cannot be activated unless it is not active.",
      "Enabled": false
    },
    {
      "DisplayText": "Deactivate",
      "Identifier": "deactivate",
      "Class": "consoleButtonEnabled",
      "HelpText": "",
      "Enabled": true
    },
    {
      "DisplayText": "Show Document Errors",
      "Identifier": "showDocumentErrors",
      "Class": "consoleLinkDisabled",
      "HelpText": "There are no errors to show.",
      "Enabled": false
    },
    {
      "DisplayText": "Show Population Statistics",
      "Identifier": "showPopulationStatistics",
      "Class": "consoleLinkEnabled",
      "HelpText": "",
      "Enabled": true
    },
    {
      "DisplayText": "Show Index Statistics",
      "Identifier": "showIndexStatistics",
      "Class": "consoleLinkEnabled",
      "HelpText": "",
      "Enabled": true
    }
  ],
  "DocLevelErrorsExist": false,
  "SearchableSetCount": 73966,
  "StatusText": "Active",
  "TrainingSetCount": 63403
}

The following exceptions may occur:

  • NotFoundException – An exception representing the case where an item isn't available or doesn't exist. For example, this exception occurs when reading or deleting an index that doesn't exist.
  • PermissionException – An exception representing the case where the user doesn't have permission to perform an attempted action. For example, this exception occurs when a user doesn't have access to a workspace, a searchable or training set, or to perform index operations.

Cancel a job

To cancel a running index job, send a DELETE request with a URL in the following format:

<host>/Relativity.REST/API/conceptual-analytics/{versionNumber}/workspaces/{workspaceID}/indexes/{indexID}/job

The body of the request is empty.

When the job is successfully canceled, the response returns the status code of 200.

The following exceptions may occur:

  • DeletionDependencyException - An exception representing the case where an object deletion fails because it's referenced by dependencies. For example, this exception occurs when attempting to delete an index that's being used.
  • NotFoundException – An exception representing the case where an item isn't available or doesn't exist. For example, this exception occurs when reading or deleting an index that doesn't exist.
  • PermissionException – An exception representing the case where the user doesn't have permission to perform an attempted action. For example, this exception occurs when a user doesn't have access to a workspace, a searchable or training set, or to perform index operations.