Relativity Transfer SDK

The Relativity Transfer SDK is designed for uploading and downloading data to and from RelativityOne fileshares.

The Relativity Transfer SDK includes the following core features:

  • Upload a file or a directory to RelativityOne fileshares.
  • Download a file or a directory from RelativityOne fileshares.
  • Track data transfer progress:
    • Shows succeeded, failed, and skipped items. Including the exact path and the reason for the error.
    • Shows transfer speed and the total amount of files and bytes to be transferred.
  • Cross-platform compatibility, transfer data from Windows, Linux, and MacOS.
  • TransferSDK works solely on HTTPS to transfer data, only a single TCP 443 port is required to be opened.
  • Enhanced performance and transfer speeds.

Version History

Release Notes
  • New features and improvements
    • Added throughput check
    • Made override policy option as a custom implementation instead of boolean
    • Exposed CalculateShaHashes to UploadDirectoryOptions, UploadFileOptions, and UploadListOfItemsOptions
    • Improved error handling
    • Added failure category for exceptions
    • Added possibility to pass WorkspaceID to TransferJobBuilder
    • Improved performance
Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes
  • Breaking changes
    • AzureTransferFullPathClientBuilder and AzureTransferJobClientBuilder now require a workspace context, which can be specified by the workspace's ID or using Staging Explorer privileges.
    • Canceled directory transfers will no longer throw an OperationCancelledException. Instead, a result with a Cancelled state will be returned.
    • If your implementation of IRelativityAuthenticationProvider throws an exception, the SDK will now throw a FailedToAcquireCredentialsException.
    • Support for legacy fileshares has been discontinued since all clients have successfully migrated to the new storage.
  • New Features and Improvements
    • Introduced OvewritePolicy in OptionsBase for implementing policies to handle file skipping and overwriting at the destination.
    • AzureTransferJobClient now supports uploading a list of files to the fileshare.
    • Added the ability to specify the location of journal files in transfer options.
    • Improved handling of I/O errors on Windows for better reliability and troubleshooting.
    • Transfer SDK will now automatically remove journal files older than 28 days to optimize storage.
Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes
  • Introducing new way of storage authentication
  • Added SkipTransferringEmptyDirectories setting for directory-based transfers
  • Enhancements in telemetry area
Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes
  • Progress updates have a new property. StepType strongly defines from which step progress report comes from.
  • TransferSDK throws meaningful exception when it fails to acquire storage path.
  • Set of improvements in observability area.
Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes
  • Fixed an issue that caused OutOfMemoryException when multiple fails or skips occur during a transfer.
  • Set of internal changes that improve TransferSDK observability.
Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes
  • Added JobBasedWorkflow to TransferJobClient, it allows to create client that will require only one path to be provided.
  • Removed vulnerable package Cleary.AsyncExtensions.
Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes

Minor improvements around statistics reporting

Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes
  • Added functionality to download a file or a directory from Relativity fileshare (legacy and ADLS based)
  • Added support for reporting of skipped/failed/succeeded files during synchronizations
  • Fixed numerous security vulnerabilities
  • Fixed a few minor issues
Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes

Fixed possible runtime issue with assembly version

Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes

Added support to download data directly to a drive as well as upload from a drive

Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes
  • Added support for exclusion policy
  • Fixed status translation of a cancelled transfer job
Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes

Improved job registration and exposed retry policy option

Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes

Fixed issue where transfer got stuck

Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Release Notes

Added skip top level directory

Supported Relativity Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Supported RAP Version Range
Lowest Version Highest Version
1.2.13.0 Latest
Release Notes

Initial Version

Supported RAP Version Range
Lowest Version Highest Version
13.2.0.0 Latest
Supported RAP Version Range
Lowest Version Highest Version
1.2.13.0 Latest

Fundamentals for the Relativity Transfer SDK

Review the following guidelines for working with the Relativity Transfer SDK.

  • Using a network share between Samba servers running Linux and Windows may cause issues. This is in relation to how Samba handles case-sensitivity. Specifically, Windows may end up copying files more than once or not at all.
  • There are no limits on transfer file path lengths in SDK. However, all destination and source paths are limited against a set of rules defined in Windows and Linux systems . See, Windows maximum file path limitations and File System Limitations for more information.
  • Use a stable internet connection, Wifi is not recommended.
  • Due to bandwidth sharing, avoid performing large and highly fragmented transfers in parallel.
  • Transfers from network drives or external disks (such as USB drives) may degrade performance or have issues maintaining connection. It is most effective to copy data to and from local storage.
  • Relativity does not store any information about file paths due to privacy concerns. See Transfer progress for more information.
  • When transferring massive data sets, it is recommended to use a local drive where Transfer SDK is executed as the transfer source and destination to avoid additional transfer steps. Using a mapped network drive is not recommended for transfer as it introduces an extra connection between the Transfer SDK and RelativityOne that may cause slowness and connection issues.

Staging Governance compliance

Within each tenant or client domain storage area, the Staging Area consists of the following four folders:

  • ARM
  • StructuredData
  • ProcessingSource
  • TenantVM

You are only able to write to and access data within these designated Staging Area folders. For more information on Staging Area, see the Staging Area topic on the RelativityOne Documentation site.

Current Relativity Transfer SDK clients will not be affected by this change, however, we strongly recommend adjusting your workflows to align with the specified four folders. New Relativity Transfer SDK clients are required to comply with this update.

Example of implementing the Relativity Transfer SDK

Use the following as an example on how to implement the Relativity Transfer SDK:

  1. Using any authentication method supported by RelativityOne, provide the Source Path, Destination Path and your Relativity Instance URL address.

    2. During the transfer you can track the job progress, this includes the percentage progress, total number of transferred GB and files, and predicted time to completion. As well as more detailed information about already transferred files paths.

    Skipped and failed files display with corresponding paths.

    Transfer summary includes the transfer job id, sum of total bytes and files transferred, quantity of empty directories, elapsed time and final status of the job.

Code samples

Use the code samples in the following sections to learn about calling the methods available in the Relativity Transfer SDK.

To use the Transfer SKD sample code and documentation in the Relativity GitHub repository, click here.

These code samples illustrate the following best practices for working with the SDK:

Create instance of SDK client

Use the following code sample to list the necessary namespaces:

Copy 
using System;
using System.Threading.Tasks;
using Relativity.Transfer.SDK.Core.ProgressReporting;
using Relativity.Transfer.SDK.Interfaces;
using Relativity.Transfer.SDK.Interfaces.Paths;
using Relativity.Transfer.SDK.Interfaces.ProgressReporting;

Implement IRelativityAuthenticationProvider to provide the RelativityOne access token. If the access token expires or is invalid, the implementation of this interface will be invoked to obtain a new token for the RelativityOne instance.

Once defined, use fluent builder to create the instance of Transfer SDK client with two different authentication types:

  • WithStagingExplorerContext – connecting in context of admin Workspace.
  • WithWorkspaceContext – connecting in context of passed WorkspaceID.
Copy 
var clientName = "<YourCompanyName>"; // Needed for troubleshooting scenarios, should you have any. No PII is collected. 
var relativityAddress = "<YourTenantAddress>"; // Regular Relativity address, like https://<YourCompaneName>.relativity.one/Relativity 
var authenticationProvider = CreateAuthenticationProvider(); // Provide us with authentication method to Relativity (e.g. OAuth) 
 
var transferClient = TransferClientBuilder.FullPathWorkflow 
    .WithAuthentication(authenticationProvider) 
    .WithClientName(clientName) 
    .WithStagingExplorerContext() // or .WithWorkspaceContext(sampleWorkspaceId)  
    .Build(); 
  • Create a unique client name that can be used for easy recognition.
  • It is suggested that the client name contains the name of the company. For example, Relativity-Transfer-Client.

Transfer progress

Use the following code sample to subscribe to all, none, or some Transfer progress updates. Subscribing is optional, but is the only way to obtain information on which individual items failed or were skipped.

Relativity does not store any information about file paths due to privacy concerns.

Copy 
ITransferProgressHandler progressHandler = TransferProgressHandlerBuilder
        .Instance
        .OnStatistics(StatisticHook.OnStatisticsReceived) // Updates about overall status (% progress, transfer rates, partial statistics)
        .OnSucceededItem(StatisticHook.OnSucceededItemReceived) // Updates on each transferred item
        .OnFailedItem(StatisticHook.OnFailedItemReceived) // Updates on each failed item (and reason for it)
        .OnSkippedItem(StatisticHook.OnSkippedItemReceived) // Updates on each skipped item (and reason for it)
        .OnProgressSteps(StatisticHook.OnProgressStepsReceived) // Updates on each job's progress steps (percentage progress and state)
        .Create();

Transfer options

Use the UploadDirectoryOptions and UploadFileOptions to upload a directory or file to a specified location with custom options.

Skip Top Level Directory

Use SkipTopLevelDirectory to upload the entire contents of a source directory but not create a directory:

  • Values- True or False
  • Default value- False
  • Can be set on a directory transfer

Transfer Retry Policy Definition

Use TransferRetryPolicyDefinition to define transfer retry behavior:

  • Values- No retry, Linear, or Exponential
  • Default value- No retry
  • Can be set on each type of transfer
Copy 
// UploadDirectoryOptions with Linear policy example.
var transferJobOptions = new UploadDirectoryOptions
{            
    TransferRetryPolicyDefinition = TransferRetryPolicyDefinition.LinearPolicy(deltaBackoff: TimeSpan.FromSeconds(1), maxAttempts: 10)
};

Exclusion Policy

Use ExclusionPolicy to determine if a file is or is not excluded from a transfer:

  • Values- define your own exclusion policy by providing your own implementation of interface.
  • Default value- by default the exclusion policy is empty, no files are being excluded.
  • Can be set on a directory transfer

If all files in a source directory are excluded by the exclusion policy, the directory will not be created.

Copy 
// Implement custom exclusion policy.
public class CustomExclusionPolicy: IFileExclusionPolicy
{
    public async Task<bool> ShouldExcludeAsync(IFileReference fileReference)
    {
        // Exclude files containing "TestCase"
        if (fileReference.AbsolutePath.Contains("TestCase"))
        {
            return await Task.FromResult(true);
        }
        
        // Exclude pdf files
        if (fileReference.SystemProperties["Extension"].ToString().EndsWith("pdf"))
        {
            return await Task.FromResult(true);
        }
        return await Task.FromResult(false);
    }
}
// Set ExclusionPolicy on options.
// UploadDirectoryOptions 
var transferJobOptions = new UploadDirectoryOptions
{           
    ExclusionPolicy = new TestCaseFilesExclusionPolicy()
};

Upload a directory

Use the UploadDirectoryAsync() method to upload a directory to a specified location.

Using the method to transfer a whole disk data set may contain hidden files and folders such as, $RECYCLE.BIN. This hidden content may change the quantity of transferred items in the statistics.

Copy 
var transferJobId = Guid.NewGuid();
var source = new DirectoryPath(@"C:\your_source_directory"); 
var destination = new DirectoryPath(@"\\files.<YourCompanyName>.relativity.one\T001\your_destination_directory"); // Destination requires path in Relativity Fileshare UNC notation
var progressHandler = CreateProgressHandler(); // Your implementation of ITransferProgressHandler. Optional, however it's strongly recommended you track and log progress of the transfer, as Relativity does not log PII data, which includes file paths

TransferJobResult result = await transferClient.UploadDirectoryAsync(transferJobId, source, destination, progressHandler, default);

Upload a directory with custom options

Use the UploadDirectoryOptions to upload a directory to a specified location with custom options.

Skip Top Level Directory

Use the SkipTopLevelDirectory option to upload the entire contents of a source directory but does not create a directory.

Copy 
private static async Task UploadDirectoryAsync(ITransferFullPathClient transferClient)
{
    DirectoryPath sourceDirectory = GetSource();
    DirectoryPath destinationDirectory = GetDestination();

    ITransferProgressHandler progressHandler = GetProgressHandler();

    var transferJobOptions = new UploadDirectoryOptions
    {
        SkipTopLevelDirectory = true
    };

    TransferJobResult result = await transferClient
                .UploadDirectoryAsync(_TRANSFER_JOB_ID, sourceDirectory, destinationDirectory,transferJobOptions, progressHandler, default)
                .ConfigureAwait(false);
}

Upload a file

Use the UploadFileAsync() method to upload a file to a specified location.

Copy 
var transferJobId = Guid.NewGuid();
var source = new FilePath(@"C:\your_source_directory\file.txt"); 
var destination = new DirectoryPath(@"\\files.<YourCompanyName>.relativity.one\T001\your_destination_directory"); // Destination requires path in Relativity Fileshare UNC notation
var progressHandler = CreateProgressHandler(); // Your implementation of ITransferProgressHandler. Optional, however it's strongly recommended you track and log progress of the transfer, as Relativity does not log PII data, which includes file paths

TransferJobResult result = await transferClient.UploadFileAsync(transferJobId, source, destination, progressHandler, default);

Upload a list of items

Use the UploadItemsAsync() method to upload a list of items.

Copy 
var transferJobId = Guid.NewGuid(); 
IAsyncEnumerable<TransferEntity> sources = ...;  // Provide a list of items to transfer, single item consists of source file path and relative transfer destination file path 
var destinationRoot = new DirectoryPath(@"\\files.<YourCompanyName>.relativity.one\T001\your_destination_directory"); // Destination requires path in Relativity Fileshare UNC notation, this is a root directory to which files are transferred
var progressHandler = CreateProgressHandler(); // Your implementation of ITransferProgressHandler. Optional, however it's strongly recommended you track and log progress of the transfer, as Relativity does not log PII data, which includes file paths 
 
TransferJobResult result = await transferClient.UploadItemsAsync(transferJobId, sources, destinationRoot, progressHandler, default);

Download a file

Use the .DownloadFileAsync() method to download a file to a specified location.

Copy 
// Every transfer job in Relativity requires a unique id
Guid _TRANSFER_JOB_ID = Guid.NewGuid();

DownloadDirectoryOptions options = new DownloadDirectoryOptions
{
    TransferRetryPolicyDefinition = TransferRetryPolicyDefinition.NoRetryPolicy()
};

FilePath sourceFile = new FilePathPath(@"\\file.<YourCompanyName>.relativity.one\T001\your_source_directory\your_file.txt"); // Source requires path in Relativity Fileshare UNC notation
DirectoryPath destinationDirectory = new DirectoryPath(@"C:\your_destination_directory");

TransferJobResult result = await transferClient
                .DownloadFileAsync(_TRANSFER_JOB_ID, sourceFile, destinationDirectory, options, progressHandler, default);
                .ConfigureAwait(false);
  • Downloading from legacy fileshare requires a synchronization step with an intermediate storage. During this step, file names will be checked against additional requirements.
  • If file names contain invalid characters, Transfer.SDK will report them as skipped and indicate that the name is invalid. To proceed, rename the files and create a new transfer.
  • The list of invalid characters is specified in Microsoft's documentation.

Download a directory

Use the .DownloadDirectoryAsync() method to download a directory to a specified location.

Copy 
// Every transfer job in Relativity requires a unique id
Guid _TRANSFER_JOB_ID = Guid.NewGuid();

DownloadDirectoryOptions options = new DownloadDirectoryOptions
{
    SkipTopLevelDirectory = true,
    TransferRetryPolicyDefinition = TransferRetryPolicyDefinition.NoRetryPolicy()
};

DirectoryPath sourceDirectory = new DirectoryPath(@"\\files.<YourCompanyName>.relativity.one\T001\your_source_directory"); // Source requires path in Relativity Fileshare UNC notation
DirectoryPath destinationDirectory = new DirectoryPath(@"C:\your_destination_directory");

TransferJobResult result = await transferClient
                .DownloadDirectoryAsync(_TRANSFER_JOB_ID, sourceDirectory, destinationDirectory, options, progressHandler, default);
                .ConfigureAwait(false);

Job based workflow

Job base is a workflow that allows users of Relativity.Transfer.SDK to perform a transfer from an existing job. This is needed when you do not know the destination/source path on Relativity Fileshare.

Job based workflow only supports transferring directories.

Complete the following steps to use the job based workflow:

  1. Declare the workflow when creating the Relativity.Transfer.SDK client
    2. Copy 
    var clientName = <Your_company_name> 
     
    var authenticationProvider = CreateAuthenticationProvider(); // Provide us with authentication method to Relativity (e.g. OAuth) 
     
    // build Transfer SDK client supporting job based workflow   
    var transferClient = TransferClientBuilder   
        .JobBasedWorkflow     
        .WithAuthentication(authenticationProvider)  
        .WithClientName(clientName)   
        .WithStagingExplorerContext() // or .WithWorkspaceContext(sampleWorkspaceId) 
        .Build();
  2. You can use an existing job ID (e.g. from previous transfers), or you can register a new job using TransferJobBuild, and providing the job identifier and destination path.
    3. Copy 
    var jobBuilder = new TransferJobBuilder(authenticationProvider);
    var transferJobId = Guid.NewGuid();
    var destinationPath = new DirectoryPath(@"\\files.<YourCompanyName>.relativity.one\T001\your_destination_directory"); // Destination requires path in Relativity Fileshare UNC notation

    // Register a new job with the specified destination path  
    await jobBuilder
            .NewJob()
            .CreateUploadDirectoryJobAsync(transferJobId, destinationPath)
            .ConfigureAwait(false);
  3. To create a new job based on an existing one, provide the previous job ID and the destination path will be taken from there.
    4. Copy 
    var existingJobId = <Job_Id_From_Previous_Transfer>
    var newJobId = Guid.NewGuid();
    // Register a new job from the existing job with the specified destination path  
    await jobBuilder
            .FromExistingJob(existingJobId)
            .CreateUploadJobAsync(newJobId)  
            .ConfigureAwait(false);
  4. Use the UploadDirectoryAsync() or DownloadDirectoryAsync() method and provide the local path.
    5. Copy 
    var transferJobId = <Your_registered_transfer_job_id> 
    var sourcePath = new DirectoryPath(@"C:\your_source_directory"); 
    var cancellationToken = <Cancellation_Token>

    var result = await transferClient
            .UploadDirectoryAsync(transferJobId, sourcePath, ConsoleStatisticHook.GetProgressHandler(), cancellationToken)
            .ConfigureAwait(false);