Creating & Updating Records via FTP (Clinical Operations)

You can now use a file transfer protocol (FTP) to load object data into your Vault. You can create and update records for a single object with a single CSV file, or use a ZIP file containing multiple CSVs to create and update records for multiple objects at once. Updating objects maintains all relationships and references to the object, such as the audit trail.

In Vault, the update operation is distinct from a delete and create operation. Instead of deleting the existing record, and then replacing it with a new record containing updated data, Vault simply updates the existing object record.

Accessing Your Vault’s FTP Server

See Accessing your Vault’s FTP Server for instructions. Once you’ve connected to your Vault’s FTP server, you can enable this feature and begin loading data.

How to Enable FTP Automation

To enable this feature, you must create a new directory on your FTP server named “Vaultloader” in each user’s directory. For example, to create the “Vaultloader” directory for Clara, a Vault user whose User ID is 5678, navigate to vault1234/u5678 and add “/vaultloader”. Load CSVs and ZIP files into this directory.

Preparing the CSV Input

The format for your CSV input is much the same as it is for Vault Loader. Label columns with the field’s Name value (field_name__v), and enter field values as one object record per row. The list below details requirements for your CSVs:

  • Create one CSV file per object.
  • Use the object field’s Name value (field_name__v, __c, __clin) as the column header.
  • Name your CSV file with the object’s Name value (object_name__v, __c, or __clin). For example, to load records into the Subject object, name the CSV “subject__clin.csv”.
  • Save as a *.csv file.
  • To transfer multiple CSVs, add each CSV to a ZIP file.

If you are updating existing records, rather than creating new ones, you can ensure that Vault updates the existing records by specifying a unique field (such as Name or ID) that Vault can use to match each row in your CSV to a record. Use the field as a column in your CSV, and then append the field name to your CSV filename in the following format:

object_name__v.field_name__v.csv

For example, if Gladys is loading subject enrollment data from an EDC report into her CTMS Vault, she might use the External ID field on the Subject object to match subject records between her report and her target Vault.

Gladys can include an External ID (external_id__v) column in her CSV. Gladys would then save with the filename subject__clin.external_id__v.csv.

Define an Order for Vault to Process CSVs

When you are loading record data to multiple objects, you may need to define an order for Vault to process the CSVs. For example, Gladys is loading record data to both the Subject and Protocol Deviation objects. Protocol Deviation has a reference relationship to Subject. Gladys must create the Subject records before she can successfully create the Protocol Violation or Deviation records that reference them. She can load both CSVs at the same time in a ZIP file, as long as she includes an order.txt file.

Create a plain text (.txt) file that lists the objects you’re loading to in the desired order. For example, Gladys’s order.txt file would include:

subject__clin

pdv__clin

Vault ignores any rows that do not reference an an object name in one of the CSV file names. So, you can reuse the same order.txt file for multiple data loads, even if you do not load the complete combination of objects each time.

If you don’t include an order.txt file in your ZIP file, Vault processes the CSVs in alphabetical order.

Sample CSVs

FTP Server Details

Vault automatically deletes your FTP loader files from the FTP server after seven (7) days.

How to Load Data to a Single Object

  1. Connect to the FTP server.
  2. Load your CSV file into the /vaultloader directory.
  3. Vault ingests the CSV and creates or updates the appropriate object records.

How to Load Data to Multiple Objects

  1. Connect to the FTP server.
  2. Load your ZIP file into the /vaultloader directory.
  3. Vault ingests the CSVs in the order specified by your order.txt file, and creates or updates the appropriate object records.

Success & Failure Log Files

After each load, Vault creates separate success and failure logs as CSV files. You can download these files from the in-Vault notification or email notification. Both files contain the system-managed ID values of each record.

Failure Logs

Failure logs include:

  • Error messages for each record that failed to load
  • Original data alongside corresponding errors
  • Row IDs from the original CSV file for the corresponding row

You can review and fix errors from within the error log. Once you correct any errors, you can re-import the failure log.

You can complete all steps in this article with the standard System Admin or Vault Owner profile.

If your Vault uses custom security profiles, your profile must grant the following permissions:

Type Permission Label Controls
Security Profile Admin: File Staging: Access Ability to access your Vault's FTP staging server.
Security Profile Admin: Access API Ability to make an API call, which is required to create and update records via FTP.
Security Profile Vault Owner Actions: Vault Loader Ability to see and use the Vault Loader tab. While creating and updating records via FTP does not use the Vault Loader tab, permission to use Vault Loader is required for automating the load with an FTP server.
Security Profile Objects: {Object}: Create, Edit Ability to create and edit records for the object you are loading into Vault.
Security Profile Objects: {Object}: Read Ability to view records related to the object you are loading into Vault. For example, when creating Subjects, users need Read access to the Study, Study Country, and Site objects.