SenNet Command-Line Transfer (CLT)

The SenNet Command-Line Transfer (CLT) tool streamlines the download of individual files and directories from multiple datasets and uploads. Files and directories are specified in a manifest file and given to the CLT.

Installation

The CLT uses the Globus Connect Personal (GCP) application to download the specified files and directories to your computer. GCP and the CLT require a SenNet account. This account is the same account used to log in to the SenNet Data Sharing Portal. Windows users should follow additional instructions to correctly setup GCP.

The CLT requires Python 3.9 or above.

pip

Install the CLT globally using pip:

pip install atlas-consortia-clt

pip with a virtual environment

Installing in a virtual environment keeps the CLT and its dependencies isolated from other Python projects.

macOS/Linux:

python3 -m venv clt-env # creates virtual environment named clt-env in the current directory
source clt-env/bin/activate # activates the virtual environment
pip install atlas-consortia-clt

Windows:

python -m venv clt-env # creates virtual environment named clt-env in the current directory
clt-env\Scripts\activate # activates the virtual environment
pip install atlas-consortia-clt

To use the CLT in the future, activate the virtual environment first:

macOS/Linux:

source clt-env/bin/activate

Windows:

clt-env\Scripts\activate

To deactivate the virtual environment, run the following command on any platform:

deactivate

pipx

pipx installs the CLT in its own isolated environment and automatically exposes the sennet-clt commands on your PATH, without affecting other Python packages.

Install pipx if you don’t have it. Check the pipx documentation for detailed installation instructions.

Then install the CLT. The CLT relies on the globus-cli package and must also be installed via pipx.

pipx install atlas-consortia-clt globus-cli

To upgrade:

pipx upgrade atlas-consortia-clt globus-cli

Manifest File

Files and directories of interest are specified in a manifest file. A manifest is a text file that contains the dataset id and the path to the dataset, separated by a space. Multiple files or directories can be specified by placing each on a separate line. A directory can be specified by ending the path with a slash /. An optional header can be specified at the top of the file. Text after the path will be ignored and can be used for comments.

Example

dataset_id file_or_dir_specifier    #optional header line is ignored
SNT123.ABCD.456 /metadata.tsv       #download the metadata.tsv file for dataset SNT123.ABCD.456
SNT345.ABCD.456 /                   #download all files in the dataset SNT345.ABCD.456
SNT378.HDGT.837 /extras             #download the extras directory from dataset SNT378.HDGT.837

Manifest files can be created from the home page of the Data Sharing Portal when Dataset is selected as a filter. Select the checkboxes next to the datasets of interest and then click the download actions button at the top left of the table.
Creating manifest

Usage

Usage documentation can also be found by running the following command:

sennet-clt -h

A one-time login is required for any download session. For non-public data, you must log in with your SenNet account. For publicly available data, you can log in with any account accepted by the login form (Google and ORCID). Login can be initiated using the following command:

sennet-clt login

By default, login will automatically open a browser window to complete authentication. If you are in a headless environment (e.g. a remote server without a browser), use the --no-browser flag. This will display a URL in the terminal that you can copy and open in a browser on another device to complete the login:

sennet-clt login --no-browser

Logout

Logout can be used to log out the current user.

sennet-clt logout

Transfer

A data transfer and download can be initiated using the transfer command and a manifest file. You must be logged in to use the transfer command.

sennet-clt transfer <PATH/TO/MANIFEST/FILE>

An optional destination argument can be specified. The destination is the directory on the user’s computer where data will be downloaded. The directory will be created if it doesn’t exist. The destination argument is relative to the user’s home directory (~). For example, --destination Desktop/sennet-data corresponds to an absolute path of ~/Desktop/sennet-data. The default destination directory is ~/sennet-downloads.

sennet-clt transfer <PATH/TO/MANIFEST/FILE> --destination <PATH/TO/DESTINATION/DIRECTORY>

An optional --from-protected-space flag can be specified to download protected data belonging to a published protected Dataset. By default, the CLT will download public data only. The user must have access to the protected data in order for the transfer to be successful.

sennet-clt transfer <PATH/TO/MANIFEST/FILE> --from-protected-space

Whoami

Whoami can be used to display the information of the currently logged in user.

sennet-clt whoami