Skip to main content

Transferring Files on Quest

This page provides information on transferring files to and from Northwestern's high performance computing cluster, Quest.

Quest supports SSH File Transfer Protocol (SFTP) and Secure Copy (SCP), which are commonly available on the Linux and Mac OSX operating systems and on Windows-based systems through third party applications such as Cyberduck.

Linux/Mac

To use SCP to transfer files from your workstation to Quest use the command (on the workstation Terminal):
scp /PathToSourceFile/file NetID@quest.it.northwestern.edu:/PathToTargetDir/file

To use SCP to transfer files from Quest to your workstation use the command (on the workstation Terminal):
scp NetID@quest.it.northwestern.edu:/PathToSourceDir/file /PathToTargerDir/file

Mac/Windows

Both Mac and Windows clients are available for downloading from the Cyberduck site. The Cyberduck Help wiki provides basic usage instructions and feature lists, and should be reviewed before using it for the first time. Be sure to establish a VPN connection from off campus before connecting to Quest.

To connect to Quest, start Cyberduck:

  1. Click Open Connection in the upper left of the Cyberduck window
  2. At the top of the Open Connection window that appears, Select SFTP (SSH File Transfer Protocol) from the drop-down menu.
  3. Enter quest.it.northwestern.edu for server specification
  4. Enter your NetID in the Username: box and leave the Password: box empty to prevent your NetID password from being saved in a file on your personal computer. Public Key Authentication is not supported.
  5. Click Connect. You will see a Login failed window.
  6. Enter your NetID password in the Password: field.
  7. Click Login.​

Download from FTP/web servers

You can download datasets directly to Quest from external ftp and web servers. wget is a command that can download files both from ftp and web servers, for example:

wget http://www.nber.org/nberces/bbg96_87.dta
wget ftp://ftp.bls.gov/pub/doc/ap.txt

With additional options wget can also download whole directories of files recursively and download from password-protected servers. Type man wget at the command prompt to get a full list of options.

Globus 

Globus is a Software-as-a-Service (SaaS) that provides a program interface for file transfer and sharing, as well as identity, profile, and group management. You can use Globus to transfer very large data sets, collaboration and data sharing, and more.

A Globus transfer handles all the difficult aspects of data transfer by optimizing parameters to maximize bandwidth usage, managing security configurations, providing automatic fault recovery, and notifying users of completion and problems. Command line and web interfaces are available for use.

Learn more about when to use globus to transfer files to and from Quest. Globus may also be used to transfer files to and from the Research Data Storage Services and FSMRESFILES. When transferring files to/from the Research Data Storage Service or FSMRESFILES, please avoid the use of symbolic links on Quest. This results in duplicate copies of your data on the Research Data Storage Service and FSMRESFILES. Please contact quest-help@northwestern.edu if you would like to use this service.

Box

To move files from Box into Quest:

  1. Make sure you have the necessary tunneling software; Northwestern IT recommends FastX2.  Download and Install FastX2. You will need to logout and login to your local machine for the tunneling software to take effect.
  2. ssh to Quest from your terminal on your local machine.
  3. Run Firefox by typing the following commands in the terminal window: “module load firefox”, followed by “firefox” to launch the Firefox browser.
  4. In the Firefox browser, type “northwestern.box.com” and log in.
  5. Select the file you wish to move onto Quest and click Download in the upper right corner. By default, the file will be placed into the Downloads directory in your home directory.

SMBCLIENT

File transfer between Quest and the Research Data Storage Service is limited to using the non-audited storage zone, resfiles.northwestern.edu. Quest does not support analysis of regulated data (such as PHI and PII). This includes data placed in FSMRESFILES and other Research Data Storage Service audited storage zones designed for handling such data.

To transfer files, it is recommended to use globus (see above) or smbclient via the command line. When transferring files to/from the Research Data Storage Service or FSMRESFILES, please avoid the use of symbolic links on Quest. This results in duplicate copies of your data on the Research Data Storage Service and FSMRESFILES. 

To check for symbolic links in a directory, use the command:



find [path-to-directory] -type l -ls

After logging into Quest:

1. Uploading files from Quest to Research Data Storage Service:

smbclient '//storage-host-name/share-name' -c 'put file-name-for-upload

/path-on-RDSS/file’ -U "ads\netid"

Example:

smbclient '//resfiles.northwestern.edu/MY_LAB' -c 'put myQuestFileToCopy

/myResfilesDestinationPath/myResfilesDestinationFile' -U "ads\abc123"

2. Downloading files to Quest from Research Data Storage Service:

smbclient '//storage-host-name/share-name' -c 'prompt;recursive;lcd

path-on-Quest; cd /path-on-RDSS/file-name; mget *' -U

"ads\netid"

Example:

smbclient '//resfiles.northwestern.edu/MY_LAB' -c 'prompt;recursive;lcd /myQuestDestinationpath/myQuestDestinationfile; cd /myResfilesPath/myResfilesFileToCopy; mget *' -U "ads\abc123"

3. Accessing SMB interactive shell at the storage:

smbclient \\\\resfiles.northwestern.edu\\MY_LAB -U "ads\abc123"

NOTE: If individual file size is over 10 GB, add the following socket options to your smbclient transfer.

smbclient YOUR-COMMAND-OPTIONS-FROM-ABOVE --socket-options='TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=131072 SO_SNDBUF=131072’

Following command will list all available commands:

smb: \> help

For more information, please visit:  https://www.samba.org/samba/docs/using_samba/ch05.html

Last Updated: 26 July 2017

Get Help Back to top