Skip to main content

File Systems and Storage

This page provides information on managing directories and project space when using Northwestern's high performance computing cluster, Quest.

Home Directories

As a Quest account holder, you will be given a home directory the first time you log in to the system. All home directories have a fixed quota of 80 GB. Your home directory is only available to you and may not be shared.

Home directories have redundancy built into the storage system. If you inadvertently delete a file from your home directory, immediately contact quest-help@northwestern.edu as it may be possible to restore it.  After 24-hours any changes made to your home directory cannot be recovered.

Note:  Northwestern IT’s home directory volume occupies 16 TB of storage on Quest. If you leave the University, your home directory will be deleted and cannot be restored. It is recommended to backup whatever you need from your home directory before leaving the University.

Project Space

The projects volume has 129 TB of high-speed storage available and should only be used when analyzing large amounts of data on the compute nodes. Your home directory is also reachable by a compute node, however the Input Output (IO) is not optimized as is the projects volume.

In project space anyone belonging to an allocation may be able to read, write, and execute files that belong to other users of that same allocation. There are exceptions to this rule. For example if a user sync’s data from their home directory into a subdirectory in project space, the ownership and group ownership will belong to you. No one else will be able to read your files in project space unless you change the file permissions or the group ownership of your files is changed.

Temporary Directories

The /tmp directory on compute nodes is intended for temporary files that are used during the execution of a job. In compute nodes that do not have local disk drives, your temporary directories are kept in memory; generally up to about 24 GB.

If you exceed 24 GB, your compute job may fail or the compute node may go into an out-of-memory condition and crash. When a compute node crashes all jobs running on that node are terminated. It is possible to redirect temporary storage to either /home or /projects space  using the $TMPDIR variable:

export TMPDIR=/projects/myproject/mytemp

You will need to use this variable for every compute node that a job occupies on Quest.

The Utilities Module

The utilities module can be used to report the resources used in project accounts such as Service Units (SUs), Storage, and disk usage in project subdirectories.

For a listing of commands, please type:

module help utilities

----------- Module Specific Help for 'utilities/1' ----------------

Adds Quest utilities to your environment:

------------------------------------------------------

checkproject - report SU balances and disk quotas for your projects

grouplist    - list members of a project group

homedu       - show disk use for HOME subdirectories

homequota    - show disk quota for HOME directory

projectdu    - show disk use for project subdirectories

------------------------------------------------------

7z           - A file archiver with highest compression ratio

duf          - produce a hierarchical listing of disk usage for a directory hierarchy

ediff        - translate diff output into plain English

els          - An Enhanced LS look-alike with many additional features

fdupes       - find and optionally delete duplicate files

jot          - print sequential or random data

lsize        - list file details sorted by size, largest first

lzma, unlzma, lzcat - LZMA compression and decompression tool

lzcmp, lzdiff - compare LZMA compressed files

lzgrep       - search LZMA compressed files for a regular expression

lzmore, lzless - view LZMA compressed (text) files

lzmadec      - small and fast LZMA decoder

lzmainfo     - show information stored in the LZMA compressed file header

pkg-config   - Return metainformation about installed libraries

rs           - reshape a data array

seepath      - see status of each component of a path

Last Updated: 26 July 2017

Get Help Back to top