Storage and Data Policy
Quest storage is available on the University’s high performance computing system Quest. This high-speed storage is available to users in active Full- and General-Access allocations. Quest has three different storage locations that are meant to be used for different purposes. This document describes the policies that are in effect to provide the best experience for all Quest users and to optimize storage utilization for active research. For information about the tools and best practices to manage these Quest storage locations, please refer to File Systems and Storage on Quest.
HOME DIRECTORIES PROJECT DIRECTORIES QUEST SCRATCH SPACE
Home directories are intended to store your personal user files, such as job submission scripts or locally installed software. The following policies apply to home directories:
- Home directories are backed up. Deleted files may be recovered within 28 days.
- Users cannot exceed their home directory 80 GB quota.
- Quest users are granted exclusive access to storage space in the home directory and should not grant access to any other user. Files shared with other users on Quest should be stored in project or scratch directories.
- Your home folder will be deleted under the following circumstances:
- If your NetID expires: If you leave the University, your NetID will expire. Home directories associated with expired Northwestern NetIDs will be deleted.
- If you have been inactive for one year: Users who are not associated with an active allocation are considered inactive users. An inactive user will have access to Quest for three months if their Northwestern NetID remains active. After one year without being associated with an active allocation, the home folder of the inactive user will be deleted.
Project Directories are meant to store files that are shared among users with access to an allocation, such as data, analysis scripts as well as locally installed software that is used by multiple allocation users. The following policies apply to project directories.
- Project directories are not backed up and deleted files cannot be recovered.
- Users cannot exceed the storage limit of their project directory.
- Data stored in the project space for Full or General Access allocations (Research I and II and Classroom/Workshop) will be deleted 90 days after the allocation expires. If the allocation is renewed within the 90 days period, no data will be deleted.
- If part of the allocation’s storage expires, the allocation’s quota will shrink by the expired amount. If the allocation occupies more storage than the new, smaller allocation quota, you will not be able to write data to project space until you reduce the amount of storage used to be below the new quota. Consequently, if your application tries to write data to an already full allocation, it will fail to run.
- If the storage used is not below the new quota within 90 days, the allocation will be locked. Then, an automated process will delete the oldest files until the storage used is less than the quota.
- Example: if 2TB of 30TB storage expires, the storage will shrink to 28TB. If the allocation occupies 29 TB, the allocation users are responsible for removing at least 1 TB of storage to get back to allocation limit, before jobs can be run under the allocation.
- Purchasing a node on Quest will automatically grant access to (i) storage on the local disk of the node and (ii) additional storage on the project space of Quest’s file system.
- When the term of ownership for a node ends, the local storage on the node will be deleted. The storage available on Quest’s project space will expire 90 days after the term of ownership ends for the node.
Quest Scratch Space
Quest scratch directories are intended for short-term storage of large data for such as temporary files from running jobs or large data transfers
- The following policies apply to scratch space:
- Scratch directories are not backed up. Deleted files cannot be recovered.
- Users apply to access Quest scratch space individually, as scratch space is assigned to individuals rather than by allocations. Access to scratch space expires with the user’s active allocation. If a user is no longer in any active Quest allocations, their scratch directory will be set to read only. Once this happens, users will retain access to any data previously written into their Quest scratch directory until that data is deleted as part of the regular scratch deletion schedule. To regain access to scratch space, users must reapply for Quest scratch space access once they are in an active allocation.
- Global scratch settings may be adjusted by Northwestern IT as necessary to manage scratch space and support availability for everyone. In such events, Quest scratch space users will receive a 5-days advance notice of any global changes. In normal operation conditions, Quest users may store up to 5 TB and up to 5 million files in their scratch directory. These settings cannot be modified on an individual user basis.
- In normal operation conditions, each file can be kept in scratch for up to a 30-day period after the last modification date of the file. Files that are not edited or changed for the entire retention period will be deleted through a continuous and automatic purge process.
- Users are responsible for monitoring the age of their files through the checkscratch utility, and transferring data that must be preserved. There are no exceptions to the scratch space deletion policy.
- Files that need to be kept longer than 30 days must be moved out of scratch and into the user's project or home directory or transferred to other storage services. See Document Sharing and Data Storage Finder for options.
- Users who run “touch” or similar commands for the purpose of altering their files' timestamps to circumvent this file deletion policy will have their access to Quest scratch space revoked.
- Users should delete files from scratch space promptly. Quest scratch space is a shared resource and should be used conscientiously by deleting files when they are no longer needed.