Skip to main content

Submitting a Job on Quest

This page provides information on how to submit jobs on Northwestern’s high performance computing cluster Quest.  Topics include:

Background Information

Login Nodes

Once you have logged into Quest, you will find yourself on one of the login nodes (quser10, quser11, quser12, quser13).  On the login nodes users can edit files, compile code, and do other relatively low compute intensive programs (e.g. transferring files, data visualization, and data analysis).  These nodes are also used for submitting, monitoring jobs, and checking job statuses.  Although it is possible to run very small jobs on the login nodes for testing code, the login nodes are NOT meant for compute or resource intensive tasks. 

Unix Shell

The shell is an environment in which you can run commands, programs, and shell scripts. The login shell interprets text from command line arguments and statements in shell scripts.  Just as there are different operating systems, there are different flavors of shells. You can check your login shell by entering the following at the command prompt:

echo $SHELL

When you enter a command at the $ prompt, the shell reads the input and interprets the text.

Shell scripts are basically a list of commands.  You can comment in a shell script by preceding your comments with a # sign.  You can also process large amounts of data using conditional statements, loops, variables, and reading and writing files.

When you write a script, such as Moab script for job submissions (discussed more below), you need to alert the system that it is a shell script by using the shebang construct (#!) at the beginning of your file, e.g.:

#!/bin/bash

This example is for the default shell on Quest (Bourne-again, i.e. bash).  By convention, shell scripts are saved with extension .sh.

Batch Jobs

Any long-running program that is compute-intense or requires large amounts of memory (>5GB) should be executed on the compute nodes. Access to the compute nodes is through the scheduler, consisting of Torque and Moab.

Through the use of command line options and/or specially formatted comments in the submission script, users can request a certain amount of resources available for your allocation on the Quest cluster, such as the number of nodes and processors per node, or specific hardware, e.g. Graphics Processing Units (GPUs).  It is recommended to use a submission script for batch jobs.

Unless you specifically request exclusive use of an entire node, other users may occupy the same node you are running on.

When you submit a job to Quest, you can directly access the nodes on which your job is physically running via SSH if needed:

ssh qnodeX

where X is the node #.  You can find the node number using the checkjob -v or showq -r commands explained in more detail at Managing Jobs on Quest.

Batch Job Submission Script

A submission script is a shell script with specially formatted comment lines starting with #MSUB which represent scheduler parameters. 

The first line of the script should specify the shell (i.e. bash in this case) used to execute the user programs:

#!/bin/bash

The following lines contain the scheduling parameters. It is recommended to specify at least the following parameter lines (valid also for other shells):

#MSUB -N name_of_job

This is used to set a descriptive name for the job which shows in the output of some utilities, such as qstat. 

#MSUB -A account_number

This is used to set the account. Users must specify their account, because it tells the scheduler from which account the SUs should be charged. Users with multiple accounts should take special care to make sure they are charging to the correct account.

#MSUB -m abe

#MSUB -M my_email_address

With these options, an e-mail is sent when the job starts (b), finishes (e), or aborts (a) to the specified email address. The letters can be given in any order and are all optional as long as one letter is present.

#MSUB -l nodes=N:ppn=p

Here we allocate N nodes with p cores each. If left out, one core on one node will be allocated. However, for clarity, nodes and ppn (processors per node) should still be specified.

#MSUB -l walltime=hh:mm:ss

This command is used to set the maximum wall clock time. For efficient scheduling, the maximum wall clock time should always be estimated, if possible, and provided in the script. Here, the walltime parameter is set to allow a maximum runtime of hh hours, mm minutes, and ss seconds. Depending on the maximum wall clock time and the number of cores requested, the job is routed to the corresponding queue on Quest that satisfies these constraints most closely.

If neither walltime nor queue name have been specified, the job is routed to the short queue. If queue name is specified, but not walltime, the maximum runtime is set to the largest allowable value for the selected queue. See Quest Queues for more information.

#MSUB -q queue_name

For full access users there is an additional buyin queue available that has no restrictions on number of cores or walltime. In order for the job to get routed to this queue, the following has to be added to the job script:

#MSUB -q buyin

See msub Commands for additional submission options. 

Before starting a job, the current working directory of the script ($HOME by default) should be changed to the intended location. This line changes to the current working directory the msub command has been executed in.

cd $PBS_O_WORKDIR

The remaining lines constitute the job script used to start the user applications.

Example Submission Script

A submission script could look like the following.  These commands would be saved in a file such as myscript.sh.


#!/bin/bash
#MSUB -A p20XXX
#MSUB -q short
#MSUB -l walltime=04:00:00
#MSUB -M my_email_address
#MSUB -j oe
#MSUB -N projectname_mysoftware
#MSUB -l nodes=1:ppn=6

# add a project directory to your PATH (if needed)
export PATH=$PATH:/projects/p20XXX/tools/

# load modules you need to use
module load python/anaconda
module load java

# Set your working directory
cd $PBS_O_WORKDIR

# A command you actually want to execute:
java -jar <someinput> <someoutput>
# Another command you actually want to execute, if needed:
python myscript.py

After you've created your submission script file, to submit your job, you would then type at the command prompt:

msub myscript.sh

If your job is successfully submitted, the above command will output your job ID number.

For more information, see Examples of Jobs on Quest.

Advanced Batch Job Options

Job Dependencies

Job dependencies are supported by Moab. Dependent jobs are useful when there is a series of jobs that need to be executed in sequence or when the execution of jobs need to be synchronized. For more information, see Moab documentation on Job Dependencies.

Job Arrays

An array of jobs can be submitted through the Moab scheduler. Job Arrays are an easy way to submit a multitude of sub-jobs that use the same runscript, but work on different input data such as a sweep of input parameters. For more information, usage, and general examples refer to Job Arrays.  

Interactive Jobs

Interactive jobs are usually for data analysis or extensive debugging (with totalview, for instance) that is resource intensive.  You can start an interactive session with the following command:

msub -I

Per default, this will allocate one core on one node, and execute the job in the short queue using your default account. If multiple cores or nodes are required, or you need more time than allowed in the short queue, the corresponding -l option needs to be passed as well. The following example will allocate 24 cores on 2 nodes for 24 hours:

msub -I -l nodes=2:ppn=12 -l walltime=24:00:00

Note that -l can be specified multiple times on the command line, and that the job is automatically routed to the appropriate queue based on the requested walltime and number of processes. Alternatively, the queue can be directly specified with the -q option. Without setting walltime, the –q option sets the runtime limit to the maximum allowable value. See msub Commands for further details and options.

Interactive Jobs with Graphical User Interface Capabilities

In order to take advantage of advanced visualization tools and other graphical applications, forwarding of the X11 (X-Windows) display must be enabled via msub's -X command line option, e.g.,

msub -I -X -l nodes=2:ppn=12 -q normal

In order to use X11 applications during interactive jobs, X forwarding must also be enabled when logging in to Quest; FastX connections have X forwarding enabled.

Last Updated: 26 July 2017

Get Help Back to top