How to Create Multiple Batch Jobs

If you need to run multiple instances of the same program with different initial parameter values you do not need to write a separate shell script for each run. Instead you can write just one shell script that will read the different sets of parameters from a text file and submit separate jobs with each set of parameters.

The examples below are for MATLAB and Stata programs. For other programs you will need to modify the line of the script that calls the program.

• Multiple job shell script for MATLAB programs
• Multiple job shell script for Stata programs
• Running the shell script
• Debugging shell scripts
• Monitoring and stopping jobs
• Writing more advanced scripts

Multiple job shell script for MATLAB programs

Suppose you have a simulation in Matlab called "my_simulation.m" which is a function that requires 4 parameters and you would like to run this program for a few different sets of parameter values:

my_simulation.m                             
function my_simulation(P1,P2,P3,P4)
      . . .

The following shell script reads the parameter values from a text file and submits a separate batch job for each set of parameters:

submit.txt (for a MATLAB program)           
#!/bin/bash
while read P1 P2 P3 P4
do
    JOB=`qsub -m abe -N Big_run - << EOJ

cd ~/matlabprograms
matlab << M_PROG
my_simulation(${P1},${P2},${P3},${P4});
M_PROG
EOJ
`

echo "JobID = ${JOB} for parameters ${P1} ${P2} ${P3} ${P4} submitted on `date`" 
done < params.txt
exit                                        

The sample script above requires a text file called "params.txt" that contains the sets of parameter values, one set per line, each parameter separated from the next by a space:

params.txt                                  
0.01 5000 0.9 1
0.02 5000 0.7 2
0.03 5000 0.6 3

The script will read each set of values and create, on the fly, the equivalent of a single job shell script. This is done in each iteration of the while ... do ... done loop — the values from the "params.txt" file are read with the redirection in the done < params.txt line.

This examples assumes that "params.txt" is located in the same directory as the shell script. You can also specify another location of the parameters file by replacing done < params.txt with one that provides the correct path to the parameters file, such as done < ~/myprograms/params.txt

Note that the definition of the job starts and ends with a "back tick" (the reverse quote, found usually in the key under the Esc in a PC keyboard).

In the script you see << EOJ and << M_PROG. These are shell "input/output redirections": they force a command that they follow (in this script we use it after qsub and matlab) to read its input from the script until it finds a line that contains only "<< EOJ" or "<< M_PROG"

Look at the matlab command: we tell MATLAB to take its input from the lines between << M_PROG and M_PROG. That line is a Matlab command that calls your my_simulation function with four parameters. The parameter values are taken from the script's local variables P1 through P4. We write ${P1} to extract the value of the script's variable P1. For the first set of parameter values MATLAB will receive the following input from our script:

my_simulation(0.01,5000,0.9,1);

Keep in mind that you will need to distinguish output files from runs with different parameter values. One simple way is to print out parameter values in the beginning of your MATLAB code. You can also redirect the output of each job into a file that has the parameter values in its title — to do that, replace the line with the matlab command in the example with:

matlab > "sim_${P1}_${P2}_${P3}_${P4}.log" << M_PROG

In this case matlab output for the jobs in this examples will be saved in files named sim_0.01_5000_0.9_1.log, sim_0.02_5000_0.7_2.log and sim_0.03_5000_0.6_3.log so matching parameter values and output files will be trivial.

You can add additional lines of MATLAB code between << M_PROG and M_PROG. This example tells matlab to perform another function after "my_simulation" using the same set of parameter values read from "params.txt":

matlab << M_PROG
my_simulation(${P1},${P2},${P3},${P4});
secondprogram(${P1},${P2});
M_PROG

Similar redirection is done for the qsub command, which is forced to take its input from the lines between "<< EOJ" and "EOJ".

The echo command prints the job ID assigned by the system for each of the runs, the parameter values used, and the date and time when the job was submitted:

JobID = 3220.seldon for parameters 0.01 5000 0.9 1  submitted on Wed Aug 10 14:29:55 CDT 2005
JobID = 3221.seldon for parameters 0.02 5000 0.7 2  submitted on Wed Aug 10 14:29:55 CDT 2005
JobID = 3222.seldon for parameters 0.03 5000 0.6 3  submitted on Wed Aug 10 14:29:55 CDT 2005

Back to Top

Multiple job shell script for Stata programs

Below is a shell script for Stata that works in a similar fashion:

submit.txt (for a Stata program)            
#!/bin/bash
while read P1 P2 P3
do
    JOB=`qsub -m abe -N Big_run - << EOJ
cd ~/statacode
stata -b do simulation ${P1} ${P2} ${P3}
EOJ
`

echo "JobID = ${JOB} for parameters ${P1} ${P2} ${P3} ${P4} submitted on `date`"
done < params.txt
exit                                        

This script also reads parameter values (3 on each line, separated by spaces) from a file called "params.txt" located in the same directory as the shell script. It passes the parameters to a stata program called simulation.do located in the ~/statacode directory. Below is the simplest do-file that takes the parameters and displays them:

simulation.do                               
args P1 P2 P3
log using sim_`P1'_`P2'_`P3'.log
di `P1'
di `P2'
di `P3'
di `P1'+`P3'
...
log close

The first line of this do-file copies the text of the three arguments into Stata "macros" called P1, P2 and P3. Wherever Stata encounters `P1' or `P2' in the code it replaces it with the text of the first or second argument. Note that the name of the macro is enclosed in different quotes - use the "back tick" before the name and the single quote after the name.

Back to Top

The second line of the do-file saves the output to a log file that contains the parameters P1, P2 and P3 in its name. By default, on each run Stata would try saving the output to a file called "simulation.log" and you will lose output from all but one job if you do not instruct Stata to do otherwise.

Back to Top

Running the shell script

Before running the script you need to make it executable. If you named the shell script "submit.txt" then type in chmod u+x submit.txt at the prompt.

To run the shell script which will submit your jobs type in ./submit.txt at the prompt. You can also save the output of the shell script in a log file by typing in ./submit.txt >log.txt

Back to Top

Debugging shell scripts

When writing these scripts, make sure to test your script by using the echo command to check that the commands generated are correct. For example, to test the stata example you would need to add echo " before the qsub command and a closing " quote in the line after the I/O redirection label EOJ.

    JOB=`echo "qsub -m abe -N Big_run - << EOJ
cd ~/statacode
stata -b do simulation ${P1} ${P2} ${P3}
EOJ
"
`

When this modified script is executed, the operating system simply will show the qsub commands generated for each line of parameters in the "params.txt" file.

Back to Top

Monitoring and stopping jobs

You can monitor the status of your jobs in the queue by typing in qstat at the prompt. To delete waiting jobs from the queue or to stop running jobs type in qdel followed by job id numbers. To delete all three jobs submitted in the example above you would type in qdel 3220 3221 3222 at the prompt.

Writing more advanced scripts

Learn more about writing shell scripts from these sources:

• BASH Programming - Introduction HOW-TO, document at the Linux Documentation Project
• Advanced Bash-Scripting Guide, document at the Linux Documentation Project

Any SSCC user: contact bef@northwestern.edu (Bruce Foster) for help.

This page is a modification of instructions prepared by Patricia Ledesma Liébana at Kellogg Research Computing.