Computing : More sophisticated job submit file examples

It should always be the main goal to submit as many jobs as possible as part of one 'cluster' instead for ex. traversing through a directory in a 'foreach-loop' and invoke a 'condor_submit' command for every single find. Here are some more sophisticated examples on how to submit several jobs as one cluster of jobs which keeps the stress on the scheduler low and the responsitivity high ! 

If you use the techniques described below it NEVER hurts to use a 'circuit-breaker' option wit condor_submit: 

condor_submit -maxjobs <number> # stop submitting after <number> jobs


Last tip befor the actual submit file examples, you can use the 'dry' option to check what condor_Submit will actually do without submitting any job:

condor_submit <your submit file> -dry -

Submit file examples:


Go over a dir in 'foreach-style'
<begin submit file>
< usual stuff like executable, logfiles etc.>
Args = $(Item)
queue 1 Item matching files (*.dat)
<end submit file>

In the submit dir:

ls *.dat
1.dat  2.dat  3.dat  4.dat


In the executable:

MY_INPUT_FILE = $1

<do something with input file>

In the queue:

[chbeyer@htc-it01]~/htcondor/testjobs% condor_submit sleep_foreach.submit Submitting job(s)....
4 job(s) submitted to cluster 2203711.

[chbeyer@htc-it01]~/htcondor/testjobs% condor_q -nobatch
-- Schedd: bird-htc-sched01.desy.de : <131.169.56.32:9618?... @ 06/29/18 12:05:07
 ID         OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
2203711.0   chbeyer         6/29 12:04   0+00:00:00 I  0    0.0 sleep_args.sh 1.dat
2203711.1   chbeyer         6/29 12:04   0+00:00:00 I  0    0.0 sleep_args.sh 2.dat
2203711.2   chbeyer         6/29 12:04   0+00:00:00 I  0    0.0 sleep_args.sh 3.dat
2203711.3   chbeyer         6/29 12:04   0+00:00:00 I  0    0.0 sleep_args.sh 4.dat
Using a script for the input of arguments, each line of the scriptoutput is treated as an item
[chbeyer@htc-it01]~/htcondor/testjobs% cat items.sh 
#!/bin/bash
ls -1 *.dat
[chbeyer@htc-it01]~/htcondor/testjobs% ./items.sh 
1.dat
2.dat
3.dat
4.dat


The submit file:
Args = $(Item)queue from ./items.sh |
Go through a list and reads the arguments from every line
[chbeyer@htc-it02]~/htcondor/testjobs% cat list.txt
60, eine
120, zwei
180, drei
240, vier


The submit file:
Args = $(var1) $(var2)
queue var1,var2 from list.txt


In the queue:
-- Schedd: bird-htc-sched02.desy.de : <131.169.56.95:9618?... @ 07/17/18 11:45:24
 ID         OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
6253267.0   chbeyer         7/17 11:45   0+00:00:00 I  0    0.0 sleep_args.sh 60 eine
6253267.1   chbeyer         7/17 11:45   0+00:00:00 I  0    0.0 sleep_args.sh 120 zwei
6253267.2   chbeyer         7/17 11:45   0+00:00:00 I  0    0.0 sleep_args.sh 180 drei
6253267.3   chbeyer         7/17 11:45   0+00:00:00 I  0    0.0 sleep_args.sh 240 vier