Using OAR for computing jobs management

OAR is the batch and job scheduler which permits to launch computing jobs on the cluster nodes.
For more details, see http://oar.imag.fr and more precisely http://oar.imag.fr/user-usecases/.
 

OAR commands

The firs command to know is

man oarsub

The cluster is organized as following :

  • chassis
  • host / nodes
  • cpu
  • core

And it is splitted as following :

  • default : all generic nodes (no smp and no visu) -> do not specify !
  • smp : all smp nodes with big memory (no generic and no visu)
  • visu : all visualization nodes (no generic and no smp)

That meant that if you don't specify the partition on which you want to work, all your jobs will be launched on generic nodes.

To reserve one core of the cluster in interactive mode (possibility to launch commands in a nutshell) :

oarsub -I

To reserve 2 cores in interactive mode :

oarsub -I -l core=2

To reserve 1 complete node in interactive mode :

oarsub -I -l nodes=1

To reserve 2 cores in interactive mode on a computer with 12 cores and not dedicated to visualization :

oarsub -I -l nodes=1/core=2 -p "nbcores=12 AND visu = 'NO'"

To reserver 2 complete nodes on the same chassis in interactive mode :

oarsub -I -l chassis=1/nodes=2

To reserve 2 complete nodes on 2 different chassis in interactive mode :

oarsub -I -l chassis=2/nodes=1

Interactive jobs have a maximum walltime of 12 hours. Beyond, you must use batch jobs (with OAR launching script).

To launch a script on 1 complete node :

oarsub -l nodes=1 ./script.sh

To launch a script on 4 nodes, to term the job and to specify a walltime of 24 hours :

oarsub -l nodes=4,walltime=24:00:00 -n nom_du_job ./script.sh


To put directly OAR directives un a launch script, here is an instance :

#!/bin/sh
#OAR -l nodes=4,walltime=2:0:00
#OAR -O file.out
echo "Debut du job ${OAR_JOB_ID}"
./monprogramme
exit $?

Thus you must submit your job with option "-S" :

oarsub -S "./script.sh var1 var2 ..."


To use SMP node called smp01 in the smp partition

oarsub -I  -p "host='smp01'" -t smp


To use SMP node in a script, you must put the 2 lines

#OAR -p host='smp01'
#OAR -t smp

To use a GPU node for computation during 2 hours

oarsub -I -l nodes=1,walltime=2:0:0 -p "visu='YES'" -t visu


Some OAR script examples are available on the cluster in the directory /softs/OAR/script_examples or here : Exemples de soumission

you can insert directly in your OAR scripts the module load or the "source" of files.

By default, the time length of a batch job is limited to 24 hours, thanks to walltime option, you can specify a time length :

walltime=X:00:00 with X > 24 can compute during X/24 days

Internal hostname are the following :

  • For generic nodes : node001 to node072
  • For SMP node : smp01
  • For hybrid node : visu01 to visu08

 

Some OAR commands to manage your jobs

# To list in progress jobs
oarstat

# To display job 483 properties
oarstat -fj 483

# To display only our own in progress jobs
oarstat -u
# To display only in progress jobs of user "user"
oarstat -u user

# To display computation time used for a given period
oarstat --accounting "2012-10-01, 2012-12-01"

# To display computation time used for a given period for the user "user"
oarstat -u user --accounting "2012-01-01, 2012-04-01"

# To delete job id 483
oardel 483

 

Address

Université de Nice Sophia-Antipolis
D.S.I.
28 Avenue de Valrose
B.P. 2135
06103 NICE Cedex 02