Use on Caparmor

Warning

This part is dedicated to advanced Caparmor users. To run a MARS configuration, informations contained in the MARS environment are sufficient.

  • OMP run : XXXX lien vers manuel
  • MPI run : XXXX
  • hybrid run : XXXX

PBS command

qsub

to submit a job (under the form of shell_script) into a batch queue:

qsub –l select=1 :ncpus=8 –o res -q parallel8 shell_script

qstat

to watch your job:

qstat -u user

qdel

to kill a job:

qstat -u user   (to get the job ID)
qdel jobid
qdel -W force jobid

QSUB

-q: specify the name of the queue according to the number of cpus you want to use:

-q queue_name
Name of the queues on Caparmor
type of the job queue name nb of nodes max nb of cpus elapsed time
sequential sequentiel 1 1 300h
OMP parallel8 1 8 72h
MPI / hybrid parallel8 1 8 72h
MPI / hybrid parallel32 2 < nb <= 4 32 48h
MPI / hybrid parallel64 5 < nb <= 8 64 24h
MPI / hybrid parallel256 9 < nb <= 32 256 18h

You get the the total number of core (and consequently the name of the queue) from select*ncpus

select: number of nodes (if ncpus=8)
ncpus and mpiprocs are defined for each node (or each select number)
ncpus: * number of OMP threads (=ncpus/mpiprocs) when using OMP
(do not specify OMP_THREADS or set it to OMP_THREADS=ncpus/mpiprocs in the batch)
: * number of cores used per node (=mpiprocs if you want to make sure not to share de node with someone else)
mpiprocs: number of mpi cores per node
-np: total number of MPI cores (select*mpiprocs)
  • MPI run (best use):

    qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=8 batch_mpi   | -np = 16 in batch_mpi
    
  • hybrid (MPI+OMP) run (best use knowing there are 8 cores per node)

    8 OMP threads, 1 MPI core per node and 4 nodes
    qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=1 batch_mpiomp   | in batch_mpiomp -np = 4
                                                                     |                 setenv OMP_THREADS 8
    
    2 OMP threads per node, 8 nodes in total, different ways with -np 16
    qsub -q parallel32 -l select=16:ncpus=2:mpiprocs=1 batch_mpiomp  | in batch_mpiomp -np = 16
                                                                     |                 setenv OMP_THREADS 2
    
    qsub -q parallel32 -l select=8:ncpus=4:mpiprocs=2 batch_mpiomp   | in batch_mpiomp -np = 16
                                                                     |                 setenv OMP_THREADS 2
    
    qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=4 batch_mpiomp   | in batch_mpiomp -np = 16
                                                                     |                 setenv OMP_THREADS 2
    with these last choice, you will share the node nobody
    
  • to specify the memory and make sure you will not share the memory with other runs (the maximum is 24Gb):

    -l select=8:ncpus=4:mpiprocs=4:mem=23.8gb
    
  • by default, if not specified mpiprocs=1

    qsub -q parallel256 -l select=64:ncpus=4           batch_mpiomp  | in batch_mpiomp -np = 64
                                                                     |                 setenv OMP_THREADS 4
    
    if you do not want to share the node, prefer :
    qsub -q parallel256 -l select=32:ncpus=4:mpiprocs=2 batch_mpiomp | in batch_mpiomp -np = 64
                                                                     |                 setenv OMP_THREADS 4
    
  • exemple with error

    qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=1 batch_mpiomp   | in batch_mpiomp -np = 32
                                                                     |                 setenv OMP_THREADS 8
    
    choose -np 4 instead
    
  • how to use only 2 threads and 1 MPI cpus per node (and I freeze other cores for any other use)

    qsub -q parallel256 -l select=16:ncpus=8:mpiprocs=1 batch_mpiomp | in batch_mpiomp -np = 16
                                                                     |                 setenv OMP_THREADS 2
    

    In this last exemple you use only 32 cores but freeze 128 cores. Do not do that !

batch for MPI jobs

  • batch for intel MPI:

    #PBS -S /bin/csh
    #--- Affectation du nom au job lance
    #PBS -N name_job
    ##affectation du nom au fichier erreurs
    #PBS -e erreur.txt
    ## Affectation du nom au fichier sorties
    #PBS -o sortie.txt
    #PBS -m e
    #PBS -l walltime=06:00:00
    #PBS -q parallel8
    #PBS -l select=1:ncpus=8:mpiprocs=8
    
    #----------------------------------------
    #-- Chargement pour accès commande module
    #----------------------------------------
    source /usr/share/modules/init/ksh
    
    #----------------------------------------
    #-- Load the module
    #----------------------------------------
    module purge
    module load intel-mpi/4.0.0.028
    module load intel-comp/11.1.073
    
    # cd to the directory from which you submitted your job
    cd $PBS_O_WORKDIR
    
    time runmpi -n 8 executable
    time runmpi [-genv I_MPI_USE_DYNAMIC_CONNECTIONS 0] -np 8 time executable
    

#denistina

  • batch for MPT (SGI MPI):

    #PBS -S /bin/csh
    #--- Affectation du nom au job lance
    #PBS -N name_job
    ##affectation du nom au fichier erreurs
    #PBS -e erreur.txt
    ## Affectation du nom au fichier sorties
    #PBS -o sortie.txt
    #PBS -m e
    #PBS -l walltime=06:00:00
    #PBS -q parallel8
    #PBS -l select=1:ncpus=8:mpiprocs=8
    
    #----------------------------------------
    #-- Chargement pour accès commande module
    #----------------------------------------
    source /usr/share/modules/init/ksh
    
    #-- Load the module
    #
    module purge
    module load mpt/2.01
    module load intel-comp/11.1.073
    
    # environment variables for MPT
    # number and depth of deribative type (do not compile if noti large enough)
    setenv MPI_TYPE_DEPTH 50
    setenv MPI_TYPE_MAX 50000
    setenv MPI_DEFAULT_SINGLE_COPY_OFF "ON"
    # optimisation
    #setenv MPI_IB_RAILS 2
    #setenv MPI_IB_XRC
    #setenv MPI_FASTSTART
    #
    # cd to the directory you submitted your job
    cd $PBS_O_WORKDIR
    
    time mpiexec_mpt -np 8 time ./mars_exe
    

batch for hybrid jobs

Same as for MPI/MPT jobs except that you must specify

  • the number of OMP threads:

    setenv OMP_THREADS 4
    
  • OMP options:

    setenv OMP_SCHEDULE "dynamic,1"
    
  • and choose the mpiprocs number in consequence