Some Flux projects contain parameters that can be solved independently. Flux comes
with a feature that allows the distribution of these parameters to speed-up this
kind of computations. To use that feature, the following environment variables must
be defined:
- FLUX_PARAMETRIC: Indicator for using parametric
distribution. Must be set to true
- FLUX_PARAM_AUTO: Indicator for using automatic or manual
computation of number of jobs. Must be set to Automatic
or Manual
- In Automatic, Flux will submit a maximum of 127
jobs (+1 for the master) to solve the project. The number of solved
parameters per job is automatically computed, and if there are less
than 127 parameters, each job will solve 1 parameter.
- In Manual, the environment variable
FLUX_PARAM_MAXCORES must be set to the
maximum number of cores you want to allocate for the job. One of
these cores will be used for the master, and the other will be jobs
submitted by Flux to solve the project. The number of solved
parameters per job is automatically computed, and if there are less
parameters than remaining cores, each job will solve 1 parameter and
the exceeding cores won’t be used and will be free for other
jobs.
# Use parametric distribution
export FLUX_PARAMETRIC=true
export FLUX_PARAM_AUTO=Manual
export FLUX_PARAM_MAXCORES=32
To submit jobs and manage files, some basic PBS and Linux commands are used by Flux.
These commands are already defined by default, but can be modified using environment
variables:
- COPYREC: recursive copy. Default: cp
-r
- SUBMITNAME: submit job with name option. Default:
qsub -N
- MOVE: move files. Default: mv
- STATGREPCOUNT: Display all jobs, filter and count option.
Default: qstat –a | grep –c
For example, using a classical Linux installation, but a batch scheduler using OAR,
the
COPYREC and
MOVE should be identical, but
the two other variables need
modifications:
#Define commands for parametric distribution /!\ Check with batch
export SUBMITNAME=oarsub -n
export STATGREPCOUNT=oarstat | grep -c