There are several partitions with preemption enabled. Preemption means, that a job running in a "preemptable" partition can be terminated by any job in a higher priority partition containing the same nodes. Terminated jobs will either be requeued - the job will go into the queue as if it was never running - or simply cancelled. The fate of preempted jobs depends on the configuration of the partition and flags used during job-submission. Please note: a job running in a reservation will never be preempted regardless of preemption-mode and partition.
The list of partitions gives an overview of available partitions and the corresponding preemption mode. Below is a table of partitions with preemption (manually generated; might be outdated!):
Partition with preemption | Higher priority partitions | Lower priority permissions | Preemption mode | explained |
---|---|---|---|---|
allcpu, allgpu | all other partitions | none | requeue | a job in a higher priority partition will terminate the job. terminated jobs will be requeued |
cfel | cfel-cdi, cfel-cmi, cfel-ux | allcpu, allgpu | cancel | termination by jobs in cfel-cdi, cfel-cmi, cfel-ux. terminates jobs in allcpu, allgpu |
exfel, upex | exfel-*, upex-* | allcpu. allgpu | requeue | termination by jobs in any other upex or exfel partition. terminates jobs in allcpu, allgpu |
xfel-guest | xfel-sim, xfel-op | allcpu, allgpu | cancel | termination by jobs in xfel-sim, xfel-op. terminates jobs in allcpu, allgpu |
xfel-sim | xfel-op | allcpu. allgpu, xfel-guest | cancel | termination by jobs in xfel-op. terminates jobs in xfel-guest, allcpu, allgpu |
For a detailed documentation visit schedmd's preemption reference.
Setting preemption mode
you can override the requeue-mode:
sbatch --no-requeue # preempted jobs will be cancelled, not requeued sbatch --requeue # preempted jobs will be requeued, not cancelled