Implement Preemption

Preempt currently running jobs in order to run higher-priority work.

Preemption Scheduling

Overview of preemption and parameters.

PBS Professional provides the ability to preempt currently running jobs in order to run higher-priority work. This is called preemption or preemptive scheduling. PBS has two different approaches to specifying preemption:
  • You can define a set of preemption priorities for all jobs. Jobs that have high preemption priority preempt those with low preemption priority. Preemption priority is mostly independent of execution priority.
  • You can specify a set of preemption targets for each job. You can also set defaults for these targets at the server and queues. Preemption targets are jobs in specific queues or that have requested specific resources.

Preemption is a primetime option, meaning that you can configure it separately for primetime and non-primetime, or you can specify it for all of the time.

If preemption is enabled, a scheduler uses preemption as part of its normal pattern of examining each job and figuring out whether or not it can run now. If a job with high preemption priority cannot run immediately, a scheduler looks for jobs with lower preemption priority. A scheduler finds jobs in the lowest preemption level that have been started the most recently. A scheduler preempts these jobs and uses their resources for the higher-priority job. A scheduler tracks resources used by lower-priority jobs, looking for enough resources to run the higher-priority job. If a scheduler cannot find enough work to preempt in order to run a given job, it will not preempt any work.

A job running in a reservation cannot be preempted.

A job’s preemption priority is determined by its preemption level.

Parameters

Enable Job Preemption
Enables job preemption. Default is enabled for both prime and non-prime time.
Sort Preemption Jobs by Start Time
Whether jobs most eligible for preemption will be sorted according to their start times. Allowable values: “min_time_since_start”, or no preempt_sort setting. If set to “min_time_since_start”, first job preempted will be that with most recent start time. If not set, preempted job will be that with longest running time. Default is Default "min_time_since_start".

Use Preemption

Preempt currently running jobs in order to run higher-priority work.

Important: The Enable Job Preemption and Sort Preemption Jobs by Start Time parameters can only be updated if the cluster was added by a user with passwordless sudo permissions and therefore may not appear as a Scheduling parameter.
  1. Click the Configure tab.
  2. Choose the HPC to configure.

    Choose an HPC
    Figure 1. Choose an HPC
  3. Click Scheduling from the PBS Professional menu located on the left-hand side of the web page.
  4. Click located in the upper right-hand corner of the web page.
  5. Enable the Display Advanced Settings check box.
  6. Scroll down to the second list of Scheduling parameters.
  7. Click located to the right of Enable Job Preemption.
  8. Choose one of the following options to enable or disable preemption:
    • To enable preemption for both prime and non-prime time, enable Enable Job Preemption and enable both Prime and Non-Prime.
    • To enable preemption for only prime time, enable Enable Job Preemption and enable Prime and disable Non-Prime.
    • To enable preemption for only non-prime time, enable Enable Job Preemption and enable Non-Prime and disable Prime.
    • To disable preemption disable Enable Job Preemption.
  9. Click to save the changes.
  10. Scroll up to the first list of Scheduling parameters.
  11. Click located to the right of Sort Preemption Jobs by Start Time.
  12. For Sort Preemption Jobs by Start Time, specify whether jobs most eligible for preemption are sorted according to their start times:
    • Select "min_time_since_start" so that the first job preempted will be that with most recent start time.
    • Select no setting so that the preempted job will be that with longest running time.
  13. Click to save the changes.