Job Submission Event Workflow

An overview of the events that occur once a job is submitted.

Once the configuration is complete, a job can be submitted and consumption of the site specific currency, as defined by the Budget Manager hook, can be tracked. The below image describe the sequence of events that occur after submitting a job. In this example, the formula defined in the Budget Manager hook calculates currency as cpu_hours = ncpus * walltime, therefore ncpus and walltime must be specified at job submission.


Figure 1. Job Submission Event Workflow

Submit a Job

Submit a job using the qsub -P <projectname> option so that PBS knows to track the HPC resource usage against a Project Account.

If you want to track HPC resource against a User Account, then use qsub -P <username>.

The hook uses the ncpus and walltime requested at job submission to calculate the amount of credit to assign to the job (Step 2).

Check Credit Availability

Budget Manager calculates the amount of credit to assign to a job based on the Service Unit assigned to the Account. In the example above, the Service Unit assigned to the Project Account is cpu_hours, therefore job credit is calculated as ncpus requested * walltime requested = 4 CPUs * 2 hours = 8 cpu_hours.

Once the job credit is calculated, Budget Manager determines if there is enough currency in the Account to run the job. When there is no available credit, two options are available (configurable by updating the am.conf file):
  1. The job is rejected.
  2. The job is placed into a queue. At run time, if there is not enough credit for the job to run, the job is returned to the queue for later consideration.
Otherwise if there is enough credit, the job acquires the credit and is scheduled to run. The Account is decremented by the amount of credit assigned to the job.

Job Completes

Once the job completes, Budget Manager determines the actual amount of currency consumed by the job based on the Service Unit assigned to the Account. In this example, the hook uses the ncpus and walltime used by the job to calculate the amount of currency consumed by the job. The currency consumed is calculated as ncpus used * walltime used = 4 CPUs * 1 hr = 4 cpu_hours.

Reconcile

The currency that was credited to the job, but unconsumed is placed back in the Account. In this example, 8 cpu_hours was assigned to the job. Only 4 cpu_hours were consumed, therefore 4 cpu_hours is credited back to the account.