NUMA Control and CPU Affinity

Non-Uniform Memory Access (NUMA) is to be used with machines that have multiple physical CPUs.

The performance of accessing RAM from a CPU depends on whether the RAM is physically attached to the same CPU or to another CPU. Therefore, application performance can be enhanced by constraining the application to stay within a single physical CPU. A Linux command, numactl, supports this control. For more information, use man numactl.

Accelerator can also automatically set the CPU affinity of an application in order to maximize its execution performance. The vovtasker automatically computes the socket, core, and memory layout of the machine on which it is executing. For each job that requests NUMA control, both the CORES and RAM resources for the job are used to determine where in the layout the job should be placed, depending on the placement type requested.
Note:
  • If a job with NUMA control requested is executed on a machine with a single socket, NUMA will be ineffective, but no problems will occur.
  • NUMA is supported only on Linux machines.

Placement Types

This version allows two different placement types for NUMA jobs: pack and spread. In both cases, the affinity of each single job is constrained to one or more physical sockets and have the following differences in behavior:
  • Pack: the placement selects CPUs in a way that minimizes the number of unused cores in each CPU.
  • Spread: the placement selects CPUs in a way that minimizes the loading of each CPU.

Examples of Job Submission with NUMA

% nc run -r CORES/2 -jpp pack -- my_job
% nc run -r CORES/4 -jpp spread -- my_job

To monitor the effect of CPU affinity, check the job property named "NUMA_AFFINITY":

For example:
% nc list -O "@ID@ @PROP.NUMA_AFFINITY@ @STATUSNC@"
283569646 NUMA pack:    0   *   *   *   *   *   *   *   *   *   *   *   *   *   *   *  Running
283569650 NUMA pack:    *   1   2   *   *   *   *   *   *   *   *   *   *   *   *   *  Running
283569654 NUMA pack:    *   *   *   *   4   5   6   7   *   *   *   *  12  13   *   *  Running
283569658 NUMA spread:  *   *   *   *   *   *   *   *   *   *   *   *   *   *  14   *  Running
283569662 NUMA spread:  *   *   *   *   *   *   *   *   *   *   *   *   *   *   *  15  Running
283569665 NUMA spread:  *   *   *   3   *   *   *   *   *   *   *   *   *   *   *   *  Running

Check Tasker NUMA Status

Each tasker sets and maintains a property on itself named "NUMA_LAYOUT". The property is updated each time a job with a NUMA request begins or ends. To check the current NUMA status for a tasker named "foo": 
% nc cmd vovselect prop.NUMA_LAYOUT from taskers where name==foo
Original:           Used/Total    _________+__
Socket:  0   RAM=    512/32089    ****oooooooo
Socket:  1   RAM=   1024/32089    ********oooo