Configure Container Integration

Container Support

Linux containers can be leveraged to constrain the amount of system resources used by jobs. Accelerator's container support is designed to be agnostic of the container solution. The examples provided are for Docker specifically.

Note: It is currently recommended to exclude contained jobs from preemption. This can be done at job submit time using the "-preemptable 0" submission option or by writing preemption rules to exclude jobs that request a "Container:X" resource.

Named Container Configurations

Containers are enabled by the administrator through named configurations that can be requested as a job resource. Each named configuration will contain a recipe of hooks to call to setup the container, and limits to impose upon it.

Named configuration files must be stored in the SWD/containers directory and end with a .cfg extension. Examples named configuration files are provided in the $VOVDIR/etc/config/containers directory, such as:

File: Container C1 definition:

### Example named-container configuration c1.

## Hook definitions
#
# containerHook <type> <mode> <path> <signature>
#   <type>      = setup | enter | cleanup | teardown
#   <privilege> = user  | root
#   <file>      = absolute path to hook file
#   <signature> = output of vovsignfile command
#
# Specifies which hooks should be called to interact with the container
# platform throughout the job's life cycle:
#
# 1. The setup hook is required if the container must be created prior to
#    spawning the job. This hook will always be used if root privileges are
#    required to create containers.
# 2. The enter hook is always required and is responsible for placing the
#    job into the container. If root privileges are not required to create
#    containers and the container platform supports it, the enter hook can
#    also create the container, as would be done via the "docker run" command,
#    for example. The enter hook must block for the duration of the job.
# 3. The cleanup hook can be used to remove temporary artifacts that are
#    generated by the job.
# 4. The teardown hook is required to stop and/or remove the container. This
#    hook would normally be used if a setup hook is used to create the
#    container.
#
# Any root-mode hook must be owned by root on the filesystem.

# containerHook setup    root "/path/to/container-setup.sh"    1211238920
containerHook enter    user "/path/to/c1-enter.sh"    2980970508
# containerHook cleanup  user "/path/to/container-cleanup.sh"  5376821904
# containerHook teardown root "/path/to/container-teardown.sh" 8237156649

## Limit definitions
#
# containerLimit <CORES | RAM | TMP>
#
# Specifies which resources should be limited in the container. For each limit
# specified, the job's resource expression will be searched for a corresponding
# request. If found, the value of the request will be set as an environment
# variable in the execution environment so that the container hooks can use it.
# The environment variable will be in the form of:
#
#   VOV_CONTAINER_<resource>

# containerLimit CORES
# containerLimit RAM
# containerLimit TMP

When a user includes Container:c1 in the resource request for their job, the request will be passed to the tasker that is selected to execute the job and the tasker will process the recipe defined in the configuration. If the recipe references a hook that does not exist, is not executable, or has a different signature than the one specified in the configuration, the job will fail.

Specify the Taskers that Support Containers

Each named container configuration will require a Container:X resource to be offered by every tasker that supports that specific container configuration. This can be done via the tasker.tcl file or the TaskerClass Table File file.

Hooks

Multiple hooks can be utilized to integrate with the container solution. At a minimum, an "enter" hook will be required, as long as the container solution provides a single command to setup the container, run the job, and remove the container afterward. For container solutions that do not provide this feature, separate setup and teardown hooks can be configured. For either method, an optional cleanup hook can be configured to remove artifacts generated by the job from the file system, for example.

Hooks must also be stored in the SWD/containers directory and can be in script or binary form. Example hook scripts are provided in the $VOVDIR/etc/config/containers directory, such as:

File: c1-enter.sh

#!/bin/bash -fxv

# Note: The enter hook must block for the duration of the job.

# Container c1 example enter hook: an all-in-one script that creates a container
# based on the "ubuntu" image, launches a job, then exits and removes the
# container.

# "$@" will contain "vw cmd args"
# The following environment variables are available and should be used to avoid
# container and host name conflicts:
#   HOST                (string)
#   VOV_PROJECT_NAME    (string)
#   VOV_JOBSLOT         (number)
#   VOV_CONTAINER_NAME  (string)

# The following environment variables are available if defined in the named
# container configuration file:
#   VOV_CONTAINER_CORES (number)
#   VOV_CONTAINER_RAM   (megabytes)
#   VOV_CONTAINER_TMP   (megabytes)

containerName=${VOV_PROJECT_NAME}_${HOST}_${VOV_CONTAINER_NAME}_${VOV_JOBSLOT}
hostName=${HOST}-${VOV_CONTAINER_NAME}
uid=`id -u ${USER}`
gid=`id -g ${USER}`

### Process limits into docker options
limitOptions=""
if [[ -n $VOV_CONTAINER_CORES && $VOV_CONTAINER_CORES > 0 ]]; then
  limitOptions+=" --cpus $VOV_CONTAINER_CORES"
fi
if [[ -n $VOV_CONTAINER_RAM && $VOV_CONTAINER_RAM > 0 ]]; then
  ramSpec="${VOV_CONTAINER_RAM}m"
  limitOptions+=" --memory $ramSpec"
fi
if [[ -n $VOV_CONTAINER_TMP && $VOV_CONTAINER_TMP > 0 ]]; then
  # Use in-memory tmpfs for /tmp in container
  tmpBytes=$(($VOV_CONTAINER_TMP*1048576))
  limitOptions+=" --mount type=tmpfs,destination=/tmp,tmpfs-size=$tmpBytes"
fi

# Capture job environment in a file for Docker to import
envFile=/tmp/${containerName}.env
env > $envFile

# Use the Docker "run" command to create a container based on the "ubuntu" image,
# setup networking, user and host handling, capture environment, and bind-mount
# required directories/files for job. Finally, the job itself is passed in for
# Docker to execute.
#
# The example below demonstrates the bind-mounts
# needed for the container to recognize the locally-defined user and group, the user's
# home directory, the project space (/data), and the Ubuntu libraries required by
# VOV software. These libraries can be included in a custom image and those bind-mounts
# would no longer be necessary.
docker run --rm \
  --network host --user "${uid}:${gid}" --hostname $hostName \
  --name $containerName --workdir $PWD --env-file $envFile \
  $limitOptions \
  --mount type=bind,source=/etc/passwd,target=/etc/passwd,readonly \
  --mount type=bind,source=/etc/group,target=/etc/group,readonly \
  --mount type=bind,source=${HOME},target=${HOME} \
  --mount type=bind,source=/data,target=/data \
  --mount type=bind,source=/lib/x86_64-linux-gnu/libexpat.so.1,target=/lib/x86_64-linux-gnu/libexpat.so.1,readonly \
  --mount type=bind,source=/lib/x86_64-linux-gnu/libpng12.so.0,target=/lib/x86_64-linux-gnu/libpng12.so.0,readonly \
  --mount type=bind,source=/usr/lib/x86_64-linux-gnu/libfontconfig.so.1,target=/usr/lib/x86_64-linux-gnu/libfontconfig.so.1,readonly \
  --mount type=bind,source=/usr/lib/x86_64-linux-gnu/libfreetype.so.6,target=/usr/lib/x86_64-linux-gnu/libfreetype.so.6,readonly \
  --mount type=bind,source=/usr/lib/x86_64-linux-gnu/libxcb.so.1,target=/usr/lib/x86_64-linux-gnu/libxcb.so.1,readonly \
  --mount type=bind,source=/usr/lib/x86_64-linux-gnu/libX11.so.6,target=/usr/lib/x86_64-linux-gnu/libX11.so.6,readonly \
  --mount type=bind,source=/usr/lib/x86_64-linux-gnu/libXau.so.6,target=/usr/lib/x86_64-linux-gnu/libXau.so.6,readonly \
  --mount type=bind,source=/usr/lib/x86_64-linux-gnu/libXdmcp.so.6,target=/usr/lib/x86_64-linux-gnu/libXdmcp.so.6,readonly \
  --mount type=bind,source=/usr/lib/x86_64-linux-gnu/libXft.so.2,target=/usr/lib/x86_64-linux-gnu/libXft.so.2,readonly \
  --mount type=bind,source=/usr/lib/x86_64-linux-gnu/libXrender.so.1,target=/usr/lib/x86_64-linux-gnu/libXrender.so.1,readonly \
  ubuntu "$@"

  rm -f $envFile

Note: Pay close attention to the comments in each hook example. Failure to follow the guidelines provided will likely result in failure to integrate with the container solution, as well as job failure. Any root-mode hook must be owned by root on the filesystem.

Hook Signatures

Each hook that is defined must contain a signature. The signature is used by the tasker to verify the hook has not been tampered with since it was defined by the administrator. The signature can be obtained by using the vovsignfile utility.

vovsignfile

Utility to obtain a security signature for files.


vovsignfile: Usage Message
  
      Utility to obtain a security signature for files.
  
      USAGE:
  
        vovsignfile [OPTIONS] <FILE>
  
      OPTIONS:
  
        -h    -- Show usage syntax.
  
      EXAMPLES:
  
        % vovsignfile -h
        % vovsignfile /path/to/file