Server Error Conditions

VOV and the jobs it runs depend on external resources such as available licenses, RAM, swap, tmp and disk space, which may become exhausted, damaged or otherwise unavailable. When users are subject to disk quotas, writes may fail when the disk is not completely full.

VOV License Violation

Listed below are possible causes for a license violation:
  • Invalid license file, such as wrong version or damaged
  • Expired license file
  • License server never started
  • No license server running, deliberately stopped or crashed
  • Licensed quantities exceeded, such as too many vovtaskers

Server License Violation Behavior

When a violation is detected, retracing and dispatching new jobs are stopped.

Disk Full and Quota - vovserver

The vovserver program tracks the free disk space on the file system where its working directory (such as PROJECTNAME.swd) is placed, and also on /usr/tmp where some temporary files are stored. The vovserver issues loud alerts when disk resources are exhausted.

The vovserver disk full warning is sticky, and persists after the disk has been cleaned up and has free space. The warning is sent to any client that tries to connect to the vovserver until the warning state has been cleared. The disk full warning causes most vovsh-based commands, such as vovconsole, to fail.

Because disk full is a severe error and can cause many cascading errors, to ensure the issue is noticed and addressed, this warning is deliberately sticky; this prevents a potentially critical alert from becoming a static message that is buried and unnoticed in a vovserver log file.

The vovproject sanity command can be used to clear the warning and return to normal operation. An example is shown below.

To clear disk full warning:
% vovproject enable your-project
thishost your-project@srvhost ENV dir> vovproject sanity
For Accelerator:
% /bin/su - FT-admin  # login as the owner of Accelerator
% nc cmd vovproject sanity

Disk Full and Quota - vovtasker

The vovtasker program also checks for free space on /usr/tmp on the host where it runs, and suspends itself (refuses to accept new jobs) if the amount falls below a configurable amount. The default is 5MB, which can be configured using the -mindisk option of vtk_tasker_define.

The tasker suspended condition is not sticky. After the tasker host disk has been cleaned up so that free space is above the threshold, the vovtasker will automatically resume and enter the ready state.