FlowTracer For UNIX Users

FlowTracer Functionality

Consider a particular situation where obtaining a result involves running a group of steps called jobs that each run a program to perform an activity. Imagine that each job may build on what other jobs have done, and each job has a proper time to run in the sequence of jobs. Some jobs have dependencies on other jobs having been run, and may in turn enable other jobs to run that depend on them. The full sequence of jobs needs to be run in order to obtain the result.

The work to schedule and dispatch the jobs can become too difficult to manage by hand as the number of jobs increase and the rules of dependency get complicated.

FlowTracer is a product that can manage such a situation to schedule and run the jobs so that all dependencies are enforced and the minimal amount of work is performed to obtain the desired result.

Although the work of FlowTracer involves the scheduling and dispatching of jobs which are computer processes and which use file system resources, this section will not be concerned with those processes. This section will be concerned with the processes that implement the FlowTracer system, which perform the scheduling and dispatching.

Command Line Programs

FlowTracer is implemented with a client/server architecture. The server component is provided by a set of background daemon processes that work together to provide functionality. These daemon processes are started and stopped as desired.

One part of the client component is a set of programs that can be run from the command line to perform specific actions. A few of these will be covered in this section.

The User Interface section of this chapter will cover the full set of client components.

When you have a command line context that is enabled to run Altair Accelerator programs, you can start the FlowTracer system for the first time. This will start up a background daemon processes that provide the FlowTracer functionality. You will do this below, but first, you will learn about three command line programs and the concept of a project.

FlowTracer provides a rich set of interactive programs that can be used from the command line to control the product. These are called the Command Line Interface (CLI). These correspond to Chapter 1 commands in UNIX. They are programs that can be run from the command line or from shell scripts. You call them to perform a specific task to accomplish a desired goal. They are controlled and adjusted by command line parameters,

The Altair Accelerator programs have a naming convention of using the prefix 'vov' in front of a root subject name ('vovproject', 'vovconsole', 'vovarch', ...). This convention is used so that the program names do not clash with names of programs from another product. There are exceptions to this naming convention ('vsr', 'vsi', ..) where the program names are acronyms for phrases having 'vov' as the first word. The use of 'vov' as the naming convention comes from an Italian liquor named 'Vov'. A very early prototype of the system was started at a place and time when projects were named after liquors, and the Italian liquor "Vov" chosen as the name for the project.

Most of the CLI command programs interact with a running FlowTracer system to interact with it and manage it, or report about it. They are most useful when FlowTracer is actively running. A few CLI programs interact with the FlowTracer system, even when it is not running. These programs manage the static parts of the product ('vovversion', 'vovproject', 'vovarch', 'vovdoc',...), and are useful even when FlowTracer is not actively running.

Verify Access to FlowTracer

You can verify that your command line environment is set correctly to access FlowTracer by running one of the interactive programs that does not require that FlowTracer be running. A good one to use for verification is the CLI command vovarch, which reports the machine architecture you are using.

Example of simple command to verify access to FlowTracer:
% vovarch 

You can also ask the system to report on all the projects that have been set up within FlowTracer using the CLI command vovproject with the option list. If you are doing this for the first time here, running vovproject list will show an empty list. If you run vovproject list at a later time, you'll see whatever projects exist then.

Example of empty list of projects:
% vovproject list 

The vovproject CLI command is a work-horse command in the FlowTracer system. There are many options for the command that qualify what project operation it should do. It is like many UNIX commands in that it has a command line option to ask for help on its usage.

View usage message from vovproject:
% vovproject -h     

This is an example of asking for help that holds true for most FlowTracer CLI commands. The -h parameter is a request to output a usage statement.

Copyright © 1995-2021, Altair Engineering   www.altair.com

    The vovproject tool is used to manage a VOV project.
    You can create new projects, start and stop
    existing ones, enable the current shell to interact with a specific
    project, etc...

    If not otherwise specified, the tool assumes that the current project
    is the one defined by the environment variables VOV_PROJECT_NAME,

    vovproject [command] [command arguments]

    archive                 # Archive projects.
    unarchive               # Unarchive projects.
    backup                  # Make a backup copy of the trace
    create                  # Create a new project
    destroy                 # Completely remove all server data
    enable                  # Enable the shell to connect to the project
    info                    # Get info and stats about the project
    list                    # List all the projects
    rename                  # Change the name of the project
    report                  # Print reports about the projects
    reread|reset|refresh    # Re-read all configuration files
    restore                 # Restore the trace from a backup file
    sanity                  # Check internal consistency of server database
    save                    # Save trace database to disk
    start                   # Start a project server
    stop                    # Stop a project server

DETAILED HELP on above commands:
    vovproject [command] -help

As a UNIX user, you may be curious about the processes that run when the FlowTracer system is active. Use the UNIX ps -f command to see what is running before the FlowTracer system becomes an active one so you can have a baseline view to compare to what you'll see later when FlowTracer is running.

When FlowTracer is actively running, it is implemented as a group of programs running in the background as services. The group of programs cooperate with each other to perform the FlowTracer function. After getting FlowTracer running, use ps -f to view the set of UNIX processes that are running to implement the functionality.

FlowTracer is Project Oriented

The FlowTracer system is "project oriented". A user establishes a project for each use case where there is a goal that is obtained by running a large set of jobs having rules of dependencies. The large set of jobs and their dependencies make up a project that FlowTracer manages. Jobs work with files, using them as input, and producing files as output. The files in the project are stored in the file system structure within the project folder.

A FlowTracer project is the focus of all work that FlowTracer does. This is why most the CLI commands need a project context to target their actions to a given project. You should always be thinking about the project context whenever you do any actions with FlowTracer.

For instance, FlowTracer is started and stopped by starting and stopping a project. This matches the underlying process architecture. Each project has a separate group of FlowTracer daemon processes that work together to take care of that project.

To get FlowTracer started for the first time, you request that a FlowTracer project be both created and started. Subsequent requests to start a project will simply be to start it. Only the first time involves the create step. It is possible to create a project, and not start it at that moment.

You create a project and default to have it start running with the program vovproject using the option create with no other control parameters.

After running that command, you can see that the project exists and is running. Here are two imaginary sequence of commands to demonstrate how vovproject list would eventually show a project after it is created and started. In the next section below, there is an actual example to try.

Imaginary commands showing a project listed after creating it:
% vovproject create myNewProject > create.log 
% vovproject list 
1   myNewProject bill  unixhost  15918 running
% ps -f > ps.log
Imaginary commands showing a project created but not started right away:
% vovproject create -nostart myNewProject > create.log 
% vovproject list 
1   myNewProject bill  unixhost  auto* never started
% vovproject start myNewProject > start.log % vovproject list 
1   myNewProject bill  unixhost  6252  running
% ps -f > ps.log

Start FlowTracer by Creating a Project

This exercise is a way to see the simplest case of FlowTracer running. There is no productive work involved in the example project. This means that FlowTracer is not being asked to do the work it would normally do with regard to accepting a set of programs to schedule and dispatch based on dependency rules.

The example simply runs the FlowTracer product with the intent to show the framework, not to get something done with it. This discussion will not include detailed explanation about options in the commands, or the full story of exactly what happens in the background.

A user of FlowTracer would normally establish a working directory to be the top level directory of a project. When doing something useful, the project directory structure would hold the files that support the work. For this example, there is no work planned for the project, so the project directory structure will have nothing in it.

A convenient way of working is to use the project name as the name of the top level working directory. For this exercise, imagine that the project name is "see-it-run" to indicate the simple ambition of the project. Examples will be as if your user name is "bill" and that the top level working directory is in your home directory (/Users/bill) and it will be named after the project (/Users/bill/see-it-run).

Session activity to try - Create a project:
% cd 
% mkdir see-it-run
% cd see-it-run
% vovproject create see-it-run
Creating a new project:
        Directory /Users/bill/vov
        Type      generic
        Name      see-it-run
        Port      automatic
vovproject 09/11/2015 15:40:23: message: Creating directory /Users/bill/vov/see-it-run.swd/.
vovproject 09/11/2015 15:40:23: message: Created setup file '/Users/bill/vov/see-it-run.swd/setup.tcl'
vovproject 09/11/2015 15:40:23: message: Copy all files from /opt/rtda/201509/linux/etc/ProjectTypes/generic
vovproject 09/11/2015 15:40:23: message: Starting a VOV server
	for project    see-it-run@unixhost
vovproject 09/28/2015 15:40:23: message: Moving old resource logs to new location
vovserver(16372) Sep 11 15:40:23 VOVserver linux/2015.09 built:Sept 10 2015 17:41:32
vovserver(16372) Sep 11 15:40:23 Copyright © 1995-2021, Altair Engineering
vovserver(16372) Sep 11 15:40:23 Checking ServerInfo lock on file ./see-it-run.swd/serverinfo.tcl
vovserver(16372) Sep 11 15:40:23 The ServerInfo file is not locked
vovserver(16372) Sep 11 15:40:23 The server port is 6401
vovserver(16372) Sep 11 15:40:23 Sanity: Version linux/2015.09 built:Sept 10 2015 17:41:32
vovserver(16372) Sep 11 15:40:23 Sanity: Copyright © 1995-2021, Altair Engineering
vovserver(16372) Sep 11 15:40:23 Setting product name from project name see-it-run -> ft
vovserver(16372) Sep 11 15:40:23 Changing seatlic to 'seat_ft_l'
vovserver(16372) Sep 11 15:40:23 Server metrics not enabled (flags=0)
vovserver(16372) Sep 11 15:40:23 Running as product ft
vovserver(16372) Sep 11 15:40:23 Project was started on: Fri Sep 11 15:40:23 2015
vovserver(16372) Sep 11 15:40:23 Running with capacity 256 file descriptors
vovserver(16372) Sep 11 15:40:23 ###########  PRIMARY SERVER LISTENING ON PORT 6401  ###########
vovserver(16372) Sep 11 15:40:23 The server pid is 16372
vovserver(16372) Sep 11 15:40:23 << Redirecting output  /Users/bill/vov/see-it-run.swd/logs/server.2015.08.28_154023.log >>

Now the system is managing the "see-it-run" project.

You can list the projects by running the CLI command vovproject list.

Session activity to try - Show list of projects after creating a project:
% vovproject list 
1   see-it-run bill unixhost 6401 running

Note that the PORT number reported is used by the web server in FlowTracer as it waits for HTTP requests to open up the browser visual console.

Background Processes of FlowTracer

Use the ps -f command to see the group of background processes that are running now that the FlowTracer system is started. Notice the server pid that was reported in the output during the start project step. The running processes are the ones that implement the FlowTracer service, one group of processes per project.

Processes that manage a running project:
  15981 15980   -bash  
  16372     1   vovserver -p automatic -jsb see-it-run
  16394     1   vovsh -f /opt/rtda/201509/linux/tcl/vtcl/vovresourced.tcl -l see-it-run
  16395     1   vovsh -n -f /opt/rtda/201509/linux/scripts/vovdailylog ./vovresourced.@DATE@.log
  16421     1   launchctl bsexec / vovtasker -p see-it-run -h unixhost -r @STD@ 
                  -l ./see-it-run.swd/logs/taskers/localhost/20150911T122253/log 
                  -I - -i 1 -V - -R - -G - -S - 
                  -a localhost -g g1 -e - -b FULL -L - -C -1 -D 5 -T -1 -U 60 
                  -n 0 -c 1 -P -1 -M -1 -s
  16422 16421   vovtasker -p see-it-run -h unixhost -r @STD@ 
                  -l ./see-it-run.swd/logs/taskers/localhost/20150911T122253/log 
                  -I - -i 1 -V - -R - -G - -S - 
                  -a localhost -g g1 -e - -b FULL -L - -C -1 -D 5 -T -1 -U 60 
                  -n 0 -c 1 -P -1 -M -1 -s

You can see that a series of programs have been started, and have become child processes of the system (PPID is "1"). These are the background daemon processes that manage the project.

The program vovserver is the main daemon process. It is run with a parameter that tells it to manage the project see-it-run. It works with the other daemon programs to coordinate their activity.

One other program is vovtasker, the dispatch handler. This process is used to run the jobs that are dispatched by the vovserver program as part of its work to schedule jobs and dispatch them as needed. In this simple case, the default configuration is used which sets up a single dispatch handler on the host machine so that dispatched jobs are run locally. That is why the vovtasker program is running on the local machine.

If you set up multiple hosts to provide resources to run jobs, you can register them with FlowTracer as additional tasker hosts where a vovtasker program can be running as part of the FlowTracer daemon process team.

Note that a few processes are run by calling the program vovsh to interpret a Tcl script file. The Tcl script provides the processing logic that implements the action for the process. One of those processes monitors activity and writes a log file to report on events.

Working Directory for Projects

You can look in your home directory and see that it contains a directory named vov. This is the default location where information about the state of the FlowTracer system is kept. When you create a project, a working directory is created having a default location within the ~/vov directory. A different location for the project's working directory can be set when creating a project. You have created a project see-it-run using the default, which leads to the creation of the project working directory ~/vov/see-it-run.swd.

This project directory holds state information about the see-it-run project. There are directories within ~/vov/see-it-run.swd that hold log files. From the output of the ps command, you may notice a command line parameter referring to the path ../logs/tasker/localhost/nnnnn/log. This is where a log file will be written by the vovtasker program.

Stop FlowTracer by Stopping a Project

You can stop the see-it-run project now, and put it into a stopped state. This shuts down this instance of the FlowTracer service. The stopped state of the project is stored in the ~/vov/see-it-run.swd folder in such a way that it can be started again when requested. Again, because FlowTracer is project oriented, you stop it by requesting to stop the project using the vovproject CLI command with the stop option and the name of the project to stop.

This stop request has a confirm step to prevent accidentally stopping a project. First you run the vovproject stop [project-name] command, then after you qualify as having privileges to stop the project, you are prompted to confirm your request with a reply of a full "y-e-s".

Session activity to try - Stop a running project:
% vovproject stop see-it-run
vovproject 08/28/2015 16:18:58: message: Checking privilege to stop project 'see-it-run'
Shut down see-it-run (yes/no)? yes

vovproject 08/28/2015 16:19:05: message: Stopping all retracing
vovleader(96135) Aug 28 16:19:05 
vovproject 08/28/2015 16:19:05: message: Retracing stopped
vovproject 08/28/2015 16:19:05: message: Stopping all running jobs
vovproject 08/28/2015 16:19:05: message: Running jobs stopped
vovtaskermgr 08/28/2015 16:19:05: message: Stopping all 1 taskers.
vovleader(96140) Aug 28 16:19:08 Server see-it-run killed without question

Even though the FlowTracer project has been stopped, and the background processes have stopped running (Did you try ps -f after stopping the project?), you may still request a list of projects, as you did earlier. Listing projects does not require the FlowTracer system to be actively running.

Session activity to try - List existing projects, expecting it to be stopped:
% vovproject list 
1   see-it-run bill unixhost 6401 stopped

This time, the request to list projects shows the one just created and shows it in the stopped state. vovproject is a CLI command that runs fine even when FlowTracer is not started.

If you run the ps -f command to see running processes, you will see the main programs are no longer running. The event monitors and loggers are still running. They do not stop immediately. They will stop after a short time if the project is not started up again.

Start FlowTracer by Starting a Project

You can start the existing see-it-run project again with a simple start command.

Session activity to try - Start a stopped project:
% vovproject start see-it-run  > start.log

The redirect of the output to a log file is shown here to reduce the clutter of what is shown. In normal use, the output would not be redirected to a log and you'd see it interactively. The output is similar to what is shown during the request to create project that is shown above.

You can list the status of projects on the server in the same way as before and can see that the project has been started and is now running. Check out the processes that have started by running the ps -f command.

Session activity to try - List existing project, expecting it to be running:
% vovproject list 
1   see-it-run bill unixhost 6401 running

This section gave you the basic experience of managing an installed FlowTracer system after you set up a proper shell environment. You can understand that CLI commands are like all UNIX commands - they are programs that you run to do a particular thing. Read the VOV Subsystem Reference Guide as a UNIX command reference guide to discover more about what each command does.

By working through this section, you have now seen the basic aspects of FlowTracer. You can verify that your shell environment is correct to access FlowTracer. You can start and stop the FlowTracer system by starting and stopping projects. You can see the processes that are running to implement an actively running FlowTracer system. You can find the directories in the file system that hold project logs and project state.