Automatic Zipping and Unzipping Files

FlowTracer can automatically compress (zip) and uncompress (unzip) files in the flow using the utilities gzip and gunzip. This service is provided for all places that satisfy the following conditions:
  • In the database "FILE"
  • Have the "zippable" flag set
Rule 1: Compression
A zippable file is automatically compressed when all the jobs that require compression have completed successfully. The check to see if compression is required is performed about every 30 seconds.
Rule 2: Decompression
A zippable file is automatically uncompressed if any of the jobs that requires it is queued and ready to be executed. Jobs will not be executed with compressed inputs.

If a file is compressable, it can be in either the normal or the compressed state. For a file called 'X', if the file does not exist but 'X.gz' does, then the file is considered compressed.

Directing vovzip Jobs to Certain Hosts

The work to compress/uncompress a file is performed by a system job that is executed by the owner of the project. These jobs are scheduled to be executed by the appropriate vovtaskers.

To direct the vovzip jobs to particular hosts, create a tool map in the project's resources.tcl file, where the right-hand side of the map is the necessary resource.

For example, the following statement could be used:
vtk_resourcemap_set Tool:vovzip UNLIMITED vovzip_host

The vovtasker resource vovzip_host can be placed on the desired hosts in the project's taskers.tcl file. The number of concurrent vovzip jobs can be limited: instead of UNLIMITED, insert an integer such as 10.

Defining Zippable Files

In a flow description, the procedure Z can be used to define the files that are zippable. Example:
J vw cp aa bb
J vw cp bb cc
Z bb cc

Another method is using the command vovset with the subcommand zippable.

% vovset zippable SETNAME 1
% vovset zippable SETNAME 0
Using Tcl, vtk_place_get and vtk_place_set can be used to test and set the zippable flag.
set id [ vtk_place_find "FILE" "aa"]
vtk_place_get $id info
puts "The current value of the zippable flag is $info(zippable)"
puts "The current value of the zipped   flag is $info(zipped)"

set  info(zippable)  1
vtk_place_set $id info

Recommendations for Zippable Flags

Using the zippable flag is recommended for relatively large files, such as greater than 100kB. A tradeoff needs to be evaluated: the space consumed and flow complexity and additional CPU/IO time, versus the degree of the compressibility of the files. For example, JPEG and PDF files are already highly compressed; zipping these files may produce little or not benefit. Test vector files, however often compresses to 3-8% of their native size.

vovzip Jobs and Log Files

The information in this section may be helpful for troubleshooting zippable files. The jobs that compress and expand the zippable files:
  • Are created automatically by the vovserver.
  • Use the tool vovzip, supplied by FlowTracer.
  • Are retraced in "FAST" mode.
  • Their log file goes into the directory projectName.swd/vovzipdir.
  • The name of the log file is of the form FILEID.log, where FILEID is the VovId of the zippable file.