Preparing the Workflow
General
To prepare VirtualFlow for a workflow, the following needs to be done:
Preparation of the input-files Folder
input-files FolderAs described here in the Background and Principles section, the input-files folder only contains the input files which are used by the external programs which process the ligands in the workflow. The only exception can be the input ligand database, which can be stored in some other centralized location (to be shareable by multiple independent VirtualFlow workflows). However, if it is stored in the VirtualFlow workflow folder, then the input-files folder is naturally the proper location. The location which is used is specified in the control file.
Currently, VFLP does not require the input-files folder as described here:
VFVS does require the input-files folder for docking input files, as described here:
Preparation of the tools Folder
tools FolderThe tools folder is only used by the user, not by the workflow. But it needs to be setup properly, as the tools folder is used to set up the workflow folder (see below).
To prepare the tools folder of VirtualFlow, the following files have to be prepared:
The tools folder is the primary working directory of VirtualFlow. All VirtualFlow user commands (starting with vf_) have to be run with this directory.
Initial Control-file
The initial control file tools/templates/all.ctrl needs to be set up. All parameters are explained in within the file itself.
If one has the choice of submitting many small jobs (such as 1-CPU jobs), or fewer large jobs (e.g. multi-node jobs), the latter is generally preferable regarding the central task list. But this usually only becomes relevant with very large numbers of jobs.
Multiple different control files for different job-lines can be employed once the workflow folder was prepared (see below).
Central Task List
The central task list tools/templates/todo.all needs to be set up.
The file contains the ligand collections which should be processed by the workflow. One ligand collection corresponds to one task for the workflow, even though one collection can contain any number of ligands. For each collection one line is used in the central task list, and it has the format:
<tranch>_<collection name> <# of ligands in the collection>
Regarding the databases which we provide, e.g. the REAL database of Enamine, a so called collection lengths file (usually it has the namelength.all) which contains the lengths of the collections is provided along with the database on the download page. The collection lengths file and the todo.all file have the same format. The todo.all file can be either the full lengths.all file (i.e. when screening the entire library), or a subset of it. If all of the collections in the file should be screened, the file can simply be copied to tools/templates/todo.all.
If one wants to screen only a subset of the collections of the database, one can do that by extracting the subset of the length file wants to screen, and store them in the tools/templates/todo.all file. The selection of subsets is most easily done with grep command.
For example, if one wants to screen all the collections of tranches with names which start with the letter "A", this can be done by the command:
Here the caret symbol "^" indicates the beginning of a line, meaning that the letter "A" has to be at the beginning in order for a match to happen. For more detailed information about the grep command, see either the corresponding man-page (which can be accessed by running the command man grep or here online), or other online resources such as here.
The Job File Template
VirtualFlow uses a job file template for the job it submits into the resource manager (batch system). The job template needs to be adjusted according to the one's preferences and according to the cluster requirements and constraints.
In the particular the following settings in the job file template should set appropriately:
Memory requirements
Special settings required by the cluster
Email notification settings
The following settings should be left unchanged, as they are handled internally by VirtualFlow and can be set partially in the control file.
job name (can be partially set in the control file)
partition (can be set in the control file)
wall-time (can be set in the control file)
number of nodes (can be set in the control file)
number of cpus and job steps (can be set in the control file)
input/output log files
Some clusters might require additional modules to be loaded, e.g. if OpenBabel which is used by VFLP should be loaded as a module it needs to go into the job template (see also VFLP - Installation - External Packages).
Preparation of the workflow and output-files Folders
workflow and output-files FoldersGeneral
The workflow and output-files folders needs to be prepared before the workflow can be started for the first time. This is done with the command vf_prepare_folders.sh. The preparation of the workflow folder is based on the current configuration/settings of the tools folder, therefore the tools folder needs to be set up at first.
The command vf_prepare_folders.sh removes previously existing output-files and workflow folders if they are present, which can be the case if test runs were carried out before. Therefore this command has to be used with care as it can delete important files.
The preparation of the workflow folder by the command vf_prepare_folders.sh does the following steps automatically:
Deletion of previously existing
output-filesandworkflowfolders (if they exist)Creation of (new)
output-filesandworkflowfoldersCreation of the required sub-folder structure of the
workflowfolder, which consists of:workflow/ligand-collections/todo/workflow/ligand-collections/current/workflow/ligand-collections/done/workflow/ligand-collections/ligand-lists/workflow/ligand-collections/var/workflow/control/workflow/output-files/jobs/workflow/output-files/queues/workflow/job-files/main/workflow/job-files/sub/
Copying of the required (previously prepared) files from the tools folder to the workflow folder:
tools/templates/one-step.sh→workflow/job-files/sub/one-step.shtools/templates/one-queue.sh→workflow/job-files/sub/one-queue.shtools/templates/todo.all→workflow/ligand-collections/var/todo.originaltools/templates/all.ctrl→workflow/control/all.ctrl
The todo-file is split into smaller pieces (as specified in the ctrl-file via the setting central_todo_list_splitting_size), and the split files are stored in the
workflow/ligand-colllections/todo/folderThe file
workflow/job-files/sub/one-step.shis made executable
Control Files
The folder workflow/control/ contains the control files with settings which are used by the jobs/workflow. The main control all.ctrlalways has to be there. It is contains the word "all", because it used by all jobs by default.
However, it is possible to define settings for specific job-lines or ranges of job-lines. For that, one simply copies the file workflow/control/all.ctrl to workflow/control/<start-jobline-id>-<end-jobline-id>.ctrl, which means that this control file is used by all the jobs which have jobline IDs between <start-joblline-id> and <end-jobline-id>. These more specific control files are called range control files. They override most of the settings in the all.ctrl file for the relevant joblines, but not all settings. Some settings are global and always read from the all.ctrl file, therefore this file always needs to be there.
For example, the range control file workflow/control/4-9.ctrl would be used by all jobs/job-lines which have a jobline ID between 4 and 9 (see also Fig. 1 below).
Job Files
What is still missing in order to start the workflow are the main job files. These are prepared on the fly when the workflow is initialized/started (see the next section, Starting the Workflow).
Last updated
