Dispatcher
Introduction
Dispatcher is a subsystem providing a proper initialization and shutdown of the TEN subsystems. It prepares the environment (creates directories for temporary files, exports environment variables, etc.), initiates building, if necessary, and initializes TEN applications according to options provided on the command line.
From user point of view Dispatcher is a BASH script that launches processes and TEN components according to specified command line options.
During its operation Dispatcher sets a few environment variables among which the most important are:
TE_BASE
Location of Test Environment sources. If the Dispatcher script is called from the source directory, this variable is exported automatically. Otherwise if building is necessary (i.e., TE is not pre-installed), TE_BASE should be exported manually.
TE_BUILD
This variable is exported automatically unless already exported. It is set to a start directory (a directory from which the Dispatcher script is called) or, if a file configure.ac is present in the start directory, to the (created if needed) build subdirectory of the start directory: [start directory]/build.
TE_INSTALL
This variable is passed as the value of the prefix option to the main configure script. Moreover, its value is used when path variables for the search of headers and libraries are constructed. It may be set manually. If it is empty, it is set to the directory where the Dispatcher script is located (if the installed Dispatcher script is used) or to ${TE_BUILD}/inst (if the Dispatcher script from the source directory is used).
TE_INSTALL_SUITE User may export this variable to specify the location of Test Suite executables (for Builder and Tester). If this variable is empty, it is set automatically to ${TE_INSTALL}/suites.
TE_TMP
This variable is set by Dispatcher to [start directory]/te_tmp by default. However, if it’s desirable to use some other directory for temporary files, it may be exported manually.
LD_LIBRARY_PATH This variable is exported by Dispatcher automatically and used for shared library search. It is set to ${TE_INSTALL}/[host platform]/lib.
PATH
Path to TEN executables is provided automatically by Dispatcher. It updates PATH variable by ${TE_INSTALL}/[host platform]/bin. Moreover, if scripts provided by Logger, Builder and storage library to Dispatcher are not installed yet, Dispatcher adds to PATH variable path to these scripts in the source directory.
TE_LOG_DIR Directory to store log files. Usually set to TE_RUN_DIR which in it’s turn is set to the current directory (PWD).
Start/stop sequence
The following sequence of events happen each time when you launch Test Environment with dispatcher.sh or run.sh script:
Dispatcher script starts with some command line options (for more information on Dispatcher options read Dispatcher Command Line Options);
Dispatcher runs te_log_init script to initialize script based logging facility. All further actions can be logged via script based interface (te_log_message script). Please note that Logger application hasn’t started yet;
Dispatcher starts Builder to prepare libraries and executables for all TE Subsystems (except Dispatcher), Test Packages, Test Agents and bootable NUT image(s). Builder is passed a configuration file that describes a set of executables to be built with a set of options for building process.
Builder configuration file name specified via conf-builder option of Dispatcher.
(For information about Builder configuration file read Builder configuration file).
Please note that traces of building process are output into the console (they are not accumulated in log file);
As soon as Builder successfully built and installed all required components, Dispatcher starts launching Test Engine componentns. First component to start is Logger. Logger is passed a configuration file whose name can be specified via conf-logger Dispatcher command line option (for information about the format of Logger configuration file refer to Configuration File).
Logger starts listening for incoming log requests that can come from tests and other TEN components;
Dispatcher starts Remote Control Facility (RCF). Remote Control Facility (RCF) is passed a configuration file that describes Test Agents to be started (for information about the format of Remote Control Facility (RCF) configuration file refer to RCF Configuration File).
As a part of initialization Remote Control Facility (RCF) establishes communication with Test Agents using Test Protocol;
As soon as Remote Control Facility (RCF) has initialized, Logger starts a thread that is responsible for polling Test Agents in order to gather log messages accumulated on Test Agent side. Polling interval is configured via Logger configuration file;
Dispatcher starts Configurator. Configurator is passed a configuration file that describes configuration objects to register as well as object instances to add (for information about the format of Configurator configuration file refer to Configurator Configuration File). On start-up Configurator retrives configuration information from Test Agents and initializes local trees of objects and instances;
Dispatcher starts Tester. Tester processes configuration file and if necessary asks Builder to build test suites (test executables). Then Tester processes test package description files and runs tests in corresponding order and with specified set of parameter values. (For information about Tester configuration file format refer to Configuration File section).
Before running tests, Tester asks Configurator to make a backup of configuration tree. When all tests are finished Tester restores the initial configuration from initial backup. To prevent tests from interfering, a backup is created and optionally restored before each test as well.
When Tester returns (all tests finished), Dispatcher stops Configurator;
Flushing of the log from all Test Agents is performed;
Dispatcher stops Remote Control Facility (RCF). During its shutdown, Remote Control Facility (RCF) performs a shutdown of all Test Agents;
Dispatcher stops Logger. Dispatcher calls Report Generator tool to convert the log from a raw format to the text and/or HTML format;
Dispatcher script finishes its work.
Publishing logs to Bublik web application
Testing logs may be published to Bublik web application using Dispatcher option –publish or a standalone script scripts/publish_logs. A path to test suite and site specific logs publishing script should be passed via –publish option of Dispatcher or –script option of scripts/publish_logs. TE will then create a tar archive containing testing metadata (meta_data.json) and raw log bundle (raw_log_bundle.tpxz), and pass it as the only argument to the script that you provide.
It is assumed that your script will then copy the tar archive to log storage server, extract it there in a proper place and request Bublik web application to import logs from the corresponding URL.
Dispatcher Command Line Options
The Dispatcher script accepts a lot of command-line options. Some of them are its own options, and the rest are passed through to other TE subsystems. Here is the complete list of the dispatcher.sh script options as well as their descriptions obtained by calling it with help option:
The Dispatcher script accepts a lot of command-line options. Some of them are
its own, and the rest are passed through to other TE subsystems: an option
starting with --tester- goes to Tester,
--logger- to Logger, --trc- to
Test Results Comparator, and so on.
There are well over a hundred of them, so the sections below name the ones you
are likely to reach for day to day. The complete list, generated from
dispatcher.sh --help, follows at the end of the page — and you can always
get the same text by running:
./dispatcher.sh --help
Choosing configuration files
--conf-dir=<directory> points at the directory holding the configuration
files, and --conf-dirs=<dir>:<dir> at a colon-separated list of them
(highest priority first). Within those directories each subsystem picks up its
own file, and each can be overridden individually:
Option |
Default file |
Subsystem |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Command lines get long quickly, so --opts=<filename> reads further options
from a file. Test suites normally wrap all of this in their own run.sh; see
TE Execution.
Building
--no-builder skips the build entirely and --build-only does the
opposite — build everything, run nothing. -n is the shorthand for building
nothing at all, neither TE nor the test suites.
--build-from-scratch throws away previous build artefacts, which is what you
want after changing the toolchain or the
Builder configuration.
--build-parallel[=num] builds with several jobs, --builder-debug makes
the build verbose when you need to see what it is actually doing, and
--profile-build=<logfile> records where the time went.
Selecting what to run
--tester-run=<testpath> is the option you will use most — it runs the
tests under the given path. Its relatives are --tester-run-from=,
--tester-run-to=, --tester-exclude= and --tester-run-while=.
--tester-fake=<testpath> walks the scenario without running anything, which
is the quickest way to check that a path selects what you expected.
A test path is more than a directory name; it can pin parameter values and iterations:
--tester-run=mysuite/mypkg/mytest:p1={a1,a2}
--tester-run=mysuite/mypkg/mytest%3*10
The first runs every iteration where parameter p1 is a1 or a2; the
second runs the third iteration ten times.
--tester-req=<expression> filters by requirements instead, and
--tester-no-reqs ignores requirements altogether. Skipped iterations are
quiet by default; --tester-verbskip logs them.
Logs
The raw log is written to tmp_raw_log in the log directory
(--log-dir=<dirname>). What you get out of it depends on which of these you
ask for: --log-txt=<filename> (text, on by default), --log-html=<dirname>
(browsable HTML, the most useful during development), --log-plain-html=,
--log-json= and --log-junit= for CI. --live-log runs
Report Generator Tool in live mode so you can watch
the run as it happens.
--publish=<script> hands the log bundle to a site-specific script; see
Publishing logs to Bublik web application.
Debugging a run
--tester-gdb=<testpath> and --tester-vg=<testpath> run the selected test
scripts under gdb or valgrind. --gdb-tester, --vg-tester, --vg-cs,
--vg-logger, --vg-rcf and --vg-engine do the same for the engine
applications themselves.
--test-wof stops before the jump to cleanup when a test fails, so you can
look at what was actually configured; --test-woc does it regardless of the
result. --cs-print-trees dumps the
Configurator object and instance trees.
Expected results
--trc-db=<filename> selects the
Test Results Comparator database to compare the run
against, --trc-tag=<TAG> picks the expectations for a particular
configuration, and --trc-html=<filename> writes the report. --trc-update
updates the database from the run — and --trc-init rewrites it from
scratch, so treat it with care.
Complete option list
Usage: dispatcher.sh [<generic options>] [[<test options> tests ]...
Generic options:
-q Suppress part of output messages.
-n Do not build TE, TA and any Test Suites
(cumulate no-builder and tester-no-build).
--force Never prompt.
--daemon[=<PID>] Run/use TE engine daemons.
--shutdown[=<PID>] Shut down TE engine daemons on exit.
--conf-dir=<directory> specify configuration file directory,
overrides previous value and --conf-dirs
(${TE_BASE}/conf or . by default)
--conf-dirs=<directories> specify list of configuration file directories
separated by colon (top priority first,
may be specified many times, appends to
--conf-dir).
In configuration files options below <filename> is full name of the
configuration file or name of the file in the configuration directory.
--conf-builder=<filename> Builder config file (builder.conf by default).
--conf-cs=<filename> Configurator config file (cs.conf by default).
--conf-logger=<filename> Logger config file (logger.conf by default).
--conf-rcf=<filename> RCF config file (rcf.conf by default).
--conf-rgt=<filename> RGT config file (rgt.conf by default).
--conf-tester=<filename> Tester config file (tester.conf by default).
--conf-nut=<filename> NUT config file (nut.conf by default).
--script=<filename> Name of the file with shell script to be
included as source.
--tester-script=<filename> Name of the file with shell script to be
included as source before starting Tester.
--finish-script=<filename> Name of the file with shell script to be
included as source right before exiting.
--live-log Run RGT in live mode.
--log-dir=<dirname> Directory where to save tmp_raw_log file
(used if TE_LOG_RAW is not set directly).
--log-html=<dirname> Name of the directory with structured HTML logs
to be generated (do not generate by default).
--log-plain-html=<filename> Name of the file with plain HTML logs
to be generated (do not generate by default).
--log-json=<dirname> Name of the directory for JSON logs
to be generated (do not generate by default).
--log-txt=<filename> Name of the file with logs in text format
to be generated (log.txt by default).
--log-txt-detailed-packets Include detailed packet dumps in text log.
--log-txt-line-prefix Add a prefix before every message line in text log.
--log-junit=<filename> Name of the file with logs in JUnit format
to be generated.
--no-builder Do not build TE and TA.
--no-nuts-build Do not build NUTs.
--no-tester Do not run Tester.
--no-cs Do not run Configurator.
--no-rcf Do not run RCF.
--no-run Do not run Logger, RCF, Configurator and Tester.
--opts=<filename> Get additional command-line options from file.
--export=<name=value> Export an environment variable.
--cs-include=<filename> CS configuration file to be included after
main configuration file(s).
--cs-print-trees Print configurator trees.
--cs-log-diff Log backup diff unconditionally.
--builder-debug Be more verbose when build.
--build-from-scratch Build everything from scratch.
--build-only Build TE, do not run RCF and Configurator,
build but do not run Test Suites.
--build-parallel[=num] Enable parallel build using num threads.
--build-colorize Preserve colouring escape codes in the build log.
The default is to strip off escape codes.
Setting TE_BUILD_COLORIZE env variable to 'yes'
has the same effect.
Note that log colourising only works with
Ninja 1.9 or later.
--profile-build=<logfile> Gather timings for the build process into <logfile>.
--no-rcf-cc-simple Do not execute simple RCF consistency checks.
--tester-suite=<name>:<path> Specify path to the Test Suite.
--tester-no-run Don't run any tests.
--tester-no-build Don't build any Test Suites.
--tester-no-trc Don't use Testing Results Comparator.
--tester-no-cs Don't interact with Configurator.
--tester-no-cfg-track Don't track configuration changes.
--tester-no-logues Disable prologues and epilogues globally.
--tester-only-req-logues Run only prologues/epilogues under which
at least one test will be run according to
requirements passed in command line. This
may not work well if your prologues can add
requirements on their own in /local:/reqs:.
--tester-req=<reqs-expr> Requirements to be tested (logical expression).
--tester-no-reqs Ignore requirements, run all possible
iterations.
--tester-quietskip Quietly skip tests which do not meet specified
requirements (default).
--tester-verbskip Force Tester to log skipped iterations.
--tester-cmd-monitor Specify command monitor to be run for all
tests in form [ta,]time_to_wait:command.
The following Tester options get test path as a value:
<testpath> := / | <path-item> | <testpath>/<path-item>
<path-item> := <test-name>[:<args>][%<iter-select>][*<repeat>]
<args> := <arg>[,<args>]
<arg> := <param-name>=<values> | <param-name>~=<values>
<values> := <value> | { <values-list> }
<values-list> := <value>[,<values-list>]
<iter-select> := <iter-number>[+<step>] | <hash>
For example,
--tester-run=mysuite/mypkg/mytest:p1={a1,a2}
requests to run all iterations of the test 'mytest' when its parameter
'p1' is equal to 'a1' or 'a2';
--tester-run=mysuite/mypkg/mytest%3*10
requests to run 10 times third iteration of the same test.
--tester-fake=<testpath> Don't run any test scripts, just emulate test
scenario.
--tester-run=<testpath> Run test under the path.
--tester-run-from=<testpath> Run tests starting from the test path.
--tester-run-to=<testpath> Run tests up to the test path.
--tester-exclude=<testpath> Exclude specified tests from campaign.
--tester-vg=<testpath> Run test scripts under specified path using
valgrind.
--tester-gdb=<testpath> Run test scripts under specified path using
gdb.
--tester-fail-on-leak Test scripts fail if valgrind detects a memory
leak (must be specified together with tester-vg).
--tester-random-seed=<number> Random seed to initialize pseudo-random number
generator.
--tester-verbose Increase verbosity of the Tester (the first
level is set by default).
--tester-quiet Decrease verbosity of the Tester.
--tester-out-tin Output Test Identification Numbers to terminal.
--tester-out-expected If result is expected (in accordance with TRC),
output the result together with OK.
--tester-interactive Interactive ask user for tests to run.
--tester-run-while=<result> Continue testing as long as tests produce results matching a given value
- passed (stop test if obtained result
is not PASSED)
- failed (stop test if obtained result
is not FAILED)
- expected (stop test if obtained result
is not the one that declared in TRC)
- unexpected (stop test if obtained
result is the one that declared in TRC).
--tester-run-until-verdict=<verdict>
Continue testing until a test produces a verdict matching the given
verdict string.
--tester-dial=<percentage> Choose randomly a given percentage of all
available test iterations.
--test-sigusr2-stop Stop all the testing when SIGUSR2 signal is received.
The default behaviour is to print a verdict in the
current test.
--test-wof Wait before jump to cleanup on test failure. Useful to
take a look at what's configured etc. Requires some
nodes in the /local:/test: tree.
--test-woc Wait before jump to cleanup regardless of test result.
--logger-foreground Run Logger in the foreground (useful for Logger debugging).
--logger-no-rcf Run Logger without interaction with RCF, i.e. without polling any
Test Agents (useful for Logger debugging).
--logger-check Check that log messages received from other TE components are
properly formatted before storing them in the raw log file.
--logger-listener=<confstr> Enable streaming live results to the specified listener.
Config string has the following format: <name>[:<runid>].
--logger-meta-file=<path> Send meta information to listeners. This option may only be specified
once.
--logger-max-size=<size> Maximum size of RAW log (4Gb by default;
negative for unlimited; may be specified in
units of G[igabytes]).
--logger-shut-timeout=<to> How long to wait for Logger shutdown, in
seconds (120 sec by default).
--trc-log=<filename> Generate bzip2-ed TRC log.
--trc-db=<filename> TRC database to be used.
--trc-tag=<TAG> Tag to get specific expected results.
--trc-key2html=<filename> File with key substitutions when output to HTML
report.
--trc-ignore-log-tags Ignore tags from log.
--trc-html=<filename> Name of the file for HTML report.
--trc-brief-html=<filename> Name of the file for brief HTML report.
--trc-html-header=<filename> Name of the file with header for all HTML
reports.
--trc-txt=<filename> Name of the file for text report
(by default, it is generated to stdout).
--trc-quiet Do not output total statistics to stdout.
--trc-comparison=<method> Specify the method to match parameter values in TRC
- exact (the default)
- casefold
- normalised (XML-style space normalization)
- tokens (the values are split into tokens which are
either sequences of XML name characters or single characters;
the matching is done on these lists; in additional, numeric
tokens are compared as numbers (so e.g. 10 and 0xa render equal).
--trc-update Update TRC database.
--trc-init Initialize TRC database (be careful,
TRC database file will be rewritten).
--vg-engine Run RCF, Configurator, Logger and Tester under
valgrind (without by default).
--vg-cs Run Configurator under valgrind.
--vg-logger Run Logger under valgrind (without by default).
--vg-rcf Run RCF under valgrind (without by default)
(without by default).
--vg-tester Run Tester under valgrind (without by default).
--gdb-tester Run Tester under gdb.
--tce Do TCE processing for all TCE-enabled components.
--tce=<list> Do TCE processing for specific components (comma-separated) or 'all'.
--no-meta Do not generate testing metadata.
--meta=<name>[:<value>] Set metadata variable.
--publish=<script> Use a given script to publish testing logs
(metadata must be enabled to do this).
--sniff-not-feed-conf Do not feed the sniffer configuration file
to Configurator.
--sniff=<TA/iface> Run sniffer on *iface* of the *TA*.
--sniff-filter=<filter> Add for the sniffer filter(tcpdump-like
syntax). See 'man 7 pcap-filter'.
--sniff-name=<name> Add for the sniffer a human-readable name.
--sniff-snaplen=<val> Add for the sniffer restriction on maximum
number of bytes to capture for one packet.
By default: unlimited.
--sniff-space=<val> Add for the sniffer restriction on maximum
overall size of temporary files, Mb.
By default: 64 Mb.
--sniff-fsize=<val> Add for the sniffer restriction on maximum
size of the one temporary file, Mb.
By default: 16 Mb.
--sniff-rotation=<x> Add for the sniffer restriction on number of
temporary files. This option excluded by
the *--sniff-ta-log-ofill-drop* option.
By default: 4.
--sniff-ofill-drop Change overfill handle method of temporary
files for the sniffer to tail drop.
By default overfill handle method is rotation.
--sniff-log-dir=<path> Path to the *TEN* side capture files.
By default used: ${TE_RUN_DIR}/caps.
--sniff-log-name=<pattern> *TEN* side log file naming pattern, the
following format specifies are supported:
- %a : agent name
- %u : user name
- %i : iface name
- %s : sniffer name
- %n : sniffer session sequence number
By default '%a_%i_%s_%n' is used. The pcap
extension will be added automatically.
--sniff-log-osize=<val> Maximum *TEN* side logs cumulative size for all
sniffers, Mb.
By default: unlimited.
--sniff-log-space=<val> Maximum *TEN* side logs cumulative size for one
sniffer, Mb. By default: 256 Mb.
--sniff-log-fsize=<val> Maximum *TEN* side capture file size for each
sniffer in Mb.
By default: 64 Mb.
--sniff-log-ofill-drop Change overfill handle method to tail drop.
By default overfill handle method is rotation.
--sniff-log-period=<val> Period of taken logs from agents, milliseconds.
By default: 200 msec.
--sniff-log-conv-disable Option to disable capture logs conversion
and merge with the main log.
Interrupt options:
Any test as well as prologue and epilogue may be stopped with a few ways:
Ctrl+C (SIGINT) Interrupt a test immediately and stop testing campaign
gracefully.
Ctrl+\ (SIGQUIT) Interrupt a test immediately and stop testing campaign
roughly (with core dump).
Environment variables defining where raw log is stored:
TE_LOG_RAW Where to save raw log file, by default tmp_raw_log
in directory specified by --log-dir (if provided)
or in the current directory.
TE_LOG_BUNDLE Where to save raw log bundle (tarball compressed
with pixz). If it is not set, raw log bundle is not
created.
The script exits with a status of zero if everything does smoothly and
all tests, if any tests are run, give expected results. A status of two
is returned, if some tests are run and give unexpected results.
A status of one indicates start up or any internal failure.