.. SPDX-License-Identifier: Apache-2.0 Copyright (C) 2020-2022 OKTET Labs Ltd. All rights reserved. te-parent: te_engine .. index:: pair: group; Dispatcher .. _doxid-group__te__engine__dispatcher: Dispatcher ========== .. toctree:: :hidden: .. _doxid-group__te__engine__dispatcher_1te_engine_dispatcher_introduction: Introduction ~~~~~~~~~~~~ Dispatcher is a subsystem providing a proper initialization and shutdown of the TEN subsystems. It prepares the environment (creates directories for temporary files, exports environment variables, etc.), initiates building, if necessary, and initializes TEN applications according to options provided on the command line. From user point of view :ref:`Dispatcher ` is a BASH script that launches processes and TEN components according to specified command line options. During its operation :ref:`Dispatcher ` sets a few environment variables among which the most important are: * TE_BASE Location of Test Environment sources. If the Dispatcher script is called from the source directory, this variable is exported automatically. Otherwise if building is necessary (i.e., TE is not pre-installed), TE_BASE should be exported manually. * TE_BUILD This variable is exported automatically unless already exported. It is set to a start directory (a directory from which the Dispatcher script is called) or, if a file configure.ac is present in the start directory, to the (created if needed) build subdirectory of the start directory: [start directory]/build. * TE_INSTALL This variable is passed as the value of the prefix option to the main configure script. Moreover, its value is used when path variables for the search of headers and libraries are constructed. It may be set manually. If it is empty, it is set to the directory where the Dispatcher script is located (if the installed Dispatcher script is used) or to ${TE_BUILD}/inst (if the Dispatcher script from the source directory is used). * TE_INSTALL_SUITE User may export this variable to specify the location of Test Suite executables (for :ref:`Builder ` and :ref:`Tester `). If this variable is empty, it is set automatically to ${TE_INSTALL}/suites. * TE_TMP This variable is set by :ref:`Dispatcher ` to [start directory]/te_tmp by default. However, if it's desirable to use some other directory for temporary files, it may be exported manually. * LD_LIBRARY_PATH This variable is exported by :ref:`Dispatcher ` automatically and used for shared library search. It is set to ${TE_INSTALL}/[host platform]/lib. * PATH Path to TEN executables is provided automatically by :ref:`Dispatcher `. It updates PATH variable by ${TE_INSTALL}/[host platform]/bin. Moreover, if scripts provided by :ref:`Logger `, :ref:`Builder ` and storage library to :ref:`Dispatcher ` are not installed yet, :ref:`Dispatcher ` adds to PATH variable path to these scripts in the source directory. * TE_LOG_DIR Directory to store log files. Usually set to TE_RUN_DIR which in it's turn is set to the current directory (PWD). .. _doxid-group__te__engine__dispatcher_1te_run_time: Start/stop sequence ~~~~~~~~~~~~~~~~~~~ The following sequence of events happen each time when you launch Test Environment with dispatcher.sh or run.sh script: #. :ref:`Dispatcher ` script starts with some command line options (for more information on :ref:`Dispatcher ` options read :ref:`Dispatcher Command Line Options `); #. :ref:`Dispatcher ` runs te_log_init script to initialize script based logging facility. All further actions can be logged via script based interface (te_log_message script). Please note that :ref:`Logger ` application hasn't started yet; #. :ref:`Dispatcher ` starts :ref:`Builder ` to prepare libraries and executables for all TE Subsystems (except :ref:`Dispatcher `), Test Packages, Test Agents and bootable NUT image(s). :ref:`Builder ` is passed a configuration file that describes a set of executables to be built with a set of options for building process. :ref:`Builder ` configuration file name specified via conf-builder option of :ref:`Dispatcher `. (For information about :ref:`Builder ` configuration file read :ref:`Builder configuration file `). Please note that traces of building process are output into the console (they are not accumulated in log file); #. As soon as :ref:`Builder ` successfully built and installed all required components, :ref:`Dispatcher ` starts launching :ref:`Test Engine ` componentns. First component to start is :ref:`Logger `. :ref:`Logger ` is passed a configuration file whose name can be specified via conf-logger :ref:`Dispatcher ` command line option (for information about the format of :ref:`Logger ` configuration file refer to :ref:`Configuration File `). :ref:`Logger ` starts listening for incoming log requests that can come from tests and other TEN components; #. :ref:`Dispatcher ` starts :ref:`Remote Control Facility (RCF) `. :ref:`Remote Control Facility (RCF) ` is passed a configuration file that describes Test Agents to be started (for information about the format of :ref:`Remote Control Facility (RCF) ` configuration file refer to :ref:`RCF Configuration File `). As a part of initialization :ref:`Remote Control Facility (RCF) ` establishes communication with Test Agents using Test Protocol; #. As soon as :ref:`Remote Control Facility (RCF) ` has initialized, :ref:`Logger ` starts a thread that is responsible for polling Test Agents in order to gather log messages accumulated on Test Agent side. Polling interval is configured via :ref:`Logger ` configuration file; #. :ref:`Dispatcher ` starts :ref:`Configurator `. :ref:`Configurator ` is passed a configuration file that describes configuration objects to register as well as object instances to add (for information about the format of :ref:`Configurator ` configuration file refer to :ref:`Configurator Configuration File `). On start-up :ref:`Configurator ` retrives configuration information from Test Agents and initializes local trees of objects and instances; #. :ref:`Dispatcher ` starts :ref:`Tester `. :ref:`Tester ` processes configuration file and if necessary asks :ref:`Builder ` to build test suites (test executables). Then :ref:`Tester ` processes test package description files and runs tests in corresponding order and with specified set of parameter values. (For information about :ref:`Tester ` configuration file format refer to :ref:`Configuration File ` section). Before running tests, :ref:`Tester ` asks :ref:`Configurator ` to make a backup of configuration tree. When all tests are finished :ref:`Tester ` restores the initial configuration from initial backup. To prevent tests from interfering, a backup is created and optionally restored before each test as well. #. When :ref:`Tester ` returns (all tests finished), :ref:`Dispatcher ` stops :ref:`Configurator `; #. Flushing of the log from all Test Agents is performed; #. :ref:`Dispatcher ` stops :ref:`Remote Control Facility (RCF) `. During its shutdown, :ref:`Remote Control Facility (RCF) ` performs a shutdown of all Test Agents; #. :ref:`Dispatcher ` stops :ref:`Logger `. :ref:`Dispatcher ` calls Report Generator tool to convert the log from a raw format to the text and/or HTML format; #. :ref:`Dispatcher ` script finishes its work. .. _doxid-group__te__engine__dispatcher_1logs_publishing: Publishing logs to Bublik web application ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Testing logs may be published to Bublik web application using Dispatcher option `--publish` or a standalone script `scripts/publish_logs`. A path to test suite and site specific logs publishing script should be passed via `--publish` option of Dispatcher or `--script` option of `scripts/publish_logs`. TE will then create a tar archive containing testing metadata (`meta_data.json`) and raw log bundle (`raw_log_bundle.tpxz`), and pass it as the only argument to the script that you provide. It is assumed that your script will then copy the tar archive to log storage server, extract it there in a proper place and request Bublik web application to import logs from the corresponding URL. .. _doxid-group__te__engine__dispatcher_1te_engine_dispatcher_options: Dispatcher Command Line Options ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The Dispatcher script accepts a lot of command-line options. Some of them are its own options, and the rest are passed through to other TE subsystems. Here is the complete list of the dispatcher.sh script options as well as their descriptions obtained by calling it with help option: The Dispatcher script accepts a lot of command-line options. Some of them are its own, and the rest are passed through to other TE subsystems: an option starting with ``--tester-`` goes to :ref:`Tester `, ``--logger-`` to :ref:`Logger `, ``--trc-`` to :ref:`Test Results Comparator `, and so on. There are well over a hundred of them, so the sections below name the ones you are likely to reach for day to day. The complete list, generated from ``dispatcher.sh --help``, follows at the end of the page --- and you can always get the same text by running: .. code-block:: none ./dispatcher.sh --help Choosing configuration files ---------------------------- ``--conf-dir=`` points at the directory holding the configuration files, and ``--conf-dirs=:`` at a colon-separated list of them (highest priority first). Within those directories each subsystem picks up its own file, and each can be overridden individually: =================== ================== ============================================================== Option Default file Subsystem =================== ================== ============================================================== ``--conf-builder=`` ``builder.conf`` :ref:`Builder ` ``--conf-cs=`` ``cs.conf`` :ref:`Configurator ` ``--conf-logger=`` ``logger.conf`` :ref:`Logger ` ``--conf-rcf=`` ``rcf.conf`` :ref:`Remote Control Facility (RCF) ` ``--conf-rgt=`` ``rgt.conf`` :ref:`Report Generator Tool ` ``--conf-tester=`` ``tester.conf`` :ref:`Tester ` =================== ================== ============================================================== Command lines get long quickly, so ``--opts=`` reads further options from a file. Test suites normally wrap all of this in their own ``run.sh``; see :ref:`TE Execution `. Building -------- ``--no-builder`` skips the build entirely and ``--build-only`` does the opposite --- build everything, run nothing. ``-n`` is the shorthand for building nothing at all, neither TE nor the test suites. ``--build-from-scratch`` throws away previous build artefacts, which is what you want after changing the toolchain or the :ref:`Builder ` configuration. ``--build-parallel[=num]`` builds with several jobs, ``--builder-debug`` makes the build verbose when you need to see what it is actually doing, and ``--profile-build=`` records where the time went. Selecting what to run --------------------- ``--tester-run=`` is the option you will use most --- it runs the tests under the given path. Its relatives are ``--tester-run-from=``, ``--tester-run-to=``, ``--tester-exclude=`` and ``--tester-run-while=``. ``--tester-fake=`` walks the scenario without running anything, which is the quickest way to check that a path selects what you expected. A test path is more than a directory name; it can pin parameter values and iterations:: --tester-run=mysuite/mypkg/mytest:p1={a1,a2} --tester-run=mysuite/mypkg/mytest%3*10 The first runs every iteration where parameter ``p1`` is ``a1`` or ``a2``; the second runs the third iteration ten times. ``--tester-req=`` filters by requirements instead, and ``--tester-no-reqs`` ignores requirements altogether. Skipped iterations are quiet by default; ``--tester-verbskip`` logs them. Logs ---- The raw log is written to ``tmp_raw_log`` in the log directory (``--log-dir=``). What you get out of it depends on which of these you ask for: ``--log-txt=`` (text, on by default), ``--log-html=`` (browsable HTML, the most useful during development), ``--log-plain-html=``, ``--log-json=`` and ``--log-junit=`` for CI. ``--live-log`` runs :ref:`Report Generator Tool ` in live mode so you can watch the run as it happens. ``--publish=