TE: User Guide

Introduction

This page gives step by step guideline on where to start with TE, how to get the sources, build them and run a simple test script.

TE can be supplied in two release types:

  • pre-installed form (external headers and binaries are available);

  • source code form (all sources of TE components are available).

If you have pre-installed binaries you can obviously skip the sources download and build part.

Getting TE sources

Test Environment is an open source project. Its sources live in https://github.com/ts-factory/test-environment and can be cloned by anyone:

git clone https://github.com/ts-factory/test-environment.git te

If you have a test suite which is not structurally embedded in the TE subtree (not in suites/) directory you also need to download its sources.

So in general case after all source code is retrieved from the repository one should have:

work_dir/
   te/
   my-ts/

Building TE components

When you get TE in source code form you need to build TE before it can be used for testing.

To build either the Test Engine or a Test Agent you first need a development toolchain. On Debian derivatives:

  • build-essential - pulls in gcc, g++, libc-dev and make;

  • pkg-config - used to locate every library listed below;

  • meson - at least 0.49.0, the version required by TE’s own meson.build;

  • ninja-build - the backend meson drives.

Same for Debian derivatives in one line:

apt-get install build-essential pkg-config meson ninja-build

Test Environment Engine dependencies

The Test Engine needs the following packages. What wants each one is named, so that a build failure is easier to place.

Package

Needed for

bash

at least 4.3; runs dispatcher.sh and the helper scripts

bison, flex

flex at least 2.5.31; the Tester test path parser, the test environment parser and logical expressions

gawk, file, wget, ssh

the build and run scripts; ssh is also how RCF reaches a Test Agent

libxml2-dev

at least 2.6.10; XML configuration files and raw log processing

xsltproc

filtering subtrees out of a Configurator backup; the command is run as it is, so the binary is what is needed, not the library headers

libpopt-dev

command line parsing in every engine application

libjansson-dev

JSON: Tester, Logger, Report Generator Tool and Test Results Comparator

libyaml-dev

the YAML configuration files: Configurator, Logger and RCF

libcurl4-openssl-dev

publishing logs from the Logger

libglib2.0-dev

Report Generator Tool

libssl-dev

Test Results Comparator

libpcre2-dev

log post-processing

libbsd-dev

string helpers in the common tools library; optional, the build falls back to its own implementations

Same for Debian derivatives in one line:

apt-get install bison flex gawk file wget ssh libxml2-dev xsltproc libpopt-dev libjansson-dev libyaml-dev libcurl4-openssl-dev libglib2.0-dev libssl-dev libpcre2-dev libbsd-dev

Optional libraries and packages:

  • libreadline-dev and libncurses-dev - enable the interactive Tester mode (the --tester-interactive option). Without them the build succeeds and the option is simply unavailable;

  • perl-Time-HiRes - package on Redhat/Fedora is very useful (it allows to avoid mixture in log because of unprecise timestamps in messages logged by Dispatcher (via logging script)).

Test Agent build dependencies

A default Test Agent build needs:

Package

Needed for

gawk, file, wget

the build and run scripts

libpcap-dev

packet capture and injection in the Traffic Application Domain

libtirpc-dev

Sun RPC; glibc no longer provides it, so this package is needed on any current distribution

libpcre2-dev

agent job control

libnl-3-dev

netlink, used for network configuration on Linux agents

openssh-server

not a build dependency: an ssh server has to run on the agent host, because RCF copies the agent there and starts it over ssh

Agents configured with extra features need more:

Package

Needed for

bison, flex

flex at least 2.5.31; the DNS and DHCP server configuration support, which is off unless the agent is built with those daemons

libssl-dev

OpenVPN daemon support

libbsd-dev

string helpers in the common tools library; optional, the build falls back to its own implementations

Same for Debian derivatives in one line:

apt-get install gawk file wget libpcap-dev libtirpc-dev libpcre2-dev libnl-3-dev openssh-server

Optional libraries and packages:

  • libelf-dev - BPF/XDP support on the agent;

  • libsnmp-dev - for SNMP support, used by the power control agent;

  • libyang-dev and libnetconf2-dev - NETCONF/RESTCONF RPCs;

  • libpam0g-dev - PAM support;

  • tcl-dev and expect - Tcl and expect support in Traffic Application Domain.

Test Agent run dependencies

Default Test Agent run depends on a set of 3-rd party libraries and packages:

  • file;

  • gawk;

  • ssh server, in Debian derivatives it is openssh-server.

Same for Debian derivatives in one line:

apt-get install file gawk openssh-server

Optional libraries and packages:

  • libsnmp - for SNMP support.

Build configuration

In order to build sources you will need a project-specific Builder configuration file, normally called builder.conf. For the details on the file format please read Builder configuration file.

If you’re dealing with an existing test suite the file was usually already written by the suite author or maintainer, and lives in the suite’s conf/ directory.

It says which TE libraries and tools to build for the engine, and what kind of Test Agent to build. Cut down to its essentials, it looks like this:

TE_PLATFORM([], [], [-D_GNU_SOURCE], [], [],
            [logger_core tools conf_oid asn ndn logic_expr ipc bsapi \
             loggerten rcfapi confapi comm_net_engine rcfunix trc tapi \
             rpcxdr rcfrpc rpc_types tapi_rpc tapi_env tapi_job])

TE_TOOLS([rgt trc])

TE_TA_TYPE([linux], [], [unix],
           [--with-rcf-rpc --with-libnetconf], [], [], [],
           [comm_net_agent asn ndn])

Every library a test links against has to appear in the TE_PLATFORM list here as well as in the suite’s meson.build. See ${TE_BASE}/suites/selftest/conf/builder.conf for a complete, working one.

Building

Before building you must export the TE_BASE environment variable pointing at the root directory of the Test Environment sources. As you usually work with one copy of TE it is worth putting

export TE_BASE=/path/to/TE_root_dir

into your ~/.bashrc. Suites that ship a scripts/guess.sh work it out themselves if TE is checked out next to them.

There is nothing to configure and no make to run: dispatcher.sh drives Builder, which drives meson. Do not invoke meson by hand.

To build TE on its own, without any test suite:

cd ${TE_BASE}
./dispatcher.sh --no-run

To build a suite together with TE, run the suite’s own run.sh and tell Tester not to run anything:

cd /path/to/my-ts
./run.sh --cfg=<configuration> --tester-no-run

run.sh is a wrapper around dispatcher.sh that adds suite-specific options and defaults; --cfg=<name> picks one of the suite’s configurations. See TE Execution or dispatcher.sh --help for the details.

Useful while working on the build:

  • --build-only — build everything, including the test suites, but run no tests;

  • --build-from-scratch — discard previous build artefacts, which is what you want after changing the toolchain or the Builder configuration;

  • --build-parallel[=num] — build with several jobs;

  • --builder-debug — be verbose about what the build is doing.

If you get errors during the build, first check that all the required packages are installed; see Test Environment Engine dependencies.

Build artefacts go to the directory named by the TE_BUILD environment variable. If it is not set, they are placed under the directory from which you started dispatcher.sh.

Configuring a run

Every TE subsystem is driven by its own configuration file. This section walks through the five of them in the order dispatcher.sh starts the subsystems.

Running RCF

Remote Control Facility (RCF) configuration file specifies the list of Test Agents to run with a a set of parameters associated with them. For the detailed information on how to write Remote Control Facility (RCF) configuration file please refer to RCF Configuration File section.

More likely you will already have some Remote Control Facility (RCF) configuration file or you will need to do your own version of configuration file based on existing one.

First thing that you need to take into account while writing Remote Control Facility (RCF) configuration file is the how and where you are going to organize testing process.

Suppose you need to test some communication API between two end-points (for example it could be Socket API) and you have the following network topology:

Sample network topology

You would like to test communication between end point pairs:

  • BSD and Windows;

  • BSD and Linux;

  • Windows and Linux.

The API to be tested is the same on all platforms, which means we can use the same test suite for each pair. The only thing specific for our test set-up is where to run Test Agent that supports interface to be tested (assume we exported interface to be tested via RPC call wrappers).

For testing BSD vs Windows configuration we should use the following set-up:

TE components location for testing BSD vs Windows configuration

In this scenario Remote Control Facility (RCF) configuration file would look like:

<?xml version="1.0"?>
<rcf>
    <ta name="Agt_A" type="bsd" rcflib="rcfunix">
        <conf name="host">gollum</conf>
        <conf name="port">5000</conf>
        <conf name="sudo"/>
    </ta>
    <ta name="Agt_B" type="win" rcflib="rcfunix">
        <conf name="host">aule</conf>
        <conf name="port">5000</conf>
        <conf name="sudo"/>
    </ta>
</rcf>

Please note that we use the same RCF UNIX Communication Library, but different Test Agent types.

For testing BSD vs Linux configuration we should use the following set-up:

TE components location for testing BSD vs Linux configuration

In this scenario Remote Control Facility (RCF) configuration file would look like:

<?xml version="1.0"?>
<rcf>
    <ta name="Agt_A" type="bsd" rcflib="rcfunix">
        <conf name="host">gollum</conf>
        <conf name="port">5000</conf>
        <conf name="sudo"/>
    </ta>
    <ta name="Agt_B" type="linux" rcflib="rcfunix">
        <conf name="port">5000</conf>
        <conf name="sudo"/>
    </ta>
</rcf>

Note that we can avoid specifying host name for Test Agent Agt_B, because it runs on the same host as Test Engine.

Similar set-up would be for testing Windows vs Linux set-up.

Now we have Remote Control Facility (RCF) configuration files ready and we can run TE with Remote Control Facility (RCF).

Our project tree has the following structure:

${PRJ_ROOT}
  +-- conf
        +-- builder.conf.mysuite
        +-- rcf.conf.mytestbed
        +-- rcf.conf.mytestbed2
        +-- rcf.conf.mytestbed3

To start TE with Remote Control Facility (RCF), but still without Configurator and Tester, run:

${TE_BASE}/dispatcher.sh --conf-dir=conf --conf-builder=builder.conf.mysuite --conf-rcf=rcf.conf.mytestbed --no-cs --no-tester

If you have some problems with copying Test Agent images to set-up hosts or if you have problems with connection to these Agents you should first check that you are able to enter these hosts without password prompt (read RCF UNIX Communication Library for more information).

Anyway when dispatcher.sh script finishes you can check results in text log file build/log.txt.

Running Logger

Logger configuration file depends on Remote Control Facility (RCF) configuration file in case we need to specify log polling intervals on per Test Agent basis, but in most cases logger configuration file specifies common polling interval to use for accessing all Test Agents.

For more information on Logger configuration file read Configuration File.

More often Logger configuration file is the same for different test set-ups, so preferably if its name is logger.conf, because Dispatcher uses this file name as the default Logger configuration file.

Running Configurator

To run Configurator you need to prepare a configuration file whose name is passed to dispatcher.sh script. For the details on Configurator configuration file read Configurator Configuration File section.

Assuming our Configurator configuration file is split into two parts we would have the following directory tree structure:

${PRJ_ROOT}
  +-- conf
        +-- builder.conf.mysuite
        +-- rcf.conf.mytestbed
        +-- rcf.conf.mytestbed2
        +-- rcf.conf.mytestbed3
        +-- logger.conf
        +-- cs.conf.common
        +-- cs.conf.mytestbed
        +-- cs.conf.mytestbed2
        +-- cs.conf.mytestbed3

Where cs.conf.mytestbed file can look as following:

<?xml version="1.0"?>
<history>
  <xi:include href="cs.conf.common" parse="xml"
              xmlns:xi="http://www.w3.org/2003/XInclude"/>
  <!-- BSD vs Win specific objects and instances descriptions -->

To start TE with Remote Control Facility (RCF), Configurator, but without Tester, run:

${TE_BASE}/dispatcher.sh --conf-dir=conf --conf-builder=builder.conf.mysuite --conf-rcf=rcf.conf.mytestbed --conf-cs=cs.conf.mytestbed --no-tester

Running Tester

Running Tester requires some test suite to be availabe.

For more information on Tester configuration file read Tester Root Configuration File section.

For information on how to create a test suite read Test Suite page.

Suppose you have the following test project directory structure:

${PRJ_ROOT}
  +-- conf
  |     +-- builder.conf.mysuite
  |     +-- rcf.conf.mytestbed
  |     +-- rcf.conf.mytestbed2
  |     +-- rcf.conf.mytestbed3
  |     +-- logger.conf
  |     +-- cs.conf.common
  |     +-- cs.conf.mytestbed
  |     +-- cs.conf.mytestbed2
  |     +-- cs.conf.mytestbed3
  |     +-- tester.conf
  +-- suite-src
        +-- configure.ac
        +-- Makefile.am
        +-- package.xml
        +-- prologue.c
        +-- test1.c
        +-- test2.c
        +-- pkg1
        |     +-- package.xml
        |     +-- test3.c
        |     +-- test4.c
        +-- pkg2
              +-- package.xml
              +-- test5.c
              +-- test6.c

The content of ${PRJ_ROOT}/conf/tester.conf is:

<?xml version="1.0"?>
<tester_cfg version="1.0">
    <maintainer mailto="te-maint@oktetlabs.ru"/>
    <description>Minimal test suite</description>

    <suite name="test-suite" src="${PRJ_ROOT}/suite-src"/>
    <run>
        <package name="test-suite"/>
    </run>
</tester_cfg>

Before we run TE with Tester we need to make sure the test suite tree has a meson.build next to every package.xml. Nothing has to be generated by hand: Builder runs meson itself.

If we need to (re-)build test suite sources at Tester start-up we should run Dispatcher as:

${TE_BASE}/dispatcher.sh --conf-dir=conf --conf-builder=builder.conf.mysuite --conf-rcf=rcf.conf.mytestbed --conf-cs=cs.conf.mytestbed --conf-tester=tester.conf

This command will build TE, build test suites specified in tester.conf file and run all tests according to Tester configuration file and test package description files.

If you need to run the particular test from a test suite (say test6) you can run:

${TE_BASE}/dispatcher.sh --conf-dir=conf --conf-builder=builder.conf.mysuite --conf-rcf=rcf.conf.mytestbed --conf-cs=cs.conf.mytestbed --no-builder --tester-no-build --tester-run=test-suite/pkg2/test6

Please note that we do not specify Tester configuration file, because Dispatcher uses tester.conf as the default name of Tester configuration file.

Also note that we ask Dispatcher to skip building TE (no-builder option) and skip building test suite (assuming we already built it, it is possible to specify tester-no-build option).

For more information on Dispatcher options related to Tester please read Dispatcher Command Line Options.

Logging results

During TE run time a number of log messages generated from different components of TE. All messages are gathered by Logger and put into a binary file that by default has tmp_raw_log name and put under a directory where dispatcher.sh run.

You can specify the location and name of binary raw log file exporting TE_LOG_RAW environment variable:

TE_LOG_RAW=/tmp/my_raw_log ${TE_BASE}/dispatcher.sh --conf-dir=conf

Alternatively you may put raw log file under a particular directory, then you should specify log-dir option:

${TE_BASE}/dispatcher.sh --conf-dir=conf --log-dir=log

In this case raw log file will be saved to ${PRJ_ROOT}/log/tmp_raw_log file.

By default Dispatcher will generate log report in plain text format (with the help of RGT tool). By default plain text log is put under run directory with name log.txt.

If you want HTML-based multi-page structured log you should pass log-html option to Dispatcher specifying directory name where to output log in HTML format:

${TE_BASE}/dispatcher.sh --conf-dir=conf --log-html=html-out

As the result HTML based log report can be found under ${PRJ_ROOT}/html-out directory (open index.html file in a browser).

For more information on log related options of Dispatcher read Dispatcher Command Line Options section.

TE Execution

To run TE one needs to have configuration files for all the TE subsystems.

The previous section, Configuring a run, covers what each of them holds.

Below we describe what happens upon dispatcher.sh execution and what artifacts are left after it.

As it was mentioned above the TE can be started by run.sh script located in you test suite directory or by direct dispatcher.sh invocation.

You should either specify all the configuration files with –conf-* options or use the default ones (see the dispatcher script):

CONF_BUILDER=builder.conf
CONF_LOGGER=logger.conf
CONF_TESTER=tester.conf
CONF_CS=cs.conf
CONF_RCF=rcf.conf
CONF_NUT=nut.conf

To avoid extremely long command lines you can use –opts option and pass a file:

$ cat conf/run.opts.defaults
--conf-builder=builder.conf.default
--conf-tester=tester.conf.default
$ ./dispatcher.sh --opts=run.opts.defaults --tester-run=foobar-ts/basic/trivial

In case you’re running with run.sh there is even faster way to start the framework.

$ ./run.sh --cfg=<cfgname> --tester-run=foobar-ts/basic/trivial

In this case the conf/run.conf.<cfgname> file is considered as an option file. This is very useful as usually project has several test configurations (i.e. apple, carrot and tomato). Although each configuration may use its own hosts they usually share tester.conf, builder.conf (if all hosts run the same operating system), cs.conf

When you start the TE the following is written to the console:

RUNDIR=/home/user/work/my_run_dir
--->>> Starting Logger...done
--->>> Starting RCF...done
--->>> Starting Configurator...done
--->>> Start Tester
Starting package foobar-ts
Starting test prologue                                               pass
Starting package basic
Starting test trivial                                                pass
Done package basic pass
Starting test epilogue                                               pass
Done package foobar-ts pass
--->>> Shutdown Configurator...done
--->>> Flush Logs
--->>> Shutdown RCF...done
--->>> Shutdown Logger...done
--->>> Logs conversion...done

Run (total)                               1
  Passed, as expected                     1
  Failed, as expected                     0
  Passed unexpectedly                     0
  Failed unexpectedly                     0
  Aborted (no useful result)              0
  New (expected result is not known)      0
Not Run (total)                         239
  Skipped, as expected                    0
  Skipped unexpectedly                    0

So:

  • Framework starts all its components; See Start/stop sequence section for details.

  • It executes prologue if exists (prologue is not a test it’s a sequence of actions specific for a given package/suite; for instance it can assign IP addresses or start certain services);

  • tests which were passed with the –tester-run option are executed; in the above case it’s {foobar-ts/basic/trivial} (see te_tester for more details on the tests specification);

  • epilogue is executed it can be used to rollback modifications done in prologue or perform arbitrary cleanup;

  • framework terminates;

  • Test Results Comparator prints some statistics, see Result explanation for details (note, that prologues and epilogues are not counted in the tests statisctics as they MUST always sucess).

After execution is complete several new files appear in the log directory (which is PWD if not specified with –log-dir option. Name of some of the files can be changed via –log-* options.

Files include:

  • tmp_raw_log main log file in binary format; can be passed to the Report Generator Tool and Test Results Comparator utils;

  • ta.* files with stderr from all of the agents, should contain only ‘Exiting’ word if all went fine;

  • if –cs-print-trees option was given then objects and instances will be created; they will contain dump of objects and instances Configurator trees.

You can read the logs in text or HTML format. See Output Formats for more info on logs generation and looks.