Tester

Introduction

Tester is an application of Test Engine that controls running tests according to configuration files.

In the context of Test Environment tests are organized in groups called Test Packages. From the Tester point of view, the Test Package is a set of executables (binary or script files can be run in shell with set of parameters passed via command line) accompanied by a file package.xml describing how to run these scripts (execution order, test parameter values, etc.). See Tester Concepts for more details.

It is possible to run the same test with different parameters. Parameters for a test are specified in package.xml file. Tests may be iterated with different combinations of parameter values. Such iterations are performed by Tester.

Tester has complex parameters iteration and multiplication logic, see Iteration and usecases.

Some tests may be marked as (associated with) checking the particular requirement(s) of the product. Tester allows to run tests checking a particular set of requirements. See Test Requirements.

Tester may be asked to build the tests from sources. In this case tests should be built using meson or GNU tools (make/autoconf/automake). All TE libraries used by the tests should be specified in the Builder configuration file and be built/installed before Tester starting.

Tester is responsible for the error recovery. It utilizes a service provided by Configurator to make a backup configuration for each package and test and to restore backups if necessary.

Tester utilizes logging facilities provided by Logger to log test/package starting and finishing events. This information is then used by Report Generator to split the log to sections corresponding to particular tests/packages.

Tester Concepts

Test is a complete sequence of actions required to achieve a specific purpose (check that tested system provides a required functionality, complies a standard, etc.) and producing a verdict pass/fail (possibly accompanied by additional data).

Test Package is a group of tightly related tests or test packages, which may share internal libraries and usually run together (one-by-one or simultaneously). Test Package may consist of one test. It may have a prologue (performing some initialization) and epilogue (releasing resources and restoring TE configuration).

Test Script is a test which is a minimal structural unit of a test harness.

Test Suite is a Test Package which may be considered as standalone entity from organisational point of view and build issues.

The hierarchy of these objects is:

• Test Suite consists of several Test Packages;

• Test Package includes several Tests;

• each Test has a corresponding Test Script.

Test Requirements

Requirements are string labels which describe:

• product features;

• configuration features;

• scenario description.

Test can be associated with a set of requirements. I.e.

• LINUX the test should be executed only on Linux;

• LINUX-3.2 specific kernel version is required;

• PRODUCTION_BUILD test should be executed only on production build (non-debug);

• SELECT test checks select() function behaviour;

• IGMP test checks IGMP-support of the product;

• LONG test scenario is really long;

• BROKEN test scenario is broken - avoid!.

Requirements can be normal and sticky. Sticky requirements are inherited by all descendants (see the hierarchy description in Tester Concepts).

Tester can be asked to run test satisfying certain requirements via tester-req option which can be set to logical expression where logical variables are test requirements and logical operators are ! (not; should be escaped in bash), | (or) and & (and). For example,

$ dispatcher.sh --tester-req=\!BROKEN : skip broken tests
$ dispatcher.sh --tester-req="MSG_MORE|TCP_CORK" : run all the tests involving MSG_MORE flag or TCP_CORK socket option

If you specify multiple tester-req options, it works as if their values are concatenated into a single logical expression by means of & (and) logical operator and passed with a single tester-req option.

$ dispatcher.sh --tester-req=IGMP --tester-req=\!LONG : run fast IGMP tests

Configuration File

Tester Root Configuration File

Tester configuration file is a root of testing scenario. This file contains some auxiliary information like description and list of maintainers as well as information about test suites.

XSD schema of Tester configuration file can be found at ${TE_BASE}/doc/xsd/tester_config.xsd file.

Tester configuration file can look like:

<?xml version="1.0"?>
<tester_cfg version="1.0">
    <maintainer name="TE Maintainers" mailto="te-maint@oktetlabs.ru"/>

    <description>IPv6 Protocol Testing</description>

    <suite name="ipv6_host" src="${TE_TS_IPV6_HOST}"/>
    <suite name="ipv6_router" bin="${TE_TS_IPV6_ROUTER_BIN}"/>

    <run>
        <package name="ipv6_host"/>
    </run>
    <run>
        <package name="ipv6_router"/>
    </run>
</tester_cfg>

Here is some description of used TAGs of configuration file:

• maintainer - specify information about the maintainer of this configuration;

• description - optional description of this configuration. Please note that this is rather configuration file description and it has nothing in common to the description of test suite(s) - you can refer to the same test suite from different Tester configuration files;

• suite - specify information about the location of top level test package (test suite). This directive tells Tester where to find sources or binaries of a test suite. The only suites mentioned in suite directives are built (if necessary) by the Tester.

  suite tag can keep the following attributes:

  • name - specifies test suite name that is used as an identifier associated with this test suite;

  • src - tells Tester where to find sources of mentioned test suite;

  • bin - tells Tester where to find binarires of mentioned test suite (please note that src and bin attributes are mutualy exclusive, only one of them can present at the same time in suite directive).

    It is possible not to have both src and bin attributes in suite tag, which means you ask Tester to build a test suite located under ${TE_BASE}/suites/[suite name] directory.

• run - a block of tests, packages to run.

In the above example we tell Tester :

• to build a test suite located under ${TE_TS_IPV6_HOST} directory. Tester will ask Builder to build and install this test suite;

• to associate a directory ${TE_TS_IPV6_ROUTER_BIN} with installed version of test suite reffered as ipv6_router;

• to run a test suite named ipv6_host (it will be run from the place where it is installed by Builder);

• to run a test suite named ipv6_router (it will be run from ${TE_TS_IPV6_ROUTER_BIN}directory that is previously associated with bin attribute of suite directive.

If you only need to build a set of test suites you can have Tester configuration file as following:

<?xml version="1.0"?>
<tester_cfg version="1.0">
    <maintainer name="TE Maintainers" mailto="te-maint@oktetlabs.ru"/>
    <description>Build IPv6 related test suites</description>

    <suite name="ipv6_host" src="${TE_TS_IPV6_HOST}"/>
    <suite name="ipv6_router"/>
</tester_cfg>

This configuration file asks Tester to:

• build a test suite whose sources located at ${TE_TS_IPV6_HOST} directory;

• build a test suite located at ${TE_BASE}/suites/ipv6_router directory.

Tester Package Description File

Test package description files contain nodes of testing scenario tree. Testing scenario tree is built of test script nodes to be executed with particular set of parameters and particular execution order.

In order to group some test scripts together there are entities called sessions and packages.

Test script is an external applications to be called by Tester. Test session is a complex structure that can include different components to run. Test sessions may have prologue that is run at the start of a session, epilogue that is executed after the session and keep-alive validation to run before and after each test script execution.

XSD schema of Package description files can be found at ${TE_BASE}/doc/xsd/test_package.xsd.

Here is an example of the simplest package description file:

<?xml version="1.0"?>
<package version="1.0">
  <description>Device LED operation Test Suite</description>

    <author mailto="te-maint@oktetlabs.ru"/>

    <session>
        <enum name="led_type">
            <value>Power</value>
            <value>USB</value>
            <value>Ethernet</value>
            <value>Error</value>
        </enum>

        <keepalive>
            <script name="notify_test_step"/>
        </keepalive>

        <prologue>
            <script name="led_prologue"/>
        </prologue>

        <epilogue>
            <script name="led_epilogue"/>
        </epilogue>

        <run>
            <script name="led_on_off"/>
            <arg name="led" type="led_type"/>
        </run>
            <script name="led_blink"/>
            <arg name="led" type="led_type"/>
        <run>
    </session>
</package>

This package description file has:

• session - mandatory part of package description file.

• prologue - prologue entry. This entry defines execution subtree that is run by Tester on package start-up. In our case we run only one script called led_prologue, which means that in test suite installation directory in corresponding place there should be a binary file or executable script with name led_prologue;

• epilogue - epilogue entry. This entry defines execution subtree that is run by Tester on package termination (after processing of all run entries);

• keepalive - this is an entry that Tester runs after processing each run entry;

• run - an item to run. run directive can contain one of the following:

  • script - to ask Tester run a single executable (a number of times with different parameters);

  • package - to ask Tester process another package, in which case Tester will process a separate package description file recursively;

  • session - to have nested session syntax;

  In our case we have plain script directives that point to particular executables with names led_on_off and led_blink;

• at the beginning of the session we define enumeration with enum TAG that we can later refer while specifying test parameters. In our sample we define enumeration type that can be reffered by name led_type;

• we refer to enumeration type inside run TAG when specify parameters for scripts. In our case we define parameter led that has values of type led_type, which means the same executable will be run as many number of times as the number of possible values in type led_type.

Tracking and rolling back configuration changes

By default Tester saves current configuration tree state before starting a test; after termination of the test, it then checks whether configuration state changed. If it changed, it restores previous configuration state and sets test result to DIRTY. Note that this does not apply to prologues; configuration changes made by a prologue are rolled back at the end of the corresponding session (including epilogue if it is present).

So, by default it is assumed that tests do not change configuration.

Default behaviour can be changed with help of track_conf attribute which can be specified in package description file like this

<run>
  <script name="some_tcp_test" track_conf="silent">
    <req id="SOCK_STREAM"/>
  </script>
</run>

track_conf can have the following values:

• no. Tester does not track configuration changes at all, so it does not try to restore configuration state if it was changed.

• yes or barf. This is the default behaviour described above.

• yes_nohistory or barf_nohistory. This differs from the default behaviour only in that Tester does not try to roll configuration changes back by executing configuration commands stored in history in reverse order. Instead it jumps directly to restoring configuration by changing only those configuration objects whose state differs from configuration backup (normally this is done only if restoring configuration from history fails).

• silent. Tester tracks and restores configuration state as usual, but it does not set to DIRTY results of the tests which changed configuration. This is convenient if you do not want to undo all changes manually in tests cleanup but instead rely on Tester to do it for you.

• nohistory or silent_nohistory. Works like silent but also does not try to restore configuration from history like yes_nohistory.

There is also sync flag which can be added to these values (for no it makes no sense though), like nohistory|sync. When the flag is set, Tester synchronizes configuration state with Test Agents before checking whether it is changed. This way it can catch fully unexpected changes (like side effects of commands such as ioctl(SIOCETHTOOL/ETHTOOL_RESET)). However synchronizing configuration with all Test Agents takes extra time. That’s why it is not done by default and extra flag is required to enable it.

track_conf attribute can be specified not only for <script> but also for <session> and <run>. In that case it can be used together with track_conf_handdown attribute which specifies how track_conf attribute is inherited by descendants.

<session track_conf="silent" track_conf_handdown="descendants">
  ...
</session>

track_conf_handdown can have the following values:

• none. track_conf is not inherited by any children.

• children. track_conf is inherited by direct children only.

• descendants. track_conf is inherited by all descendants.

By default track_conf is inherited by all descendants of <run> item but only by direct children of <session> due to historical reasons.

Package File Syntax

XML schema for Tester configuration file may be found in doc/xsd/tester_config.xsd file.

Iteration and usecases

Parameters

Every test has a set of parameters and every parameter has certain values. In the package.xml file it’s represented as:

<session>
  <run>
    <script name="test1"/>
    <arg name="arg1">
       <value>val1</value>
       <value>val2</value>
       <value>val3</value>
    </arg>
   </run>
</session>

So test test1 has one argument arg1 which has three possible values val1, val2 and val3.

When executed it will result in:

$ ./run.sh --tester-run=myts/mypackage/test1
....
Staring package mypackage
Starting test test1             PASSED     <- arg1=val1
Starting test test1             PASSED     <- arg1=val2
Starting test test1             PASSED     <- arg1=val3
Done package mypackage PASSED

You can run test with specific argument value or values:

$ ./run.sh --tester-run=mysuite/mypkg/test1:arg1=val2
or
$ ./run.sh --tester-run=mysuite/mypkg/test1:arg1={val1, val3}

–tester-run-from and –tester-run-to can be used to run part of the sequence.

To run the second iteration one should call:

$ ./run.sh --tester-run=mysuite/mypkg/test1%2
...
Staring package mypackage
Starting test test1             PASSED     <- arg1=val2
Done package mypackage PASSED

You can also specify a step with (test1%1+2) sytax, this will run only odd iterations.

Iteration is specified with an asterisk:

$ ./run.sh --tester-run=mysuite/mypkg/test1%2*3
...
Staring package mypackage
Starting test test1             PASSED     <- arg1=val2
Starting test test1             PASSED     <- arg1=val2
Starting test test1             PASSED     <- arg1=val2
Done package mypackage PASSED

Negation with tilda:

$ ./run.sh --tester-run=myts/mypackage/test1:arg1~=val2
....
Staring package mypackage
Starting test test1             PASSED     <- arg1=val1
Starting test test1             PASSED     <- arg1=val3
Done package mypackage PASSED

Types

Arguments can have certain types. If explicit values are specified:

<session>
  <run>
    <script name="test1"/>
    <arg name="arg1" type="boolean">
       <value>True</value>
       <value>False</value>
    </arg>
   </run>
</session>

then they are checked agains given type. If no values then all values of the type are iterated:

<enum name="sock_type">
  <value>SOCK_STREAM</value>
  <value>SOCK_DGRAM</value>
</enum>
<session>
  <run>
    <script name="test1"/>
    <arg name="socket_type" type="sock_type"/>
   </run>
</session>

Multiplication

Set of argument values are multiplied:

<session>
  <run>
    <script name="test1"/>
    <arg name="arg1">
       <value>val1</value>
       <value>val2</value>
    </arg>
    <arg name="arg2">
       <value>val3</value>
       <value>val4</value>
    </arg>
   </run>
</session>

results into

$ ./run.sh --tester-run=mysuite/mypkg/test1
...
Staring package mypackage
Starting test test1             PASSED     <- arg1=val1, arg2=val3
Starting test test1             PASSED     <- arg1=val1, arg2=val4
Starting test test1             PASSED     <- arg1=val2, arg2=val3
Starting test test1             PASSED     <- arg1=val2, arg2=val4
Done package mypackage PASSED

If full multiplication should not be performed then list attribute should be used:

<session>
  <run>
    <script name="test1"/>
    <arg name="arg1" list="mylist">
       <value>val1</value>
       <value>val2</value>
    </arg>
    <arg name="arg2" list="mylist">
       <value>val3</value>
       <value>val4</value>
    </arg>
   </run>
</session>

results into

$ ./run.sh --tester-run=mysuite/mypkg/test1
...
Staring package mypackage
Starting test test1             PASSED     <- arg1=val1, arg2=val3
Starting test test1             PASSED     <- arg1=val2, arg2=val4
Done package mypackage PASSED

Important points:

• number of values for all arguments in the same list should be equal;

• multiplication and lists can be combined;

• multiplication significantly increases number of test iterations, so one should be sure that he wants to check all combinations.

Variables

You can specify variables in package description file and later refer to values of these variables when specifying values for test arguments.

<session>
  <var name="long_path" global="true">
    <value>/this/is/very/long/path/to/a/file/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</value>
  </var>
  <var name="up_case_name">
    <value>AAAAAA.TXT</value>
  </var>

  <run>
    <script name="readdir"/>
    <arg name="path">
       <value>/</value>
       <value>/dir</value>
       <value ref="long_path"/>
    </arg>
  </run>

  <run>
    <script name="file_open"/>
    <arg name="path">
       <value ref="up_case_name"/>
    </arg>
  </run>
</session>

In this sample we create two variables long_path and up_case_name with particular values. Later we can refer to these variables using ref attribute - as one of values for path parameter of readdir test or as the value for path parameter of file_open test.

Todo More details should be added.