Test Suite

Terminology

Term

Definition

Test Package

Group of tightly related tests or test packages, which may share internal libraries and usually run together (one-by-one or simultaneously). Test Package may consist of one test. It may have a prologue (performing some initialization) and epilogue (releasing resources and restoring TE configuration).

Test Script

A test which is a minimal structural unit of a test harness.

Test Suite

Test Package which may be considered as standalone entity from organisational point of view and build issues.

Directory tree structure

Test suite can be distributed in two forms:

  1. pre-installed binary form;

  2. source based form.

For pre-installed binary test suite does not require building procedure, which is why there is no need to have build related files.

Pre-installed binary test suite has the following directory structure:

${TS_ROOT}
  +-- package.xml
  +-- prologue
  +-- epilogue
  +-- p1_test1
  +   ...
  +-- p1_testN
  +-- subpackage
       +-- package.xml
       +-- prologue
       +-- epilogue
       +-- p2_test1
       +   ...
       +-- p2_testN

A test suite consists of a set of packages each containing a number of test executables and package description file. For the details on the format of package.xml files refer to Tester Package Description File section.

Source based test suite additionally has build-specific files. As any other component of TE, a source based test suite is expected to be built with the help of Builder. Builder uses autotools in the backgroud, which means a top level package directory shall include:

  • configure script.

And all directories shall include:

  • Makefile.in file;

More often source based test suite will have base files of autotools:

  • Makefile.am;

  • configure.ac.

And before running TE we there will be need to generate configure script and Makefile.in files with the following command:

autoreconf --install --force

The directory structure of source-based test suite is following:

${TS_ROOT}
  +-- package.xml
  +-- configure.ac
  +-- Makefile.am
  +-- prologue.c
  +-- epilogue.c
  +-- p1_test1.c
  +   ...
  +-- p1_testN.c
  +-- subpackage
       +-- package.xml
       +-- Makefile.am
       +-- prologue.c
       +-- epilogue.c
       +-- p2_test1.c
       +   ...
       +-- p2_testN.c

Minimal Test Suite

Minimal test suite can be found under ${TE_BASE}/suites/minimal/suite directory.

The structure of minimal project is:

${TE_BASE}/suites/minimal
  +-- conf
  |     +-- builder.conf
  |     +-- cs.conf
  |     +-- logger.conf
  |     +-- rcf.conf
  |     +-- tester.conf
  +-- suite
        +-- configure.ac
        +-- Makefile.am
        +-- package.xml
        +-- sample1.c

Tester configuration file (tester.conf) refers to suite sources and adds this suite for Tester run:


This test suite has simplest package description file (package.xml):


Package description file specifies one test executable sample1 to run.

To run this test suite do the following steps:

  1. under ${TE_BASE}/suites/minimal/suite run:

    autoreconf --install --force
    

    This will generate configure script and Makefile.in file under ${TE_BASE}/suites/minimal/suite directory.

  2. export TS_BASE environment variable, because Tester configuration file (tester.conf) uses it as the value of src attribute of suite TAG.

    export TS_BASE=${TE_BASE}/suites/minimal
    
  3. set TE_BUILD variable to specify allocation of build artifacts.

    export TE_BUILD=${TS_BASE}/build
    

    If this variable is not set, all generated files will be created in working directory.

  4. run TE:

    ${TE_BASE}/dispatcher.sh --conf-dir=${TS_BASE}/conf
    

    In our test project all configuration file names match with the default names, which is why we do not specify them with separate conf-<prog> options, but only tell where to find configuration files with conf-dir option.

As the result of these operations a number of directories are created that have TE and suite build and installation files.

${TE_BASE}/suites/minimal
  +-- conf
  +-- suite
  +-- build - build directory
        +-- engine - :ref:`Test Engine <doxid-group__te__engine>` build directory
        +-- agents - :ref:`Test Agents <doxid-group__te__agents>` build directory
        +-- lib - build directory for TE libraries
        +-- include - build directory for includes
        +-- platforms - platforms build
        +-- suites - test suites build directory
              +-- minimal -- build directory of our minimal test suite
                    +-- sample1 - test executable
        +-- inst - installation directory
              +-- agents - :ref:`Test Agents <doxid-group__te__agents>` installation directory
              +-- default - te_test_engine installation directory
              +-- suites - test suites installation directory
                    +-- minimal -- installation directory of our minimal test suite
                          +-- package.xml - installed package description file
                          +-- sample1 - test executable

Please note that Tester runs a test suite from inst/suites/<suite_name> directory.

Autotools files

The content of configure.ac of our minimal test suite is following:


The content of Makefile.am is following:


Test scenario file

A test scenario file used in minimal test suite is a C source file:


Test scenario layout

Each test scenario shall have:

  1. Definition of TE_TEST_NAME macro in order to specify Entity name of log messages generated from test scenario. Usually this is set to test name.

    #define TE_TEST_NAME "sample1"
  2. Inclusion of te_config.h file. This file defines macros generated by TE configure script after verification of header files, structure fields, structure sizes (i.e. it has definitions of HAVE_xxx_H, SIZEOF_xxx macros).

    #include "te_config.h"
  3. Inclusion of tapi_test.h file - basic API for test scenarios.

    #include "tapi_test.h"
  4. main() function. A test scenario is an executable, so in case of C source file we should have program entry point, which is main() function for the default linker script. main() function has to have argc, argv parameters, because macros defined at tapi_test.h depend on them;

  5. Mandatory test points:

    • TEST_START;

    • TEST_END;

    • TEST_SUCCESS.

    The mandatory test structure is:

    {
        TEST_START;
        ...
        TEST_SUCCESS;
    cleanup:
        TEST_END;
    }

The following things should be taken into account while writing a test scenario:

  • If you need to add local variables to your test scenario, put them BEFORE TEST_START macro:

    {
        /* Local variables go before TEST_START macro */
        int              sock;
        struct sockaddr *addr4;
        struct sockaddr *addr6;
        int              opt_val;
    
        TEST_START;
        ...
        TEST_SUCCESS;
    cleanup:
        TEST_END;
    }
  • If a set of tests require definition of the same set of local variables we can avoid duplication these variables from test to test by using TEST_START_VARS macro:

    /* test_suite.h */
    #define TEST_START_VARS \
        int sock;                 \
        struct sockaddr *addr4;   \
        struct sockaddr *addr6

    In each test scenarion we will have:

    ...
    #include "test_suite.h"
    ...
    int main(int argc, char **argv)
    {
        /* Define test-specific local variables */
        int              opt_val;
    
        TEST_START;
        ...
        sock = rpc_socket(rpc_srv,
                          RPC_AF_INET, RPC_SOCK_STREAM, RPC_IPPROTO_TCP);
        ...
    }
  • Another useful macros are:

    • TEST_START_SPECIFIC;

    • TEST_END_SPECIFIC.

    They can be defined if you need some common parts of code to be executed during TEST_START and TEST_END procedures. For example tests suites that use tapi_env library may define these macros as:

    #define TEST_START_VARS TEST_START_ENV_VARS
    #define TEST_START_SPECIFIC TEST_START_ENV
    #define TEST_END_SPECIFIC TEST_END_ENV

Test parameters

The main function to process test parameters in test scenario context is test_get_param(). It gets parameter name as an argument value and returns string value associated with that parameter.

Apart from base function test_get_param() there are a number of macros that process type-specific parameters:

For example for the following test run (from package.xml):

<run>
  <script name="comm_sender"/>
    <arg name="size">
      <value>1</value>
      <value>100</value>
    </arg>
    <arg name="oob">
      <value>TRUE</value>
      <value>FALSE</value>
    </arg>
    <arg name="msg">
      <value>Test message</value>
    </arg>
</run>

we can have the following test scenario:

int main(int argc, char **argv)
{
    int      size;
    te_bool  oob;
    char    *msg;

    TEST_START;

    TEST_GET_INT_PARAM(size);
    TEST_GET_BOOL_PARAM(oob);
    TEST_GET_STRING_PARAM(msg);
    ...
}

Please note that variable name passed to TEST_GET_xxx_PARAM() macro shall be the same as expected parameter name.

Test suite can also define parameters of enumeration type. For this kind of parameters you will need to define a macro based on TEST_GET_ENUM_PARAM().

For example if you want to specify something like the following in your package.xml files:

<enum name="ledtype">
  <value>POWER</value>
  <value>USB</value>
  <value>ETHERNET</value>
  <value>WIFI</value>
</enum>

<run>
  <script name="led_test"/>
    <arg name="led" type="ledtype"/>
</run>

You can define something like this in your test suite header file (test_suite.h):

enum ts_led {
    TS_LED_POWER,
    TS_LED_USB,
    TS_LED_ETH,
    TS_LED_WIFI,
};
#define LEDTYPE_MAPPING_LIST \
           { "POWER", (int)TS_LED_POWER },  \
           { "USB", (int)TS_LED_USB },      \
           { "ETHERNET", (int)TS_LED_ETH }, \
           { "WIFI", (int)TS_LED_WIFI }

#define TEST_GET_LED_PARAM(var_name_) \
            TEST_GET_ENUM_PARAM(var_name_, LEDTYPE_MAPPING_LIST)

Then in your test scenario you can write the following:

int main(int argc, char **argv)
{
    enum ts_led led;

    TEST_START;

    TEST_GET_LED_PARAM(led);
    ...
    switch (led)
    {
        case TS_LED_POWER:
    ...
    }
    ...
}
Test API