Test Suite
Terminology
Term |
Definition |
|---|---|
Test Package |
Group of tightly related tests or test packages, which may share internal libraries and usually run together (one-by-one or simultaneously). Test Package may consist of one test. It may have a prologue (performing some initialization) and epilogue (releasing resources and restoring TE configuration). |
Test Script |
A test which is a minimal structural unit of a test harness. |
Test Suite |
Test Package which may be considered as standalone entity from organisational point of view and build issues. |
Directory tree structure
Test suite can be distributed in two forms:
pre-installed binary form;
source based form.
For pre-installed binary test suite does not require building procedure, which is why there is no need to have build related files.
Pre-installed binary test suite has the following directory structure:
${TS_ROOT}
+-- package.xml
+-- prologue
+-- epilogue
+-- p1_test1
+ ...
+-- p1_testN
+-- subpackage
+-- package.xml
+-- prologue
+-- epilogue
+-- p2_test1
+ ...
+-- p2_testN
A test suite consists of a set of packages each containing a number of test executables and package description file. For the details on the format of package.xml files refer to Tester Package Description File section.
Source based test suite additionally has build-specific files. As any other component of TE, a source based test suite is expected to be built with the help of Builder. Builder uses autotools in the backgroud, which means a top level package directory shall include:
configure script.
And all directories shall include:
Makefile.in file;
More often source based test suite will have base files of autotools:
Makefile.am;
configure.ac.
And before running TE we there will be need to generate configure script and Makefile.in files with the following command:
autoreconf --install --force
The directory structure of source-based test suite is following:
${TS_ROOT}
+-- package.xml
+-- configure.ac
+-- Makefile.am
+-- prologue.c
+-- epilogue.c
+-- p1_test1.c
+ ...
+-- p1_testN.c
+-- subpackage
+-- package.xml
+-- Makefile.am
+-- prologue.c
+-- epilogue.c
+-- p2_test1.c
+ ...
+-- p2_testN.c
Minimal Test Suite
Minimal test suite can be found under ${TE_BASE}/suites/minimal/suite directory.
The structure of minimal project is:
${TE_BASE}/suites/minimal
+-- conf
| +-- builder.conf
| +-- cs.conf
| +-- logger.conf
| +-- rcf.conf
| +-- tester.conf
+-- suite
+-- configure.ac
+-- Makefile.am
+-- package.xml
+-- sample1.c
Tester configuration file (tester.conf) refers to suite sources and adds this suite for Tester run:
This test suite has simplest package description file (package.xml):
Package description file specifies one test executable sample1 to run.
To run this test suite do the following steps:
under ${TE_BASE}/suites/minimal/suite run:
autoreconf --install --force
This will generate configure script and Makefile.in file under ${TE_BASE}/suites/minimal/suite directory.
export TS_BASE environment variable, because Tester configuration file (tester.conf) uses it as the value of src attribute of suite TAG.
export TS_BASE=${TE_BASE}/suites/minimalset TE_BUILD variable to specify allocation of build artifacts.
export TE_BUILD=${TS_BASE}/buildIf this variable is not set, all generated files will be created in working directory.
run TE:
${TE_BASE}/dispatcher.sh --conf-dir=${TS_BASE}/confIn our test project all configuration file names match with the default names, which is why we do not specify them with separate conf-<prog> options, but only tell where to find configuration files with conf-dir option.
As the result of these operations a number of directories are created that have TE and suite build and installation files.
${TE_BASE}/suites/minimal
+-- conf
+-- suite
+-- build - build directory
+-- engine - :ref:`Test Engine <doxid-group__te__engine>` build directory
+-- agents - :ref:`Test Agents <doxid-group__te__agents>` build directory
+-- lib - build directory for TE libraries
+-- include - build directory for includes
+-- platforms - platforms build
+-- suites - test suites build directory
+-- minimal -- build directory of our minimal test suite
+-- sample1 - test executable
+-- inst - installation directory
+-- agents - :ref:`Test Agents <doxid-group__te__agents>` installation directory
+-- default - te_test_engine installation directory
+-- suites - test suites installation directory
+-- minimal -- installation directory of our minimal test suite
+-- package.xml - installed package description file
+-- sample1 - test executable
Please note that Tester runs a test suite from inst/suites/<suite_name> directory.
Autotools files
The content of configure.ac of our minimal test suite is following:
The content of Makefile.am is following:
Test scenario file
A test scenario file used in minimal test suite is a C source file:
Test scenario layout
Each test scenario shall have:
Definition of
TE_TEST_NAMEmacro in order to specify Entity name of log messages generated from test scenario. Usually this is set to test name.#define TE_TEST_NAME "sample1"Inclusion of
te_config.hfile. This file defines macros generated by TE configure script after verification of header files, structure fields, structure sizes (i.e. it has definitions ofHAVE_xxx_H,SIZEOF_xxxmacros).#include "te_config.h"
Inclusion of
tapi_test.hfile - basic API for test scenarios.#include "tapi_test.h"
main() function. A test scenario is an executable, so in case of C source file we should have program entry point, which is main() function for the default linker script. main() function has to have argc, argv parameters, because macros defined at
tapi_test.hdepend on them;Mandatory test points:
TEST_START;
TEST_END;
TEST_SUCCESS.
The mandatory test structure is:
{ TEST_START; ... TEST_SUCCESS; cleanup: TEST_END; }
The following things should be taken into account while writing a test scenario:
If you need to add local variables to your test scenario, put them BEFORE TEST_START macro:
{ /* Local variables go before TEST_START macro */ int sock; struct sockaddr *addr4; struct sockaddr *addr6; int opt_val; TEST_START; ... TEST_SUCCESS; cleanup: TEST_END; }
If a set of tests require definition of the same set of local variables we can avoid duplication these variables from test to test by using TEST_START_VARS macro:
/* test_suite.h */ #define TEST_START_VARS \ int sock; \ struct sockaddr *addr4; \ struct sockaddr *addr6
In each test scenarion we will have:
... #include "test_suite.h" ... int main(int argc, char **argv) { /* Define test-specific local variables */ int opt_val; TEST_START; ... sock = rpc_socket(rpc_srv, RPC_AF_INET, RPC_SOCK_STREAM, RPC_IPPROTO_TCP); ... }
Another useful macros are:
TEST_START_SPECIFIC;
TEST_END_SPECIFIC.
They can be defined if you need some common parts of code to be executed during TEST_START and TEST_END procedures. For example tests suites that use tapi_env library may define these macros as:
#define TEST_START_VARS TEST_START_ENV_VARS #define TEST_START_SPECIFIC TEST_START_ENV #define TEST_END_SPECIFIC TEST_END_ENV
Test parameters
The main function to process test parameters in test scenario context is test_get_param(). It gets parameter name as an argument value and returns string value associated with that parameter.
Apart from base function test_get_param() there are a number of macros that process type-specific parameters:
For example for the following test run (from package.xml):
<run> <script name="comm_sender"/> <arg name="size"> <value>1</value> <value>100</value> </arg> <arg name="oob"> <value>TRUE</value> <value>FALSE</value> </arg> <arg name="msg"> <value>Test message</value> </arg> </run>
we can have the following test scenario:
int main(int argc, char **argv) { int size; bool oob; char *msg; TEST_START; TEST_GET_INT_PARAM(size); TEST_GET_BOOL_PARAM(oob); TEST_GET_STRING_PARAM(msg); ... }
Please note that variable name passed to TEST_GET_xxx_PARAM() macro shall be the same as expected parameter name.
Test suite can also define parameters of enumeration type. For this kind of parameters you will need to define a macro based on TEST_GET_ENUM_PARAM().
For example if you want to specify something like the following in your package.xml files:
<enum name="ledtype"> <value>POWER</value> <value>USB</value> <value>ETHERNET</value> <value>WIFI</value> </enum> <run> <script name="led_test"/> <arg name="led" type="ledtype"/> </run>
You can define something like this in your test suite header file (test_suite.h):
enum ts_led { TS_LED_POWER, TS_LED_USB, TS_LED_ETH, TS_LED_WIFI, }; #define LEDTYPE_MAPPING_LIST \ { "POWER", (int)TS_LED_POWER }, \ { "USB", (int)TS_LED_USB }, \ { "ETHERNET", (int)TS_LED_ETH }, \ { "WIFI", (int)TS_LED_WIFI } #define TEST_GET_LED_PARAM(var_name_) \ TEST_GET_ENUM_PARAM(var_name_, LEDTYPE_MAPPING_LIST)
Then in your test scenario you can write the following:
int main(int argc, char **argv) { enum ts_led led; TEST_START; TEST_GET_LED_PARAM(led); ... switch (led) { case TS_LED_POWER: ... } ... }