Remote Control Facility (RCF)

Introduction

Remote Control Facility (RCF) is a distributed entity allowing interactions between Test Engine and Test Agents.

RCF is responsible for:

starting Test Agents during initialization or after reloading the NUT or after Test Agent death;
time synchronization;
low-level interactions between Test Agents and Test Engine (serial port, telnet, ssh, etc.);
construction and passing commands from Test Engine to Test Agents and processing answers;
providing convenient API to other TE subsystems and tests;
interpreting commands on the Test Agents calling appropriate handles;
logging commands and answers.

The following diagram illustrates data flows in RCF:

Interaction with Test Agents

On start-up RCF gets from RCF configuration file the list of Test Agents to be started. For each Test Agent the file contains its name, type and additional configuration parameters.

After TA was run, RCF tries to connect to it several times (3 by default) before exiting with failure if connection cannot be established. The number of attempts to connect can be changed with RCF_TA_MAX_CONN_ATTEMPTS environment variable.

Test Agent type value is used by the RCF to find out which shared library should be dynamically linked to the RCF process to interact with the Test Agent. This library “knows” how to start and stop the Test Agent and how to send/receive commands to/from it. It provides to RCF the strictly specified set of functions.

For example, the library rcfunix is responsible for interacting with UNIX-like Test Agents (Linux, BSD, Cygwin). It uses scp to copy Test Agent image to the remote host and ssh to start or stop it. TCP connection is used to send/receive commands.

Test Protocol

RCF interacts with the Test Agents using text-based Test Protocol. Some text commands may be accompanied by binary attachments. When the RCF is compiled with high log level, all Test Protocol commands and answers appear in the log.

It is possible to create logical sub-channels (“sessions”) in the connection to the Test Agent. Inside the particular session, the commands are sent to the Test Agent one-by-one (i.e., next command is sent only after receiving the answer to the previous one). However, if several sessions are created, the Test Agent may be asked to do several actions simultaneously (for example, generate traffic, catch traffic and call a remote routine).

Efficient using of this feature is possible on multi-thread Test Agents only.

RCF awaits a command acknowledgement before sending the next one.

Portable Command Handler

Test Agents are usually based on the RCF library “Portable Command Handler” (lib/rcfpch) which is responsible for parsing Test Protocol commands and calling TA-specific handlers for the command processing. It also provides default handlers and functions for TA-specific handlers implementation.

Application Domains

All functionality provided by RCF can be divided into several areas – Application Domains. This chapter describes shortly the functionality provided by each AD.

Test Agent Control

RCF is able to start the Test Agent, shutdown it, restart Test Agent itself as well as reboot the host on which the Test Agent is running or NUT controlled by the Test Agent. RCF allows to retrieve the list of all running Test Agents as well as their types.

RCF discovers a TA death by breaking of the connection or a command time-out. In this case, the TA application is restarted. Moreover, RCF may be asked to force a check of the Test Agent validity to discover if all Test Agents are still alive or not.

Finally RCF provides to Logger get log command allowing retrieval of the log as a binary attachment.

TA Configuring

A service of this application domain is used by Configurator as the previous one and allows to add/delete/change object instance on the TA as well as obtain the value of existing instance.

It is also possible to perform wildcard get operations.

Usually Configurator issues commands like get *:* or get /a:a1/b:b1/… to retrieve identifiers of all instances (or only instances of a particular subtree) present on the Test Agent. Then it may separately get value of each instance by its identifier.

Variables

RCF allows to get or change the value of a global variable of the Test Agent program (integer or string) or an environment variable of the Test Agent (string). This feature is used rarely and usually for environment variables only.

Files

Following services are included in the application domain:

copy file from the Test Engine to the Test Agents;
copy file from the Test Agents to the Test Engine;
delete file on the Test Agents.

Piece of memory in the TA address space may be copied as well using /memory/<address>:<length> path.

Files are transferred between TA and Test Engine as attachments of protocol commands.

Traffic

Following services are included in the application domain:

create/destroy CSAPs (Communication Service Access Points) on the Test Agent;
obtain/change CSAP parameters;
transmit packets from the Test Agent according to a specified template;
receive packets matched to a specified pattern on the Test Agent.

CSAP should be created before any traffic sending or receiving. It is similar to socket in Berkeley Socket API, but may provide an access to CLI or other communication facilities as well.

CSAP may consist of several layers to allow send/receive packets corresponding to a protocol stack. For example, it’s possible to create CSAP “dhcp.udp.ipv4.aal5.atm”.

Parameters of the CSAP as well as packet templates and patterns are described using ASN.1. Each level (protocol) is described according to its rules.

CSAP parameters and packets on the Test Engine are stored as files on the TEN and passed to/from Test Agent as binary attachments (compiling of RCF with high log level however allows to observe content of these files in the log).

Test API libraries are provided to work with TAD. They allow to create CSAP and send/receive packets using functions adapted for particular media and/or protocol.

Routines and Processes

RCF allows to call a C routine on the Test Agent in different ways:

call it in the context of the Test Agent;
start the thread in the same address space with the routine as an entry point;
start the process with the routine as an entry point.

The routine is specified by the name and should correspond to a global routine defined in the Test Agent or a library linked with it (symbol table is built for the Test Agent). For process starting, it is also possible to specify the name of the program instead of the name of the routine.

The mechanism described above allows very restricted routine parameters (only integers and strings) and only integer return code. It also does not assume output parameters.

Another mechanism based on SUN RPC is provided by RCF for API testing (white box testing). It allows:

create a separate process: RPC server;
fork one RPC server from another;
create a thread in the RPC server;
destroy the RPC server;
call the routing with arbitrary input and output parameters.

RCF Configuration File

Remote Control Facility (RCF) has its own configuration file that specifies the list of Test Agents to run. Examples of Remote Control Facility (RCF) configuration file can be found under ${TE_BASE}/conf directory.

Simple configuration file looks like the following:

<?xml version="1.0"?>
<rcf>
    <ta name="Agt_B" type="linux_snmp" rcflib="rcfunix">
        <conf name="host">olwe</conf>
        <conf name="port">5012</conf>
        <conf name="sudo"/>
    </ta>
</rcf>

This configuration file tells Remote Control Facility (RCF) to run one Test Agent, where the meaning of attributes is following:

name - specifies the name to be associated with running Test Agent. While using functions of API: RCF you will need to pass the value of this attribute (Agt_B in our sample) as the value of the first parameter.
```
ta_name = "Agt_B";
rc = rcf_ta_del_file(ta_name, session_id, file_path);
```

type - specifies type of Test Agent. This value shall match with any type of Test Agent specified via TE_TA_TYPE directive of Builder configuration file (See TE_TA_TYPE for additional information). For example:

# From builder.conf
# Agent with WiFi support for PRISM 54 Wireless card
TE_TA_TYPE([linux_wifi_prism54], [], [unix],
           [--with-rcf-rpc --enable-wifi --enable-8021x
            --with-tad=dummy_tad],
           [-DWIFI_CARD_PRISM54],
           [], [], [comm_net_agent ndn asn])

<?xml version="1.0"?>
<rcf>
    <ta name="Agt_A" type="linux_wifi_prism54" rcflib="rcfunix"/>
</rcf>

rcflib - specifies the name of shared library to be used by Remote Control Facility (RCF) in order to communicate with this particular Test Agent. rcfunix corresponds to a library located under ${TE_BASE}/lib/rcfunix directory.

The Test Agent configuration parameters that are passed to communication library (to a library specified via rcflib attribute) should be written in separated subitems “attr:name” (see RCF Communication Library), like this:

<?xml version="1.0"?>
<rcf>
    <ta name="Agt_A" type="linux_wifi_prism54" rcflib="rcfunix">
        <conf name="host">marcus</conf>
        <conf name="port">5012</conf>
        <conf name="sudo"/>
    </ta>
</rcf>

Other attributes that you can specify for a Test Agent are:

synch_time - Enable/disable time synchronization between Test Engine and Test Agent (possile values are yes or no, the default is no);
rebootable - Tell RCF whether Test Agent can be rebooted or not (possile values are yes or no, the default is no). If this attribute is not enabled, rcf_ta_reboot() function from API: RCF returns TE_EPERM error code;

disabled - Whether RCF shall ignore this Test Agent (act as if there was no such Test Agent in configuration file). This attribute can be used to tune RCF configuration via environment variables. For example:

<ta name="Agt_A" type="${TE_TST_TA_TYPE:-linux}" disabled="${TE_TST:-yes}" rcflib="rcfunix"
    rebootable="${TE_TST_REBOOTABLE}">
    <conf name="host">${TE_TST}</conf>
    <conf name="port">${TE_TST_PORT:-${TE_RCF_PORT:-50000}}</conf>
    <conf name="sudo" cond="${TE_TST_SUDO:-true}"/>
    <conf name="shell">${TE_TST_VG}</conf>
</ta>

This is an example of tunable RCF configuration. Regarding our disabled attribute we can skip running Test Agent Agt_A if TE_TST environment variable is not set;

fake - Tells RCF that this Test Agent is already running (for example because it is run under gdb);

cold_reboot - Specifies Power Control Test Agent name and its parameters that can be used to perform cold reboot ot this TA. Usually has the form [power TA name]:[outlet name], like:

<?xml version="1.0"?>
<rcf>
    <ta name="Agt_A" type="bsd" rcflib="rcfunix" cold_reboot="Agt_Power:5">
        <conf name="host">192.168.1.2</conf>
        <conf name="port">5050</conf>
        <conf name="sudo"/>
    </ta>
    <ta name="Agt_Power" type="power_ctl" rcflib="rcfunix">
        <conf name="host">192.168.1.120</conf>
        <conf name="port">${TE_POWER_PORT:-${TE_RCF_PORT:-50000}}</conf>
    </ta>
</rcf>

Agent side threads/processes

RCF configuration file allows to specify a number of threads or processes to create on Test Agent at start-up. Also it is possible to specify the list of functions to be called on Test Agent at start-up. These features are implemented via standard RCF mechanisms that can be done from test scenarios with calls to:

rcf_ta_start_task();
rcf_ta_start_thread();
rcf_ta_call().

<?xml version="1.0"?>
<rcf>
    <ta name="Agt_C" type="linux" rcflib="rcfunix">
        <conf name="host">olwe</conf>
        <conf name="port">5000</conf>
        <conf name="sudo"/>
        <thread name="log_serial">
            <arg value="${TE_TST2_LOG_SERIAL}"/>
            <arg value="WARN"/>
            <arg value="10"/>
            <arg value="3109:${TE_LOG_SERIAL_USER:-tester}:${TE_TST2_LOG_SERIAL}"/>
        </thread>
        <task name="temp_init" when="${TE_TST_TEMP_TASK}"/>
        <function name="usb_init">
            <arg value="ehci"/>
        </function>
    </ta>
</rcf>

This configuration file tells RCF to run one Test Agent named Agt_C and on start-up do:

create a thread in Test Agent application passing control to log_serial function with four arguments;
if TE_TST_TEMP_TASK is not empty, create a separate task on Test Agent node using temp_init function as task entry point;
call function usb_init passing one argument with value ehci.

This feature can be useful if we need to do some system initialization on Test Agent side and we do not want to modify test suite sources.

Another good example is serial logs collection:

<thread name="log_serial" when="${TE_AGT_LOG_SERIAL}">
    <arg value="${TE_AGT_LOG_SERIAL}"/>
    <arg value="WARN"/>
    <arg value="10"/>
    <arg value="3109:${TE_LOG_SERIAL_USER:-tester}:${TE_AGT_LOG_SERIAL}"/>
</thread>

Below element tells RCF that an agent should have a thread running log_serial() function if TE_AGT_LOG_SERIAL is set. Arguments are passed to the thead function.

RCF Configuration file XSD

XSD schema of Remote Control Facility (RCF) configuration file can be found at ${TE_BASE}/doc/xsd/rcf_config.xsd file.

RCF Communication Library

For each Test Agent listed in Remote Control Facility (RCF) configuration file we specify rcflib attribute whose value define the name of communication library that Remote Control Facility (RCF) shall use while sending or receiving commands to/from that Test Agent. Currently TE supports only one type of Remote Control Facility (RCF) communication library - RCF UNIX Communication Library.

RCF UNIX Communication Library

Test Environment: RCF library for UNIX Test Agents

RCF UNIX Communication Library should be used to control and interact with Test Agents on Unix-like hosts. It uses ssh and scp utilities to run commands on other hosts and to copy files (Test Agent executables) to other hosts. Output from Test Agent (stdout and stderr) is directed to TE log with Remote Control Facility (RCF) as logging entity name and Test Agent name as logging user name and to ta.<name> file in run directory.

Apart from rcflib attribute for each Test Agent Remote Control Facility (RCF) configuration file contains confstr attribute that specifies configuration string passed to communication library. The format of this configuration string is library specific.

Configuration string of the RCF UNIX Communication Library has the following format:

[[user@]<IP_address_or_hostname>]:<port>
[:key=<ssh_private_key_file>]
[:ssh_port=<port>]
[:ssh_proxy=<ssh-proxy>]
[:copy_timeout=<timeout>]
[:copy_tries=<number_of_tries>]
[:kill_timeout=<timeout>]
[:sudo|su][:<shell>][:<parameters>]

where elements in square brackets are optional and may be skipped.

<IP_address_or_hostname> - is IPv4, IPv6 or DNS address of the host where to run Test Agent. If the value is an empty string (skipped), then the Test Agent runs on local host (on a host where Test Engine runs);
user - if specified it is the user to log in as on the <IP_address_or_hostname>;
<port> - is TCP port to bind TCP server on the Test Agent (RCF UNIX Communication Library is based on TCP sockets and Test Agent plays role of TCP server in connection establishment with Remote Control Facility (RCF), which means Remote Control Facility (RCF) side shall know to which address and port to connect);
key - specifies file from which the identity (private key) for RSA or DSA authentication is read;
ssh_port - specifies TCP port to be used by SSH. May be unspecified or 0 to use standard SSH port 22;
ssh_proxy - specifies SSH proxy to be used, it may include SSH options and must include proxy host name or IP address;
copy_timeout - specifies the maximum time duration (in seconds) that is allowed for image copy operation. If image copy takes more than this timeout, Test Agent start-up procedure fails, provided that the copy operation was the last try (see copy_tries);
copy_tries - specifies the number of tries to perform image copy operation. The time to wait between tries is doubled from RCFUNIX_COPY_RETRY_SLEEP_FIRST_SEC to RCFUNIX_COPY_RETRY_SLEEP_MAX_SEC with every next try. If all copy tries fail, Test Agent start-up procedure fails;
kill_timeout - specifies the maximum time duration (in seconds) that is allowed for Test Agent termination procedure;
sudo\|su - specify this option when we need to run agent under sudo|su (with root privileges). This can be necessary if Test Agent access resources that require privileged permissions (for example network interface configuration);
<shell> - is usually used to run the Test Agent under valgrind tool with a set of options (e.g. valgrind tool=memcheck). Note that this part of configuration string CANNOT contain collons;
<parameters> - string value that is transparently passed to the Test Agent executables as command-line parameters (each token separated with spaces will go as a separate command line parameter).

Here are few examples of configuration strings suitable for RCF UNIX Communication Library :

```
<ta name="Agt_A" type="linux" rcflib="rcfunix"
    confstr=":12000::"/>
```
will run Test Agent on localhost host with user privileges, TCP server will be bound to port 12000;
```
<ta name="Agt_A" type="linux" rcflib="rcfunix"
    confstr=":12000::valgrind --tool=helgrind:"/>
```
will run Test Agent on localhost host with user privileges, TCP server will be bound to port 12000, the Test Agent will be run under valgrind with a data-race detection tool;
```
<ta name="Agt_A" type="linux" rcflib="rcfunix"
    confstr="loki:12000:sudo:valgrind --tool=memcheck --num-callers=32:"/>
```
will run Test Agent on loki host with root privileges. The Test Agent will be run under valgrind with memory checking tool and 32-depth call backtrace.

It is worth noting that a user who runs Dispatcher script should be able to enter hosts specified in Remote Control Facility (RCF) configuration file without password prompt (e.g. using public key). It requires special tunings of SSH daemon on remote host as well. File name that keeps private key for a particular Test Agent can be specified with key option of configuration string.

If sudo element is specified in the configuration string of a Test Agent, then it is assumed that user is sudoer without password.

Building RCF UNIX library

In order to build RCF UNIX library you should just mention rcfunix in the list of platform libraries:

TE_PLATFORM([], [], [-D_GNU_SOURCE], [-D_GNU_SOURCE], [], [tools ... rcfunix ...])

This will build default version of rcfunix suitable for Linux, but there can be some UNIX specific issues that need to be resolved with an alternative build of rcfunix. For example there is a minor difference ofthis library for Solaris (On Solaris we should use ‘coreadm’ utility to configure the name of core files). To sort this out we can build a version of rcfunix for Solaris:

TE_PLATFORM([], [], [-D_GNU_SOURCE], [-D_GNU_SOURCE], [], [tools ... rcfunix rcfsolaris ...]) TE_LIB_PARMS([rcfsolaris], [], [rcfunix], [], [-DRCF_UNIX_SOLARIS])

Then in RCF configuration file we can specify ‘rcfsolaris’ as the value of ‘rcflib’ attribute.

API Usage: Remote Control Facility API

API: RCF

API Usage: Remote Procedure Calls (RPC)