System Architecture

Module Design

This document contains a short description of the basic terms and principles of the Test Execution Environment.

• Test Executor: Performs the necessary tests based on the instructions received by the testing framework.
• Event Manager: Facilitates the internal event based communication between the Reporter and the Test Executor.
• TEE Proxy: Handles the dependencies and loading of test case classes.
• Reporter: Aggregates the data and creates the reports in XML, HTML, etc. formats.

Basic Concepts

Test

A test is the smallest element of the test hierarchy of the TEE. Each test represents a Java method that executes some actions which are purposed to check whether methods of registered OSGi services (or exported classes) are working correctly.

Test Case

A test case consists of a series of tests (test methods) performed in a predefined order. OSGi-aware test cases are typically Java classes which consist of sets of test methods (tests). An OSGi-aware test case is usually packaged as an OSGi bundle which has specific manifest headers containing information about this OSGi-aware test case.

The OSGi-aware test case bundles are placed in the testcases directory which is specified by the tee.core.base.testcases system property. TEE uses this directory for storing the bundles of the test cases which can be executed.

Test cases generate results which describe clearly and in detail whether the tested functionality is working as expected or not. Test case result generation is usually done in two ways:

• By using reference log files: The test case generates results of its execution as logs. These result logs are compared to some reference logs containing the expected results of the test case. Results are generated in this way for the Reference Model.
• By using assertion: The test case checks in its code whether the results from some action, such as a method call, match to some expected results.

Test cases can allocate resources, e.g. open sockets or files, get OSGi services, etc. This is usually done in a special test case initialization method. All occupied resources must be freed after test execution in order to leave the target system in the state prior to test case execution. This is also done in a special test case method.

Any exceptions, thrown during the test execution but not expected by the running test are caught by TEE and logged.

The way a test case is written and its contents completely depend on the test case model.

Each test case contains three components, not all of which are compulsory:

• Class files containing the code of the test case.
• A manifest file which contains specific information needed by the TEE.
• A configuration file which should be situated in the main directory of the test case JAR file. For more information, refer to the "Test Configuration" part of this document.

Test Project

A test project represents a collection of test cases which are executed together. It is defined in a XML file which contains information about the test cases included in the project, the execution mode and duration of the project, etc. By default, test projects are placed in the tee/sa/projects directory of the target framework. For further information about the test project concept, refer to Defining a New Project.

Test Case Model

A Test Case Model defines the rules for writing and executing a test case. These are typically sets of abstract classes which define abstractions for test cases and common utility classes.

Following are listed the currently supported Test Case Models:

• Assertion Model: It is based on assertions about the state of the tested component. It is recommended that you use this Test Case Model for developing new test cases. Refer to the Create a New Test Case guide for more information about how to create test cases for the Assertion Model.
• JUnit Model: It is based on the well-known JUnit model. TEE supports the 3.8, 4.11 and 5.2 versions of the JUnit testing framework. There are some specific rules for packing JUnit test cases as bundles.
• Reference Model: It is based on comparison of the generated result logs and the expected results of the test cases. The expected output is preliminary defined in a reference file which is situated in the main directory of the of the test case JAR file. This is the old Test Case Model and it is recommended that you use the new Assertion Model instead.

Implementations of the above-mentioned test case models are stored in the tee/sa/tfws directory which is created after you install the TEE.

Test Configuration

A test configuration is a XML file containing some prepositions needed for the proper test case execution. For example, you can configure the system properties, the configurations for the Configuration Admin, or the installation of some bundles before test case execution. Depending on the place of the test configurations, they are:

• Internal: It is packed in the test case bundle JAR and its name depends on the test case model.
• External: It is placed by default in the tee/sa/configs directory which is specified with the tee.core.base.configs system property. In this case, it is compulsory to use the config_<test_case_name>_.xml naming convention.

There are also two test configuration models provided by the TEE to meet the needs of the different test case models – the OSGi Framework Configuration Model and the Reference Test Case Configuration Model. They define the schemes for writing new test configurations.

Refer to Defining a New Test Configuration for guidelines on creating a new test configuration following the OSGi Framework Configuration Model.

Result Report

A result report contains information of the generated logs provided in HTML or JUnit XML format. It is placed in the tee/sa/logs directory and has the following structure by default:

• <project_name> – In case no project name is defined, the default System Project name is used
  • <current_date_and_time>
    • summary.html – Contains information about the name and the status of the executed test case(s), additional project information and a link to the env.html
    • env.html – Holds a list of all system properties with their values which are used during the test project execution
    • <test_case_name_and_version>: <current_date_and_time >.html – Contains the full description of the test case execution results, as well as information about the elapsed time and memory delta – it specifies the difference between the memory at the beginning and at the end of the test case execution

    <current_date_and_time> file could be also in .xml format

• levels_<test_case_model>.html - Contains description of the different test levels specific for the used test case model

A test case has four basic test levels:

• Passed: The test case was executed successfully and all preliminary defined test conditions were met.
• Failed: The test case was successfully executed but some of the conditions of the target were not met as expected.
• Error: The test case crashed unexpectedly during the execution. Additionally this level implies the Failed level.
• Skipped: The test case was not executed due to some problems with the preparation of the target framework for its processing or because it was switched off manually. This level does not imply any of the other levels.

A test project has a Passed level only if all test cases it contains were executed successfully. If one test case of the project results with a Failed or Error level, the whole project is respectively with a Failed or Error level. In case one or more test cases are skipped, the result of the project is not influenced. If all test cases are with a Skipped test level, the test project is assumed as Skipped.