This page is a collection of ideas how Franca interfaces could be used to generate test cases.
Collection of ideas
Here are some aspects which have to be discussed further:
- Franca interfaces with contracts vs. without contracts
- a Franca interface contract defines the allowed order of events for any instance of the interface
- a Franca interface without contract allows any order of events, this results in a huge space of possible test sequences
- for a Franca interface with contract: the contract can be used to restrict the number of possible test sequences
- moreover, from a contract tests can be generated which deliberately violate the contract
- testing provided interfaces vs. required interfaces
- a test for a provided interface is a test sequence stimulating a service which is provided by the CUT
- a test for a required interface is a mock component which provides the required interface and reacts to the same stimuli in different ways
- testing a single interface instance vs. testing a component instance with multiple provided/required interfaces
- see section "Scope: One interface vs. many interfaces" below
- generating tests from Franca interfaces alone vs. generating from Franca models plus additional models (i.e., functional models, constraints, ..)
- maybe additional constraints have to be provided for a proper testcase generation
- implementation-agnostic representation of generated tests
- the core of the test generator should be independent of any unit test runner and other implementation technologies
- a format (or metamodel) is needed in order to represent the generated tests
- this format will resemble sequence diagrams, where one lifeline is dedicated to the CUT
- how to get executable tests from these implementation-agnostic representations
Scope: One interface vs. many interfaces
It seems that generating tests just based on a single Franca interface is not sufficient. Many components or subsystems under test will have more dependencies than one provided interface. The following diagram shows a typical scenario for component tests:
The CUT will have several provided interfaces as well as several required interfaces. The test harness has to provide mock implementations for the interfaces required by the CUT and has to stimulate a subset or all of the provided interfaces. The interfaces in the diagram should be modeled as interface instances, because two or more of the interfaces provided/required by the CUT might be defined by the same Franca interface. Nevertheless, it is necessary t distinguish these instances during the test.
Franca system contracts could be used in addition to the Franca definitions of the CUT's interfaces in order to restrict the space of possible sequences even further.
Franca and fMBT
The model-based testing tool fMBT is "set of tools for fully automatic test generation and execution, and a collection of utilities and libraries that support high level of test automation" (source: fMBT homepage). It can be used for GUI testing as well as for component / subsystem testing. Here we will look at the latter and analyse how Franca models can be utilized to speed up testing with fMBT. A test model in fMBT consists of the following main aspects:
- test steps: each test step defines some stimulus which will change the state of the CUT. fMBT's test generator will create tests as sequences of test steps. A test step has a guard (specifying the condition when the test step can be executed) and a body (specifying the state change in the test model).
- adapters: each test step should have an adapter which actually executes (Python) code to stimulate the CUT and check the resulting state
- test configuration: a (huge) set of options for defining how tests are generated from a collection of test steps
It doesn't seem useful to re-build parts of fMBT specific for Franca interfaces, esp. the test sequence generation from a collection of test steps. Via fMBT's test configuration the test generator is highly configurable and can also measure the resulting test coverage (on test model level). The tool is highly flexible and can be tailored and extended by Python code. It offers nice visualization features.
Thus, we should investigate the interplay between Franca and fMBT instead of reimplementing fMBT's functionality. How can Franca interface specifications (and maybe even Franca system contracts) used to create one or more of the above mentioned aspects?
Test Steps: From a Franca interface, a collection of test steps could be generated. This will depend on if the interface has a contract or not: Without a contract, each Franca interface element will be mapped to one or more test steps. A contract would add a number of restrictions, i.e., more complex guard conditions in the fMBT specification.
Adapters: fMBT adapters (in Python) could be generated out of Franca interface specifications. For each Franca interface, a Python test adapter class could be generated which can connect to a provided port of the CUT and interact via Franca attributes, methods and broadcasts. This Python test adapter class could then be used from test steps to interact with the CUT easily. TODO: Generate CommonAPI generic code or use e.g. the Python D-Bus binding to generate D-Bus specific code (what about SOME/IP?)...
Franca and ...?
There are many model-based testing and test generation frameworks, see "Related work" section for some hints. There might be alternatives to fMBT - we should look at those, too. Please feel free to add your experiences here.
The (experimental) tracegen plug-in for Franca can generate legal traces from any Franca interface with contract. This could be used as a starting point for a test case generator based on Franca interfaces. The tracegen plug-in is not part of the Franca release yet, it is available on some dev-branch. This has been developed in 2013 and hasn't been improved since.
CUT: Component under Test