Skip to end of metadata
Go to start of metadata

(warning) UPDATE: We now primarily use the standard Yocto image test framework which contains several basic tests.

Please refer to the parent page for information.  The test runner/exporter is part of Yocto/Poky so we're reusing that instead of test_harness.pl now.

Output using the  TAP protocol for reporting could however be useful.   That's something to be discussed with Yocto community.




Historical information kept here.

This test suite for the GENIVI Demo platform currently is a proof of concept.

Description

The test suite is composed of a test harness and a number of tests to run. The harness runs the tests and reports back the results in a final report. There are currently two different tests but they're just designed to demonstrate the PoC and how tests can be added. Example tests in different programming languages will be included as well. The test harness uses a protocol called "Test Anything Protocol" or TAP. TAP is supported by many libraries in different programming languages making it quite flexible to create new tests with your favorite language. More info on TAP here: http://testanything.org/

Test suite requirements

The test suite is currently written in perl and requires a fairly new version of perl as well as potentially additional perl modules depending on your set up. If these requirements are problematic, then its possible that the test suite can be re-written in shell for easier use and deployment.

 Assumptions

You have set up a running GDP with needed components as follows;

 Setting up test environment

Using this approach to testing requires adding additional components to the GDP. This is because the required TAP functionality is not installed by default on the GDP. 

To get the components we need, we'll need to edit two files;

    1. ./meta-genivi-demo/recipes-demo-platform/images/genivi-demo-platform.bb
    2. meta-genivi-demo/conf/bblayers.conf.sample

In the first file, genivi-demo-platform, we'll need to add a recipe so that our test harness dependencies get brought in. Add this line;

libtest-harness-perl 

to the file under IMAGE_INSTALL_append line. In the second file (bblayers.conf.sample), add this line;

##OEROOT##/../meta-openembedded/meta-perl \ 

to the file under the BBLAYERS line.

Note that the GDP still needs additional modules to complete the tests! This is still a work in progress.

You have configured ssh for your test user on the target and on the host for passwordless logins

Instructions 

    1.  Get the test suite

      The test suite is kept in a git repository along with the GENIVI Demo platform. It is available separately here: http://github.com/GENIVI/gdp-test-suite.git

    2.  Run the tests

      The tests are kept in a directory called "t". To run the test harness, which automatically scoops up the test directory, simply run
      ./test-harness.pl which is a perl executable. It makes some assumptions about the environment its running so it will likely fail if it doesn't have needed dependencies. Installation information is provided.

    3.  Analyze the output

      The report is created in plain text, here's an example;

      <pre> Report 09/01/15

      ---

      /home/jeremiah/code/perl/get_os.pl results:
      ---
      1..3
      ok 1 - require Test::Harness;
      ok 2 - Perl version 5.20.0
      ok 3 - Operating system is correct

       

      Passed: 3
      Failed: 0

       

      /home/jeremiah/code/shell/tap-bash-ex.sh results:
      ---
      1..2
      ok 1 ssh to porter
      ok 2 NSM is active

       

      Passed: 2
      Failed: 0
      </pre>
      The report above shows TAP protocol output which is "ok" or "not ok" for passed or failed tests respectively. The first line is the report's date, then the next line (/home/jeremiah/code/perl/get_os.pl results:) holds the name of the specific test file. Below that line is the output of that test file in "ok" or "not ok" as well as the enumeration of the number of tests, in this case 3. Finally, there is a summary which lists the number of passed and failed tests.

Roadmap

    • Test against Qemu emulated platform



 

  • No labels

4 Comments

  1. Dear Jeremiah Foster,

    Does this test only available only on Host at this moment?

    Will you try this on Qemu at next time? I just want to know current status about your progress.

    When I tried this on GDP, there were some errors like below.

  2. Yes, it really only runs on the host at the moment, and as such I don't know if that is a good approach? What do you think?

    I will try this on Qemu, that might be a way to run the test suite as a "validation" or "smoke" test.

    The errors you're seeing are missing dependencies. This means that either you can install the dependencies, or, we can perhaps move the test suite to a shell script that doesn't require depedencies on perl? Would that be easier for your use?

    Thanks for your feedback.

  3. Nicholas Contino, Jeremiah Foster


    Hi, I find this interesting and would like to hear some stories / history as it has been almost 1.5 years since last update.

    Could you please share?

    (Specifically, I would like to hear difficulties you had during the project (if you have)

    and the reason you stopped the project (whether it is finished or you met big issues )).


    Thanks.

  4. The work on a test suite is being done in the Baseline Integration Team: https://collab.genivi.org/wiki/display/genivi/Baseline+Testing I will put our discussion here in the agenda for the next BIT meeting and they'll be able to provide more information.