Dynamic host test loader (from directory location)

[PrzemekWirkus]

Description

This feature allows mbedhtrun to load and register additional host test scripts from given directory.

In many cases users will add host tests to their yotta modules preferably under /test/host_tests/module directory.

Host tests script files are enumerated and registered so they can be used with local module test cases.

Rationale

Not all host tests can be stored with mbedhtrun package. Some of them may and will be only used locally, for prototyping. Some host tests may just be very module dependent and should not be stored with `mbedhtrun``.

Switch -e (enumerate host tests from directory)

New CLI switch -e / --enum-host-tests defines (yotta module) directory in which we will store local host tests. You can also point mbedhtrun to different directory with pool of host tests.

Note: Directory ytmodule/test/host_tests will be default local host test location used by test tools such as greentea.

Example

In this example we have simple test basic located under /test/basic (C++ code) and corresponding host test under /test/host_tests,

In this example we will add to yotta module ./test directory sub directory host_tests and put simple host test basic.py.

Test /test/basic is connected with host test by name mbed_drivers_basic defined both in test case C++ source code (see listing below) and as name of host test.

Directory structure with local host test(s):

ytmodule
├───test
...
│   ├───host_tests
│       ├───basic.py

We can now point mbedhtrun (switch --enum-host-tests .\test\host_tests\) to new location within yotta module and use --list switch to just registered host tests:

$ cd ytmodule
$ mbedhtrun --enum-host-tests .\test\host_tests\ --list

'default'                : mbed_host_tests.host_tests.default_auto.DefaultAuto()
'default_auto'           : mbed_host_tests.host_tests.default_auto.DefaultAuto()
'detect_auto'            : mbed_host_tests.host_tests.detect_auto.DetectPlatformTest()
'dev_null_auto'          : mbed_host_tests.host_tests.dev_null_auto.DevNullTest()
'echo'                   : mbed_host_tests.host_tests.echo.EchoTest()
'hello_auto'             : mbed_host_tests.host_tests.hello_auto.HelloTest()
'mbed_client_auto'       : mbed_host_tests.host_tests.mbed_client_auto.LWM2MClientAutoTest()
'mbed_drivers_basic'     : basic.Basic()
...

New locally defined host test was registered as mbed_drivers_basic (module basic, class basic.Basic().
Where:

• mbed_drivers_basic is host test name loaded from BaseHostTest.Basic.name
• module basic name is Python file name (no extension)
• class basic.Basic() name is composed from module and class (inherited from BaseHostTest) name inside host test file

Example test (+ new host test) execution command

$ mbedhtrun -d F: -p COM172 -f "/path/to/basic.bin" -C 4 -c shell -m K64F -e ./test/host_tests

Note: Add switch -v to increase -e switch's verboseness.

basic.cpp listing

#include "mbed-drivers/test_env.h"

void runTest() {
    MBED_HOSTTEST_TIMEOUT(20);
    MBED_HOSTTEST_SELECT(mbed_drivers_basic);
    MBED_HOSTTEST_DESCRIPTION(Basic);
    MBED_HOSTTEST_START("MBED_A1");
    MBED_HOSTTEST_RESULT(true);
}

void app_start(int, char*[]) {
    minar::Scheduler::postCallback(&runTest);
}

Note: MBED_HOSTTEST_SELECT(mbed_drivers_basic); macro usage.

basic.py listing

from mbed_host_tests import BaseHostTest

class Basic(BaseHostTest):
    """ Simple, basic host test's test runner waiting for serial port
        output from MUT, no supervision over test running in MUT is executed.
    """

    name = 'mbed_drivers_basic'

    def test(self, selftest):
        result = selftest.RESULT_SUCCESS
        # test case implementation
        return result

Note: name = 'mbed_drivers_basic' host test name definition.

[autopulated]

Would this work with --digest (e.g. here https://github.com/ARMmbed/mbed-drivers/blob/master/module.json#L42)

[PrzemekWirkus]

@autopulated It should, I've added -e support to all paths for host test execution.
I will anyway double check with yt test.

[PrzemekWirkus]

@autopulated There is a problem with resolving of default host test directory. I've filed issue on myself: #33