Usage

The program is excuted from commandline
/usr/bin/testrunner-lite [options]

Options

-h, --help

Display short help and exit

-V, --version

Display version information and exit

-f FILE, --file FILE

Input file with test definitions in XML (required)

-o FILE, --output FILE

Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]

Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL

Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic

Enable only automatic tests to be executed

-S, --syslog

Enable logging to syslog.

-P, --print-step-output\fR

Output standard streams from test steps.

-m, --manual

Enable only manual tests to be executed

-c, --ci

Disable validation of test definition against schema

-s, --semantic

Enable validation of test definition against stricter (semantics) schema

-A, --validate-only

Do only input xml validation, do not execute tests

-H, --no-hwinfo

Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS

Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

Filtering options

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.
testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'
The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.
testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'
Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,
testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'
whereas the following will accept tests from test sets "set1" and "set2".
testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

Filtering Based on Hardware Identifier

If different tests sets for different hardware are required then hwiddetect feature can be utilised. User can define a command used to get a hardware identifier within hwiddetect tag. The hardware identifier returned by the command is matched with optional hwid attribute of a test set. If not equal, test cases in the set are skipped and are not written to the result file. A test set will be never skipped if hwid attribute has not been defined for it.

Command defined by hwiddetect can be a shell command or a separate executable. The executable should be included in the test package. Testrunner-lite removes extra whitespace and linefeeds from the the output of the hwiddetect command so that test developer does not need to care about it.

Example usage of hwiddetect:

<testdefinition version="1.0">
  <hwiddetect>/usr/bin/getmyhwid</hwiddetect>
  <suite name="suite1">
    <set name="test_feature_X_on_hw_bar" hwid="bar">
      <case name="test_X_1">
	<step>echo "hwid is bar"</step>
      </case>
    </set>
    <set name="test_feature_X_on_hw_foo" hwid="foo">
      <case name="test_X_1">
	<step>echo "hwid is foo"</step>
      </case>
    </set>
  </suite>
</testdefinition>

Manual Test Cases

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

Generated on Wed Nov 10 14:13:01 2010 for testrunner-lite by  doxygen 1.5.6