The program is excuted from commandline
/usr/bin/testrunner-lite [options]
Display short help and exit
Display version information and exit
Input file with test definitions in XML (required)
Output file format; FORMAT can be xml or text (Default xml)
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [
http://]host[:port][/path/], where host may be a hostname or an IPv4 address.
Enable only automatic tests to be executed
Enable logging to syslog.
Enable only manual tests to be executed
Disable validation of test definition against schema
Enable validation of test definition against stricter (semantics) schema
Do only input xml validation, do not execute tests
Do not try to obtain hardware information
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.
Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).
How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.
If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.
'filtername' can be any of the following:
- feature
- requirement
- testset
- type
- testcase
The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.
testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'
The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.
testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'
Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,
testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'
whereas the following will accept tests from test sets "set1" and "set2".
testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'
It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.
In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.
Example output when running manual case:
[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.
Please enter the result ([P/p]ass or [F/f]ail):
P
--- Execute test step ---
Description:
Stop calculator
Please enter the result ([P/p]ass or [F/f]ail):
P
--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.
[INFO] 15:16:41 Finished test case. Result: PASS