Runner

Introduction

The runner is a command-line Testerum component that makes it possible to run tests without the graphical interface.

The command-line interface is especially appropriate for running on CI/CD servers:

Where can I find it?

The runner is part of the Testerum package.

The shell script can be found in the runner/bin/ directory.

Use testerum-runner.sh on Mac and Linux, and testerum-runner.bat on Windows.

testerum
├── basic_steps/
├── bin/
├── conf/
├── jre/
├── relational_database_drivers/
├── repo/
├── runner/
│   ├── bin/
│   │   ├── testerum-runner.bat
│   │   └── testerum-runner.sh
│   ├── node/
│   ├── repo/
│   └── report_templates/
└── start.sh

Command-line arguments

Generic arguments

--help or -h Displays the built-in command-line arguments help info, without running any tests.
--version Display version information, without running any tests.

Test selection

--repository-directory <dir-path>

The path to the repository directory.
The repository directory is where the Testerum files (features, tests, external resources, etc.) are stored.

It's recommended to keep this directory as part of the project you are testing, and check-in the files together with your project.
Versioning your project files with your code has many advantages:

  • it makes it easy to run the tests on the CI/CD server, as part of the project build
  • if you use a feature-branch workflow, checking-in the test together with the project allows you to develop the tests in parallel with the code, and merge the feature branch only when all tests pass

This argument is required.

Example:
--repository-directory "/path/to/the repo dir"

Note: you need to use double quotes around paths that contain spaces.

--test-path <file-or-dir-path>

The path to a test file or directory.

If the path is a directory, all tests in that directory will be run.
If this argument is missing, all tests will be run.

This argument can be specified multiple times.

If the paths are relative, they will be resolved relative to the features directory.

Example:
--test-path "/path/to/some feature/" \
--test-path "/path/to/another-feature/File 1.test" \
--test-path "another-feature/File 2.test"

Reporting

By default, the runner will output minimal information to the console, and only show details for failed tests (the CONSOLE report below).
If you want to configure other reports, you can use the following arguments.

--managed-reports-directory <dir-path>

Use this argument to specify the path to root directory of managed reports

Normally, if you use the --report argument (see below), you need to specify a different report directory every time you invoke the runner, otherwise the new report will overwrite the previous one.
If this argument is present, the runner will create a separate directory at every execution. This makes automation easier (e.g. from shell scripts or CI/CD servers).
In addition, a latest-report.html file will be created that always shows the latest report.

Note: you can specify both this argument and --report (see below), if you want to. For example you may want to specify both this argument and --report CONSOLE to see some kind of progress in the console, even when creating managed reports.

--report <report-type-and-properties>

What report(s) to produce.

If this argument is missing. the default value will be CONSOLE.

The value of this argument has the following generic syntax:
<report-type>[:prop1=val1,prop2=val2,...]
where

  • report-type is one of the following:
    CONSOLE
    CONSOLE_DEBUG
    PRETTY
  • the properties are optional, and they depend on the report type
Multiple properties are separated by comma ,.
Keys are separated from values by equal =.
If the value is empty, the equal sign is optional.
If the key or the value must contain comma , or equal =, they must be escaped using backslash.
For example:
equation=1+2\=3,greeting=Hello\\, world!
will produce 2 properties:
  • equation with the value 1+2=3
  • greeting with the value Hello, world!


CONSOLE report type
This report shows the test execution progress and the results to the console.
Logs are only shown for failed tests.
This report doesn't have any properties.

Example:
--report CONSOLE


CONSOLE_DEBUG report type
This report shows as much information as possible to aid in debugging problems.
This report doesn't have any properties.

Example:
--report CONSOLE_DEBUG


PRETTY report type
This produces an interactive HTML report.
This report has one required property called destinationDirectory, which myst be the path to the directory where the report will be written.

Example:
--report PRETTY:destinationDirectory=/some/path
If the destination directory doesn't exist yet, it will be created.

Multiple reports can be specified at the same time:
--report CONSOLE --report PRETTY:destinationDirectory=/some/path

Note: Do not specify multiple reports that use the console (e.g. --report CONSOLE --report CONSOLE_DEBUG).
If you do this, the results will be very hard to understand, since the reports will be interleaved.

--execution-name <name>

Human-readable title of this execution, displayed in the reports.

Variables

Use these arguments to override Testerum variables.

Variables are user-defined key-value pairs that can be used to extract information used multiple times in tests, composed steps, etc.

--var-env <environment-name>

The name of the variables environment to use.

Example:
--var-env acceptance

--var <key>=<value>

Override Testerum variables.

This argument can be specified multiple times.

This argument takes precedence over --var-env, if the same key is found in both.

Example:
--var URL=http://localhost:4200

Settings

Use these arguments to override Testerum settings.

Settings are used to configure Testerum itself or specific steps.

--setting <key>=<value>

Override Testerum settings.

This argument can be specified multiple times.

This argument takes precedence over --settings-file, if the same key is found in both.

Example:
--setting testerum.selenium.afterStepDelayMillis=0

--settings-file <file-path>

Path to a settings file.

The settings in this file override Testerum settings.

The file must be in the standard Java .properties format.

Example:
--settings-file /path/to/file.properties

Selenium settings

The following settings are used to control the Selenium steps.


--setting testerum.selenium.driver.browserType=<browserType>

Tells Testerum against what browser to run Selenium tests.

Possible values are: CHROME, FIREFOX, OPERA, EDGE, INTERNET_EXPLORER, SAFARI, or REMOTE.

The default value is CHROME.

REMOTE is a special value. If specified, Testerum assumes that a Selenium driver server is already running. Its URL should be specified using the testerum.selenium.driver.remoteUrl setting (see below).

Example:
--setting testerum.selenium.driver.browserType=FIREFOX


--setting testerum.selenium.driver.browserExecutablePath=/path/to/browser/executable-file

Tells Testerum where to find the browser to run. If this setting is missing, the browser specified by testerum.selenium.driver.browserType will be looked up in the system PATH.

Examples:


--setting testerum.selenium.driver.headless=<true or false>

Tells Testerum whether to run the browser in headless mode or not. When running in headless mode, no browser window or any other UI will be displayed, but the tests will still run as normal. This is very useful when running in a CI/CD environment, where the operating system usually doesn't have a graphical user interface.

The default value is false.

IMPORTANT: only CHROME and FIREFOX support the headless mode.

Example:
--setting testerum.selenium.driver.browserType=FIREFOX
--setting testerum.selenium.driver.headless=true


--setting testerum.selenium.driver.driverVersion=<version>

Tells Testerum what version of Selenium driver to use.

The default value is to use the most recent driver. Because of this, this setting is usually useful when running against an older version of a particular browser (that needs an older version of the Selenium driver).

Example:
--setting testerum.selenium.driver.driverVersion=75.0.3770.8


--setting testerum.selenium.driver.remoteUrl=<URL>

Tells Testerum to what Selenium driver server to connect.

This setting is taken into consideration only if testerum.selenium.driver.browserType=REMOTE.
Instead of running the Selenium driver server on the local machine, this setting tells Testerum to connect to an already-running Selenium server.
This allows us to run tests against a browser that is running on a different machine than the one running the tests.

To understand how this works, you need to understand how Testerum communicates with the browser:
how Testerum communicates with the browser
Normally, Testerum will start the Selenium driver. This is a separate process that opens an HTTP server. Testerum will then communicate with the Selenium driver server using the WebDriver protocol (over HTTP).
The Selenium driver will then start the browser (separate process) and communicate with it using a proprietary browser protocol - each browser vendor is allowed to implement their own protocol between the driver and the browser.
When using testerum.selenium.driver.browserType=REMOTE in combination with testerum.selenium.driver.remoteUrl, Testerum will no longer start a local Selenium driver process, but will attempt to connect to an already-running one.
The Selenium driver is not required to run on the same machine as Testerum. This is very useful, since it allows us to run the Testerum tests in a CI/CD environment (where the operating system may not have a graphical user interface) and still test browsers like Microsoft Edge or Safari, which require a graphical user interface.
One way to make this work is to start a Vagrant machine containing the Selenium driver and the browser before the tests, tell Testerum to connect to this machine, using it's IP, and then, after all the tests have finished running, stop the Vagrant machine.

Example:
--setting testerum.selenium.driver.browserType=REMOTE
--setting testerum.selenium.driver.remoteUrl=http://192.168.1.123:9515


Other

--basic-steps-directory <dir-path>

Path to the directory containing the basic steps jar files.

If missing, the path will be the basic_steps directory in the Testerum installation directory.

Note: you will very rarely need to use this argument.

Example:
--basic-steps-directory /path/to/testerum/basic_steps

Examples

The following examples assume the repository directory at /testerum-repo.

Run all tests

./testerum-runner.sh --repository-directory /testerum-repo

Run a single test

./testerum-runner.sh --repository-directory /testerum-repo --test-path "login/Correct login.test"

Run all the tests in a feature directory

./testerum-runner.sh --repository-directory /testerum-repo --test-path "login/"

Run multiple tests

./testerum-runner.sh --repository-directory /testerum-repo --test-path "login/Correct login.test" --test-path "checkout/"

Produce managed reports (different sub-directory for each execution)

./testerum-runner.sh --repository-directory /testerum-repo --managed-reports-directory /test-reports