Runner

Displays the built-in command-line arguments help info, without running any tests.

Display version information, without running any tests.

The path to the repository directory.
The repository directory is where the Testerum files (features, tests, external resources, etc.) are stored.

It's recommended to keep this directory as part of the project you are testing, and check-in the files together with your project.
Versioning your project files with your code has many advantages:

• it makes it easy to run the tests on the CI/CD server, as part of the project build
• if you use a feature-branch workflow, checking-in the test together with the project allows you to develop the tests in parallel with the code, and merge the feature branch only when all tests pass

This argument is required.

Example:
--repository-directory "/path/to/the repo dir"

Note: you need to use double quotes around paths that contain spaces.

The path to a test file or directory.

If the path is a directory, all tests in that directory will be run.
If this argument is missing, all tests will be run.

This argument can be specified multiple times.

If the paths are relative, they will be resolved relative to the features directory.

Example:
--test-path "/path/to/some feature/" \
--test-path "/path/to/another-feature/File 1.test" \
--test-path "another-feature/File 2.test"

Test or feature tag used to select what tests will run.

If no tag will be specified, the other options will be used to determine what to run (e.g. --test-path).

If a test has one of these tags, the test will be executed.

If a feature has one of these tags, all tests inside it (recursively) will be executed.

Exclude tags have priority over include tags, i.e. if a feature or a test have both an include tag and an exclude tag, the feature or test will NOT be executed.

Example:
--include-tag web
--include-tag a_tag --include-tag a_second_tag
--include-tag "tag with spaces"

Test or feature tag used to select what tests will NOT run.

If a test has one of these tags, the test will NOT be executed.

If a feature has one of these tags, all tests inside it (recursively) will NOT be executed.

Exclude tags have priority over include tags, i.e. if a feature or a test have both an include tag and an exclude tag, the feature or test will NOT be executed.

Example:
--exclude-tag web
--exclude-tag a_tag --exclude-tag a_second_tag
--exclude-tag "tag with spaces"

Use this argument to specify the path to root directory of managed reports

Normally, if you use the --report argument (see below), you need to specify a different report directory every time you invoke the runner, otherwise the new report will overwrite the previous one.
If this argument is present, the runner will create a separate directory at every execution. This makes automation easier (e.g. from shell scripts or CI/CD servers).
In addition, a latest-report.html file will be created that always shows the latest report.

Note: you can specify both this argument and --report (see below), if you want to. For example you may want to specify both this argument and --report CONSOLE to see some kind of progress in the console, even when creating managed reports.

What report(s) to produce.

If this argument is missing. the default value will be CONSOLE.

The value of this argument has the following generic syntax:
<report-type>[:prop1=val1,prop2=val2,...]
where

• report-type is one of the following:
  CONSOLE
CONSOLE_DEBUG
PRETTY
REMOTE_SERVER
  
• the properties are optional, and they depend on the report type

Multiple properties are separated by comma ,.
Keys are separated from values by equal =.
If the value is empty, the equal sign is optional.
If the key or the value must contain comma , or equal =, they must be escaped using backslash.
For example:
equation=1+2\=3,greeting=Hello\\, world!
will produce 2 properties:

• equation with the value 1+2=3
• greeting with the value Hello, world!

CONSOLE report type
This report shows the test execution progress and the results to the console.
Logs are only shown for failed tests.
This report doesn't have any properties.

Example:
--report CONSOLE

CONSOLE_DEBUG report type
This report shows as much information as possible to aid in debugging problems.
This report doesn't have any properties.

Example:
--report CONSOLE_DEBUG

PRETTY report type
This produces an interactive HTML report.
This report has one required property called destinationDirectory, which myst be the path to the directory where the report will be written.

Example:
--report PRETTY:destinationDirectory=/some/path
If the destination directory doesn't exist yet, it will be created.

REMOTE_SERVER report type
This sends the reports to the Testerum Reports Server.
This report has one required property called "reportServerUrl", which must be the URL to the report server.

Example:
--report REMOTE_SERVER:reportServerUrl=http://report-server-hostname:7788

Important!
Multiple reports can be specified at the same time:
--report CONSOLE --report PRETTY:destinationDirectory=/some/path

Note: Do not specify multiple reports that use the console (e.g. --report CONSOLE --report CONSOLE_DEBUG).
If you do this, the results will be very hard to understand, since the reports will be interleaved.

Human-readable title of this execution, displayed in the reports.

The name of the variables environment to use.

Example:
--var-env acceptance

Override Testerum variables.

This argument can be specified multiple times.

This argument takes precedence over --var-env, if the same key is found in both.

Example:
--var URL=http://localhost:4200

Override Testerum settings.

This argument can be specified multiple times.

This argument takes precedence over --settings-file, if the same key is found in both.

Example:
--setting testerum.selenium.afterStepDelayMillis=0

Path to a settings file.

The settings in this file override Testerum settings.

The file must be in the standard Java .properties format.

Example:
--settings-file /path/to/file.properties

Tells Testerum against what browser to run Selenium tests.

Possible values are: CHROME, FIREFOX, OPERA, EDGE, INTERNET_EXPLORER, SAFARI, or REMOTE.

The default value is CHROME.

REMOTE is a special value. If specified, Testerum assumes that a Selenium driver server is already running. Its URL should be specified using the testerum.selenium.driver.remoteUrl setting (see below).

Example:
--setting testerum.selenium.driver.browserType=FIREFOX

Tells Testerum where to find the browser to run. If this setting is missing, the browser specified by testerum.selenium.driver.browserType will be looked up in the system PATH.

Examples:

• --setting testerum.selenium.driver.browserExecutablePath=/usr/bin/firefox
• --setting testerum.selenium.driver.browserExecutablePath=C:\Programs\Firefox\firefox.exe

Tells Testerum whether to run the browser in headless mode or not. When running in headless mode, no browser window or any other UI will be displayed, but the tests will still run as normal. This is very useful when running in a CI/CD environment, where the operating system usually doesn't have a graphical user interface.

The default value is false.

IMPORTANT: only CHROME and FIREFOX support the headless mode.

Example:
--setting testerum.selenium.driver.browserType=FIREFOX
--setting testerum.selenium.driver.headless=true

Tells Testerum whether to leave the browser open or not after each test execution. When running using this option, there are three possibilities:

The browser will remain open after each test execution, no matter what the test result is.

The browser will be closed after each test execution, no matter what the test result is.

The browser will remain open only after the failed tests.

The default value is onFailure.

Example:
--setting testerum.selenium.leaveBrowserOpenAfterTest=true
--setting testerum.selenium.leaveBrowserOpenAfterTest=false
--setting testerum.selenium.leaveBrowserOpenAfterTest=onFailure

With this option you can maximize the browser before the test execution.

The default value is false.

Example:
--setting testerum.selenium.maximizeWindowBeforeTest=true

With this option you can set up the time duration in milliseconds between Selenium steps.

The default value is 0.

Example:
--setting testerum.selenium.afterStepDelayMillis=15000

With this option you can set up the maximum time duration in milliseconds for wait steps(e.g. wait until an element is present on the page).

The default value is 5000.

Example:
--setting testerum.selenium.waitTimeoutMillis=15000

Tells Testerum what version of Selenium driver to use.

The default value is to use the most recent driver. Because of this, this setting is usually useful when running against an older version of a particular browser (that needs an older version of the Selenium driver).

Example:
--setting testerum.selenium.driver.driverVersion=75.0.3770.8

Tells Testerum to what Selenium driver server to connect.

This setting is taken into consideration only if testerum.selenium.driver.browserType=REMOTE.
Instead of running the Selenium driver server on the local machine, this setting tells Testerum to connect to an already-running Selenium server.
This allows us to run tests against a browser that is running on a different machine than the one running the tests.

To understand how this works, you need to understand how Testerum communicates with the browser:

Normally, Testerum will start the Selenium driver. This is a separate process that opens an HTTP server. Testerum will then communicate with the Selenium driver server using the WebDriver protocol (over HTTP).
The Selenium driver will then start the browser (separate process) and communicate with it using a proprietary browser protocol - each browser vendor is allowed to implement their own protocol between the driver and the browser.
When using testerum.selenium.driver.browserType=REMOTE in combination with testerum.selenium.driver.remoteUrl, Testerum will no longer start a local Selenium driver process, but will attempt to connect to an already-running one.
The Selenium driver is not required to run on the same machine as Testerum. This is very useful, since it allows us to run the Testerum tests in a CI/CD environment (where the operating system may not have a graphical user interface) and still test browsers like Microsoft Edge or Safari, which require a graphical user interface.
One way to make this work is to start a Vagrant machine containing the Selenium driver and the browser before the tests, tell Testerum to connect to this machine, using it's IP, and then, after all the tests have finished running, stop the Vagrant machine.

Example:
--setting testerum.selenium.driver.browserType=REMOTE
--setting testerum.selenium.driver.remoteUrl=http://192.168.1.123:9515

Enables advanced customization of the WebDriver> instance, like for example setting DesiredCapabilities.

This can be done by calling methods on the capabilities> object, which is available in the context of this script.

The type of this object depends on the selected browser, as follows:

• Google Chrome
• Firefox
• Opera
• Microsoft Edge
• Microsoft Internet Explorer
• Safari
• remote WebDriver

Example script (generic):

capabilities.setCapability("os_version", "11");
capabilities.setCapability("device", "iPhone 8 Plus");
capabilities.setCapability("real_mobile", "true");
capabilities.setCapability("browserstack.local", "false");

Another example (for Chrome):

capabilities.addArguments("start-maximized");

var Proxy = Java.type("org.openqa.selenium.Proxy")
var proxy = new Proxy();
proxy.setHttpProxy("myhttpproxy:3337");
capabilities.setCapability("proxy", proxy);

Another example (for [BrowserStack](https://www.browserstack.com/)):

var HashMap = Java.type('java.util.HashMap');
var ArrayList = Java.type('java.util.ArrayList');

capabilities.setCapability('os','OS X');
capabilities.setCapability('os_version', 'Catalina');
capabilities.setCapability('browser', 'Chrome');
capabilities.setCapability('browser_version', '86');
capabilities.setCapability('name', 'Chrome test');

var chromeOptions = new HashMap();
var chromeArgs = new ArrayList();
chromeArgs.add('window-size=400,300');
chromeOptions.put('args', chromeArgs);
capabilities.setCapability('goog:chromeOptions', chromeOptions);

More information for browser-specific options:

• for Google Chrome
• for Firefox
• for Opera
• for Microsoft Edge (EdgeHTML engine)
• for Microsoft Edge (Chromium engine)
• for Microsoft Internet Explorer
• for Safari

Path to the directory containing the basic steps jar files.

If missing, the path will be the basic_steps directory in the Testerum installation directory.

Note: you will very rarely need to use this argument.

Example:
--basic-steps-directory /path/to/testerum/basic_steps