Runner

Introduction

The runner is a command-line Testerum component that makes it possible to run tests without the graphical interface.

The command-line interface is especially appropriate for running on CI/CD servers:

Where can I find it?

The runner is part of the Testerum package.

The shell script can be found in the runner/bin/ directory.

Use testerum-runner.sh on Mac and Linux, and testerum-runner.bat on Windows.

testerum
├── basic_steps/
├── bin/
├── conf/
├── jre/
├── relational_database_drivers/
├── repo/
├── runner/
│   ├── bin/
│   │   ├── testerum-runner.bat
│   │   └── testerum-runner.sh
│   ├── node/
│   ├── repo/
│   └── report_templates/
└── start.sh

Command-line arguments

Generic arguments

--help or -h
Displays the built-in command-line arguments help info, without running any tests.
--version
Display version information, without running any tests.

Test selection

--repository-directory <dir-path>

The path to the repository directory.
The repository directory is where the Testerum files (features, tests, external resources, etc.) are stored.

It's recommended to keep this directory as part of the project you are testing, and check-in the files together with your project.
Versioning your project files with your code has many advantages:

  • it makes it easy to run the tests on the CI/CD server, as part of the project build
  • if you use a feature-branch workflow, checking-in the test together with the project allows you to develop the tests in parallel with the code, and merge the feature branch only when all tests pass

This argument is required.

Example:
--repository-directory "/path/to/the repo dir"

Note: you need to use double quotes around paths that contain spaces.

--test-path <file-or-dir-path>

The path to a test file or directory.

If the path is a directory, all tests in that directory will be run.
If this argument is missing, all tests will be run.

This argument can be specified multiple times.

If the paths are relative, they will be resolved relative to the features directory.

Example:
--test-path "/path/to/some feature/" \
--test-path "/path/to/another-feature/File 1.test" \
--test-path "another-feature/File 2.test"

--include-tag <tag-name>

Test or feature tag used to select what tests will run.

If no tag will be specified, the other options will be used to determine what to run (e.g. --test-path).

If a test has one of these tags, the test will be executed.

If a feature has one of these tags, all tests inside it (recursively) will be executed.

Exclude tags have priority over include tags, i.e. if a feature or a test have both an include tag and an exclude tag, the feature or test will NOT be executed.

Example:
--include-tag web
--include-tag a_tag --include-tag a_second_tag
--include-tag "tag with spaces"

--exclude-tag <tag-name>

Test or feature tag used to select what tests will NOT run.

If a test has one of these tags, the test will NOT be executed.

If a feature has one of these tags, all tests inside it (recursively) will NOT be executed.

Exclude tags have priority over include tags, i.e. if a feature or a test have both an include tag and an exclude tag, the feature or test will NOT be executed.

Example:
--exclude-tag web
--exclude-tag a_tag --exclude-tag a_second_tag
--exclude-tag "tag with spaces"

Reporting

By default, the runner will output minimal information to the console, and only show details for failed tests (the CONSOLE report below).
If you want to configure other reports, you can use the following arguments.

--managed-reports-directory <dir-path>

Use this argument to specify the path to root directory of managed reports

Normally, if you use the --report argument (see below), you need to specify a different report directory every time you invoke the runner, otherwise the new report will overwrite the previous one.
If this argument is present, the runner will create a separate directory at every execution. This makes automation easier (e.g. from shell scripts or CI/CD servers).
In addition, a latest-report.html file will be created that always shows the latest report.

Note: you can specify both this argument and --report (see below), if you want to. For example you may want to specify both this argument and --report CONSOLE to see some kind of progress in the console, even when creating managed reports.

--report <report-type-and-properties>

What report(s) to produce.

If this argument is missing. the default value will be CONSOLE.

The value of this argument has the following generic syntax:
<report-type>[:prop1=val1,prop2=val2,...]
where

  • report-type is one of the following:
    CONSOLE
    CONSOLE_DEBUG
    PRETTY
    REMOTE_SERVER
  • the properties are optional, and they depend on the report type
Multiple properties are separated by comma ,.
Keys are separated from values by equal =.
If the value is empty, the equal sign is optional.
If the key or the value must contain comma , or equal =, they must be escaped using backslash.
For example:
equation=1+2\=3,greeting=Hello\\, world!
will produce 2 properties:
  • equation with the value 1+2=3
  • greeting with the value Hello, world!


CONSOLE report type
This report shows the test execution progress and the results to the console.
Logs are only shown for failed tests.
This report doesn't have any properties.

Example:
--report CONSOLE


CONSOLE_DEBUG report type
This report shows as much information as possible to aid in debugging problems.
This report doesn't have any properties.

Example:
--report CONSOLE_DEBUG


PRETTY report type
This produces an interactive HTML report.
This report has one required property called destinationDirectory, which myst be the path to the directory where the report will be written.

Example:
--report PRETTY:destinationDirectory=/some/path
If the destination directory doesn't exist yet, it will be created.


REMOTE_SERVER report type
This sends the reports to the Testerum Reports Server.
This report has one required property called "reportServerUrl", which must be the URL to the report server.

Example:
--report REMOTE_SERVER:reportServerUrl=http://report-server-hostname:7788

Important!
Multiple reports can be specified at the same time:
--report CONSOLE --report PRETTY:destinationDirectory=/some/path

Note: Do not specify multiple reports that use the console (e.g. --report CONSOLE --report CONSOLE_DEBUG).
If you do this, the results will be very hard to understand, since the reports will be interleaved.

--execution-name <name>

Human-readable title of this execution, displayed in the reports.

Variables

Use these arguments to override Testerum variables.

Variables are user-defined key-value pairs that can be used to extract information used multiple times in tests, composed steps, etc.

--var-env <environment-name>

The name of the variables environment to use.

Example:
--var-env acceptance

--var <key>=<value>

Override Testerum variables.

This argument can be specified multiple times.

This argument takes precedence over --var-env, if the same key is found in both.

Example:
--var URL=http://localhost:4200

Settings

Use these arguments to override Testerum settings.

Settings are used to configure Testerum itself or specific steps.

--setting <key>=<value>

Override Testerum settings.

This argument can be specified multiple times.

This argument takes precedence over --settings-file, if the same key is found in both.

Example:
--setting testerum.selenium.afterStepDelayMillis=0

--settings-file <file-path>

Path to a settings file.

The settings in this file override Testerum settings.

The file must be in the standard Java .properties format.

Example:
--settings-file /path/to/file.properties

Selenium settings

The following settings are used to control the Selenium steps.


--setting testerum.selenium.driver.browserType=<browserType>

Tells Testerum against what browser to run Selenium tests.

Possible values are: CHROME, FIREFOX, OPERA, EDGE, INTERNET_EXPLORER, SAFARI, or REMOTE.

The default value is CHROME.

REMOTE is a special value. If specified, Testerum assumes that a Selenium driver server is already running. Its URL should be specified using the testerum.selenium.driver.remoteUrl setting (see below).

Example:
--setting testerum.selenium.driver.browserType=FIREFOX

--setting testerum.selenium.driver.browserExecutablePath=/path/to/browser/executable-file

Tells Testerum where to find the browser to run. If this setting is missing, the browser specified by testerum.selenium.driver.browserType will be looked up in the system PATH.

Examples:

  • --setting testerum.selenium.driver.browserExecutablePath=/usr/bin/firefox
  • --setting testerum.selenium.driver.browserExecutablePath=C:\Programs\Firefox\firefox.exe

--setting testerum.selenium.driver.headless=<true or false>

Tells Testerum whether to run the browser in headless mode or not. When running in headless mode, no browser window or any other UI will be displayed, but the tests will still run as normal. This is very useful when running in a CI/CD environment, where the operating system usually doesn't have a graphical user interface.

The default value is false.

IMPORTANT: only CHROME and FIREFOX support the headless mode.

Example:
--setting testerum.selenium.driver.browserType=FIREFOX
--setting testerum.selenium.driver.headless=true

--setting testerum.selenium.leaveBrowserOpenAfterTest=<true, false or onFailure>

Tells Testerum whether to leave the browser open or not after each test execution. When running using this option, there are three possibilities:

  • The browser will remain open after each test execution, no matter what the test result is.
  • The browser will be closed after each test execution, no matter what the test result is.
  • The browser will remain open only after the failed tests.
  • The default value is onFailure.

    Example:
    --setting testerum.selenium.leaveBrowserOpenAfterTest=true
    --setting testerum.selenium.leaveBrowserOpenAfterTest=false
    --setting testerum.selenium.leaveBrowserOpenAfterTest=onFailure

    --setting testerum.selenium.maximizeWindowBeforeTest=<true or false>

    With this option you can maximize the browser before the test execution.

    The default value is false.

    Example:
    --setting testerum.selenium.maximizeWindowBeforeTest=true

    --setting testerum.selenium.afterStepDelayMillis=<5000>

    With this option you can set up the time duration in milliseconds between Selenium steps.

    The default value is 0.

    Example:
    --setting testerum.selenium.afterStepDelayMillis=15000

    --setting testerum.selenium.waitTimeoutMillis=<5000>

    With this option you can set up the maximum time duration in milliseconds for wait steps(e.g. wait until an element is present on the page).

    The default value is 5000.

    Example:
    --setting testerum.selenium.waitTimeoutMillis=15000

    --setting testerum.selenium.driver.driverVersion=<version>

    Tells Testerum what version of Selenium driver to use.

    The default value is to use the most recent driver. Because of this, this setting is usually useful when running against an older version of a particular browser (that needs an older version of the Selenium driver).

    Example:
    --setting testerum.selenium.driver.driverVersion=75.0.3770.8

    --setting testerum.selenium.driver.remoteUrl=<URL>

    Tells Testerum to what Selenium driver server to connect.

    This setting is taken into consideration only if testerum.selenium.driver.browserType=REMOTE.
    Instead of running the Selenium driver server on the local machine, this setting tells Testerum to connect to an already-running Selenium server.
    This allows us to run tests against a browser that is running on a different machine than the one running the tests.

    To understand how this works, you need to understand how Testerum communicates with the browser:
    how Testerum communicates with the browser
    Normally, Testerum will start the Selenium driver. This is a separate process that opens an HTTP server. Testerum will then communicate with the Selenium driver server using the WebDriver protocol (over HTTP).
    The Selenium driver will then start the browser (separate process) and communicate with it using a proprietary browser protocol - each browser vendor is allowed to implement their own protocol between the driver and the browser.
    When using testerum.selenium.driver.browserType=REMOTE in combination with testerum.selenium.driver.remoteUrl, Testerum will no longer start a local Selenium driver process, but will attempt to connect to an already-running one.
    The Selenium driver is not required to run on the same machine as Testerum. This is very useful, since it allows us to run the Testerum tests in a CI/CD environment (where the operating system may not have a graphical user interface) and still test browsers like Microsoft Edge or Safari, which require a graphical user interface.
    One way to make this work is to start a Vagrant machine containing the Selenium driver and the browser before the tests, tell Testerum to connect to this machine, using it's IP, and then, after all the tests have finished running, stop the Vagrant machine.

    Example:
    --setting testerum.selenium.driver.browserType=REMOTE
    --setting testerum.selenium.driver.remoteUrl=http://192.168.1.123:9515

    --setting
    testerum.selenium.webDriverCustomizationScript=<Script>

    Enables advanced customization of the WebDriver> instance, like for example setting DesiredCapabilities.

    This can be done by calling methods on the capabilities> object, which is available in the context of this script.

    The type of this object depends on the selected browser, as follows:

    Example script (generic):
    capabilities.setCapability("os_version", "11");
    capabilities.setCapability("device", "iPhone 8 Plus");
    capabilities.setCapability("real_mobile", "true");
    capabilities.setCapability("browserstack.local", "false");
    
    Another example (for Chrome):
    capabilities.addArguments("start-maximized");
    
    var Proxy = Java.type("org.openqa.selenium.Proxy")
    var proxy = new Proxy();
    proxy.setHttpProxy("myhttpproxy:3337");
    capabilities.setCapability("proxy", proxy);
    
    Another example (for [BrowserStack](https://www.browserstack.com/)):
    var HashMap = Java.type('java.util.HashMap');
    var ArrayList = Java.type('java.util.ArrayList');
    
    capabilities.setCapability('os','OS X');
    capabilities.setCapability('os_version', 'Catalina');
    capabilities.setCapability('browser', 'Chrome');
    capabilities.setCapability('browser_version', '86');
    capabilities.setCapability('name', 'Chrome test');
    
    var chromeOptions = new HashMap();
    var chromeArgs = new ArrayList();
    chromeArgs.add('window-size=400,300');
    chromeOptions.put('args', chromeArgs);
    capabilities.setCapability('goog:chromeOptions', chromeOptions);
    
    More information for browser-specific options:

    Other

    --basic-steps-directory <dir-path>

    Path to the directory containing the basic steps jar files.

    If missing, the path will be the basic_steps directory in the Testerum installation directory.

    Note: you will very rarely need to use this argument.

    Example:
    --basic-steps-directory /path/to/testerum/basic_steps

    Examples

    The following examples assume the repository directory at /testerum-repo.

    Run all tests

    ./testerum-runner.sh --repository-directory /testerum-repo

    Run a single test

    ./testerum-runner.sh --repository-directory /testerum-repo --test-path "login/Correct login.test"

    Run all the tests in a feature directory

    ./testerum-runner.sh --repository-directory /testerum-repo --test-path "login/"

    Run multiple tests

    ./testerum-runner.sh --repository-directory /testerum-repo --test-path "login/Correct login.test" --test-path "checkout/"

    Produce managed reports (different sub-directory for each execution)

    ./testerum-runner.sh --repository-directory /testerum-repo --managed-reports-directory /test-reports