Tests and runners form the backbone of this testing platform.

Background

See this doc for background on how this fits into the testing platform as a whole.

For help running your first test, see this doc.

What do they do?

The runnerConfig object allows you to provide important test-specific information, including information about what to include in the final reports.

Why do we need them?

In short, to run a single file/browser combination at a time from a larger, list-based definition.

There are a few benefits to that approach - if we want to use a proxy to capture/change network traffic, we need to add this level of abstraction. webdriver.io (our test runner, a wrapper for Selenium) would happily run them in parallel to the extent allowed by Sauce Labs (our Selenium grid). But a proxy would be added for ALL the tests, instead of each of them (it can't be added later than the onPrepare hook).

And since they'd run in parallel, so we wouldn't really be able to tell which entries in the network logs belonged to which steps of which tests. So, we run them one-at-a-time to allow for clean proxy logs when that service is used.

Also, Sauce Labs charges far more for more parallel connections, so using a single connection per engineer is much cheaper for Tealium.

How do I run them?

You can run them the command line:

$ npm run-script runner {PATH-TO-YOUR-RUNNER-CONFIG-FILE-NAME}.js

For instance, the integration test can be run with

$ npm run-script runner runners/configuration-files/self-tests/integration-test.js

See here to get started running tests.

Configuration Options

By default, the tests will run in a single browser (Chrome), on your local machine, without starting a Trace or LiveConnect (proxy) run, or providing much context in the final reports.

The runnerConfig object supports the following options:

Test Files and Context

• specs (required) - an array containing at least one entry, EITHER:
  • a string, which is the path to a spec file to run (like 'tests/examples/most-basic.js'), OR
  • an object with the following properties:
    • testFile (required) - the path to a spec file
    • testContext (optional) - a description of the test, which will appear in the report if populated. Supports Markdown.
    • testTitle (optional) - the title to use for this test in the report - defaults to the filename if not populated. Supports Markdown.
    • ANY RUNNER OPTIONS (below) are also accepted here, as per-file overrides.
• helpers (optional) - an array of helper files used by your specs. This array is only used to indicate which files (other than the specs) should be copies into the reports root folder for transparency - you still need to use require inside the relevant files to use the helpers.

Report Options and Description Fields

• summaryReportTitle
  • (optional) - the title for the Summary Report (defaults to 'Summary Report' if blank or missing).
• summaryContext
  • (optional) - a description of the group of tests, to be displayed in the report. Supports Markdown.
• reportContext
  • (optional) - if defined and non-empty, will override the general information about the testing platform that displays at the bottom of each page. An empty string will suppress that output entirely. Supports Markdown.
• showDurations
  • (optional) - if true, will show per-suite and per-test durations (run times), to help with performance optimization.
• showInBrowser
  • (optional) - if true, displays the report in a browser window.
  • (default) - true

Runner Options

• enableTrace (optional) - if true, a Tealium Trace will be started for each file/browser combination, provided traceAccount and traceProfile are also provided

• traceAccount (required if enableTrace is true) - the Tealium CDH account to use for the Trace(s). The same value is used for all file/browser combinations.

• traceProfile (required if enableTrace is true) - the Tealium CDH profile to use for the Trace(s). The same value is used for all file/browser combinations.

• enableProxy (optional) - if true, the network traffic for your test will be captured and made available for assertions. Will activate enableSauceConnect

• extraProxyCommands (optional) - any extra mitmproxy commands to use for this particular test. For instance, --map-remote @tags\.tiqcdn\.com\/utag\/tealium-solutions\/test-example\/prod\/utag\.js@\/prod\/@/dev/ will switch TiQ environments from prod to dev for tealium-solutions/test-example (though that can be done more easily with urlRewrites).

• urlRewrites (optional) - an array of url-rewriting configuration objects, each with the following properties:

  • pattern (required) - a match pattern string, evaluated against the URL, and interpreted as a string or regex based on the value of type
  • type (required) - the type of pattern, accepts string or regex
  • target (required) - the string that should be replaced in matching URLs
  • replacement (required) - the string with which each occurrence of target in matching URLs should be replaced

• blockPatterns (optional) - an array of block pattern configuration objects, each with the following properties:

  • pattern (required) - a match pattern string - interpreted as a string or regex based on the value of type
  • type (required) - the type of pattern, accepts string or regex
  • responseCode (optional) - the response code to use, defaults to 403
  • responseBody (optional) - the response body to use, defaults to 'Request blocked by the Tealium testing tool.'
  • responseContentType (optional) - the response content type header to use, defaults to text/html

• runRemotely (optional) - if true, runs tests in a single on Sauce Labs instead of locally. Can be overridden by runCrossBrowser and runFullCrossBrowser.

• runCrossBrowser (optional) - if true, uses a pre-defined (smaller) cross-browser list on Sauce Labs instead of locally. Can be overridden by runFullCrossBrowser

• runFullCrossBrowser (optional) - if true, uses a pre-defined (extensive and slow) cross-browser list on Sauce Labs instead of locally.

• runRetries (optional) - how many times to retry runs that fail (for any reason) - defaults to 0

• reportOnly (optional) - if true runs tests in browser headless mode. Usefull to run test scripts with API calls or Javascript only, usually used on tests which do not required a browser instance - defaults to false

Examples

A runner file is basically just an configuration object, which is passed to a runner method. Only the runnerConfig object should be changed for each specific test.

Here's the simplest possible example:

module.exports = {
  specs: [
    'tests/examples/most-basic.js'
  ]
}

And here's a more complex example:

module.exports = {

  specs: [
    {
      testFile: 'tests/self-tests/integration-tests/standard-integration-test.js',
      testContext: 'The core integration test, which uses all features.',
      // the rest of these override the runner-level settings for this file
      enableTrace: true,
      traceProfile: 'test-example',
      runRemotely: true,
      runRetries: 0,
      extraProxyCommands: '',
      blockPatterns: [{
        'pattern': 'utag.v.js?',
        'type': 'string'
      }],
      // switch to QA to activate the consent manager to make sure /event trace injection is working, as well as the environment switching
      urlRewrites: [{
        'pattern': 'tags.tiqcdn.com/utag/tealium-solutions/test-example/prod/utag.js',
        'type': 'string',
        'target': '/prod/',
        'replacement': '/qa/'
      }]
    }
  ],

  helpers: ['platform-helpers/standard-integration-test-helper.js'],

  summaryReportTitle: 'Integration Test Report',

  summaryContext: 'This integration test is designed to ensure that all components of the testing platform are working as expected.\n\nIt\'s intended to provide a way for operators to sanity check when working on tests (_is my test broken, or is this tool broken?_), and also helps platform development by ensuring new features don\'t break existing ones.',

  enableProxy: true,
  enableTrace: false,

  runCrossBrowser: false,
  runRemotely: false,
  runRetries: 2,

  // would switch to dev, but will be overridden by the entry in the 'specs' object
  extraProxyCommands: '--map-remote @tags\.tiqcdn\.com\/utag\/tealium-solutions\/test-example\/prod\/utag\.js@\/prod\/@/dev/',

  showDurations: true,

  traceAccount: 'tealium-solutions',
  traceProfile: 'test-exampleBROKEN'
}