Reporting on scenario runs

To report on scenario activity, reporters can be added to the scenario.

Scenario reporters execution

Messages can be sent from anywhere in a scenario run:

  • to send a message right before a scenario run starts, add a reporter to the scenario in the Settings pane, and set Send on scenario... to Start
  • to send a message during the scenario run, add a Send message step (in step-based scenario) or use get_message_sender() on the Scenario object (in a script scenario)
  • to send a message right after a scenario run ends, add a reporter to the scenario in the Settings pane, and set Send on scenario... to End

For a reporter running at the end of the scenario run, it is possible to further control whether the reporter is run or not using a Run condition. Typically the message reported at the end of the run depends on its outcome, so the user can set up several reporters using as Run condition the expression outcome == ‘SUCCESS’ or outcome == ‘FAILED’ or outcome == ‘ABORTED’, each reporter holding a message corresponding to the filtered outcome.

Building the message contents

Each reporter needs to define a message to broadcast on the selected messaging channel, and/or for some channels variables to pass to the reporter itself. In terms of message building abilities, reporting to mail is the most versatile.

Mail reporter

The message sent by mail can be html or plain text, and 2 engines are available to build it:

  • Freemarker-based templating
  • variables-based formatting using DSS formulas

In both cases, the scenario run results are offered to the engine as named variables.

DSS comes with a default Freemarker-based template, and changing the Message source to Inline gives the user the possibility to write its own.

Hipchat, Slack, Webhook and Twilio reporters

These reporters only offer variables-based formatting using DSS formulas for the message body. These reporters also take additional parameters, like message color or sender alias, that can be computed using DSS formulas. Note that Slack uses a specific format for its messages, and leaves html as-is.

Slack integration setup

Slack provides 2 ways to automatically send messages on a channel. Both are available in DSS:

  • using an incoming webhook. Using an incoming webhook in a DSS integration means selecting the WEBHOOK mode and specifying the webhook’s URL, that you can see by navigating to Apps & Integrations, Manage, Custom integrations and finally Incoming webhooks. To create an incoming webhook in your Slack channel, you need to go to Apps & Integrations, Build (top right corner), Make a Custom integration, and from there you can create a new Incoming webhook.
  • using a bot user. The advantage over a simple incoming webhook is that the bot can have a preset appearance in the Slack channel. Using a bot user in a DSS integration means selecting the API mode and specifying the bot’s API token as authentication token. The API token can be found by navigating to Apps & Integrations, Build, Make a Custom integration and finally Bots.

It is also possible to use a testing token instead of a bot user, since they rely on the same API token mechanism. Once they are created, you can access your bot’s or incoming webhook’s tokens by going to Apps & Integrations, Manage (top right corner) and Custom integrations.

Shell reporter

The scenario run results are sent to the shell script defined in the administration section by means of environment variables whose value the user can set up using DSS formulas.

Common variables

Variables, from the DSS instance, from the project and from the scenario run, can be used in the message. Two mechanisms are also available to help customizing the message:

  • new variables can be defined with a custom Python script
  • the variables in the message are handled as DSS formulas