1 Overview
2 Getting Started
3 YAML Definition
4 Integrating with Release/Pipeline
5 Using variables in YAML test definitions
6 Configuration with a custom workflow
7 Workflow Inputs
- 7.1 TEST_INSTANCE Workflow Input

Overview

Test Automation provides a way for you to define tests. These are often used as smoke tests to ensure that the deployment completed successfully. They can also be used to run regression tests and any other testing that is targeted to the deployed target. Tests can be executed as part of build, deploy or utility workflow executions as well.

The Test Automation configurations, defined in YAML, are found in the project configuration.

The YAML that is shown here can also be stored in source control in the projet_root/fdtests folder to allow for versioning of the test configuration. For example:

It would be better to use project configuration or source control to avoid confusion. If both configuration files are used then they will be combined such that project configuration will override source control file for same testName. We recommend using project configuration as it would be much easier to manage.
Inactivate test automation configuration on project will not be used.
Files stored in project_root/fdtests are discovered and automatically saved to artifacts directory on build execution. This applies to Package-based projects as well.
Changes in project_root/fdtests folder does not trigger CI build for Package-based projects as these files are not considered part of Package.

Once the YAML file is configured on project and/or saved in your projects source code you are ready to begin executing tests!

Getting Started

Open the project you would like to add automated testing to and click Get Started.

A guide to getting started will be shown, similar to the following:

# To get started type 'test-' and select the template you want to use.
# YAML uses backslash as its escape character and eats the first layer of quote marks.
# Use Double Quotes around values containing '$' or '{'.

# To reference a property, use this syntax: 
# without groovy - value: "${PROPERTY_CODE}"
#    with groovy - value: "PROPERTY_CODE" 

# To reference a secure property on the deploy target group and hide the value from the UI, use this syntax: 
# without groovy - value: "\\$${FD_INSTANCE_CODE}_PROPERTY_CODE"

Armed with this information, type test- and select a tool you want to use for running tests.

Once selected, sample YAML configuration will be inserted that you can modify as you see fit. See the YAML definition below for more details.

Below is the generated YAML for Postman

In this particular example it can run as is provided the newman cli is installed, but we will make one modification to specify a specific Test Instance for it to run on. ctrl+space or simply start typing to see suggestions of Test Instances.

Note that in our project source control we have sample postman collections under the fdtests folder at the root of our repository.

For trying out postman testing you can use this sample repository https://github.com/flexdeploy/postman-tests-sample

After saving our configuration we are ready to run some tests! Click on the TEST button on the main execution screen or use a Test All step in your pipeline.

YAML Definition

While modifying the test configuration in the editor, available fields and values will always be suggested to you. As such understanding the YAML definition below is certainly NOT REQUIRED but certainly can be helpful in understanding how things work.

Field	Data type	Description	Notes

Field	Data type	Description	Notes
[testRoot] denoted `---`	Array	Multiple tests can be defined in the YAML configuration and are separated by `---`	Every test in the file must start with `---` specified on its own line
testName	String	Unique name of the test within the YAML configuration	When executing tests you can specify a specific name filter to execute individual tests
description	String	Description of the test	-
workflow	Object	The workflow that will be executed when executing this test
workflow/name	String	The name of the workflow as it appears in the workflow UI	Valid test workflow names show as editor suggestions. Can be triggered by typing or hitting `ctrl+space` while on the name field
workflow/inputs	Array	A list of workflow inputs for the specified workflow. Only inputs configured on the workflow are valid here	All available inputs for the specified workflow will be auto inserted when auto completing the inputs field, provided the name has already been specified
workflow/inputs/code	String	Workflow Input Code for the specified workflow. It should match the code as seen on the workflow screen	Valid workflow input codes will be suggested provided the workflow name has been specified
workflow/inputs/value	String	The value of the current workflow input	Can be plain text or variables can be used.
workflow/inputs/groovy	Boolean	Flag indicating if the provided value should be treated as a groovy expression
tags	String	@Since 7.0.0.1 - A comma separated list of tags used for filtering tests on execution	Quick selection of tags can be accessed by hitting `ctrl+space`. This will display a list of tags already in used within the YAML
filters	Array	A list of filters that limit the conditions under which the given test will execute	For example an `env` filter with a value of DEVELOPMENT will ensure this test is only run when executing in the DEVELOPMENT environment
filters/type	String	The type of filter to use. Valid values are: `branch, env, file, pkg, targetGroup`
filters/value	String	The value of the supplied filter	Possible values are suggested based on the filter/type specified

Integrating with Release/Pipeline

The real power of test automation comes when integrated within FlexDeploy Release Orchestration platform. Here we can allow or block further deployments based on the test results of a project.

The above example is a minimalistic pipeline just to demonstrate how it integrates with FlexDeploy Test Automation framework.

Executes any tests on projects in the Development Environment after deployment to the Development Environment. This means any tests with an env filter with a value of DEVELOPMENT or no filters at all would be executed as seen below.
- If using Package-based projects, only unique Tests (by Test Name) will be executed when multiple Packages from same Project are included in Release.

---
testName: Postman Test
description: Generated Postman Test
filters: 
  - type: env
    value: DEVELOPMENT
workflow:
  name: PostmanDocker
  inputs: 
    - code: FDPMN_COLLECTION_FILE
      value: .*collection.json
     # (Optional) null
    - code: FDPMN_ENVIRONMENT_FILE
      value: fdtests/postman_environment_TOMCATPOSTGRES.json

A Gate that will block or allow progression of deployments to the Test Stage.
In the configuration for Step 2 we have said that All Tests must PASS in order for progression to continue.

Note that the criteria specified applies to each project on an individual basis, however all Projects must pass the criteria for the gate to succeed.

For example suppose the gate criteria is Percentage of Tests PASSED greater than 80% and 10 projects have tests in the Release. This means that all 10 projects need their individual passed test case percentage to be greater than 80 and not the combined percentage of all projects.

Using variables in YAML test definitions

YAML uses backslash as its escape character and eats the first layer of quote marks.
Use Double Quotes around values containing '$' or '{'.

To reference a property, use this syntax:
without groovy - value is replaced in linux during plugin execution, in the ui, you will see ${PROPERTY_CODE}

value: "${PROPERTY_CODE}"

with groovy - value is replaced before plugin execution, in the ui, you will see the property value.

To reference a secure property on the deploy target group and hide the value from the UI, use this syntax:
without groovy - value is replaced in linux during plugin execution, in the ui, you will see something like $POSTMAN_PROP_CD, but the value will be used.

Configuration with a custom workflow

Other workflows can be designed and used in the same manner as this one. Create workflow inputs for any options you want to expose to the YAML file. Map the workflow inputs into plugin inputs as groovy values. Activate the workflow. Fill in the values in the YAML file, referencing the workflow input codes.

When you choose a workflow name, consider including one of the values from the workflow prefix table

Workflow Inputs

Workflow inputs specified in the YAML configuration directly correspond with the inputs as seen in the workflow definition.

Any naming change in the definition of the workflow inputs will need to be updated in test configurations as well.

TEST_INSTANCE Workflow Input

TEST_INSTANCE is a special workflow input. It’s available values are filtered by the workflow name using the table below.

If your workflow contains (case insensitively)	Test Instances will be filtered to type

If your workflow contains (case insensitively)	Test Instances will be filtered to type
soapui	SOAP_UI
postman	POSTMAN
junit	JUNIT
testng	TEST_NG
hpt	HP_UFT
apifortress	API_FORTRESS
automationany	AUTOMATION_ANYWHERE
cucumber	CUCUMBER
jmeter	APACHE_JMETER
oats	OATS
salesforce	SALESFORCE
tricentis	TRICENTIS
utpl	UTPLSQL

Then, add it to your workflow inputs and map it to your endpoint selection override. It controls where the test will be run.