Overview

Test Automation provides a way for you to define tests. These are often used as smoke tests to ensure that the deployment completed successfully. They can also be used to run regression tests and any other testing that is targeted to the deployed target. Tests can be executed as part of build, deploy or utility workflow executions as well.

The Test Automation configurations, defined in YAML, are found in the project configuration.

The YAML that is shown here can also be stored in source control in the projet_root/fdtests folder to allow for versioning of the test configuration. For example:

Once the YAML file is configured on project and/or saved in your projects source code you are ready to begin executing tests!

Getting Started

Open the project you would like to add automated testing to and click Get Started.

A guide to getting started will be shown, similar to the following:

# To get started type 'test-' and select the template you want to use.
# YAML uses backslash as its escape character and eats the first layer of quote marks.
# Use Double Quotes around values containing '$' or '{'.

# To reference a property, use this syntax: 
# without groovy - value: "${PROPERTY_CODE}"
#    with groovy - value: "PROPERTY_CODE" 

# To reference a secure property on the deploy target group and hide the value from the UI, use this syntax: 
# without groovy - value: "\\$${FD_INSTANCE_CODE}_PROPERTY_CODE"

Armed with this information, type test- and select a tool you want to use for running tests.

Once selected, sample YAML configuration will be inserted that you can modify as you see fit. See the YAML definition below for more details.

Below is the generated YAML for Postman

In this particular example it can run as is provided the newman cli is installed, but we will make one modification to specify a specific Test Instance for it to run on. ctrl+space or simply start typing to see suggestions of Test Instances.

After saving our configuration we are ready to run some tests! Click on the TEST button on the main execution screen or use a Test All step in your pipeline.

YAML Definition

While modifying the test configuration in the editor, available fields and values will always be suggested to you. As such understanding the YAML definition below is certainly NOT REQUIRED but certainly can be helpful in understanding how things work.

Integrating with Release/Pipeline

The real power of test automation comes when integrated within FlexDeploy Release Orchestration platform. Here we can allow or block further deployments based on the test results of a project.

The above example is a minimalistic pipeline just to demonstrate how it integrates with FlexDeploy Test Automation framework.

• Executes any tests on projects in the Development Environment after deployment to the Development Environment. This means any tests with an env filter with a value of DEVELOPMENT or no filters at all would be executed as seen below.

  • If using Package-based projects, only unique Tests (by Test Name) will be executed when multiple Packages from same Project are included in Release.

---
testName: Postman Test
description: Generated Postman Test
filters: 
  - type: env
    value: DEVELOPMENT
workflow:
  name: PostmanDocker
  inputs: 
    - code: FDPMN_COLLECTION_FILE
      value: .*collection.json
     # (Optional) null
    - code: FDPMN_ENVIRONMENT_FILE
      value: fdtests/postman_environment_TOMCATPOSTGRES.json

• A Gate that will block or allow progression of deployments to the Test Stage.

• In the configuration for Step 2 we have said that All Tests must PASS in order for progression to continue.

Using variables in YAML test definitions

YAML uses backslash as its escape character and eats the first layer of quote marks.
Use Double Quotes around values containing '$' or '{'.

To reference a property, use this syntax:
without groovy - value is replaced in linux during plugin execution, in the ui, you will see ${PROPERTY_CODE}

value: "${PROPERTY_CODE}"

with groovy - value is replaced before plugin execution, in the ui, you will see the property value.

groovy: true
value: "PROPERTY_CODE"

To reference a secure property on the deploy target group and hide the value from the UI, use this syntax:
without groovy - value is replaced in linux during plugin execution, in the ui, you will see something like $POSTMAN_PROP_CD, but the value will be used.

value: "\$${FD_INSTANCE_CODE}_PROPERTY_CODE"

Configuration with a custom workflow

Other workflows can be designed and used in the same manner as this one. Create workflow inputs for any options you want to expose to the YAML file. Map the workflow inputs into plugin inputs as groovy values. Activate the workflow. Fill in the values in the YAML file, referencing the workflow input codes.

When you choose a workflow name, consider including one of the values from the workflow prefix table

Workflow Inputs

Workflow inputs specified in the YAML configuration directly correspond with the inputs as seen in the workflow definition.

Any naming change in the definition of the workflow inputs will need to be updated in test configurations as well.

TEST_INSTANCE Workflow Input

workflow:
  name: 
  inputs:
    - code: TEST_INSTANCE   
      value: 

TEST_INSTANCE is a special workflow input. It’s available values are filtered by the workflow name using the table below.

Then, add it to your workflow inputs and map it to your endpoint selection override. It controls where the test will be run.