Azure Synapse Deployment - Package Based Approach

Objective

We have previously specified how Azure Synapse Artifacts deployment can be achieved using FlexDeploy in this article. In the current article we are going to elaborate more on how to implement package-based deployment for the same.

Design

There are two approaches we follow here:

Approach A:

Configuring a Generic Package based project, create and deploy the ARM templates being generated based on those specific resources.

Approach B:

First Phase - Configuring a Generic Package based project, create the ARM template based on the specific resources and store it in a repository. This repository may contain all the ARM templates corresponding to different packages. Will showcase it later in this tutorial.

Second Phase - Configuring another package-based project that will fetching the ARM template from the repository and initiate the Synapse Release pipeline.

The benefit of the second approach is there would be more control over the ARM template which can be edited as needed by adding a new parameter or variables or something else.

Flow (Approach A)

Workflow configuration

Build Workflow:

steps:
  - id: '1'
    name: publishArtifacts
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonAzurePlugin
      pluginOperation: publishArtifacts
      inputs:
        - name: FDAZ_DEVOPS_INP_FEED
          value:
            value: FEED_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PACKAGE_NAME
          value:
            value: FD_PACKAGE_NAME + "-" + ARTIFACTS_PACKAGE_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PACKAGE_VERSION
          value:
            value: PACKAGE_VERSION
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PACKAGE_DESCRIPTION
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PROJECT_NAME
          value:
            value: PROJECT_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PACKAGE_PATH
          value:
            value: FD_TEMP_DIR
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_ADDITIONAL_PARAMS
          value:
            isExpression: false
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs:
        - output: FDAZ_DEVOPS_OUT_ERR
        - output: FDAZ_DEVOPS_OUT_RESP
      userInputs: []
      userOutputs: []
  - id: '2'
    name: buildPipeline
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonAzurePlugin
      pluginOperation: buildPipeline
      inputs:
        - name: FDAZ_DEVOPS_INP_DEFINITION_NAME
          value:
            value: BUILD_DEFINITION_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_BUILD_DEFINITION_ID
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_QUEUE_ID
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_COMMIT_ID
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_BRANCH_NAME
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PROJECT_NAME
          value:
            value: PROJECT_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_VARIABLES_LIST
          value:
            value: >-
              VARIABLES_LIST + "##artifactsPackageName:" + FD_PACKAGE_NAME + "-"
              + ARTIFACTS_PACKAGE_NAME + "##feedName:" + PROJECT_NAME + "/" +
              FEED_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PACKAGE_NAME
          value:
            value: FD_PACKAGE_NAME + "-" + PACKAGE_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_ADDITIONAL_PARAMS
          value:
            isExpression: false
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs:
        - output: FDAZ_DEVOPS_OUT_ERR
        - output: FDAZ_DEVOPS_OUT_RESP
        - output: FDAZ_DEVOPS_OUT_BUILD_PIPELINE_WEB_URL
      userInputs: []
      userOutputs: []
  - id: '3'
    name: downloadArtifacts
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonAzurePlugin
      pluginOperation: downloadArtifacts
      inputs:
        - name: FDAZ_DEVOPS_INP_FEED
          value:
            value: FEED_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PACKAGE_NAME
          value:
            value: FD_PACKAGE_NAME + "-" + PACKAGE_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PACKAGE_VERSION
          value:
            value: PACKAGE_VERSION
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PROJECT_NAME
          value:
            value: PROJECT_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PACKAGE_PATH
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_ADDITIONAL_PARAMS
          value:
            isExpression: false
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: true
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs:
        - output: FDAZ_DEVOPS_OUT_ERR
        - output: FDAZ_DEVOPS_OUT_RESP
      userInputs: []
      userOutputs: []

Deploy Workflow:

variables:
  - code: ENV_DETAILS
    dataType: String
    returnAsOutput: true
    constant: false
    encrypted: false
    scope: LOCAL
  - code: RELEASE_ID
    dataType: String
    returnAsOutput: true
    constant: false
    encrypted: false
    scope: LOCAL
  - code: ENVIRONMENT_ID
    dataType: String
    returnAsOutput: true
    constant: false
    encrypted: false
    scope: LOCAL
  - code: RELEASE_URL
    dataType: String
    returnAsOutput: true
    constant: false
    encrypted: false
    scope: LOCAL
steps:
  - id: '1'
    name: createRelease
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonAzurePlugin
      pluginOperation: createRelease
      inputs:
        - name: FDAZ_DEVOPS_INP_RELEASE_DEFINITION_ID
          value:
            value: RELEASE_DEFINITION_ID
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_ARTIFACTS_PAYLOAD
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PROJECT_NAME
          value:
            value: PROJECT_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_VARIABLES_LIST
          value:
            value: >-
              "feedName:" + PROJECT_NAME + "/" + FEED_NAME + "##packageName:" +
              FD_PACKAGE_NAME + "-" + PACKAGE_NAME + "##packageVersion:" +
              PACKAGE_VERSION
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PROPERTIES
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_MANUAL_ENVIRONMENTS
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_RELEASE_STATUS_CHECK
          value:
            value: 'false'
            isExpression: false
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs:
        - output: FDAZ_DEVOPS_OUT_RESP
        - output: FDAZ_DEVOPS_OUT_RELEASE_WEB_URL
        - output: FDAZ_DEVOPS_OUT_REL_ENV_STAGE_DETAILS
          variable: ENV_DETAILS
        - output: FDAZ_DEVOPS_OUT_RELEASE_ID
          variable: RELEASE_ID
      userInputs: []
      userOutputs: []
  - id: '2'
    name: Parse Environment specific Id
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonXPathPlugin
      pluginOperation: parseJson
      inputs:
        - name: FDP_SOURCE
          value:
            value: ENV_DETAILS
            isExpression: true
          isEncrypted: false
        - name: FDP_XPATH
          value:
            value: '''$..[?(@.name==\''''+FD_ENVIRONMENT_NAME+''\'')].id'''
            isExpression: true
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs:
        - output: FDP_VALUE
          variable: ENVIRONMENT_ID
      userInputs: []
      userOutputs: []
  - id: '3'
    name: updateReleaseEnvStatus
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonAzurePlugin
      pluginOperation: updateReleaseEnvStatus
      inputs:
        - name: FDAZ_DEVOPS_INP_RELEASE_ID
          value:
            value: RELEASE_ID
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_RELEASE_ENVIRONMENT_ID
          value:
            value: ENVIRONMENT_ID.substring(1,ENVIRONMENT_ID.length()-1)
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_PROJECT_NAME
          value:
            value: PROJECT_NAME
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_RELEASE_ENVIRONMENT_STATUS
          value:
            value: InProgress
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_VARIABLES_LIST
          value:
            value: ENVIRONMENT_VARIABLES_LIST
            isExpression: true
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_RELEASE_COMMENT
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_SCHEDULED_DEPLOYMENT_TIME
          value:
            isExpression: false
          isEncrypted: false
        - name: FDAZ_DEVOPS_INP_REL_ENV_STATUS_CHECK
          value:
            value: 'true'
            isExpression: false
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs:
        - output: FDAZ_DEVOPS_OUT_RESP
        - output: FDAZ_DEVOPS_OUT_REL_ENV_WEB_URL
          variable: RELEASE_URL
      userInputs: []
      userOutputs: []

Project Configuration

Create a Project by choosing the Classification: Package-based and Project Types as Generic

Once it is created the Source Control needs to be configured, so that we can discover the files from the respective repository.

For example, this is how the repository looks like where we have all the artifacts uploaded from Synapse Dev Workspace.

Discover the files using File Catalog tab

Now we can create Package upon selecting specific files for which the ARM template should get generated and the specific resources would be deployed to the Target Synapse Workspace.

Say we have two pipelines in repository but want to deploy only one to the target workspace, in that case we can create a package and add the pipeline file in it.

Click on the Configuration tab, select General and update the Build and Deploy workflows along with respective Target groups.

Execution

Initiate Build on the selected Package “pipeline”.

Note: The ARM templates are published to Azure DevOps Feed as Artifacts with the package name as a keyword in it. Example: if the FD project name is SynapseOrchestration-Incremental and the package name is “pipeline” then the artifacts package name would be “pipeline-synapseorchestration-incremental-armtemplate“. Now Microsoft has certain restrictions on the naming of the Azure Artifacts, hence please follow the guide before you name the package that you create in FD or else adjust your workflow to be in compliance. Restrictions and conventions for naming objects - Azure DevOps

To verify, clicked on the Artifacts tab and verified the ARM template and it shows up with just one selected pipeline.

Now initiated the Deploy.

Flow (Approach B)

Workflow configuration for creating ARM template and upload to Repository

Build Workflow:

Note: For uploading the ARM template to repository, we have created another DevOps repository as az-synapse-arm and configured it in FlexDeploy-Integration with the code SYNAPSEARM.

If you have some other repository present in GITHUB or Azure DevOps anywhere and want to use that, then please create accordingly and use the correct Instance Code in the workflow pasted below.

steps:
  - id: '1'
    name: clone
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonGITPlugin
      pluginOperation: clone
      inputs:
        - name: FDGIT_INP_INSTANCE_CODE
          value:
            value: SYNAPSEARM
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_DEST_SUBFOLDER
          value:
            value: ARMTemplates
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_BRANCH
          value:
            value: flexdeploy
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_TREELESS_CLONE
          value:
            value: 'false'
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_DEPTH
          value:
            value: ''
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_SPARSE_CHECKOUT_FOLDERS
          value:
            isExpression: false
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs: []
      userInputs: []
      userOutputs: []
  - id: '2'
    name: Invoke Workflow
    type: INVOKE_WORKFLOW
    data:
      inputs: []
      workflowId: 7080183
      outputs: []
  - id: '3'
    name: copy
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonFilePlugin
      pluginOperation: copy
      inputs:
        - name: FDFILE_INP_FILE_FILTER
          value:
            value: Template*.json
            isExpression: false
          isEncrypted: false
        - name: FDFILE_INP_FILE_FILTER_EXCLUDED
          value:
            isExpression: false
          isEncrypted: false
        - name: FDFILE_INP_SOURCE_PATH
          value:
            value: FD_ARTIFACTS_DIR
            isExpression: true
          isEncrypted: false
        - name: FDFILE_INP_TARGET_PATH
          value:
            value: FD_TEMP_DIR+"/ARMTemplates/"+FD_PACKAGE_NAME
            isExpression: true
          isEncrypted: false
        - name: FDFILE_INP_CLEAN_DIRECTORY
          value:
            value: 'true'
            isExpression: false
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs: []
      userInputs: []
      userOutputs: []
  - id: '4'
    name: add
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonGITPlugin
      pluginOperation: add
      inputs:
        - name: FDGIT_INP_INSTANCE_CODE
          value:
            value: SYNAPSEARM
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_DEST_SUBFOLDER
          value:
            value: ARMTemplates
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_FILELIST
          value:
            value: .
            isExpression: false
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs: []
      userInputs: []
      userOutputs: []
  - id: '5'
    name: commit
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonGITPlugin
      pluginOperation: commit
      inputs:
        - name: FDGIT_INP_INSTANCE_CODE
          value:
            value: SYNAPSEARM
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_DEST_SUBFOLDER
          value:
            value: ARMTemplates
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_NAME
          value:
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_EMAIL
          value:
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_REVISION_MESSAGE
          value:
            value: '"ARM Template updated: "+ FD_PROJECT_VERSION'
            isExpression: true
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs: []
      userInputs: []
      userOutputs: []
  - id: '6'
    name: push
    type: INVOKE_PLUGIN
    data:
      pluginName: FlexagonGITPlugin
      pluginOperation: push
      inputs:
        - name: FDGIT_INP_INSTANCE_CODE
          value:
            value: SYNAPSEARM
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_DEST_SUBFOLDER
          value:
            value: ARMTemplates
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_BRANCH
          value:
            value: flexdeploy
            isExpression: false
          isEncrypted: false
        - name: FDGIT_INP_FOLLOW_TAGS
          value:
            value: 'false'
            isExpression: false
          isEncrypted: false
      endpointInstanceOverride:
        isExpression: false
      consumesArtifacts: false
      producesArtifacts: false
      endpointSelection:
        choice: All
      endpointExecution:
        choice: Any
        stopOnError: false
      outputs: []
      userInputs: []
      userOutputs: []

Project Configuration

Create a Project by choosing the Classification: Package-based and Project Types as Generic

Once it is created the Source Control needs to be configured, so that we can discover the files from the respective repository.

For example, this is how the repository looks like where we have all the artifacts uploaded from Synapse Dev Workspace.

Discover the files using File Catalog tab

Now we can create Package upon selecting specific files for which the ARM template should get generated and the specific resources would be deployed to the Target Synapse Workspace.

Say we want to selectively migrate the linkedservice AzureBlobStorageDemo to the target workspace, in that case we can create a package linkedservice and add it in.

Click on the Configuration tab, select General and update the Build workflow with respective Target groups.

We are not configuring any Deploy workflow in this Project for the purpose of this Tutorial, as all we need is to create the ARM template and upload it to GIT.

Execution

Initiate Build on the selected Package “linkedservice”.

Note: The ARM templates are published to Azure DevOps Feed as Artifacts with the package name as a keyword in it. Example: if the FD project name is ExtractSynapseArtifacts and the package name is “linkedservice” then the artifacts package name would be “linkedservice-extractsynapseartifacts-armtemplate“. Now Microsoft has certain restrictions on the naming of the Azure Artifacts, hence please follow the guide before you name the package that you create in FD or else adjust your workflow to be in compliance. Restrictions and conventions for naming objects - Azure DevOps

To verify, clicked on the Artifacts tab and verified the ARM template and it shows up with just one selected pipeline.

The ARM template is now uploaded to repository

The folder name is following the naming we selected for the package name in FD for right control. If any parameter is to be added/updated or some other manipulation is to be done in the ARM template json in terms of variables or parameters, it can be done. But the integrity of the json file should remain intact.

Workflow configuration for fetching the ARM template from the repository and initiate the Synapse Release pipeline

Build Workflow:

Deploy Workflow:

Project Configuration

Create a Project by choosing the Classification: Package-based and Project Types as Generic

Once it is created the Source Control needs to be configured, so that we can discover the files from the respective repository. This time the source control is the one where we have uploaded the ARM template.

Discover the files using File Catalog tab

Now we can create Package upon selecting specific ARM template that to be deployed to the Target Synapse Workspace.

Click on the Configuration tab, select General and update the Build and Deploy workflows with respective Target groups.

Parameterizing the values

Replacement configuration on Project

Starting 7.0 we have a new feature that allows us to replace a string in any file with another string based on as it is executed to different environment. Ref: Replacements

For our use-case a specific parameter as below is created in the Linked Service:

Our objective is that when this linked service gets deployed to target cicd-demo-uat workspace, the value would get replaced with “UAT”.

In the ARM template that we stored in repository, it shows up like this:

Using Environment Variables for the parameters

Say the linked service we are using is having the connection params in the source workspace as the following:

The parameter file in ARM template folder looks like:

If the value for the Storage Account is to be changed then the parameter needs to be passed in AzureBlobStorageDemo_connectionString.

Hence, we can configure the below connection string with the parameter name in the Target Group properties for the code: ENVIRONMENT_VARIABLES_LIST.

Anything that is specified as parameter in the ARM templates, can be parameterized like above during deployment to Target Workspace

Execution

Initiate Build on the selected Package “linkedservice”.

To verify, clicked on the Artifacts tab and verified the ARM template.

Click on Deploy for the same

Post deployment if we verify the Linked Service in UAT workspace, we see the updated values in there: