Azure Synapse Deployment - Package Based Approach

Objective

We have previously specified how Azure Synapse Artifacts deployment can be achieved using FlexDeploy in this article. In the current article we are going to elaborate more on how to implement package-based deployment for the same.

Design

There are two approaches we follow here:

  • Approach A: (Recommended)

First Phase - Configuring a Generic Package based project. Select the JSON objects (say Linked Services, Pipeline etc.), generate the ARM template based on the selected resources and store it in a repository. This repository may contain all the ARM templates corresponding to different packages. Will showcase it later in this tutorial.

Second Phase - Configuring another package-based project that will fetching the ARM template from the repository as specified in previous phase and initiate the Synapse Release pipeline to deploy them to target workspaces.

The benefit of the second approach is there would be more control over the ARM template which can be edited as needed by adding a new parameter or variables etc.

  • Approach B:

Configuring a Generic Package based project. Select the JSON objects (say Linked Services, Pipeline etc.), generate the ARM template based on the selected resources and deploy the ARM template to target workspaces. In this case the customization on parameters needs to be directly done on the JSON files by the user.

Flow (Approach A)

Workflow configuration for generating ARM template and upload to Repository

Build Workflow:

Note: For uploading the ARM template to repository, we have created another DevOps repository as az-synapse-arm and configured it in FlexDeploy-Integration with the code SYNAPSEARM.

If you have some other repository present in GITHUB or Azure DevOps anywhere and want to use that, then please create accordingly and use the correct Instance Code in the workflow pasted below.

image-20240329-080026.png

 

description: GenerateARMAndSyncToGIT steps: - id: '1' name: execute type: INVOKE_PLUGIN data: pluginName: FlexagonShellPlugin pluginOperation: execute inputs: - name: FDSHELL_INP_CODE_SNIPPET value: value: "# Source directory (parent directory A)\r\nSRC_DIR=$FD_TEMP_DIR\r\n\r\n# Target directory (parent directory B)\r\nDEST_DIR=$FD_TEMP_DIR/JSON\r\n\r\nmkdir -p $FD_TEMP_DIR/JSON\r\nchmod -R 777 $FD_TEMP_DIR/JSON\r\n\r\n# Copy .json files while preserving directory structure\r\nfind \"$SRC_DIR\" -type f -name \"*.json\" | while read -r file; do\r\n # Get the directory path relative to the source\r\n rel_path=$(dirname \"${file#$SRC_DIR/}\")\r\n \r\n # Create the same directory structure in the target\r\n mkdir -p \"$DEST_DIR/$rel_path\"\r\n \r\n # Copy the file to the target directory\r\n cp \"$file\" \"$DEST_DIR/$rel_path\"\r\ndone\r\n\r\necho \"All JSON files have been copied successfully.\"" isExpression: false isEncrypted: false - name: FDSHELL_INP_STOP_ON_ERROR value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_DISABLE_ECHO value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_RESTRICT_ENVIRONMENT value: isExpression: false isEncrypted: false - name: FDSHELL_INP_NO_SECURE_VARIABLES value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_LOCK value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: [] - id: '2' name: publishArtifacts type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: publishArtifacts inputs: - name: FDAZ_DEVOPS_INP_FEED value: value: FEED_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_NAME value: value: (FD_PACKAGE_NAME + "-" + ARTIFACTS_PACKAGE_NAME).toLowerCase() isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_VERSION value: value: PACKAGE_VERSION isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_DESCRIPTION value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_PATH value: value: FD_TEMP_DIR + "/JSON" isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_ADDITIONAL_PARAMS value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_ERR - output: FDAZ_DEVOPS_OUT_RESP userInputs: [] userOutputs: [] - id: '3' name: buildPipeline type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: buildPipeline inputs: - name: FDAZ_DEVOPS_INP_DEFINITION_NAME value: value: BUILD_DEFINITION_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_BUILD_DEFINITION_ID value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_QUEUE_ID value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_COMMIT_ID value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_BRANCH_NAME value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_VARIABLES_LIST value: value: >- "workspaceName:cicd-demo-test##artifactsPackageName:" + (FD_PACKAGE_NAME + "-" + ARTIFACTS_PACKAGE_NAME).toLowerCase() + "##feedName:" + PROJECT_NAME + "/" + FEED_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_NAME value: value: (FD_PACKAGE_NAME + "-" + PACKAGE_NAME).toLowerCase() isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_ADDITIONAL_PARAMS value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_ERR - output: FDAZ_DEVOPS_OUT_RESP - output: FDAZ_DEVOPS_OUT_BUILD_PIPELINE_WEB_URL userInputs: [] userOutputs: [] - id: '4' name: downloadArtifacts type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: downloadArtifacts inputs: - name: FDAZ_DEVOPS_INP_FEED value: value: FEED_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_NAME value: value: (FD_PACKAGE_NAME + "-" + PACKAGE_NAME).toLowerCase() isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_VERSION value: value: PACKAGE_VERSION isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_PATH value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_ADDITIONAL_PARAMS value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: true endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_ERR - output: FDAZ_DEVOPS_OUT_RESP userInputs: [] userOutputs: [] - id: '5' name: clone type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: clone inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: SYNAPSEARM isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: value: ARMTemplates isExpression: false isEncrypted: false - name: FDGIT_INP_BRANCH value: value: importedartifacts isExpression: false isEncrypted: false - name: FDGIT_INP_TREELESS_CLONE value: value: 'false' isExpression: false isEncrypted: false - name: FDGIT_INP_DEPTH value: value: '' isExpression: false isEncrypted: false - name: FDGIT_INP_SPARSE_CHECKOUT_FOLDERS value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: [] - id: '6' name: copy type: INVOKE_PLUGIN data: pluginName: FlexagonFilePlugin pluginOperation: copy inputs: - name: FDFILE_INP_FILE_FILTER value: value: Template*.json isExpression: false isEncrypted: false - name: FDFILE_INP_FILE_FILTER_EXCLUDED value: isExpression: false isEncrypted: false - name: FDFILE_INP_SOURCE_PATH value: value: FD_ARTIFACTS_DIR isExpression: true isEncrypted: false - name: FDFILE_INP_TARGET_PATH value: value: FD_TEMP_DIR+"/ARMTemplates/"+FD_PACKAGE_NAME isExpression: true isEncrypted: false - name: FDFILE_INP_CLEAN_DIRECTORY value: value: 'true' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: [] - id: '7' name: add type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: add inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: SYNAPSEARM isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: value: ARMTemplates isExpression: false isEncrypted: false - name: FDGIT_INP_FILELIST value: value: . isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: [] - id: '8' name: commit type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: commit inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: SYNAPSEARM isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: value: ARMTemplates isExpression: false isEncrypted: false - name: FDGIT_INP_NAME value: isExpression: false isEncrypted: false - name: FDGIT_INP_EMAIL value: isExpression: false isEncrypted: false - name: FDGIT_INP_REVISION_MESSAGE value: value: '"ARM Template updated: "+ FD_PROJECT_VERSION' isExpression: true isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: [] - id: '9' name: push type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: push inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: SYNAPSEARM isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: value: ARMTemplates isExpression: false isEncrypted: false - name: FDGIT_INP_BRANCH value: value: importedartifacts isExpression: false isEncrypted: false - name: FDGIT_INP_FOLLOW_TAGS value: value: 'false' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: []
image-20241021-160459.png

Project Configuration

Create a Project by choosing the Classification: Package-based and Project Types as Generic

Once it is created the Source Control needs to be configured, so that we can discover the files from the respective repository.

For example, this is how the source control looks like:

Discover the files using File Catalog tab

Now we can create Package upon selecting specific JSON files.

Click on the Configuration tab, select General and update the Build workflow with respective Target group.

We are not configuring any Deploy workflow in this Project for the purpose of this Tutorial, as all we need is to generate the ARM template and upload it to GIT.

Project properties

Follow this link to know about the Azure DevOps Build Pipeline configuration. This must be configured prior to configuring the FlexDeploy workflows and project properties.

Execution

Initiate Build on the selected Package.

Note: The ARM templates are published to Azure DevOps Feed as Artifacts with the package name as a keyword in it. Example: <packagename>-<fdprojectname>-json & <packagename>-<fdprojectname>-arm. Now Microsoft has certain restrictions on the naming of the Azure Artifacts, hence please follow the guide before you name the package that you create in FD or else adjust your workflow to be in compliance. https://learn.microsoft.com/en-us/azure/devops/organizations/settings/naming-restrictions?view=azure-devops

The ARM template is now uploaded to repository

The folder name is following the naming we selected for the package name in FD. If any parameter is to be added/updated or some other manipulation is to be done in the ARM template json in terms of variables or parameters, it can be done. But the integrity of the ARM template should remain intact.

Workflow configuration for fetching the ARM template from the repository and initiate the Synapse Release pipeline

Build Workflow:
description: SynapsePackageBuildARM steps: - id: '1' name: Execute type: INVOKE_PLUGIN data: pluginName: FlexagonShellPlugin pluginOperation: execute inputs: - name: FDSHELL_INP_CODE_SNIPPET value: value: "# Define the name of the zip file to be created\r\nZIP_NAME=$FD_PROJECT_NAME.zip\r\n\r\nmkdir -p $FD_TEMP_DIR/ARMTEMPLATES\r\nfind $FD_TEMP_DIR -type f -name \"Template*.json\" -exec cp {} $FD_TEMP_DIR/ARMTEMPLATES \\;\r\ncd $FD_TEMP_DIR/ARMTEMPLATES\r\n\r\n# Find and zip all Template*.json files into a single zip file\r\nzip \"$ZIP_NAME\" *\r\ncp $FD_TEMP_DIR/ARMTEMPLATES/$ZIP_NAME $FD_ARTIFACTS_DIR\r\n\r\necho \"Zip created and moved to artifacts.\"\r\n" isExpression: false isEncrypted: false - name: FDSHELL_INP_STOP_ON_ERROR value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_DISABLE_ECHO value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_RESTRICT_ENVIRONMENT value: isExpression: false isEncrypted: false - name: FDSHELL_INP_NO_SECURE_VARIABLES value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_LOCK value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: true endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: []
Deploy Workflow:
name: SynapsePackageDeployARM description: SynapsePackageDeployARM variables: - code: ENV_DETAILS dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL - code: RELEASE_ID dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL - code: ENVIRONMENT_ID dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL - code: RELEASE_URL dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL - code: PKG_NAME dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL - code: FINALVERSION dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL steps: - id: '1' name: execute type: INVOKE_PLUGIN data: pluginName: FlexagonShellPlugin pluginOperation: execute inputs: - name: FDSHELL_INP_CODE_SNIPPET value: value: "mkdir -p $FD_TEMP_DIR/ARMFiles\r\nunzip $FD_ARTIFACTS_DIR/*.zip -d $FD_TEMP_DIR/ARMFiles" isExpression: false isEncrypted: false - name: FDSHELL_INP_STOP_ON_ERROR value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_DISABLE_ECHO value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_RESTRICT_ENVIRONMENT value: isExpression: false isEncrypted: false - name: FDSHELL_INP_NO_SECURE_VARIABLES value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_LOCK value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: true producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: [] - id: '2' name: propertyReplacement type: INVOKE_PLUGIN data: pluginName: FlexagonFilePlugin pluginOperation: propertyReplacement inputs: - name: FDFILE_INP_SOURCE_PATH value: value: FD_TEMP_DIR + "/ARMFiles" isExpression: true isEncrypted: false - name: FDFILE_INP_FILE_FILTER value: isExpression: false isEncrypted: false - name: FDFILE_INP_FILE_FILTER_EXCLUDED value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: [] - id: '3' name: Assign type: ASSIGN data: from: >- PACKAGE_VERSION.substring(0,PACKAGE_VERSION.lastIndexOf(".") +1) + FD_WORKFLOW_EXECUTION_ID to: FINALVERSION - id: '4' name: publishArtifacts type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: publishArtifacts inputs: - name: FDAZ_DEVOPS_INP_FEED value: value: FEED_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_NAME value: value: >- (FD_PACKAGE_NAME + "-" + PACKAGE_NAME + "-" + FD_ENVIRONMENT_CODE).toLowerCase() isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_VERSION value: value: FINALVERSION isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_DESCRIPTION value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PACKAGE_PATH value: value: FD_TEMP_DIR + "/ARMFiles" isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_ADDITIONAL_PARAMS value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_ERR - output: FDAZ_DEVOPS_OUT_RESP userInputs: [] userOutputs: [] - id: '5' name: createRelease type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: createRelease inputs: - name: FDAZ_DEVOPS_INP_RELEASE_DEFINITION_ID value: value: RELEASE_DEFINITION_ID isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_ARTIFACTS_PAYLOAD value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_VARIABLES_LIST value: value: >- "{\"feedName\": {\"value\": \"" + PROJECT_NAME + "/" + FEED_NAME + "\"},\"packageName\": {\"value\": \"" + (FD_PACKAGE_NAME + "-" + PACKAGE_NAME + "-" + FD_ENVIRONMENT_CODE).toLowerCase() + "\"},\"packageVersion\": {\"value\": \"" + FINALVERSION + "\"}}" isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PROPERTIES value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_MANUAL_ENVIRONMENTS value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_STATUS_CHECK value: value: 'false' isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_PARTIALLY_SUCCEEDED_ENVIRONMENTS value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_RESP - output: FDAZ_DEVOPS_OUT_RELEASE_WEB_URL - output: FDAZ_DEVOPS_OUT_REL_ENV_STAGE_DETAILS variable: ENV_DETAILS - output: FDAZ_DEVOPS_OUT_RELEASE_ID variable: RELEASE_ID userInputs: [] userOutputs: [] - id: '6' name: parseJson type: INVOKE_PLUGIN data: pluginName: FlexagonXPathPlugin pluginOperation: parseJson inputs: - name: FDP_SOURCE value: value: ENV_DETAILS isExpression: true isEncrypted: false - name: FDP_XPATH value: value: '''$..[?(@.name==\''''+FD_ENVIRONMENT_NAME+''\'')].id''' isExpression: true isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDP_VALUE variable: ENVIRONMENT_ID userInputs: [] userOutputs: [] - id: '7' name: updateReleaseEnvStatus type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: updateReleaseEnvStatus inputs: - name: FDAZ_DEVOPS_INP_RELEASE_ID value: value: RELEASE_ID isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_ENVIRONMENT_ID value: value: ENVIRONMENT_ID.substring(1,ENVIRONMENT_ID.length()-1) isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_ENVIRONMENT_STATUS value: value: InProgress isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_VARIABLES_LIST value: value: ENVIRONMENT_VARIABLES_LIST isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_COMMENT value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_SCHEDULED_DEPLOYMENT_TIME value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_REL_ENV_STATUS_CHECK value: value: 'true' isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_PARTIALLY_SUCCEEDED_STATUS_CHECK value: value: 'false' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_RESP - output: FDAZ_DEVOPS_OUT_REL_ENV_WEB_URL variable: RELEASE_URL userInputs: [] userOutputs: [] - id: '8' name: packageExecute type: INVOKE_PLUGIN data: pluginName: FlexagonShellPlugin pluginOperation: packageExecute inputs: - name: FDSHELL_INP_CODE_SNIPPET value: value: echo "Deployment Completed" isExpression: false isEncrypted: false - name: FDSHELL_INP_STOP_ON_ERROR value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_DISABLE_ECHO value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_RESTRICT_ENVIRONMENT value: isExpression: false isEncrypted: false - name: FDSHELL_INP_NO_SECURE_VARIABLES value: value: 'false' isExpression: false isEncrypted: false - name: FDSHELL_INP_PERFORM_PROPERTY_REPLACEMENT value: value: 'false' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: true producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: [] userInputs: [] userOutputs: []

Project Configuration

Create a Project by choosing the Classification: Package-based and Project Types as Generic

Once it is created the Source Control needs to be configured, so that we can discover the files from the respective repository. This time the source control is the one where we have uploaded the ARM template.

Discover the files using File Catalog tab

Now we can create Package upon selecting specific ARM template which is to be deployed to the Target Synapse Workspace.

Click on the Configuration tab, select General and update the Build and Deploy workflows with respective Target group.

Parameterizing the values

There are many ways to do environment specific parameter value customization. 1) Using Replacement configuration of Project 2) Specifying the parameter as ${{}} in the source json file itself or the generated ARM template 3) Edit the generated ARM template to include the parameter as ARM template parameter. We will show option 2 in the following example.

Replacement configuration on Project

Starting 7.0 we have a new feature that allows us to replace a string in any file with another string based on as it is executed to different environment. Ref: Replacements - FlexDeploy 7.0 - Confluence (atlassian.net)

For our use-case a specific parameter as below is created in the Linked Service:

When this source workspace is integrated with GIT, the linked service json file gets auto-synced there. One can edit the parameter manually to hold the target group property as

Once this is done as shown earlier, the target group property will take care of replacing it with right environment specific value during deployment.

Target Group configuration

Build = for build execution; UAT = for deploy execution.

Build

UAT

The Environment Specific Variables field controls the target workspace name and several other environment specific parameters. To know more follow this link.

Follow this link to know about the Azure DevOps Release Pipeline configuration. This must be configured prior to configuring the FlexDeploy workflows and project properties.

Execution

Initiate Build on the selected Package. Once done initiate the Deploy. Build produces the ARM templates as a zip bundle, which Deploy further passes on to the Azure DevOps Release Pipeline to deploy to target workspace.

The Output tab of createRelease step would show the Azue DevOps Release URL to further see the log in more detail:

 

Post deployment if we verify the Linked Service in UAT workspace, we see the updated values in there.

Flow (Approach B)

Azure Devops pipeline configuration

More information is here

Workflow configuration

Build Workflow:
Deploy Workflow:

Project Configuration

Create a Project say SelectiveJsonDeployment, by choosing the Classification: Package-based and Project Types as Generic

Once it is created the Source Control needs to be configured, so that we can discover the files from the respective repository.

Note: These files are from the GIT integrated workspace that we use in Synapse for DEV environment.

For example, this is how the source control in Project looks like:

Project Properties

 

Discover the files using File Catalog tab

Now we can create Package upon selecting specific files.

For example, here we chose two pipelines and a linked service to form the package called abstractdemo.

Click on the Configuration tab, select General and map the Build and Deploy workflows along with respective Target group.

Target Group configuration

Build = for build execution; UAT = for deploy execution.

Build

UAT

The Environment Specific Variables field controls the target workspace name and several other environment specific parameters. To know more follow this link.

Execution

Initiate Build on the selected Package abstractdemo.

It creates the artifacts with the zip bundle of json files.

Initiate Deploy on the built version.

Clicking on the createRelease step’s Output tab, you would find the release url of Azure DevOps Release pipeline. Navigating to that shows more log:

The following macros are not currently supported in the footer:
  • style