Azure Synapse Workspace Artifacts Deployment

Introduction on Azure Synapse Analytics

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless or dedicated resources—at scale.

Objective

This tutorial is going to describe how to deploy synapse workspace artifacts in an automated way following CI/CD best practices. We are going to utilize FlexDeploy's capability of integrating with Azure DevOps. Microsoft has released Azure DevOps free extension "Synapse workspace deployment" for synapse artifacts deployment from one workspace to another.

Flow

Azure DevOps Pipelines

Build pipeline will be created which invokes the “validate” operation provided by the Microsoft Synapse extension(Azure DevOps). This will generate ARM template(along with parameter file) corresponding to the code base. The ARM templates will be published to the feed in Azure DevOps as package.
Release pipeline will grab the ARM template package from the feed and invoke the “deploy” operation provided by the Microsoft Synapse extension.

FlexDeploy Orchestration

We are just leveraging Azure DevOps just to execute validate and deploy operation, the rest of the orchestration will be performed in FlexDeploy. It includes the below steps:

Developer will commit the workspace artifacts to the Source Control Repository (example: Azure DevOps Repo or GitHub etc.) from the Workspace through Synapse Studio.
FlexDeploy Build Workflow will checkout the codebase and initiate the build pipeline that is created in Azure DevOps. This build pipeline will create the ARM templates based on the codebase and publish to Azure DevOps feed.
FlexDeploy will also store the ARM templates to it’s own Artifact Repository for versioning.
The Deploy workflow in FlexDeploy will create a release in Azure DevOps that will initiate the release pipeline there and deploy the workspace artifacts to the target workspace.
Any dynamic values such as workspace name, resource group name or package name where the ARM templates would be published, will be controlled by FlexDeploy.

Pre-requisite setup in Azure DevOps

The official extension “Synapse workspace deployment” must be installed in the Azure DevOps Organization, that we will use for pipeline creation in DevOps.

You can visit the marketplace and install the extension following this link:

https://marketplace.visualstudio.com/items?itemName=AzureSynapseWorkspace.synapsecicd-deploy

Create Personal Access Token in Azure DevOps and Cloud Account in FlexDeploy, as that will be used for deployment. Please follow this steps.
Project needs to be created:

feed needs to be created within this project, that will store the packages with their respective versions.

Create a new repository that will store the build pipeline yaml. For our tutorial we used az-pipeline-cicd.

Synapse Workspace and committing code to Repository

For this tutorial we will start with a simple example. We have created the Synapse workspace: cicd-demo-dev and included a SQL script, that will query data from a linked service in datalake storage account.

We have created a repository in Azure DevOps as az-synapse-cicd (you can choose any SCM type of your choice) and as we created the artifacts in the workspace, committed them to the branch synapse-codebase to that repository.

For this you can link the GIT through the navigation: Manage → Git configuration

Note: Here we are utilizing the recommended practice for keeping the source code in repository and not using the “Publish” feature here, as that generates the ARM templates directly and uploads to the Git publish branch. But as per CI/CD best practice to leverage the PR feature and review the code before deployment and to support deployment to happen from any branch, we are following this route.

GIT structure

Azure DevOps Pipeline Creation

Build Pipeline

Navigate to the DevOps project and create a new pipeline pointing to the GIT repository az-pipeline-cicd. Choose the starter template.

Once created it would appear as shown below.

Edit the pipeline az-pipeline-cicd (Build Pipeline Definition name) and add the below yaml.

# Synapse Build pipeline

trigger: none

pool:
  vmImage: ubuntu-latest

steps:
- script: echo Initiating Build!!
  displayName: 'Initiated'

- task: UniversalPackages@0
  inputs:
    command: 'download'
    downloadDirectory: '$(System.DefaultWorkingDirectory)'
    feedsToUse: 'internal'
    vstsfeed: $(feedName)
    vstsfeedPackage: $(artifactsPackageName)
    vstsPackageVersion: $(packageVersion)
- task: Synapse workspace deployment@2
  continueOnError: false    
  inputs:
    operation: 'validate'
    TargetWorkspaceName: $(workspaceName)
- task: UniversalPackages@0
  inputs:
    command: 'publish'
    publishDirectory: '$(System.DefaultWorkingDirectory)/ExportedArtifacts'
    feedsToUsePublish: 'internal'
    vstsfeedPublish: $(feedName)
    vstsfeedPackagePublish: $(packageName)
    versionOption: 'custom'
    versionPublish: $(packageVersion)

We are not hardcoding the important elements here, hence the specified variables also need to be created as the mentioned in the following screenshot:

Click on Save and this will save the yaml to the repository : az-pipeline-cicd.

Release Pipeline

Navigate to the DevOps project → Release and create new release pipeline.

Select Empty job and create it.

We are going to name our release pipeline “DeployToSynapseWS” and the basic structure is as followed:

We will be using variables, hence we created the below ones from the Variables tab:

Note: The property “Settable at release time” should be ticked.

For adding Artifacts we need to click on the “+Add” symbol and choose Azure Artifacts as option. It should show as below after providing the respective values:

Similar to Build pipeline we are using variables everywhere. How the values are being passed will be discussed when we will talk about the FlexDeploy configurations.

We have added the UAT stage (to think logically, it is the target environment) and added the job Deploy to UAT as:

Respective to this job, added the task by clicking the + symbol.

Template: $(System.DefaultWorkingDirectory)/_ARMTemplates/TemplateForWorkspace.json

Template parameters: $(System.DefaultWorkingDirectory)/_ARMTemplates/TemplateParametersForWorkspace.json

Note: When the ARM template is created by default it is named as TemplateForWorkspace.json and the parameters file as TemplateParametersForWorkspace.json. The same are being provided here without any modifications.

Another important configuration is we need to enable Manual Only for the Stage Execution.

Click on the symbol manual symbol and select Manual Only option. This ensures that from FlexDeploy we can selectively choose which environment’s deployment to be triggered.

Once this pipeline is created, we are all set in Azure DevOps configuration perspective.

Now we can talk about FlexDeploy configurations.

Workflow Configuration in FlexDeploy

Build Workflow

Creating build workflow CreateARMTemplate as:

Project scoped workflow properties are added as: FEED_NAME, PROJECT_NAME, ARTIFACTS_PACKAGE_NAME, PACKAGE_VERSION, VARIABLES_LIST, BUILD_DEFINITION_NAME, PACKAGE_NAME.

Explanation

Step 1: It clones the code from the az-synapse-cicd repository.

Step 2: It uses FlexagonAzurePlugin’s publishArtifacts operation to upload the checked out code base as package to Azure DevOps feed (Synapsefeed).

Step 3: This step initiates the Build Pipeline that we created in Azure DevOps. Operation buildPipeline is used for this.

The expression for Variables List input is:

VARIABLES_LIST + "##artifactsPackageName:" + ARTIFACTS_PACKAGE_NAME + "##feedName:" + PROJECT_NAME + "/" + FEED_NAME

Step 4: The generated ARM package in Azure DevOps feed gets downloaded.

Deploy Workflow

Creating deploy workflow DeploySynapseArtifacts as:

Project scoped workflow property RELEASE_DEFINITION_ID and Target scoped ENVIRONMENT_VARIABLES_LIST are added.

Variables

Explanation

Step 1: Using FlexagonAzurePlugin’s createRelease operation, it creates the Release in Azure DevOps and passes the values for the respective variables through these plugin inputs.

Expression for Azure DevOps Release Variables List:

"feedName:" + PROJECT_NAME + "/" + FEED_NAME + "##packageName:" + PACKAGE_NAME + "##packageVersion:" + PACKAGE_VERSION

Step 2: Step 1 returns (we will see it when we will explain the real Execution later in this tutorial) Environment Id that we configured in Azure DevOps release pipeline. For our case we had named it as UAT. So the output of Step 1 will have UAT’s corresponding Environment ID returned and gets assigned to the variable ENVIRONMENT_ID.

Expression for JSONPath:

'$..[?(@.name==\''+FD_ENVIRONMENT_NAME+'\') ].id'

Step 3: updateReleaseEnvStatus initiates the deployment step in Azure DevOps Release pipeline for the selected Environment that we select during the Deploy execution of the FlexDeploy Project.

Expression for Azure DevOps Release Environment Id:

ENVIRONMENT_ID.substring(1,ENVIRONMENT_ID.length()-1)

This also returns the executed Release’s URL from Azure DevOps as output so that we can directly track it from FlexDeploy plugin execution’s output.

Project Configuration

Create Blank standard project:

We named ours as SynapseOrchestration.

Now the source control needs to be configured as below. This will pull the synapse workspace artifacts that we committed from studio to GIT. Refer: GIT structure

We need to select both these workflow’s now in the Project.

The Target Group is selected as Synapse which basically supports is tied with Localhost endpoint.

Note: We need to ensure the az CLI in installed on the endpoint, in this case Localhost(FlexDeploy Server).

Project properties

ARTIFACTS_PACKAGE_NAME = This is the package name with which Build Workflow uploads the codebase of Workspace artifacts to Azure DevOps feed.
BUILD_DEFINITION_NAME = Azure DevOps Build pipeline definition name.
FEED_NAME = Azure DevOps feed Name
PACKAGE_NAME = This is the package name with which Azure DevOps Build pipeline packages the ARM templates into.
PACKAGE_VERSION = Version of the packages. We are versioning it as per FlexDeploy Project version.
PROJECT_NAME = Azure DevOps Project Name.
RELEASE_DEFINITION_ID = Release Pipeline’s definition id from DevOps. It is shown in the URL in Azure DevOps upon clicking on the Release pipeline:

VARIABLES_LIST = Sending the variable’s values through this to the Build Pipeline in DevOps. All the variables those were defined in the Build Pipeline there, are getting the values through this step from FlexDeploy.

Target Group Properties

ENVIRONMENT_VARIABLES_LIST = This is used to pass the environment specific variables to the Azure DevOps release pipeline for the UAT stage’s deployment.

FDAZ_DEVOPS_CLOUD_ACCOUNT_CODE = This is the Cloud Account configured in FlexDeploy with PAT, to interact with DevOps.

Note: The account used for this must have the permission to write packages to feed as well.

FlexDeploy Release and Pipeline Configuration

We have created a simple pipeline for deployment to UAT:

Note: We must use the same name for the Stage in FlexDeploy pipeline and DevOps’s release pipeline.

Here we used the environment as UAT, so the same name we used in Release pipeline in Azure DevOps too. Since this is the value with which we have parsing logic in the Deploy workflow to trigger deployment to.

Added the pipeline to a Release and added the project as well.

Monitoring the Execution

Click on the create button to create a snapshot

This will trigger the Build in FlexDeploy

Upon clicking on the Output of the buildPipeline step, it gives the link to the Azure DevOps Build job that ran:

It opens up the below page in DevOps:

The Job row takes to the execution details of the job, which looks like below:

Also the Artifacts tab shows the packages got published:

Once everything looks fine for Build, we can now initiate the Deployment.

Similar to Build execution, deploy execution as well publishes the direct link for the DevOps’s execution

It directs here

It takes us to the Logs page:

Clicking on each task shows the detailed log of each steps.

Note: For the first time execution one will encounter error as:

In such cases, the mentioned principal needs to be given Synapse Artifact Publisher access to the Target Workspace, in this case cicd-demo-uat. Copy the specified principal from the log message and add.

This is the Service Principal used by the Release pipeline to deploy the artifacts to the target workspace through Azure DevOps.