Power BI Reports and Dataset Deployment

Introduction to Azure Power BI

Power BI is a unified self-service and enterprise analytics solution. It lets you visualize your data and share insights across your organization and embed them in your app or website. Azure Analytics and Power BI together provide insights at scale, allowing you to develop the data-driven culture needed to thrive in a fast-paced, competitive environment.

Objective

In this tutorial we are going to talk about deploying Power BI Reports and Dataset. As per best practice and standard approach, the deployment process will depict shared dataset deployment along with the report that is built using it.

We are going to utilize FlexDeploy's capability of integrating with Azure DevOps. Power BI Actions extension that is available in Azure DevOps Marketplace, has the ideal support for Dataset and Reports deployment. FlexDeploy will do the entire orchestration to make the deployment happen.

Flow

Azure DevOps Pipelines

  • DevOps release pipeline will fetch the pbix formatted file for dataset and report and deploy them to the target workspace in PowerBI. As a pre-requisite, the pbix files must be uploaded to SCM repository of user’s choice.

  • In case, one chooses to deploy one single report pbix file, the pipeline flow will remain same. So, it is flexible enough to support both the scenarios.

  • Dataset files can grow large in size depending on how much data is wrapped inside it and Azure artifacts feeds become paid service if more than 2GB space is utilized. Hence, we will maintain the dataset pbix files in DevOps build pipeline as part of build artifacts. Release pipeline will consume from there. The benefit is one doesn’t have to much worry about space occupancy in Feed.

FlexDeploy Orchestration

  • One has the flexibility to selectively choose the dataset and report files for deployment. We are going to brief about the package-based deployment here, which will enable this scope.

  • The build workflow will take the chosen artifacts and send them to Azure DevOps build pipeline via FlexDeploy’s Azure DevOps plugin.

  • The deploy workflow will trigger the deployment for those artifacts to target workspace provided by user.

  • All the user’s choice parameters such as workspace name etc will be set and passed from FlexDeploy.

Setting up Service Principal with necessary privileges

The deployment of datasets and reports to target workspace will be done using a service principal. For this one must ensure right permissions are given to it. We will talk about where exactly we are pointing to this SP from the Azure DevOps Release pipeline in the later sections.

Create a Microsoft Entra app in the Azure portal

Please follow the instructions in Step 2 in this page: https://learn.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal#create-a-microsoft-entra-app-in-the-azure-portal

For this tutorial we are naming our SP as PowerBICICDApp.

image-20240423-060810.png

Create a Microsoft Entra security group

Please follow the instructions in Step 2 in this page: https://learn.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal#step-2---create-a-microsoft-entra-security-group

Enable the Power BI service admin settings

Please follow the instructions in Step 3 in this page: https://learn.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal#step-3---enable-the-power-bi-service-admin-settings

Add the service principal to your workspace

Please follow the instructions in Step 4 in this page:

If there are multiple such target workspaces where the resources will be deployed, then the SP must be given access to every one of them following this step. The other previous 3 steps are one-time work.

In order to continue with our tutorial, we have selected the SP as member type and provided Admin access.

Note: The admin access is given as it is observed to overwrite same resource again and again during multiple deployments attempt, this is necessary.

Preparing the PBIX files for dataset, reports and uploading them to GIT

As SCM repository we are going to take an example of Azure DevOps repository here. But it can be anything of user’s choice, say Github, gitlab etc.

Created a repository named az-powerbi-artifacts under newly created project FD-PowerBIDeploy. As developer creates the dataset and reports, they must save them in the format of .pbix files and store in the repository under respective folders such as dataset, reports. This will help in identifying and choosing easily the respective objects while deploying. Will talk about it in details next.

image-20240423-065817.png

dataset:

reports:

Azure DevOps Pipeline

Build Pipeline

Navigate to the Pipelines option under the Project FD-PowerBIDeploy and create a new pipeline as below:

Add the variables as highlighted.

# Starter pipeline # Start with a minimal pipeline that you can customize to build and deploy your code. # Add steps that build, run tests, deploy, and more: # https://aka.ms/yaml trigger: - main pool:   vmImage: ubuntu-latest steps: - script: echo Initiating Build!!   displayName: 'Initiated' - task: UniversalPackages@0   inputs:     command: 'download'     downloadDirectory: '$(System.DefaultWorkingDirectory)/PBIArtifacts'     feedsToUse: 'internal'     vstsFeed: $(feedName)     vstsFeedPackage: $(packageName)     vstsPackageVersion: $(packageVersion) - task: PublishBuildArtifacts@1   inputs:     PathtoPublish: '$(System.DefaultWorkingDirectory)/PBIArtifacts'     ArtifactName: $(packageName)     publishLocation: 'Container'

Release Pipeline

Navigate to the Pipelines option under the Project FD-PowerBIDeploy, select Releases and create a new release pipeline as below:

Add the variables as highlighted.

For this tutorial, we are naming it as DeployToPowerBIWS.

We are going to deploy resources to UAT target workspace, hence added the stage UAT. You may add as many stages as applicable to the deployment. Ensure selecting Manual Only bullet option.

Variables

Click on the Variables and add the below ones. We will be using them throughout the pipeline.

Tasks

  • First task will be grabbing the pbix files from the build pipeline. To be noted here is that we are considering the build artifacts and not the artifacts from Feed.

Download artifacts produced by: Select “Specific build”

Project: FD-PowerBIDeploy

Build pipeline: Select the build pipeline created in previous step

Build version to download: Select “Specific version”

Build: $(buildNumber)

Artifact name: $(buildArtifactName)

Matching pattern: **/*.pbix

Destination directory: $(System.ArtifactsDirectory)

Tick the option “Clean destination folder

  • Second step will be executed when user chose dataset to be deployed, hence it has a pre-condition configured. The type of task is Power BI Actions.

Power BI service connection

We are configuring the service principal that we created earlier, in here.

Click on the Control Options and add the custom condition as: or(eq(variables['resourceType'], 'dataset'), eq(variables['resourceType'], 'all'))

Choose the Action as Upload Power BI report and tick the option Skip report uploading, since we are going to use this task for deploying datasets.

Workspace name: $(datasetWorkspaceName)

Source file: $(System.ArtifactsDirectory)/$(buildArtifactName)/dataset/*.pbix

Tick “Overwrite Power BI file”.

  • Third step will be executed when user chose reports to be deployed, hence it has a pre-condition configured. The type of task is Power BI Actions.

Similarly add the custom condition under Control options as: or(eq(variables['resourceType'], 'reports'), eq(variables['resourceType'], 'all'))

Workspace name: $(reportsWorkspaceName)

Source file: $(System.ArtifactsDirectory)/$(buildArtifactName)/reports/*.pbix

Tick “Overwrite Power BI file”.

For this task we are not selecting “Skip report uploading

Note: Power BI Action extension currently supports pbix file less than 2GB in size.

Additional Pipelines

As we have undertaken the shared dataset concept here, hence there would be two more additional pipelines required: 1. Deleting the existing report from the target workspace 2. Rebinding the report to the dataset

Let’s say user has a shared dataset present in the source dataset workspace A, based on which he created a report. Now the same dataset is promoted to higher workspace B. When that report is going to get deployed to Target Workspace, it will still point to the source workspace A for referencing to the dataset, since the pbix file is pointing there. Hence, the rebinding feature is needed in this situation to rebind the target workspace’s report to the dataset present in workspace B. Now when we are continuously revising a report and redeploying it multiple times, in order to make this rebinding work properly, before deploying the report to the target workspace, it should be deleted, so that we don’t end up in having two reports with the same name in the target workspace, one pointing to the workspace A’s dataset and another to B’s.

You don’t need to worry about these two, if you are not following Enterprise architecture of shared dataset and keeping the dataset embedded inside the same pbix file report.

  • We have named the first one as DeleteReportFromWS.

  • We have named the pipeling for rebinding as RebindReportToDataset

Both the pipelines will have trigger type as “After release

 

FlexDeploy Project Configuration

We are going to create generic package-based project in order to have the flexibility to select which dataset and report are going to be deployed.

In this tutorial, we will follow the name as: BuildAndDeployPowerBIArtifacts

Source Control will point to the repository that contains the pbix files as mentioned earlier.

Build Workflow

Purpose

This workflow will pick the selected dataset/report file, upload to Azure artifacts feed for the DevOps build pipeline to continue. The DevOps build pipeline will then store those pbix files as part of build artifacts. This way even if someone deleted the artifacts from the DevOps feed to reclaim space, the deploy workflow will still work. The same pbix files are also stored in FlexDeploy artifact repository.

 

name: BuildPowerBIArtifacts description: '' variables: - code: BuildURL   dataType: String   returnAsOutput: false   constant: false   encrypted: false   scope: LOCAL - code: BuildNumber   dataType: String   returnAsOutput: true   constant: false   encrypted: false   scope: LOCAL steps: - id: '1'   name: copy   type: INVOKE_PLUGIN   data:     pluginName: FlexagonFilePlugin     pluginOperation: copy     endpointInstanceOverride:       isExpression: false     consumesArtifacts: false     producesArtifacts: false     endpointSelection:       choice: All     endpointExecution:       choice: Any       stopOnError: false     inputs:     - name: FDFILE_INP_FILE_FILTER       value:         value: '*.pbix'         isExpression: false       isEncrypted: false     - name: FDFILE_INP_FILE_FILTER_EXCLUDED       value:         isExpression: false       isEncrypted: false     - name: FDFILE_INP_SOURCE_PATH       value:         value: FD_TEMP_DIR         isExpression: true       isEncrypted: false     - name: FDFILE_INP_TARGET_PATH       value:         value: FD_TEMP_DIR + "/PBIArtifacts"         isExpression: true       isEncrypted: false     - name: FDFILE_INP_CLEAN_DIRECTORY       value:         value: 'true'         isExpression: false       isEncrypted: false     outputs: []     userInputs: []     userOutputs: [] - id: '3'   name: publishArtifacts   type: INVOKE_PLUGIN   data:     pluginName: FlexagonAzurePlugin     pluginOperation: publishArtifacts     endpointInstanceOverride:       isExpression: false     consumesArtifacts: false     producesArtifacts: false     endpointSelection:       choice: All     endpointExecution:       choice: Any       stopOnError: false     inputs:     - name: FDAZ_DEVOPS_INP_FEED       value:         value: FEED_NAME         isExpression: true       isEncrypted: false     - name: FDAZ_DEVOPS_INP_PACKAGE_NAME       value:         value: FD_PACKAGE_NAME + "-" + PACKAGE_NAME         isExpression: true       isEncrypted: false     - name: FDAZ_DEVOPS_INP_PACKAGE_VERSION       value:         value: PACKAGE_VERSION         isExpression: true       isEncrypted: false     - name: FDAZ_DEVOPS_INP_PACKAGE_DESCRIPTION       value:         isExpression: false       isEncrypted: false     - name: FDAZ_DEVOPS_INP_PROJECT_NAME       value:         value: PROJECT_NAME         isExpression: true       isEncrypted: false     - name: FDAZ_DEVOPS_INP_PACKAGE_PATH       value:         value: FD_TEMP_DIR + "/PBIArtifacts"         isExpression: true       isEncrypted: false     - name: FDAZ_DEVOPS_INP_ADDITIONAL_PARAMS       value:         isExpression: false       isEncrypted: false     outputs:     - output: FDAZ_DEVOPS_OUT_ERR     - output: FDAZ_DEVOPS_OUT_RESP     userInputs: []     userOutputs: [] - id: '4'   name: buildPipeline   type: INVOKE_PLUGIN   data:     pluginName: FlexagonAzurePlugin     pluginOperation: buildPipeline     endpointInstanceOverride:       isExpression: false     consumesArtifacts: false     producesArtifacts: false     endpointSelection:       choice: All     endpointExecution:       choice: Any       stopOnError: false     inputs:     - name: FDAZ_DEVOPS_INP_BUILD_DEFINITION_ID       value:         isExpression: false       isEncrypted: false     - name: FDAZ_DEVOPS_INP_DEFINITION_NAME       value:         value: BUILD_DEFINITION_NAME         isExpression: true       isEncrypted: false     - name: FDAZ_DEVOPS_INP_QUEUE_ID       value:         isExpression: false       isEncrypted: false     - name: FDAZ_DEVOPS_INP_COMMIT_ID       value:         isExpression: false       isEncrypted: false     - name: FDAZ_DEVOPS_INP_BRANCH_NAME       value:         isExpression: false       isEncrypted: false     - name: FDAZ_DEVOPS_INP_PROJECT_NAME       value:         value: PROJECT_NAME         isExpression: true       isEncrypted: false     - name: FDAZ_DEVOPS_INP_VARIABLES_LIST       value:         value: VARIABLES_LIST         isExpression: true       isEncrypted: false     - name: FDAZ_DEVOPS_INP_PACKAGE_NAME       value:         value: FD_PACKAGE_NAME + "-" + PACKAGE_NAME         isExpression: true       isEncrypted: false     - name: FDAZ_DEVOPS_INP_ADDITIONAL_PARAMS       value:         isExpression: false       isEncrypted: false     outputs:     - output: FDAZ_DEVOPS_OUT_ERR     - output: FDAZ_DEVOPS_OUT_RESP       variable: ''     - output: FDAZ_DEVOPS_OUT_BUILD_PIPELINE_WEB_URL       variable: BuildURL     userInputs: []     userOutputs: [] - id: '5'   name: Assign   type: ASSIGN   data:     from: BuildURL.substring((BuildURL.indexOf("buildId=")+8))     to: BuildNumber - id: '6'   name: saveArtifacts   type: INVOKE_PLUGIN   data:     pluginName: FlexagonFilePlugin     pluginOperation: saveArtifacts     endpointInstanceOverride:       isExpression: false     consumesArtifacts: false     producesArtifacts: true     endpointSelection:       choice: All     endpointExecution:       choice: Any       stopOnError: false     inputs:     - name: FDFILE_INP_SOURCE_SUBFOLDER       value:         value: PBIArtifacts         isExpression: false       isEncrypted: false     - name: FDFILE_INP_FILE_FILTER       value:         isExpression: false       isEncrypted: false     - name: FDFILE_INP_FILE_FILTER_EXCLUDED       value:         isExpression: false       isEncrypted: false     - name: FDFILE_INP_TARGET_SUBFOLDER       value:         isExpression: false       isEncrypted: false     outputs: []     userInputs: []     userOutputs: []

Deploy Workflow

Purpose

Deploy workflow will have an optional step for deleting the report from the target workspace, performing the deployment of the report/dataset as chosen by user and rebinding the report to the dataset.

Design Considerations

  • For simplicity purpose, we are considering the approach of deploying the one or multiple datasets within the package to one workspace, defined by parameter key name datasetWorkspaceName. To support multiple workspaces, additional coding needs to be done that will specify which dataset should go to which workspace.

  • Similarly for reports as well, they will get deployed to one workspace at a time, defined by the parameter key name reportsWorkspaceName

  • The given workflow will showcase the deletion and rebinding step with a hardcoded report and dataset name, in order to demonstrate how the functionality works. One might reconfigure them to handle multiple values as needed. If those two steps are not needed at all then one may delete them altogether as well, in case when report and dataset are in one single pbix file and rebinding not needed.

We will consider singular value for the simplicity purpose of this tutorial while specifying the deletion and rebinding step. One might reconfigure them to handle multiple values as needed. If those two steps are not needed at all then one may delete them altogether as well.

name: DeployPowerBIArtifacts description: '' inputs: - code: DELETE dataType: String subDataType: TextField defaultValue: 'NO' defaultValueExpression: false listData: YES,NO listDataExpression: false multiselect: false required: false encrypted: false variables: - code: ENV_DETAILS dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL - code: RELEASE_ID dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL - code: ENVIRONMENT_ID dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL - code: RELEASE_URL dataType: String returnAsOutput: true constant: false encrypted: false scope: LOCAL steps: - id: '1' name: If type: IF data: steps: - id: '1.1' name: DeleteReportFromWS type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: createRelease inputs: - name: FDAZ_DEVOPS_INP_RELEASE_DEFINITION_ID value: value: '2' isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_ARTIFACTS_PAYLOAD value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_VARIABLES_LIST value: value: REPORTSTOBEDELETED isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PROPERTIES value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_MANUAL_ENVIRONMENTS value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_STATUS_CHECK value: value: 'true' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_RESP - output: FDAZ_DEVOPS_OUT_RELEASE_WEB_URL - output: FDAZ_DEVOPS_OUT_REL_ENV_STAGE_DETAILS - output: FDAZ_DEVOPS_OUT_RELEASE_ID userInputs: [] userOutputs: [] condition: ((DELETE == "YES") && (!POWERBIRESOURCETYPE.contains("dataset"))) elseIfBlock: [] - id: '2' name: createRelease type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: createRelease inputs: - name: FDAZ_DEVOPS_INP_RELEASE_DEFINITION_ID value: value: RELEASE_DEFINITION_ID isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_ARTIFACTS_PAYLOAD value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_VARIABLES_LIST value: value: >- "buildArtifactName:" + FD_PACKAGE_NAME + "-" + PACKAGE_NAME + "##buildNumber:" + BuildNumber + "##resourceType:" + POWERBIRESOURCETYPE isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PROPERTIES value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_MANUAL_ENVIRONMENTS value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_STATUS_CHECK value: value: 'false' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_RESP - output: FDAZ_DEVOPS_OUT_RELEASE_WEB_URL - output: FDAZ_DEVOPS_OUT_REL_ENV_STAGE_DETAILS variable: ENV_DETAILS - output: FDAZ_DEVOPS_OUT_RELEASE_ID variable: RELEASE_ID userInputs: [] userOutputs: [] - id: '3' name: Parse Environment specific Id type: INVOKE_PLUGIN data: pluginName: FlexagonXPathPlugin pluginOperation: parseJson inputs: - name: FDP_SOURCE value: value: ENV_DETAILS isExpression: true isEncrypted: false - name: FDP_XPATH value: value: '''$..[?(@.name==\''''+FD_ENVIRONMENT_NAME+''\'') ].id''' isExpression: true isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDP_VALUE variable: ENVIRONMENT_ID userInputs: [] userOutputs: [] - id: '4' name: updateReleaseEnvStatus type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: updateReleaseEnvStatus inputs: - name: FDAZ_DEVOPS_INP_RELEASE_ID value: value: RELEASE_ID isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_ENVIRONMENT_ID value: value: ENVIRONMENT_ID.substring(1,ENVIRONMENT_ID.length()-1) isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_ENVIRONMENT_STATUS value: value: InProgress isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_VARIABLES_LIST value: value: ENVIRONMENT_VARIABLES_LIST isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_COMMENT value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_SCHEDULED_DEPLOYMENT_TIME value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_REL_ENV_STATUS_CHECK value: value: 'true' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_RESP - output: FDAZ_DEVOPS_OUT_REL_ENV_WEB_URL variable: RELEASE_URL userInputs: [] userOutputs: [] - id: '5' name: If type: IF data: steps: - id: '5.1' name: RebindReports type: INVOKE_PLUGIN data: pluginName: FlexagonAzurePlugin pluginOperation: createRelease inputs: - name: FDAZ_DEVOPS_INP_RELEASE_DEFINITION_ID value: value: '3' isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_ARTIFACTS_PAYLOAD value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_PROJECT_NAME value: value: PROJECT_NAME isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_VARIABLES_LIST value: value: REBINDREPORTS isExpression: true isEncrypted: false - name: FDAZ_DEVOPS_INP_PROPERTIES value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_MANUAL_ENVIRONMENTS value: isExpression: false isEncrypted: false - name: FDAZ_DEVOPS_INP_RELEASE_STATUS_CHECK value: value: 'true' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any stopOnError: false outputs: - output: FDAZ_DEVOPS_OUT_RESP - output: FDAZ_DEVOPS_OUT_RELEASE_WEB_URL - output: FDAZ_DEVOPS_OUT_REL_ENV_STAGE_DETAILS - output: FDAZ_DEVOPS_OUT_RELEASE_ID userInputs: [] userOutputs: [] condition: '!POWERBIRESOURCETYPE.contains("dataset")' elseIfBlock: []

Execution

Discover the files from SCM and create package selecting the intended ones.

Upon submitting the Build, the build ID from Devops is returned, which denotes the Id against which the resources are stored. As report points to dataset in our case, hence we chose the strategy of deplying dataset before report.

This value gets saved as Output as well, so that can be utilized in the Deploy workflow.

Below are the examples of how to pass the Release Variables List in json format for denoting the parameters related to Delete Report and Rebind Report step:

Delete Report:

Rebind Report:

Once the dataset and report are deployed to respective workspaces, in Power BI console it would look like:

The following macros are not currently supported in the footer:
  • style