Automate check-in of development catalog items to SCM

Owned by Karl Henselin
Last updated: Mar 07, 2024 by Blake Westmoreland
25 min read

Oracle OBIEE projects can either source catalog objects from a source code repository or Development OBIEE instances. In your OBIEE project used to build and deploy WebCatalog objects you will either have the option to Populate from Git (or other SCM type) or Populate from OBIEE Web Catalog on files tab of your project. Similar to other technologies, FlexDeploy can sync files from a development server to a source code repository. This helps implement DevOps best practices to source control developer code.

The OBIEE plugin manipulates the file extensions and paths of exported/downloaded objects in order to establish metadata about each object.  For this reason, you cannot simply download the .catalog files from the OBIEE catalog manager and place them in source control.  This would not be compatible with the build and deploy operations of the OBIEE plugin. FlexDeploy provides OBIEE export and download operations to simplify so that you do not need to manipulate the files manually with the required structure and extensions.

obiExtractForSCM

This operation is useful when you want export specific files from OBIEE.  This is managed through a package within a package-based project (separate from the project that is used for build/deploy).

Ensure project classification is Package-Based Deployments and project type is Oracle Transactional BI. Your project for this purpose may look something like this. 

 

Similar to other projects, you will setup project properties to indicate folder that you will be working with from catalog. Our purpose is to extract files from development server and commit to Git with necessary changes in file and/or extension names. Note that you are using this as part of build workflow. You can implement one of two options - 1) standard build/deploy from development server with check-in to SCM for historical reference. or 2) standard build/deploy from SCM with this project executing extract from Development and check-in to SCM. With either option, you end up with code being managed in SCM of your choice.

Now let's talk about Extract WebCatalog workflow. This workflow will execute 6 steps as shown below.

  1. Clone git repository where objects are stored.

  2. Extract WebCatalog files into local git repository folder. Extract will be generally done from development environment. Number of objects extracted will depend on whether All Files or Package build is executing.

  3. Add any new files in local git repository.

  4. Commit any changes to local git repository. It is possible that nothing is found to commit if there were no changes on development server.

  5. Push changes to remote repository.

 

 

Here is source code for the workflow. You will need to update this workflow to indicate project Git instance code and sparse checkout folder for catalog objects. Also, keep in mind that we are running this build against master branch(stream), if you want to commit on different branch then create stream on project appropriately.



steps: - id: '1' name: clone type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: clone inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: DEMOGIT isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: value: '' isExpression: false isEncrypted: false - name: FDGIT_INP_BRANCH value: value: FDBLD_STREAM_NAME isExpression: true isEncrypted: false - name: FDGIT_INP_TREELESS_CLONE value: value: 'false' isExpression: false isEncrypted: false - name: FDGIT_INP_DEPTH value: isExpression: false isEncrypted: false - name: FDGIT_INP_SPARSE_CHECKOUT_FOLDERS value: isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '2' name: obiExtractForSCM type: INVOKE_PLUGIN data: pluginName: FlexagonOBIEEPlugin pluginOperation: obiExtractForSCM inputs: - name: FDOBIEE_INP_EXPORT_FOLDER value: value: ../obieefiles/OBIEECatalog isExpression: false isEncrypted: false - name: FDOBIEE_INP_COPY_PERMISSIONS value: value: 'true' isExpression: false isEncrypted: false - name: FDOBIEE_INP_PRESERVE_TIMESTAMPS value: value: 'false' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '3' name: Add any new Files type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: add inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: DEMOGIT isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: isExpression: false isEncrypted: false - name: FDGIT_INP_FILELIST value: value: '--all .' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '4' name: commit type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: commit inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: DEMOGIT isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: isExpression: false isEncrypted: false - name: FDGIT_INP_NAME value: value: FD_WF_USER isExpression: true isEncrypted: false - name: FDGIT_INP_EMAIL value: value: FD_WF_USER_EMAIL isExpression: true isEncrypted: false - name: FDGIT_INP_REVISION_MESSAGE value: value: new obiee files isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '5' name: push type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: push inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: DEMOGIT isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: isExpression: false isEncrypted: false - name: FDGIT_INP_BRANCH value: value: FDBLD_STREAM_NAME isExpression: true isEncrypted: false - name: FDGIT_INP_FOLLOW_TAGS value: value: 'false' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: []

We are referencing Git in the example above, but plugins exist for many other popular source control management systems as well and a similar workflow can be created to accomplish the same task.

downloadWebCatalog

This operation will also export/download objects from the OBIEE WebCatolog.  The difference is that this operation does not require a package-based project and can be called from within a utility workflow.

Ensure project classification is Utility. Your project for this purpose may look something like this:

 

Now let's talk about Download WebCatalog workflow. This workflow will execute 6 steps as shown below.

  1. Clone the git repository where the objects are stored.

  2. Download WebCatalog files into local git repository folder. The download will be generally from a development environment. All files from the specified path will be downloaded.

  3. Add any new files in local the git repository.

  4. Commit any changes in the local git repository to the remote repository. It is possible that nothing is found to commit if there were no changes on development server.

  5. Push changes to remote repository.

Here is source code for the utility workflow. You will need to update the steps in the workflow according to your Git configuration (e.g. Git Instance, sparse checkout folder, etc.).

 

<?xml version="1.0" encoding="UTF-8"?> <ns0:WorkFlow xmlns:ns1="http://flexagon.com/flexdeploy/workflow/common" xmlns:ns0="http://flexagon.com/flexdeploy/workflow"> <ns1:Name>syncOBIEEWebCatalogToGit</ns1:Name> <ns1:Description></ns1:Description> <ns0:Steps> <ns0:Step> <ns1:Name>Git Clone</ns1:Name> <ns1:StepId>1</ns1:StepId> <ns0:InvokePlugin> <ns0:PluginName>FlexagonGITPlugin</ns0:PluginName> <ns0:PluginOperation>clone</ns0:PluginOperation> <ns0:InvokePluginTypeVersion>v2</ns0:InvokePluginTypeVersion> <ns0:consumesArtifacts>false</ns0:consumesArtifacts> <ns0:producesArtifacts>false</ns0:producesArtifacts> <ns0:EndpointSelection> <ns0:EndpointSelectionChoice>All</ns0:EndpointSelectionChoice> </ns0:EndpointSelection> <ns0:EndpointExecutionChoice>Any</ns0:EndpointExecutionChoice> <ns0:PluginInputs> <ns0:PluginInput> <ns0:Name>FDGIT_INP_INSTANCE_CODE</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>FLEXTEST</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_DEST_SUBFOLDER</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text></ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_BRANCH</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>master</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_DEPTH</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>Integer</ns1:Type> <ns0:Text>1</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_SPARSE_CHECKOUT_FOLDERS</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>OBIEE_12c</ns0:Text> </ns0:PluginInput> </ns0:PluginInputs> <ns0:UserInputs/> <ns0:PluginOutputs/> <ns0:UserOutputs/> </ns0:InvokePlugin> </ns0:Step> <ns0:Step> <ns1:Name>downloadWebCatalog</ns1:Name> <ns1:StepId>2</ns1:StepId> <ns0:InvokePlugin> <ns0:PluginName>FlexagonOBIEEPlugin</ns0:PluginName> <ns0:PluginOperation>downloadWebCatalog</ns0:PluginOperation> <ns0:InvokePluginTypeVersion>v2</ns0:InvokePluginTypeVersion> <ns0:consumesArtifacts>false</ns0:consumesArtifacts> <ns0:producesArtifacts>false</ns0:producesArtifacts> <ns0:EndpointSelection> <ns0:EndpointSelectionChoice>All</ns0:EndpointSelectionChoice> </ns0:EndpointSelection> <ns0:EndpointExecutionChoice>Any</ns0:EndpointExecutionChoice> <ns0:PluginInputs> <ns0:PluginInput> <ns0:Name>FDOBIEE_INP_OBJECT_PATH</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>/shared/Sample Lite</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDOBIEE_INP_DOWNLOAD_PRJ_TYPE</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>Oracle BI 2.0</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDOBIEE_INP_DOWNLOAD_FOLDER</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>OBIEE_12c</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDOBIEE_INP_COPY_PERMISSIONS</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>Boolean</ns1:Type> <ns0:Text>true</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDOBIEE_INP_PRESERVE_TIMESTAMPS</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>Boolean</ns1:Type> <ns0:Text>false</ns0:Text> </ns0:PluginInput> </ns0:PluginInputs> <ns0:UserInputs/> <ns0:PluginOutputs/> <ns0:UserOutputs/> </ns0:InvokePlugin> </ns0:Step> <ns0:Step> <ns1:Name>Add any new Files</ns1:Name> <ns1:StepId>3</ns1:StepId> <ns0:InvokePlugin> <ns0:PluginName>FlexagonGITPlugin</ns0:PluginName> <ns0:PluginOperation>add</ns0:PluginOperation> <ns0:InvokePluginTypeVersion>v2</ns0:InvokePluginTypeVersion> <ns0:consumesArtifacts>false</ns0:consumesArtifacts> <ns0:producesArtifacts>false</ns0:producesArtifacts> <ns0:EndpointSelection> <ns0:EndpointSelectionChoice>All</ns0:EndpointSelectionChoice> </ns0:EndpointSelection> <ns0:EndpointExecutionChoice>Any</ns0:EndpointExecutionChoice> <ns0:PluginInputs> <ns0:PluginInput> <ns0:Name>FDGIT_INP_INSTANCE_CODE</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>FLEXTEST</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_DEST_SUBFOLDER</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text></ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_FILELIST</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>--all .</ns0:Text> </ns0:PluginInput> </ns0:PluginInputs> <ns0:UserInputs/> <ns0:PluginOutputs/> <ns0:UserOutputs/> </ns0:InvokePlugin> </ns0:Step> <ns0:Step> <ns1:Name>Commit to Git</ns1:Name> <ns1:StepId>4</ns1:StepId> <ns0:InvokePlugin> <ns0:PluginName>FlexagonGITPlugin</ns0:PluginName> <ns0:PluginOperation>commit</ns0:PluginOperation> <ns0:InvokePluginTypeVersion>v2</ns0:InvokePluginTypeVersion> <ns0:consumesArtifacts>false</ns0:consumesArtifacts> <ns0:producesArtifacts>false</ns0:producesArtifacts> <ns0:EndpointSelection> <ns0:EndpointSelectionChoice>All</ns0:EndpointSelectionChoice> </ns0:EndpointSelection> <ns0:EndpointExecutionChoice>Any</ns0:EndpointExecutionChoice> <ns0:PluginInputs> <ns0:PluginInput> <ns0:Name>FDGIT_INP_INSTANCE_CODE</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>FLEXTEST</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_DEST_SUBFOLDER</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text></ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_NAME</ns0:Name> <ns0:ValueType>Expression</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Value> <ns0:Expression>FD_WF_USER</ns0:Expression> </ns0:Value> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_EMAIL</ns0:Name> <ns0:ValueType>Expression</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Value> <ns0:Expression>FD_WF_USER_EMAIL</ns0:Expression> </ns0:Value> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_REVISION_MESSAGE</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>Files committed by FlexDeploy workflow</ns0:Text> </ns0:PluginInput> </ns0:PluginInputs> <ns0:UserInputs/> <ns0:PluginOutputs/> <ns0:UserOutputs/> </ns0:InvokePlugin> </ns0:Step> <ns0:Step> <ns1:Name>Push to Central Repository</ns1:Name> <ns1:StepId>5</ns1:StepId> <ns0:InvokePlugin> <ns0:PluginName>FlexagonGITPlugin</ns0:PluginName> <ns0:PluginOperation>push</ns0:PluginOperation> <ns0:InvokePluginTypeVersion>v2</ns0:InvokePluginTypeVersion> <ns0:consumesArtifacts>false</ns0:consumesArtifacts> <ns0:producesArtifacts>false</ns0:producesArtifacts> <ns0:EndpointSelection> <ns0:EndpointSelectionChoice>All</ns0:EndpointSelectionChoice> </ns0:EndpointSelection> <ns0:EndpointExecutionChoice>Any</ns0:EndpointExecutionChoice> <ns0:PluginInputs> <ns0:PluginInput> <ns0:Name>FDGIT_INP_INSTANCE_CODE</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>FLEXTEST</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_DEST_SUBFOLDER</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text></ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_BRANCH</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>String</ns1:Type> <ns0:Text>master</ns0:Text> </ns0:PluginInput> <ns0:PluginInput> <ns0:Name>FDGIT_INP_FOLLOW_TAGS</ns0:Name> <ns0:ValueType>Text</ns0:ValueType> <ns0:Encrypted>false</ns0:Encrypted> <ns1:Type>Boolean</ns1:Type> <ns0:Text>false</ns0:Text> </ns0:PluginInput> </ns0:PluginInputs> <ns0:UserInputs/> <ns0:PluginOutputs/> <ns0:UserOutputs/> </ns0:InvokePlugin> </ns0:Step> </ns0:Steps> </ns0:WorkFlow>

We are referencing Git in the example above, but plugins exist for many other popular source control management systems as well and a similar workflow can be created to accomplish the same task.

Summary

Using the exportWebCatalog and downloadWebCatalog plugin operations enable syncing the OBIEE WebCatalog objects from a development instance to a source code repository.  You can then create your build/deploy project to source from the source code repository instead of an OBIEE instance.  This allows control over when objects are available/eligible for build and deployment.

The following macros are not currently supported in the footer:
  • style