Automate check-in of development OTBI catalog items to SCM

Owned by Karl Henselin
Last updated: Nov 22, 2024 by Muthu Sankaralingam
8 min read

Oracle OTBI projects can either source catalog objects from a source code repository or Development OTBI instances. In your OTBI project used to build and deploy WebCatalog objects you will either have the option to Populate from Git (or other SCM type) or Populate from OTBI Web Catalog on files. Similar to other technologies, FlexDeploy can sync files from a development server to a source code repository. This helps implement DevOps best practices to source control developer code.

The OTBI plugin manipulates the file extensions and paths of exported/downloaded objects in order to establish metadata about each object.  For this reason, you cannot simply download the .catalog files from the OTBI catalog manager and place them in source control.  This would not be compatible with the build and deploy operations of the OTBI plugin. FlexDeploy provides OTBI export and download operations to simplify so that you do not need to manipulate the files manually with the required structure and extensions.

exportWebCatalog

This operation is useful when you want export specific files from OTBI.  This is managed through a package within a package-based deployment project (separate from the project that is used for build/deploy).

Ensure project classification is Package-Based Deployments and project type is Oracle Transactional BI. Your project for this purpose may look something like this. 

Similar to other projects, you will setup project properties to indicate folder that you will be working with from the catalog. Our purpose is to extract files from a development server and commit to Git with necessary changes in file and/or extension names. Note that you are using this as part of build workflow. You can implement one of two options - 1) standard build/deploy from development server with check-in to SCM for historical reference. or 2) standard build/deploy from SCM with this project executing extract from Development and check-in to SCM. With either option, you end up with code being managed in SCM of your choice.

Now let's talk about Extract WebCatalog workflow. This workflow will execute 6 steps as shown below.

  1. Clone the git repository where objects are stored.

  2. Extract WebCatalog files into local git repository folder. Extract will be generally done from development environment. Number of objects extracted will depend on whether All Files or Package build is executing.

  3. Add any new files in the local git repository.

  4. Commit any changes to local git repository. It is possible that nothing is found to commit if there were no changes on development server.

  5. Push changes to remote repository.

Here is source code for the build workflow. You will need to update the steps in the workflow according to your Git configuration (e.g. Git Instance, sparse checkout folder, etc.). Also, keep in mind that we are running this build against the master branch (stream), if you want to commit on a different branch then create the stream on the project as appropriate.

name: Extract WebCatalog description: '' steps: - id: '1' name: Git Clone type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: clone inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: FLEXTEST isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: value: ../obieefiles isExpression: false isEncrypted: false - name: FDGIT_INP_BRANCH value: value: main isExpression: false isEncrypted: false - name: FDGIT_INP_DEPTH value: value: '1' isExpression: false isEncrypted: false - name: FDGIT_INP_SPARSE_CHECKOUT_FOLDERS value: value: '' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '2' name: exportWebCatalog type: INVOKE_PLUGIN data: pluginName: FlexagonOTBIPlugin pluginOperation: exportWebCatalog inputs: - name: FDOTBI_INP_EXPORT_FOLDER value: value: ../obieefiles/OBIEECatalog isExpression: false isEncrypted: false - name: FDOTBI_INP_COPY_PERMISSIONS value: value: 'true' isExpression: false isEncrypted: false - name: FDOTBI_INP_PRESERVE_TIMESTAMPS value: value: 'false' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '3' name: Add any new Files type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: add inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: FLEXTEST isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: value: ../obieefiles isExpression: false isEncrypted: false - name: FDGIT_INP_FILELIST value: value: '--all .' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '4' name: Commit to Git type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: commit inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: FLEXTEST isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: value: ../obieefiles isExpression: false isEncrypted: false - name: FDGIT_INP_NAME value: value: FD_WF_USER isExpression: true isEncrypted: false - name: FDGIT_INP_EMAIL value: value: FD_WF_USER_EMAIL isExpression: true isEncrypted: false - name: FDGIT_INP_REVISION_MESSAGE value: value: OBIEE files added from flexdeploy workflow. isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '5' name: Push to Central Repository type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: push inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: FLEXTEST isExpression: false isEncrypted: false - name: FDGIT_INP_DEST_SUBFOLDER value: value: ../obieefiles isExpression: false isEncrypted: false - name: FDGIT_INP_BRANCH value: value: main isExpression: false isEncrypted: false - name: FDGIT_INP_FOLLOW_TAGS value: value: 'false' isExpression: false isEncrypted: false endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: []

We are referencing Git in the example above, but plugins exist for many other popular source control management systems as well and a similar workflow can be created to accomplish the same task.

downloadWebCatalog

This operation will also export/download objects from the OTBI WebCatolog.  The difference is that this operation does not require a package-based deployment project and can be called from within a utility workflow.

Ensure project classification is Utility. Your project for this purpose may look something like this:

Now let's talk about Download WebCatalog workflow. This workflow will execute 6 steps as shown below.

  1. Clone the git repository where objects are stored.

  2. Download WebCatalog files into the local git repository folder. The download will be generally from a development environment. All files from the specified path will be downloaded.

  3. Add any new files in local git repository.

  4. Commit any changes to the local git repository. It is possible that nothing is found to commit if there were no changes on development server.

  5. Push changes to the remote repository.

 

Here is source code for the utility workflow. You will need to update this workflow to indicate project Git instance code and sparse checkout folder for catalog objects:

name: downloadWebCatalogs description: '' inputs: - code: INPUT_WEB_CATALOG_ITEMS displayName: Web Catalog Items dataType: String subDataType: TextField defaultValueExpression: false listDataExpression: false multiselect: false required: false encrypted: false hideWhen: type: equals negate: false values: [] disableWhen: type: equals negate: false values: [] steps: - id: '1' name: Git Clone type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: clone inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: FLEXTEST isExpression: false isEncrypted: 'false' - name: FDGIT_INP_DEST_SUBFOLDER value: value: ../otbifiles isExpression: false isEncrypted: 'false' - name: FDGIT_INP_BRANCH value: value: main isExpression: false isEncrypted: 'false' - name: FDGIT_INP_TREELESS_CLONE value: value: 'false' isExpression: false isEncrypted: 'false' - name: FDGIT_INP_DEPTH value: value: '1' isExpression: false isEncrypted: 'false' - name: FDGIT_INP_SPARSE_CHECKOUT_FOLDERS value: value: '' isExpression: false isEncrypted: 'false' endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '2' name: downloadWebCatalog type: INVOKE_PLUGIN data: pluginName: FlexagonOTBIPlugin pluginOperation: downloadWebCatalog inputs: - name: FDOTBI_INP_OBJECT_PATH value: value: INPUT_WEB_CATALOG_ITEMS isExpression: true isEncrypted: 'false' - name: FDOTBI_INP_DOWNLOAD_FOLDER value: value: ../otbifiles/OTBICatalog isExpression: false isEncrypted: 'false' - name: FDOTBI_INP_COPY_PERMISSIONS value: value: 'true' isExpression: false isEncrypted: 'false' - name: FDOTBI_INP_PRESERVE_TIMESTAMPS value: value: 'false' isExpression: false isEncrypted: 'false' - name: FDOTBI_INP_DISCOVER_SUB_ITEMS value: value: Scorecard, Dashboard Group isExpression: false isEncrypted: 'false' endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '3' name: Add any new Files type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: add inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: FLEXTEST isExpression: false isEncrypted: 'false' - name: FDGIT_INP_DEST_SUBFOLDER value: value: ../otbifiles isExpression: false isEncrypted: 'false' - name: FDGIT_INP_FILELIST value: value: '--all .' isExpression: false isEncrypted: 'false' endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '4' name: Commit to Git type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: commit inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: FLEXTEST isExpression: false isEncrypted: 'false' - name: FDGIT_INP_DEST_SUBFOLDER value: value: ../otbifiles isExpression: false isEncrypted: 'false' - name: FDGIT_INP_NAME value: value: FD_WF_USER isExpression: true isEncrypted: 'false' - name: FDGIT_INP_EMAIL value: value: FD_WF_USER_EMAIL isExpression: true isEncrypted: 'false' - name: FDGIT_INP_REVISION_MESSAGE value: value: OTBI files added from flexdeploy workflow. isExpression: false isEncrypted: 'false' endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: [] - id: '5' name: Push to Central Repository type: INVOKE_PLUGIN data: pluginName: FlexagonGITPlugin pluginOperation: push inputs: - name: FDGIT_INP_INSTANCE_CODE value: value: FLEXTEST isExpression: false isEncrypted: 'false' - name: FDGIT_INP_DEST_SUBFOLDER value: value: ../otbifiles isExpression: false isEncrypted: 'false' - name: FDGIT_INP_BRANCH value: value: main isExpression: false isEncrypted: 'false' - name: FDGIT_INP_FOLLOW_TAGS value: value: 'false' isExpression: false isEncrypted: 'false' endpointInstanceOverride: isExpression: false consumesArtifacts: false producesArtifacts: false endpointSelection: choice: All endpointExecution: choice: Any outputs: [] userInputs: [] userOutputs: []

We are referencing Git in the example above, but plugins exist for many other popular source control management systems as well, and a similar workflow can be created to accomplish the same task.

Summary

Using the exportWebCatalog and downloadWebCatalog plugin operations enables syncing the OTBI WebCatalog objects from a development instance to a source code repository.  You can create your build/deploy project to source from the source code repository instead of an OTBI instance.  This allows control over when objects are available/eligible for build and deployment.

The following macros are not currently supported in the footer:
  • style