Automate check-in of development OTBI catalog items to SCM
Oracle OTBI projects can source catalog objects either from a source code repository or from development OTBI instance. Use "Discover from Git" (or another source control management system) or "Discover from OTBI Web Catalog" on your FlexDeploy Package-based project used for building and deploying WebCatalog objects. Once catalog objects are discovered (or rediscovered), you can create Package with specific catalog objects for build and deploy.
The OTBI plugin modifies the file extensions and paths of exported or downloaded objects to establish metadata for each object. As a result, simply downloading the .catalog files from the OTBI catalog manager and placing them in source control is not a viable option, as it would be incompatible with the build and deployment operations of the OTBI plugin. To streamline this process, FlexDeploy offers OTBI export and download operations, eliminating the need for manual manipulation of files and ensuring they follow the correct structure and extensions.
There are two options, see below for more details. Using the exportWebCatalog
and downloadWebCatalog
plugin operations allows you to synchronize OTBI WebCatalog objects from a development instance to a source code repository. By configuring your build and deploy project to source from the repository rather than an OTBI instance, you gain control over when objects become available and eligible for build and deployment. This ensures more flexibility and management over the deployment process.
exportWebCatalog
This process is managed through a package within a Package-based deployment project. Users can create a package as needed, or it can be managed via an incoming webhook. It's important to note that this is a separate project from the one used for build and deployment, and its sole purpose is to synchronize files from the development server to Git or any other source control management (SCM) system.
Ensure project classification is Package-based and project type is Oracle Transactional BI. Project SCM type should be None. Your project for this purpose may look something like this.
As with other projects, you will configure project properties to specify the folder in the catalog that you will be working with. The goal is to extract files from the development server, and then commit the modified files to Git. Note that you are using this as part of build workflow.
Now let's talk about Extract WebCatalog workflow. This workflow will execute 6 steps as shown below.
Clone the git repository where objects are stored.
Extract WebCatalog files into local git repository folder. Extract will be generally done from development environment. Number of objects extracted will depend on whether All Files or Package build is executing.
Add any new files in the local git repository.
Commit any changes to local git repository. It is possible that nothing is found to commit if there were no changes on development server.
Push changes to remote repository.
Here is source code for the build workflow. You will need to update the steps in the workflow according to your Git configuration (e.g. Git Instance, sparse checkout folder, etc.). Also, keep in mind that we are running this build against the master branch (stream), if you want to commit on a different branch then create the stream on the project as appropriate.
We are referencing Git in the example above, but plugins exist for many other popular source control management systems as well and a similar workflow can be created to accomplish the same task.
name: Extract WebCatalog
description: ''
steps:
- id: '1'
name: Git Clone
type: INVOKE_PLUGIN
data:
pluginName: FlexagonGITPlugin
pluginOperation: clone
inputs:
- name: FDGIT_INP_INSTANCE_CODE
value:
value: FLEXTEST
isExpression: false
isEncrypted: false
- name: FDGIT_INP_DEST_SUBFOLDER
value:
value: ../obieefiles
isExpression: false
isEncrypted: false
- name: FDGIT_INP_BRANCH
value:
value: main
isExpression: false
isEncrypted: false
- name: FDGIT_INP_DEPTH
value:
value: '1'
isExpression: false
isEncrypted: false
- name: FDGIT_INP_SPARSE_CHECKOUT_FOLDERS
value:
value: ''
isExpression: false
isEncrypted: false
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
- id: '2'
name: exportWebCatalog
type: INVOKE_PLUGIN
data:
pluginName: FlexagonOTBIPlugin
pluginOperation: exportWebCatalog
inputs:
- name: FDOTBI_INP_EXPORT_FOLDER
value:
value: ../obieefiles/OBIEECatalog
isExpression: false
isEncrypted: false
- name: FDOTBI_INP_COPY_PERMISSIONS
value:
value: 'true'
isExpression: false
isEncrypted: false
- name: FDOTBI_INP_PRESERVE_TIMESTAMPS
value:
value: 'false'
isExpression: false
isEncrypted: false
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
- id: '3'
name: Add any new Files
type: INVOKE_PLUGIN
data:
pluginName: FlexagonGITPlugin
pluginOperation: add
inputs:
- name: FDGIT_INP_INSTANCE_CODE
value:
value: FLEXTEST
isExpression: false
isEncrypted: false
- name: FDGIT_INP_DEST_SUBFOLDER
value:
value: ../obieefiles
isExpression: false
isEncrypted: false
- name: FDGIT_INP_FILELIST
value:
value: '--all .'
isExpression: false
isEncrypted: false
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
- id: '4'
name: Commit to Git
type: INVOKE_PLUGIN
data:
pluginName: FlexagonGITPlugin
pluginOperation: commit
inputs:
- name: FDGIT_INP_INSTANCE_CODE
value:
value: FLEXTEST
isExpression: false
isEncrypted: false
- name: FDGIT_INP_DEST_SUBFOLDER
value:
value: ../obieefiles
isExpression: false
isEncrypted: false
- name: FDGIT_INP_NAME
value:
value: FD_WF_USER
isExpression: true
isEncrypted: false
- name: FDGIT_INP_EMAIL
value:
value: FD_WF_USER_EMAIL
isExpression: true
isEncrypted: false
- name: FDGIT_INP_REVISION_MESSAGE
value:
value: OBIEE files added from flexdeploy workflow.
isExpression: false
isEncrypted: false
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
- id: '5'
name: Push to Central Repository
type: INVOKE_PLUGIN
data:
pluginName: FlexagonGITPlugin
pluginOperation: push
inputs:
- name: FDGIT_INP_INSTANCE_CODE
value:
value: FLEXTEST
isExpression: false
isEncrypted: false
- name: FDGIT_INP_DEST_SUBFOLDER
value:
value: ../obieefiles
isExpression: false
isEncrypted: false
- name: FDGIT_INP_BRANCH
value:
value: main
isExpression: false
isEncrypted: false
- name: FDGIT_INP_FOLLOW_TAGS
value:
value: 'false'
isExpression: false
isEncrypted: false
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
downloadWebCatalog
This operation also exports or downloads objects from the OTBI WebCatalog. However, unlike the package-based deployment project, this operation does not require such a project and can instead be called directly from within a utility workflow.
Ensure project classification is Utility. Your project for this purpose may look something like this:
Now let's talk about Download WebCatalog workflow. This workflow will execute 6 steps as shown below.
Clone the git repository where objects are stored.
Download WebCatalog files into the local git repository folder. The download will be generally from a development environment. All files from the specified path will be downloaded.
Add any new files in local git repository.
Commit any changes to the local git repository. It is possible that nothing is found to commit if there were no changes on development server.
Push changes to the remote repository.
Here is source code for the utility workflow. You will need to update this workflow to indicate project Git instance code and sparse checkout folder for catalog objects:
We are referencing Git in the example above, but plugins exist for many other popular source control management systems as well, and a similar workflow can be created to accomplish the same task.
For Git, if you want the files placed in a sub-directory rather than the root of the repository, be sure to set the following on your workflow:
Clone step - Sparse Checkout Folder
downloadWebCatalog step - Download Folder (value should be same as Sparse Checkout Folder)
name: downloadWebCatalogs
description: ''
inputs:
- code: INPUT_WEB_CATALOG_ITEMS
displayName: Web Catalog Items
dataType: String
subDataType: TextField
defaultValueExpression: false
listDataExpression: false
multiselect: false
required: false
encrypted: false
hideWhen:
type: equals
negate: false
values: []
disableWhen:
type: equals
negate: false
values: []
steps:
- id: '1'
name: Git Clone
type: INVOKE_PLUGIN
data:
pluginName: FlexagonGITPlugin
pluginOperation: clone
inputs:
- name: FDGIT_INP_INSTANCE_CODE
value:
value: FLEXTEST
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_DEST_SUBFOLDER
value:
value: ../otbifiles
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_BRANCH
value:
value: main
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_TREELESS_CLONE
value:
value: 'false'
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_DEPTH
value:
value: '1'
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_SPARSE_CHECKOUT_FOLDERS
value:
value: ''
isExpression: false
isEncrypted: 'false'
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
- id: '2'
name: downloadWebCatalog
type: INVOKE_PLUGIN
data:
pluginName: FlexagonOTBIPlugin
pluginOperation: downloadWebCatalog
inputs:
- name: FDOTBI_INP_OBJECT_PATH
value:
value: INPUT_WEB_CATALOG_ITEMS
isExpression: true
isEncrypted: 'false'
- name: FDOTBI_INP_DOWNLOAD_FOLDER
value:
value: ../otbifiles/OTBICatalog
isExpression: false
isEncrypted: 'false'
- name: FDOTBI_INP_COPY_PERMISSIONS
value:
value: 'true'
isExpression: false
isEncrypted: 'false'
- name: FDOTBI_INP_PRESERVE_TIMESTAMPS
value:
value: 'false'
isExpression: false
isEncrypted: 'false'
- name: FDOTBI_INP_DISCOVER_SUB_ITEMS
value:
value: Scorecard, Dashboard Group
isExpression: false
isEncrypted: 'false'
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
- id: '3'
name: Add any new Files
type: INVOKE_PLUGIN
data:
pluginName: FlexagonGITPlugin
pluginOperation: add
inputs:
- name: FDGIT_INP_INSTANCE_CODE
value:
value: FLEXTEST
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_DEST_SUBFOLDER
value:
value: ../otbifiles
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_FILELIST
value:
value: '--all .'
isExpression: false
isEncrypted: 'false'
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
- id: '4'
name: Commit to Git
type: INVOKE_PLUGIN
data:
pluginName: FlexagonGITPlugin
pluginOperation: commit
inputs:
- name: FDGIT_INP_INSTANCE_CODE
value:
value: FLEXTEST
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_DEST_SUBFOLDER
value:
value: ../otbifiles
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_NAME
value:
value: FD_WF_USER
isExpression: true
isEncrypted: 'false'
- name: FDGIT_INP_EMAIL
value:
value: FD_WF_USER_EMAIL
isExpression: true
isEncrypted: 'false'
- name: FDGIT_INP_REVISION_MESSAGE
value:
value: OTBI files added from flexdeploy workflow.
isExpression: false
isEncrypted: 'false'
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
- id: '5'
name: Push to Central Repository
type: INVOKE_PLUGIN
data:
pluginName: FlexagonGITPlugin
pluginOperation: push
inputs:
- name: FDGIT_INP_INSTANCE_CODE
value:
value: FLEXTEST
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_DEST_SUBFOLDER
value:
value: ../otbifiles
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_BRANCH
value:
value: main
isExpression: false
isEncrypted: 'false'
- name: FDGIT_INP_FOLLOW_TAGS
value:
value: 'false'
isExpression: false
isEncrypted: 'false'
endpointInstanceOverride:
isExpression: false
consumesArtifacts: false
producesArtifacts: false
endpointSelection:
choice: All
endpointExecution:
choice: Any
outputs: []
userInputs: []
userOutputs: []
- style