tencent cloud

CI/CD — Continuous Integration and Delivery
Download PDF
Last updated:2026-03-13 16:41:28
CI/CD — Continuous Integration and Delivery
Last updated: 2026-03-13 16:41:28
Download PDF

Usage Scenario

Continuous integration and continuous delivery (CI/CD) refers to the process of developing and delivering software via automated pipelines with short durations and high-frequency. CI/CD is common in software development and increasingly important for data engineering and data science. Compared with manual processes, automation enables development teams to deliver versions more reliably by building, testing, and deploying code.
WeData uses the WeData bundle feature to implement CI/CD capacity. The bundle describes WeData resources (workflows, tasks, resource files, project parameters, alarm rules, events) as source files. These packages can be used for code control, and you can use external CI/CD automation (such as GitHub Actions) to trigger execution, test, and deployment.
Feature Introduction
Define resource configuration information as a yml file in the project. Modify resource configuration information by changing the yml file.
Provide bundle-related commands to achieve resource operation, testing, and deployment capabilities.
Configure development and production project information in the bundle to achieve development-production isolation.
Combine with external version control systems (such as GitHub, GitLab) to achieve automation for triggering execution, testing, and deployment.

Feature Architecture



Developing with Bundle

Step 1: Install and Log In

1. Install client.
Execute the installation command in the command line:
#Install the WeData client (no installation package required, simply run the program).
#Singapore area
#macOS system installation
curl https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/linux/install.sh |sudo sh
#windows system installation
curl.exe -fsSL https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/windows/install.bat  -o install.bat && install.bat
Installation successful indicates the installation is successful.

2. Log in to client.
Execute the login command. After the command is executed, navigate to the product login interface. Complete login to automatically perform client login.
For detailed steps, see Log In
wedata auth login
The webpage shows verification is successful, indicating client login is successful.


Step 2: Initialize the Bundle Package

1. Initialize and configure the bundle package.
Execute the initialization bundle command.
For detailed steps, see Initialization
wedata bundle init
2. Configure the wedata.yml file.
In the targets of the wedata.yml file, modify the project URLs for the development and production environments.
mode: development means development environment, production means production environment.
host: refers to the project url of an environment, select any url with regional information and project ID.
root_path: production projects can specify the storage location of bundle packages in wedata studio.
targets:
  dev:
    mode: development
    workspace:
      host: https://wedata-studio-fusion.cloud.tencent.com/studio/clone/packages/edit/11336a82-dd7c-4ba6-b3a6-74bda71cc237?ProjectId=2998786495253803008
    default: true
  prod:
    mode: production
    workspace:
      host: https://wedata-studio-fusion.cloud.tencent.com/studio/datadev/workflow?ProjectId=2995753587734867968
      root_path: /Workspace/Users/test5-auto-test1/.bundle/${bundle.name}/${bundle.target}
3. Fetch resources and bind.
Pull required resources from the product and bind them. After binding, proceed with local development as needed (modify the local resource yml file).
Resource fetch: After pulling the workflow, tasks under the workflow will also be pulled to your local system bundle.
${profile_name}: The name configured upon login, indicating usage of the corresponding login account for operation authentication.
For detailed steps, see Initialization
wedata bundle generate workflow --existing-workflow-id ${workflow_id} -t dev -p ${profile_name}
If the mode is workflow scheduling, the command is:
wedata bundle generate triggerworkflow --existing-triggerworkflow-id ${workflow_id} -t dev -p ${profile_name}
Bind Resource
Execute the following command to bind the workflow and tasks.
${resource key}: The unique key of the current resource entity in the local bundle, located in the corresponding yaml file as shown in the figure below.
For detailed steps, see Binding and Unbinding

wedata bundle deployment bind task  ${resource key}  ${task_id} -t dev -p ${profile_name}           
wedata bundle deployment bind workflow  ${resource key}  ${workflow_id} -t dev -p ${profile_name}
If the mode is workflow scheduling, the command is:
wedata bundle deployment bind triggertask  ${resource key}  ${task_id} -t dev -p ${profile_name}           
wedata bundle deployment bind triggerworkflow  ${resource key}  ${workflow_id} -t dev -p ${profile_name}

Step 3: Verify, Run and Deploy

Execute the verification command to verify whether there are exceptions in the bundle before deployment.
Execute the command to run tasks for verification.
Execute the deployment command. Deployment completed.
wedata bundle validate -t dev
wedata bundle run task ${resource key} -t dev -p ${profile_name}
wedata bundle deploy -t dev
If the mode is workflow scheduling, the command is:
wedata bundle validate -t dev
wedata bundle run triggertask ${resource key} -t dev -p ${profile_name}
wedata bundle deploy -t dev

Practical Tutorial (Linking to GitLab to Automatically Trigger Pipeline Deployment)

Step 1: Install and Log In

1. Install client.
Execute the installation command in the command line.
#Install the WeData client (no installation package required, simply run the program).
#Singapore area
#macOS system installation
curl https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/linux/install.sh |sudo sh
#windows system installation
curl.exe -fsSL https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/windows/install.bat  -o install.bat && install.bat
Installation successful indicates the installation is successful.

2. Log in to client.
Execute the login command. After the command is executed, navigate to the product login interface. Complete login to automatically perform client login.
For detailed steps, see Log In
wedata auth login
The webpage shows verification is successful, indicating client login is successful.


Step 2: Initialize the Bundle Package

1. Initialize and configure the bundle package.
Execute the initialization bundle command.
For detailed steps, see Initialization
wedata bundle init
2. Configure the wedata.yml file.
In the targets of the wedata.yml file, modify the project URLs for the development and production environments.
mode: development means development environment, production means production environment.
host: refers to the project url of an environment, select any url with regional information and project ID.
root_path: production projects can specify the storage location of bundle packages in wedata studio.
targets:
  dev:
    mode: development
    workspace:
      host: https://wedata-studio-fusion.cloud.tencent.com/studio/clone/packages/edit/11336a82-dd7c-4ba6-b3a6-74bda71cc237?ProjectId=2998786495253803008
    default: true
  prod:
    mode: production
    workspace:
      host: https://wedata-studio-fusion.cloud.tencent.com/studio/datadev/workflow?ProjectId=2995753587734867968
      root_path: /Workspace/Users/test5-auto-test1/.bundle/${bundle.name}/${bundle.target}

Step 3: Bind a Git Repository and Build a Pipeline

1. Build a git repository, use the git repository tool, and bind the local repository to the remote git repository.
Build a remote git repository with two branches: dev and prod.
Here uses the ugit tool. Customer can also use other tools to perform local and remote repository binding.

2. Configure the pipeline file and place it in the git repository.
Write a pipeline triggering file with the logic: if pushing to the dev branch, then execute verification and deployment to the dev environment bundle command; if merging to the prod branch, then execute verification and deployment to the prod environment bundle command.
Note:
Here is the triggering logic recommended by the system. Alternatively, you can define the required environment and trigger pipeline logic on demand.
The command is as follows:
dev environment command:
wedata bundle validate -t dev
wedata bundle deploy -t dev
prod environment command:
wedata bundle validate -t prod
wedata bundle deploy -t prod
Git pipeline triggering file: Place the current file under the first layer directory of the git repository.
Note:
The content needs to be modified as per the note, and may differ across different GitLab versions. It requires modification and adjustment based on the situation to use.
stages:
  - build

# The build logic of the dev branch
build_dev:
  stage: build
  tags:
    - docker      #CICD machine label, modify as needed
  script:
    - |
      set -ex
      apk add --no-cache curl sudo
      curl https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/linux/install.sh |sudo sh -s -- --region 'https://api-ap-singapore.wedata.tencentcloud.com' 
      #Download the client script; it needs to be modified to suit different environments.
      # Create configuration file
      config_file="$HOME/.wedatacfg"
      if [ ! -f "$config_file" ]; then
          echo "The configuration file does not exist; create a new configuration file..."
          echo "host =https://ap-singapore.wedata.cloud.tencent.com/studio/datadev/workflow?ProjectId=2430455587205529600" > "$config_file"   
          #It needs to be modified to any wedata URL in the current environment.
          echo "secret_id =******" >> "$config_file"    
          #current user's secret_id
          echo "secret_key =******" >> "$config_file"        
          #current user's secret_key
          echo "region     =ap-singapore" >> "$config_file"               
          #Modify to the corresponding area
          echo "is_intl    =false" >> "$config_file"
          echo "The content is as follows:"
          cat "$config_file"
      else
          echo "Current configuration details:"
          cat "$config_file"
      fi
      cd wedata-cicd-test/cicd_bjjr/v_vyuyudu_bjjr/           
      # Path to the corresponding bundle

      echo "This is the build logic for the dev branch."
      wedata bundle validate -t dev
      echo "Start pushing to the dev environment:"
      wedata bundle deploy -t dev
  only:
    - dev       #Change to the git branch name

# The construction logic of the prod branch
build_prod:
  stage: build
  tags:
    - docker
  script:
    - |
      set -ex

      apk add --no-cache curl sudo
      curl https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/linux/install.sh |sudo sh -s -- --region 'https://api-ap-singapore.wedata.tencentcloud.com' 
      #Download the client script; you'll need to modify it to suit different environments.
      # Create configuration file
      config_file="$HOME/.wedatacfg"
      if [ ! -f "$config_file" ]; then
          echo "The configuration file does not exist; create a new configuration file..."
          echo "host =https://ap-singapore.wedata.cloud.tencent.com/studio/datadev/workflow?ProjectId=2430455587205529600" > "$config_file"
          #It needs to be modified to any wedata URL in the current environment.
          echo "secret_id =******" >> "$config_file"
          echo "secret_key =******" >> "$config_file"
          echo "region     =ap-singapore" >> "$config_file"
          echo "is_intl    =false" >> "$config_file"
          echo "The content is as follows:"
          cat "$config_file"
      else
          echo "Current configuration details:"
          cat "$config_file"
      fi
      cd wedata-cicd-test/cicd_bjjr/v_vyuyudu_bjjr/

      echo "This is the build logic for the prod branch."
      wedata bundle validate -t prod
      echo "Start pushing to the prod environment:"
      wedata bundle deploy -t prod
  only:
    - merge_requests              # Triggered for merge scenarios
    - prod                        # Triggered on the prod branch

Step 4: Develop in the Dev Environment and Submit

Modify the bundle package content in the local editor to develop, then use ugit to submit to the dev branch after development.
After submission, the pipeline will be triggered automatically and the changes will be deployed to the corresponding project in the dev environment.


Step 5: Merge to the Prod Branch

Merge changes from the dev branch to the prod branch, and automatically trigger the pipeline to deploy the modifications to the corresponding project in the prod environment.




Bundle Package Definition

1. wedata.yml

1.1 Basic Configuration

Define bundle basic information and target info, contain content:
bundle basic information: name, uuid, client version.
include: file path and scope of resources in the bundle. resources/*/*.yml means all yaml files under the resource directory. Customers can self-define the resource range of the bundle.
target: for defining target projects, define on demand for different environments.
mode: development means development environment, production means production environment.
host: refers to the project url of an environment, select any url with regional information and project ID.
root_path: production projects can specify the storage location of bundle packages in wedata studio.
variables: define parameters used in the bundle. A common scenario is when configuration messages vary across different environments. This can be addressed by defining different parameter values for each environment.
# This is a Wedata asset bundle definition
bundle:
  name: wedataBundleDemo
  uuid: e1e95042-9e18-4806-a2e6-8de5e535881c
  wedata_cli_version: '0.18.*'
  # allow any Wedata CLI version between 0.218.0 and 1.0.0, inclusive
  # wedata_cli_version: '>= 0.218.0, <= 1.0.0'

# Paths to match based on root directory
include:
  - resources/*/*.yml

# Deploy synchronizes default+include folders that are not filtered by exclude rules, using Glob pattern
sync:
  include:
    - "resources/**"
    - "src/**"
    - "tests/**"
  exclude:
    - "tests/README.md"

targets:
  dev:
    # The default target uses 'mode: development' to create a development copy.
    # - Deployed resources get prefixed with '[dev my_user_name]'
    # - Any job schedules and triggers are paused by default.
    mode: development
    # Specify the environment configuration for the current bundle
    default: true
    variables: #You can define different environment variables here
      test_worker_group_id: "20241120210318772642"
      cluster_id: "123"
    workspace:
      # projectAddr
      host: https://ap-beijing.wedata.cloud.tencent.com/studio/datadev/workflow?ProjectId=1460726726769418240
      # root_path: ~/.bundle/${bundle.name}/${bundle.target}
      # ~/ also represents the user's personal path, equivalent to root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}
      # Note: root_path must start with '~/' or contain the current username to ensure uniqueness when using 'mode: development'
  prod:
    mode: production
    variables: #You can define different environment variables here
      test_worker_group_id: "2024112021031872222"
      cluster_id: "111"
    workspace:
      # projectAddr
      host: https://ap-beijing.wedata.cloud.tencent.com/studio/datadev/workflow?ProjectId=1460726726769418240
      # We explicitly deploy to /Workspace/Users/asherxu@tencent.com to make sure we only have a single copy.
      root_path: /Workspace/Users/demo@tencent.com/.bundle/${bundle.name}/${bundle.target}

# Global custom variables, usage definition: ${var.<variable_name>}
variables:
  test_worker_group_id:
    description: defaultGroupID
    default: "20241120210318772644"
    type: string
  cluster_id:
    description: defaultClusterID
    default: "234"
    type: string

1.2 Parameter Definition Method

Usage scenario: When different projects require different configurations, the parameter method can be used. For example: in a development project, tasks use scheduling resource group resource_group_A, while in a production project, the corresponding scheduling resource group is resource_group_B.
Parameter definition: Define the parameter scheduling_resource_group in the wedata.yml file.
variables:
  scheduling_resource_group:
    description: "scheduling_resource_group_name"
    default: "resource_group_A"
    type: string
Parameter value definition: Define parameter values used in different projects within different targets in the wedata.yml file.
targets:
    dev:
      mode: development
      variables: 
        scheduling_resource_group: "resource_group_A"
    prod:
      mode: production
      variables: 
        scheduling_resource_group: "resource_group_B"
Parameter usage: Use parameters in any yml file of the bundle with the format: ${var.parameter name}.
resources:
  tasks:
    taskResourceKey:this_is_task_key
      name: taskName
      resourceGroupName: ${var.scheduling_resource_group}

2. Resource Entity

The resource types that can be included in a bundle are: workflow, task, resource file, project parameter, alarm rule, and event. The following will introduce how to configure different resources in a bundle.
Most configurations are identical to the input parameter names defined in the OpenAPI for creating entities. See the definition in the OpenAPI.

2.1 Workflow (Task Scheduling Mode)

Divide into workflow scheduling projects and task scheduling projects by project. The current resource is a workflow in task scheduling mode.
resources:
  workflows:
    bundle_example_workflow:
      name: "bundle_example_workflow"
      remoteDirPath: Default/temp/
      ownerName: ${var.ownerName}
      workflowDesc: ""
      workflowType: "cycle"
      workflowParams:
        paramDs: []
      workflowSchedulerConfiguration:
        startTime: "2025-10-22 11:37:57"
        endTime: "2099-12-31 23:59:59"
        executionStartTime: "00:00"
        executionEndTime: "23:59"
        crontabExpression: "0 0 0 * * ? *"
        scheduleTimeZone: "UTC+8"
        selfDepend: "serial"
        dependencyWorkflow: "no"
        cycleStep: 0
        cycleType: "DAY_CYCLE"
        delayTime: 0


2.2 Task (Task Scheduling Mode)

Projects are divided into workflow scheduling projects and task scheduling projects. The current resource is a task in task scheduling mode.
Configuration items vary by task type. Below is an example of a Python Task. For other task types, refer to the corresponding OpenAPI.
resources:
  tasks:
    bundle_example_task_python_task:
      name: "bundle_example_task_python"
      filePath: "../../src/task/bundle_example_task_python.py"
      remoteDirPath: Default/temp/
      taskBaseAttribute:
        taskTypeName: "PYTHON"
        workflowName: "demo"
        taskDescription: "This is a python task"
        ownerName: lolayqzhang
      taskConfiguration:
        resourceGroupName: test_worker
        brokerIp: "any"
        taskExtConfigurationList:
        - paramKey: "python_type"
          paramValue: "python3"
        - paramKey: "python_sub_version"
          paramValue: "python3"
        - paramKey: "extraInfo"
          paramValue: "{\\"fromMapping\\":false}"
        taskSchedulingParameterList: []
      taskSchedulerConfiguration:
        crontabExpression: "0 0 0 * * ? *"
        cycleType: "DAY_CYCLE"
        scheduleTimeZone: "UTC+8"
        startTime: "2025-10-24 00:00:00"
        endTime: "2099-12-31 23:59:59"
        executionStartTime: "00:00"
        executionEndTime: "23:59"
        scheduleRunType: 0
        selfDepend: "serial"
        upstreamDependencyConfigList: []
        downstreamDependencyConfigList: []
        runPriority: "low"
        retryWait: 5
        maxRetryAttempts: 4
        executionTTL: -1
        waitExecutionTotalTTL: "-1"
        allowRedoType: "ALL"

Offline Integration Task - Script Mode Description: The script location for offline integration is src/task/, and other configuration is in resource/task/.
For-Each node description
Configuration content of the For-Each primary node is placed in resource/task/
In the yml file of the For-Each primary node, add childPath to identify the location of sub-nodes. For example: the primary node name is test_foreach, test_foreach_end_foreach refers to the end sub-node, test_foreach_start_foreach refers to the start sub-node, and test_dlcsql represents a user-configured task sub-node.
childPath:
    - "../../src/forEachChild/test_foreach_end_foreach.yml"
    - "../../src/forEachChild/test_foreach_start_foreach.yml"
    - "../../src/forEachChild/test_dlcsql.yml"
Configuration content of For-Each sub-nodes is placed in src/forEachChild
Code content of For-Each sub-nodes is placed in src/task
If a sub-node is downstream of the start node, configure the upstream node as start_foreach in the sub-node. For example:
triggerTaskSchedulerConfiguration:
   upstreamDependencyConfigList:
   - taskName: "start_foreach"
If a sub-node is upstream of the end Node, configure the upstream node as this sub-node in the end Node.
Notebook task description
Notebook Task only supports the reference method configured from studio. The code is placed under the src folder. Code modification will sync the modifications to the corresponding studio script code.

2.3 Workflow (Workflow Scheduling Mode)

Projects are divided into workflow scheduling projects and task scheduling projects. The current resource is a workflow in workflow scheduling mode.
resources:
  triggerWorkflows:
    bundle_example_triggerWorkflow_triggerWorkflow:
      name: "bundle_example_triggerWorkflow"
      remoteDirPath: Default/temp
      ownerName: ${var.ownerName}
      workflowDesc: "this is a triggerWorkflow"
      workflowParams:
        paramDs: []
      triggerWorkflowSchedulerConfigurations:
      - schedulerStatus: "ACTIVE"
        triggerMode: "TIME_TRIGGER"
        scheduleTimeZone: "UTC+8"
        startTime: "2025-12-17 20:29:55"
        endTime: "2099-12-31 23:59:59"
        configMode: "COMMON"
        cycleStep: 0
        cycleType: "DAY_CYCLE"
        crontabExpression: "0 0 1 * * ? *"
        delayTime: 0

2.4 Task (Workflow Scheduling Mode)

Projects are divided into workflow scheduling projects and task scheduling projects. The current resource is a task in workflow scheduling mode.
resources:
  triggerTasks:
    bundle_example_triggerTask_python_triggerTask:
      name: "bundle_example_triggerTask_python"
      filePath: "../../src/triggerTask/bundle_example_triggerTask_python.py"
      remoteDirPath: Default/temp
      triggerTaskBaseAttribute:
        taskTypeName: "PYTHON"
        workflowName: "bundle_example_triggerWorkflow"
        taskDescription: "this is a python triggerTask"
        ownerName: ${var.ownerName}
      triggerTaskConfiguration:
        resourceGroupName: ${var.resourceGroupName}
        brokerIp: "any"
        taskExtConfigurationList:
        - paramKey: "python_sub_version"
          paramValue: "python3"
        - paramKey: "python_type"
          paramValue: "python3"
        - paramKey: "waitExecutionTotalTTL"
          paramValue: "-1"
        taskSchedulingParameterList: []
      triggerTaskSchedulerConfiguration:
        upstreamDependencyConfigList: []
        allowRedoType: "ALL"

2.5 Resource File

resources:
  resourceFiles:
    resourceFileResourceKey:
      name: resourceFileName
      # Wedata remote directory
      remoteDirPath: /fd1/test/
      # Parameter name
      filePath: ../../src/resourceFile/datastudio-service-5.2.0.1.jar
      # COS bucket name
      bucketName: wedata-cos
      # COS bucket region
      cosRegion: ap-shanghai

2.6 Project Parameter

resources:
  projectParams:
    paramResourceKey:
      # Parameter name key
      name: paramName
      # Parameter description
      paramDescription: paramDescription
      # Debug run value
      paramDefineTest: dryRunVal
      # Cycle scheduling value
      paramDefine: scheduleRunVal

2.7 Alarm Rule

resources:
  alarms:
    alarmResourceKey:
      # Alarm name
      name: alarmName
      # Alarm rule description
      description: description
      # List of monitored object business names (task names)
      monitorObjectNames:
        - taskName1
        - taskName2
      # Monitor type: task, workflow, project
      monitorObjectType: task
      # Alarm type: 1.failure alarm, 2.timeout alarm, 3.success alarm, 4.baseline breach, 5.baseline warning, 6.baseline task failure, 7.alarm monitoring, alarm type code list (default 1.failure alarm)
      alarmTypes:
       - failAlarm
      alarmGroups:
        - alarmRecipientType: Task Owner # Alarm recipient type: 1.specified personnel, 2.task owner, 3.duty roster (default 1.specified personnel)
          # Alarm level, 1.normal, 2.important, 3.urgent (default 1.normal)
          alarmLevel: normal
          # Alarm escalation user names
          alarmEscalationRecipientNames:
            - asherxu@tencent.com
          # Alarm escalation interval
          alarmEscalationInterval: 86400
          # Alarm user names
          alarmRecipientNames:
            - asherxu@tencent.com
          # Alarm methods: 1.email, 2.SMS, 3.WeChat, 4.voice, 5.WeCom, 6.Http, 7.WeCom group, 8.Feishu group (default 1.email)
          alarmWays:
            - email
          webHooks:
            alarmWay: "Teams"
            webHooks: "http://open/group"
          # Alarm notification frequency
          notificationFatigue:
            # Notification count (3 times)
            notifyCount: 3
            # Notification interval, in minutes
            notifyInterval: 5
            # Do not disturb time
            quietIntervals:
              - daysOfWeek: [ 1, 2 ] # Days of week
                startTime: 02:00:00 # Start time, precise to hour, minute, second
                endTime: 06:00:00 # End time, precise to hour, minute, second
              - daysOfWeek: [ 3 ] # Days of week
                startTime: 01:00:00 # Start time, precise to hour, minute, second
                endTime: 05:00:00 # End time, precise to hour, minute, second

      # Alarm rule details
      alarmRuleDetail:
        # Failure trigger timing 1 - First failure trigger 2 - All retries completed trigger (default)
        trigger: 1
        # Rerun/backfill failure trigger timing 1 - First failure trigger 2 - All retries completed trigger (default)
        dataBackfillOrRerunTrigger: 1
        # Timeout configuration details
        timeOutExtInfo:
          - ruleType: 1 # Timeout strategy: 1.estimated runtime, 2.estimated completion time, 3.estimated scheduling time, 4.not completed within cycle, 5.total waiting time, 6.waiting runtime
            type: 1 # Timeout value type 1.specified value (default) 2.average value
            hour: 1 # Timeout specified value hours (default 0)
            min: 30 # Timeout specified value minutes (default 1)
            scheduleTimeZone: UTC+7 # Configured timezone, defaults to project timezone
        # Backfill/rerun timeout configuration details
        dataBackfillOrRerunTimeOutExtInfo:
          - ruleType: 1 # Timeout strategy: 1.estimated runtime, 2.estimated completion time, 3.estimated scheduling time, 4.not completed within cycle, 5.total waiting time, 6.waiting runtime
            type: 1 # Timeout value type 1.specified value (default) 2.average value
            hour: 1 # Timeout specified value hours (default 0)
            min: 30 # Timeout specified value minutes (default 1)
            scheduleTimeZone: UTC+7 # Configured timezone, defaults to project timezone
        # Project fluctuation alarm configuration details
        projectInstanceStatisticsAlarmInfoList:
          alarmType: projectFailureInstanceUpwardFluctuationAlarm # Failed instance upward fluctuation alarm, successful instance downward fluctuation alarm
          instanceThresholdCountPercent: 10 # Percentage of failed instances for the day
          instanceThresholdCount: 20 # Total threshold: daily failed instances, daily cumulative instances, daily failed instances upward fluctuation, cumulative instances fluctuation threshold
          instanceCount: 20 # Daily cumulative instances, daily failed instances downward fluctuation
          stabilizeStatisticsCycle: 3 # Stability statistics cycle (anti-jitter configuration statistics cycle count)
          isCumulate: false # Whether to calculate cumulatively, false: consecutive, true: cumulative
          stabilizeThreshold: 3 # Stability count threshold (anti-jitter configuration statistics cycle count)
        reconciliationExtInfo:
          - ruleType: reconciliationFailure
            mismatchCount: 1
            hour: 1
            min: 30

2.8 Event

resources:
  events:
    eventResourceKey:
      name: eventName
      # Types include: MIN, HOUR, DAY
      eventSubType: MIN
      # Optional, if not filled, use default user
      ownerName: asherxu
      # Event description
      description: "description"
      # Event time to live
      timeToLive: 1
      # Time to live unit: DAY, MIN
      timeUnit: DAYS
      # Event type: GENERAL, TIME_SERIES
      eventType: "TIME_SERIES"
      # Event broadcast type: SINGLE, BROADCAST
      eventBroadcastType: "BROADCAST"
      # Time format
      dimensionFormat: "yyyymmdd"

3. Src File

Used to store task code information and resource file upload packages. Task code changes or resource file upload package modifications can be performed in the current file.

Bundle Command

1. Installing & Uninstalling Client

1. Install client.
Feature: Install WeData Client locally.
Command:
#Install the WeData client (you need to download the installation package to your local machine and then run it).
sudo  sh install.sh 

#Install the WeData client (no installation package required, simply run the program).
#Singapore area
#macOS system installation
curl https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/linux/install.sh |sudo sh
#windows system installation
curl.exe -fsSL https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/windows/install.bat  -o install.bat && install.bat
2. Uninstall client.
Feature: Uninstall WeData Client locally.
Command:
#Uninstall WeData client
sudo  sh install.sh  --uninstall
3. Upgrade to the latest version.
Feature: Upgrade the currently installed client version to the latest version.
Command:
#Upgrade the currently installed client version to the latest version.
sudo  sh install.sh  --upgrade

#Update to the latest version
#Singapore area
#macOS system upgrade
curl https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/linux/install.sh |sudo sh -s -- --upgrade
#windows system upgrate
curl.exe -fsSL https://wedata-agent-sg-1257305158.cos.ap-singapore.myqcloud.com/bundle/install-script/windows/install.bat  -o install.bat && install.bat   --upgrade
4. Get the installed version.
Feature: Get the current installed WeData Client version.
Command:
#Get the currently installed client version
wedata -v     
wedata --version

2. Log In

Support two client login methods: redirect to product login verification, or use fixed key login.
1. Redirect product login verification.
Step 1: Enter the login command in the local terminal, replace ${workspace-url} with any project link under the login account. The link must include region and project ID information.
#Both commands can be used to log in.
wedata auth login --host '${workspace-url}'    
wedata auth login
Step 2: Define the login profile and input the project url information.
enter the profile name here (press enter to use the current configuration).
Please enter the Wedata project address: If you see no project url in the command, you need to specify the url here.
Step 3: Navigate to the product and log in.
After refining the login command info, if you are not logged in to the product, navigate to the login page. You need to complete login. If already logged in, enter the third step.
Step 4: Go to Cloud Access Management (CAM) to authorize on initial login.
For the first login, according to page guidance, go to the CAM module to authorize. After authorization complete, return to the login page and refresh to verify again.
2. Log in using a fixed key.
Users logged in via EntraID are recommended to use a fixed key for logon.
Step 1: Use EntraID to log in to the WeData product.
If you have never used EntraID for login, you must complete the initial login:
On the WeData product interface, select EntraID login method to log in
Once logged in successfully, execute the following command in the terminal to complete CLI authentication (this command will write EntraID's temporary credentials to the local configuration file ~/.wedata/wedata.json):
wedata auth login
Step 2: Set Token information for EntraID
Open the local credential file:
# Open the configuration file with an editor
code ~/.wedata/wedata.json
# Or.
vim ~/.wedata/wedata.json
Locate your account's corresponding position in the file (use your UIN as the key), and write EntraID's Token, SecretId, SecretKey to the corresponding field.
The file structure example is as follows:
{
  "version": 1,
  "tokens": {
    "<your UIN>": {
      "Token": "<EntraID token value>",
      "SecretId": "<EntraID SecretId value>",
      "SecretKey": "<EntraID SecretKey value>",
      "ExpireTime": 1770242769,
      "AppId": "<Your AppId>",
      "Uin": "<your UIN>",
      "UserName": "<your Entra ID email address>",
      "OwnerUserId": "<OwnerUserId>",
      "Region": "ap-guangzhou",
      "Language": "zh-CN",
      "IsIntl": false
    }
  }
}
Ways to get temporary credentials:
Turn on the credential view access point: Tencent Cloud Organization > Identity Center Management > Settings > Login page. Get temporary key quickly: turn on the switch.

Obtain credential: identity center login > find the account and click access credential > copy temporary credential.

Step 3: Set ExpireTime to 0
In the wedata.json file, change the ExpireTime field value to **0**:
Set the ExpireTime to 0 to avoid the CLI tool denying the use of the Token due to expiration check, underwrite credential availability in CI/CD process.
{
  "Token": "<your token>",
  "SecretId": "<your SecretId>",
  "SecretKey": "<your SecretKey>",
  "ExpireTime": 0,
  ...
}
Step 4: Use CI/CD commands
The login configuration is completed. You can now use client commands normally.
Step 5: Configure session duration
If adjustments are needed for the EntraID account's session valid duration, it can be configured on the Tencent Cloud Group Account Management (TCO) console.
Enter Organization Account Management > CAM Synchronization > Permission Configuration
On the configuration detail page, find the maximum session duration field
Click edit button, set appropriate duration (for example 12 hr), after 12 hour, modify again Token, SecretId, SecretKey in the configuration file.

Note：
Log in via redirection product login verification with a login status retention time of 12 hours. Exceeding 12 hours requires logging in again. Use fixed key method for permanent login status retention.

3. Initialization

1. bundle init
Feature: Initialize a bundle package. The initialized bundle package includes a wedata.yml file and resource files. The wedata.yml file contains basic information of the bundle package, target (for defining corresponding projects in different environments), etc. You can manually select resources to initialize, such as workflows, tasks, resource files, project parameters, alarm rules, and events. If selected, the system defaults to initializing the corresponding resources, and you can modify the resource configuration as required.
in the command:
Configure bundle package name
Select a scheduling mode: workflow_trigger/task_trigger. If your project is a workflow scheduling project, select workflow_trigger. If your project is a task scheduling project, select task_trigger.
Select the required resource demo type
Command:
#Initialize a bundle
wedata bundle init -p ${profile_name}
2. bundle generate
Feature: Pull resources already created in the project to your local system bundle. Pulling resources locally does not mean they are bound to remote resources. To establish a binding relationship, execute the bind command.
Note: For tasks created via git binding method, generate only retrieves configuration, not code content. Code content changes depend on the git path specified by the customer.
-t: target environment.
-p: specify the execution command profile.
#Bundle initialization related commands
wedata bundle generate task --existing-task-id ${task_id} -t dev/prod -p ${profile_name}                       
wedata bundle generate workflow --existing-workflow-id ${workflow_id} -t dev/prod -p ${profile_name}           
wedata bundle generate event --existing-event-id ${event_id} -t dev/prod -p ${profile_name}                     
wedata bundle generate alarm --existing-alarm-id ${alarm_id} -t dev/prod -p ${profile_name}                   
wedata bundle generate resourceFile --existing-resourceFile-id ${resourceFile_id} -t dev/prod -p ${profile_name} 
wedata bundle generate projectParam --existing-projectParam-name ${projectParam_name} -t dev/prod -p ${profile_name}
In workflow scheduling mode, the task and workflow commands are:
#Bundle initialization related commands
wedata bundle generate triggertask --existing-triggertask-id ${task_id} -t dev/prod -p ${profile_name}                       
wedata bundle generate triggerworkflow --existing-triggerworkflow-id ${workflow_id} -t dev/prod -p ${profile_name}

4. Bind and Unbind

1. bundle deployment bind
Feature: Bind the generated resources to the current bundle. After binding, local modifications will synchronize to the remote end.
-t: target environment.
-p: specify the execution command profile.
Command:
#Resources are bound to local bundles
wedata bundle deployment bind task  ${resource key}  ${task_id} -t dev/prod -p ${profile_name}                   
wedata bundle deployment bind workflow ${resource key}  ${workflow_id} -t dev/prod -p ${profile_name}            
wedata bundle deployment bind event  ${resource key} ${event_id} -t dev/prod -p ${profile_name}                  
wedata bundle deployment bind alarm  ${resource key} ${alarm_id} -t dev/prod -p ${profile_name}                  
wedata bundle deployment bind resourceFile ${resource key}  ${resourceFile_id} -t dev/prod -p ${profile_name}   
wedata bundle deployment bind projectParam ${resource key}  ${projectParam_name} -t dev/prod -p ${profile_name}
In workflow scheduling mode, the task and workflow commands are:
#Resources are bound to local bundles
wedata bundle deployment bind triggertask  ${resource key}  ${task_id} -t dev/prod -p ${profile_name}                    
wedata bundle deployment bind triggerworkflow ${resource key}  ${workflow_id} -t dev/prod -p ${profile_name}
2. bundle deployment unbind
Feature: Unbind resources from the local bundle.
-t: target environment.
-p: specify the execution command profile.
Command:
#Unbind resources from local bundle
wedata bundle deployment unbind task   ${resource key} -t dev/prod -p ${profile_name}          
wedata bundle deployment unbind workflow   ${resource key} -t dev/prod -p ${profile_name}      
wedata bundle deployment unbind event   ${resource key} -t dev/prod -p ${profile_name}         
wedata bundle deployment unbind alarm   ${resource key} -t dev/prod -p ${profile_name}         
wedata bundle deployment unbind resourceFile   ${resource key} -t dev/prod -p ${profile_name}  
wedata bundle deployment unbind wedataStudio   ${resource key} -t dev/prod -p ${profile_name}  
wedata bundle deployment unbind projectParam   ${resource key} -t dev/prod -p ${profile_name}
In workflow scheduling mode, the task and workflow commands are:
#Unbind resources from local bundle
wedata bundle deployment unbind triggertask   ${resource key} -t dev/prod -p ${profile_name}           
wedata bundle deployment unbind triggerworkflow   ${resource key} -t dev/prod -p ${profile_name}

5. Running

Feature: Run a certain task or all tasks under the workflow. Execution status and results can be viewed in the orchestration space.
-t: target environment.
-p: specify the execution command profile.
Command:
wedata bundle run task ${resource key} -t dev/prod -p ${profile_name}       #Run a specific task
wedata bundle run workflow ${resource key} -t dev/prod -p ${profile_name}   #Run a specific workflow
In workflow scheduling mode, the task and workflow commands are:
wedata bundle run triggertask ${resource key} -t dev/prod -p ${profile_name}        #Run a specific task
wedata bundle run triggerworkflow ${resource key} -t dev/prod -p ${profile_name}    #Run a specific workflow

6. Verification

Feature: Verify whether there are issues with the bundle and its resources. It can verify before deployment for early problem detection and handle problems.
-t: target environment.
-p: specify the execution command profile.
wedata bundle validate -t dev/prod -p ${profile_name}    #Verify bundle and resource configuration

7. Deployment

Feature: Deploy the content of the bundle to the target environment.
-t: target environment.
-p: specify the execution command profile.
-s: configure -s means change submission to scheduling, without configuring means only generate save version in orchestration space.
Command:
wedata bundle deploy -t dev/prod -p ${profile_name} -s true    # Deploy the contents of the bundle to the corresponding environment

8. Termination

Feature: Delete the bound task at the remote end and remove the bundle package and resource information in remote storage.
-t: target environment.
-p: specify the execution command profile.
Command:
wedata bundle destroy -t dev/prod -p ${profile_name}  #Destroy the contents deployed via bundle in the corresponding environment.

9.Obtaining Identity Information

Feature: Describe the configuration used for identity authentication in Databricks CLI, including credentials and its source.
-p: retrieve the corresponding information of the specified profile.
--host: retrieve the corresponding profile info of the current sub-account.
--sensitive: Output sensitive information (secret_id and secret_key). If not specified, secret_id and key show as ******.
Command:
wedata auth describe   #By default, it retrieves the information corresponding to the default profile.
wedata auth describe -p   ${profile_name}   #Retrieve information corresponding to the specified profile.
wedata auth describe  --host sub_account   #Retrieve the profile information corresponding to the current sub-account.
wedata auth describe  --sensitive        #Outputs sensitive information (secret_id and secret_key). If not specified, secret_id and key will be displayed as ******

10. Summary

Feature: Get the resource information contained in the current bundle.
-t: target environment.
Command:
wedata bundle summary  #Summary of the current package binding task status

11. Help

Feature: Get help information, including the usage rules and help information for the bundle command.
Command:
wedata bundle -help    # Display command help information
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback