Workflow Scheduling Universal Development Process
Last updated:2026-03-04 09:43:27
Usage Scenario
Workflow scheduling means the scheduling configuration is uniformly set at the workflow granularity, and tasks under the workflow run directly in the workflow.
Use Limits
A project can only be configured with one scheduling mode, either workflow scheduling or task scheduling. Once configured, the scheduling mode cannot be modified.
Workflow scheduling mode projects do not support cross-workflow dependency but support nested workflow capacity. You can nest another workflow in a workflow, and the nested workflow will run with the scheduling configuration of the external workflow.
Workflow scheduling and task scheduling projects do not support cross-project dependency for tasks.
The workflow scheduling project supports the following task types: Offline Integration, Python, Shell, branch, Notebook, For-Each, DLC SQL, DLC Spark, DLC PySpark, DLC Spark Streaming.
Workflow scheduling projects do not support operation dashboard and other features.
Workflow scheduling projects do not currently support standard mode, cross-project cloning, or import and export features.
Workflow Scheduling Configuration
During project creation, you can select the scheduling mode. Task scheduling means the scheduling is configured at the task granularity, while workflow scheduling means the scheduling is configured at the workflow granularity. Tasks under a task scheduling project can have mutual dependence, but cannot be mutually dependent with tasks under a workflow scheduling project. Workflows under a workflow scheduling project can nest each other. Once configured, it cannot be modified, so choose carefully.
Workflow Scheduling Development
Create Workflow
Workflow Orchestration — Orchestration Space — Create Workflow
Configure workflow name, select folder
Workflow Configuration
General Configurations
Toggle on the Right Sidebar General Settings to complete the configuration of basic information such as workflow name, workflow owner, description, workflow parameters, and Spark parameters.
Scheduler Configuration
Toggle on the Right Sidebar scheduling configuration. If not configured, it will not run periodically by default after submission. You can manually trigger the operation in the Ops center (similar to the ability of manual workflow in task scheduling mode).
If you need the workflow to run periodic tasks, you can add scheduling in the scheduling configuration.
Scheduling status: startup/suspension. If you need periodic tasks, set it to startup. If you don't need periodic tasks but want to reserve the configuration content, set it to suspension.
Trigger mode: Only scheduled triggering is supported in the current period.
Effective time: the running time range of the workflow.
Setting method: General and cron expression.
Normal mode supports scheduling configuration for day, week, month, year, hr, and minute cycle types.
cron expression supports configuration or manually entering the expression.
Daily task: Supports two configuration modes: single time point and multiple time points. Single time point means running at a fixed time each day. Multiple time points mean selecting hr and minute, running at the corresponding hr and minute each day. For example, if hr is set to 2 and 3, and minute to 5 and 35, it will run at 2:05, 2:35, 3:05, and 3:35 daily.
Hourly task: Supports two configuration modes: specify interval and specify time. Specify interval means setting how many hr intervals to run within a time interval each day (end time only supports selecting 59 minutes). Specify time means selecting the corresponding hr and minute.
Minute-based task: Supports selection of start time, end time, and time interval. From the start time, if the calculation reaches the next hour on the hour, it will not continue execution at the interval. Subsequently, the minute time points for hourly runs remain identical to those prior, until the end time. For example, executing every 30 minutes with a time interval from 05:29 to 23:59. It will start at 05:29, running at 29 and 59 minutes every hour. The specific running times are 05:29, 05:59, 06:29, 06:59, and so on.
Weekly task: Supports selecting the day of the week and time to run. Execution time point also supports single time point and multiple time points, with the same setting method as daily task.
Monthly task: Supports selecting the day of the month and time to run. Execution time point also supports single time point and multiple time points, with the same setting method as daily task.
Annual task: Supports selecting the month, day, and time to run each year. Execution time point also supports single time point and multiple time points, with the same setting method as daily task.
Scheduling preview: After completing the scheduling configuration, you can view the last 5 scheduling times to help determine whether it meets expectations.
Configured scheduling can also be deleted, meaning no periodic scheduling, and can be manually triggered.
The workflow's features, such as functioning, advanced execution, and pause running, are the same as those in task scheduling mode.
Creating Task
Click the task name in the workflow canvas - create task to create a corresponding type task.
Creating Task
Set the task name, task type, and workflow directory to complete task creation.
Workflow scheduling currently supports the following task types: Offline Integration, Python, Shell, branch, Notebook, For-Each, DLC SQL, DLC Spark, DLC PySpark, DLC Spark Streaming.
Configuring Tasks
Task Attribute
Configure the basic information such as task name, task owner, description, and scheduling parameters.
Scheduling Setting
At the task granularity, the ability to configure scheduling time is no longer provided. Tasks run as periodic tasks based on workflow configuration.
Task scheduling settings include upstream task configuration (only support selection of other tasks in the workflow, cross-workflow dependency is not supported), output registration, parameter passing, and advanced setting feature.
Run condition configuration: If the current task is configured with upstream, the run condition can take effect. The run condition means what status the upstream task is in for the current task to operate normally, otherwise it will skip running or be in a failed state. The run condition supports the following options. By default, select all successful, which means the direct upstream must be all successful to run.
Run Condition | Condition Description | Is Upstream Mandatory to Be All in Final State | Upstream Status Description | Downstream Status When Failing to Satisfy Conditions |
all successful | Only when ALL direct upstream tasks run successfully will this task be triggered. | Yes | all successful | all failed/upstream failed is upstream failed. skip running if at least one skip running |
all failed | Only when ALL direct upstream tasks fail to run will this task be triggered. | Yes | all failed, upstream failed | skip running |
all completed | This task will be triggered as long as ALL direct upstream tasks reach running end (success, failure, upstream failure or skip running). | Yes | all succeeded, failed, upstream failed or skip running | waiting for upstream |
all completed and at least one successful | ALL upstream tasks are stopped, and at least one is successful. | Yes | all succeeded, failed, upstream failed or skip running, and at least one is successful | skip running |
all completed and at least one failed | ALL upstream tasks are stopped, and at least one is failed. | Yes | all succeeded, failed, upstream failed or skip running, and at least one is failed or upstream failed | skip running |
all skip running | Only when ALL direct upstream tasks skip running due to logical judgments will this task be triggered. | Yes | all skip running | skip running |
all completed and no failure | ALL upstream tasks are stopped, status is successful or skip running. | Yes | all successful or skip running | upstream failed |
no failure and at least one successful | The upstream task did not fail, and at least one task was executed successfully. Commonly used for scenarios where the downstream of branch node aggregates node operation. | Yes | all successful or skip running and at least one succeed | skip running |
not skip running | ALL upstream tasks must be actual running (success or fail, upstream failure), cannot have any task skip. | Yes | all successful, failed, upstream failed | skip running |
at least one failed | Once an upstream task fails, this task triggers immediately with no waiting for other tasks to complete. | No | at least one failed or upstream failed | Wait for upstream until failure occurs. all completed and no failure: skip running |
at least one successful | Once an upstream task succeeds, this task triggers immediately with no waiting for other tasks to complete. | No | at least one successful | Wait for upstream until success occurs. all completed and no success: skip running |
at least one completed | Once an upstream task is completed (successful, unsuccessful, upstream failure or skip running), this task triggers immediately with no waiting for other tasks to complete. | No | at least one succeeded, failed, upstream failed or skip running | waiting for upstream |
The task's features, such as saving, submission, run, and advanced execution, are identical to the corresponding operation in task scheduling mode.
Configuring Run Workflow Task
Run workflow usage scenario: A run workflow is used when one workflow has upstream and downstream dependencies with another workflow in the workflow and must follow the periodic tasks of another workflow.
Run workflow usage limits: Loop nesting and exceeding 3 levels of nesting are not supported.
Loop nesting example: Task A in workflow A nests workflow B, and task B in workflow B nests workflow A.
Example of exceeding 3 levels of nesting: Task A in workflow A nests workflow B, task B in workflow B nests workflow C, and task C in workflow C nests workflow D.
Create a run workflow task: Select the task type as workflow, choose a workflow to nest, and only workflows under the same project are supported for selection.
Run workflow task configuration: Supports configuration of run workflows. Scheduling configuration includes upstream dependency tasks and parameter passing.
Upstream dependency tasks only support selecting other tasks in the workflow.
Parameter passing rules are as follows
Parameters passed upstream to the run workflow task node can take effect and function in each task under the run workflow when the run workflow task node follows the public workflow running. If the run workflow node runs separately, parameter passing cannot be achieved.
Output parameters of the run workflow task node can be passed to downstream nodes and can be used normally.
If the passed parameter has the same name as the configured parameter in the task, the parameter configured in the public workflow running takes precedence. When running inside the run workflow, only internal parameters are considered, and external parameters cannot be used.
Multi-level nesting supports multi-layer transmission. Priority: outer layer is higher than inner-layer.
Workflow Scheduling Operation and Maintenance
Workflow Operation
workflow list
You can view the basic situation and perform basic operations of workflows submitted for scheduling in the workflow list.
Basic information: view workflow name, folder, owner, scheduling type, trigger mode, number of tasks, etc.
Basic operations: run, modify owner, delete workflow, workflow development
Run: run ALL or some tasks in the workflow
Execution scope: support selecting the entire workflow or specify some tasks to run in the workflow.
Parameter configuration: support re-specifying parameters at runtime. If not specified, the configured parameters at task submission apply. Here, defining new parameters is not supported, only re-specifying existing parameter values is supported.
Execution resource group: support re-specifying scheduling resource group and integration resource group.
Workflow details: Click the workflow name in the workflow list to navigate to the workflow details page and view the workflow run history and task DAG in the workflow.
Workflow Operation
Workflow run list: Records of periodic tasks or manual operations will be displayed in the workflow run list. You can view basic information and perform basic operations.
Basic information: workflow name, running ID, start time, end time, execution status, execution duration, trigger mode, owner, folder.
workflow execution status
Waiting: All task nodes in the workflow are not started.
Running: At least one task in the workflow is running.
Successful: All tasks in the workflow run successfully.
Failed: All tasks in the workflow reach the final state, and at least one node operation fails.
Terminating: The workflow is terminating.
Basic operations
View detail: Click to support redirection to the workflow running detail page.
Termination: Only workflows in "Waiting" or "running" status support operation.
Rerun:
Only workflows in "success" or "failure" status support operation.
If the task dependency in the workflow changes (such as adding, deleting nodes or modifying existing node dependencies), the re-run operation is not supported.
DLC Spark Streaming tasks do not support rerun. If you select to rerun the entire workflow, other tasks except DLC Spark Streaming are rerun by default.
Workflow run details page: On the workflow run details page, you can view the operating status of task nodes in the workflow run based on graph mode, list mode, or Gantt Chart mode.
Graph mode: Click nodes in the DAG graph to navigate to the task execution details page.
List mode: Click the task name in list mode to redirect to the task execution details page.
Gantt Chart mode: Click the task name or chart in Gantt Chart mode to redirect to the task execution details page.
Task execution details page: Support viewing task execution details, including running time, logs, code, etc.
Task running status.
Waiting for upstream: Instance generation exists and upstream has unfinished tasks.
Pending execution: Upstream passed and waiting for scheduling resource.
Running: The task instance is running.
Retry on failure: Failed to configure the task and retry, the task is in wait for retry on failure status.
Failure: The task running failed or was terminated.
Successful: The task ran successfully.
Skip running: Skip running if failing to satisfy run condition or branch condition.
Upstream failure: The current node is in upstream failed state if at least one failed task is running directly or indirectly upstream.
Alarm Monitoring
Support configuring workflow and task alert to monitor task execution status and duration. Alarm rule configuration and alarm information feature are basically the same as those in task scheduling project.
Workflow scheduling mode project supports the following alarm conditions
Monitoring Object | Alarm Condition | Remarks |
Task | Task execution failed | Execution failure no longer distinguishes between scheduling trigger and manual triggering. |
Task | Task running time | - |
Task | Task completion time. | - |
Task | Task waiting total time | - |
Task | Task execution success | - |
Workflow | Workflow execution trigger (cycle) | Indicates that triggering workflow periodic execution sends alarm |
Workflow | Workflow execution failed | Indicates that workflow execution failure triggers alarm, including period and manual triggering scenarios. |
Workflow | Workflow execution success | Indicates that workflow execution success triggers alarm, including period and manual triggering scenarios. |
Workflow | Workflow completion time (cycle) | Indicates that workflow execution exceeding the set threshold triggers alarm, taking effect only in periodic task scenarios. |
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No
Feedback