Product Introduction
The Workflow service in Data Engineering is meticulously developed based on a Serverless architecture. By optimizing the performance and deployment parameters of the open-source Argo Workflows engine, it supports the efficient and elastic scheduling of large-scale workflows without requiring users to handle the Ops work. The service features a built-in Argo-function native plugin, allowing you to directly orchestrate and invoke Serverless Cloud Function (SCF) functions within workflows. No resources remain idle. Resources are released immediately after function execution, achieving efficient optimization of business costs.
Scenarios
As a core component in the cloud native domain, workflow orchestration is widely used in fields such as autonomous driving, scientific computing, and large language model fine-tuning. It can run compute-intensive jobs related to large-scale data processing within a short period of time, making it suitable for scenarios such as machine learning, AI inference, and data batch processing.
Strengths
Tencent Cloud SCF has built a managed workflow product based on open-source Argo Workflows. By deeply integrating standardized encapsulation with cloud-native capabilities, it provides enterprises with a flexible and secure workflow management solution. Embracing the Serverless architecture concept, it offers fully managed Ops, aiming to solve three core pain points enterprises face in workflow orchestration:
1. Infrastructure management burden: Users do not need to focus on underlying tasks such as Kubernetes cluster deployment, node maintenance, or version upgrades. The platform automatically handles the creation, scaling, and Ops monitoring of the Argo cluster.
2. Complexity of multi-scenario adaptation: The Workflow service meets the resource isolation and collaboration requirements of multiple teams and environments through regional distribution, namespace isolation, and fine-grained permission control.
3. Building upon native container orchestration, it deeply integrates with SCF. The built-in SCF invocation plugin supports directly invoking functions within workflows, enabling a "function + container" hybrid orchestration model.
Usage Process
2. Configure secrets，such as the image permanent key, COS storage secret used in YAML, database credentials, etc.; 3. Configure storage，such as COS object storage and CFS file storage used in YAML, etc.;