Dependency Package Management
Last updated: 2025-10-24 09:21:31
Core Concepts
Hierarchical Model
To ensure consistency and controllability of the dependency environment, above the “Standard Engine” layer, DLC maintains a protected set of dependencies preinstalled by the engine kernel and standard images, referred to as the Engine Kernel Baseline Dependencies (hereinafter, “Baseline Dependencies”). This collection serves as the starting point for all dependency resolution and merging. For details on baseline dependencies of different images, see Runtime Environment.
Engine Kernel Baseline Dependency (Baseline Dependency)
Description: Preset dependency list delivered with the engine kernel version and standard image.
Features: Non-removable; Cannot be directly overwritten via PyPI. Version adjustment requires controlled "version pinning/rewrite" at the engine level through requirements.txt.
Role: Acts as the global baseline's upper limit to underwrite platform operation stability and consistency.
Standard engine
Role: Performs limited dependency supplementation and controlled rewrite on top of "baseline dependency" to form a tenant engine-level global dependency baseline.
Impact range: all resource groups, jobs, and Notebooks inherited by default.
Resource group
Resource group: SQL analysis resource group, machine learning resource group.
Role: isolation layer for teams/business lines to handle common but non-global dependency configuration.
Notebooks and jobs
Role: personalized and experimental minimum incremental dependency, close to development and debugging scenarios.
Inheritance and Custom Rules
Inheritance relationship: Child resources inherit the parent's dependency configuration by default.
such as: resource groups inherit standard engine dependencies by default.
Customize additions: Dependencies can be added at any level; newly-added dependencies only take effect at that level and its child levels.
such as: Python packages added to the "machine learning resource group" are only applicable to jobs/Notebooks within that resource group.
The final dependent environment of a task instance = baseline dependency + engine dependency + resource group/job dependency + Notebook dependency
Installation Sequence
Installation Sequence: Newly-added dependencies at the engine/resource group/job level are installed in ascending order of Addition Time.
Conflict Rule
Conflict semantic alignment native ecosystem:
Maven (Jar): Follow Maven resolution and conflict processing semantics.
PyPI (Python): Follow pip/PEP standards.
For details, see dependency conflict resolution.
Overwrite and Rewrite
Cannot directly rewrite: The built-in engine dependency cannot be rewritten via the PyPI method.
Controllable rewrite: Engine-level support through
requirements.txt to perform version rewrite for built-in dependencies.Effective Timing and Installation Status
Effective actions: Install, Uninstall, and Clone take effect during instance startup.
Trigger scenario:
Start job
Restart SQL resource group
Machine learning resource group:
ML open-source framework and Python dependency: Click
Restart on the Wedata Notebook Exploration webpage.Spark MLlib dependency: Recreate a Spark session on the Wedata Notebook Exploration webpage.
View installation status:
Engine level: Display the status of any compute instance after the last installation (for example, if only the SQL resource group and job A run sequentially, show the corresponding status of job A).
Resource group/job/Notebook level: Display installation status by respective dimension.
Support click dependency entry to view logs for successful installation or failure.
Prerequisites
Note:
The dependency package feature is an allowlist feature. To use this feature, submit a ticket to contact after-sales for enablement.
Purchase standard engine Spark.
To use existing engines, submit a ticket to contact after-sales for upgrading engine image and gateway image to version 2025-09-30 or later.
Operation Guide
Engine-Level Dependency Management
Navigation path: Resource management → Standard engine → Select engine → Dependency package management
Installing
COS / Local File
Support Jar and Python packages; upload from COS bucket or local.
PyPI
Package name and version (compliant with PyPI naming); default repository is Tencent Cloud, supports external source.
Maven
Fill in Maven coordinates; by default uses Tencent Cloud Repository, supports custom remote repositories and dependency exclusion.
Uninstalling
Select dependency → Click "Uninstall". Uninstalling is an asynchronous operation and will take effect when the compute instance starts next time.
Cloning
Quick clone the dependent environment from another engine; duplicate dependencies are automatically skipped.
2. Resource Group Level Dependency Management
Navigation path: Resource management → Standard engine → Select engine → Resource group management → Select resource group → Dependency package management.
Install/Uninstall: Same as the engine level.
3. Job Level Dependency Management
Navigation path: Data development and exploration → Data job → Create/Edit job → Dependency resources
Description: Customize dependencies for a single job, only applicable to that job.
Installation method: Same as the engine level.
4. Task Dependency Environment Overview
Navigation path: Ops management → History task instance → Select task → Dependent Environment.
Description: Shows the final dependent collection and installation status (covering engine/resource group/job level) for rapid troubleshooting.
Resolving Dependency Conflicts
Maven(Jar)
Direct dependency conflict: Post-installation package installation failure.
Transitive dependency conflict: The system automatically removes conflicting transitive dependencies, successfully installed; viewable dependency removal record.
Class conflict: Installation successful but may conflict at runtime.
Unused conflict class: task running properly.
Use conflict class: task failure with the reason displayed in logs.
PyPI(Python)
Duplicate installation at the same level: Installation with the same name or different versions failed due to dependency failure.
Duplicate installation at different levels:
Standard engine/resource group/job level: Install by add time in sequential execution, dependency failed for post-installation.
Notebook level: Can rewrite the version of existing packages at the parent level.
Best Practices
Core Principle
Minimize engine-level dependencies
The engine level is the global baseline at the tenant engine level, impacting the entire system. Unless it is confirmed as a public dependency required for all scenarios, adding excessive dependencies at the engine level is not recommended. Reduce global coupling and cross-team compatibility costs.
Prioritize using resource groups for isolation and hierarchical evolution
Place common dependencies of teams/business lines at the resource group level to naturally isolate differentiated needs of different teams/projects and avoid global pollution.
Make minimal additions at the job/Notebook level
Place only unique, short-term verification, or small-range personalized dependencies for the job/Notebook. When multiple jobs reuse them, move them to the resource group level for unified management.
Scenario-Based Recommendations
Multiple teams share the same engine
The engine maintains a "common business base".
Teams with differentiated dependencies are placed in their respective resource groups to avoid mutual constraints.
Algorithm team quick test
Notebook stage allows quick trial installation; dependencies are fixed at the resource group level before release.
FAQ
Q: How to handle Maven transitive dependency conflicts?
A: The system automatically removes conflicting transitive dependencies and records the details. Please confirm the impact in the installation log and "remove record".
Q: Why does installation/uninstallation not take effect immediately?
A: Dependency changes take effect when the compute instance starts. Restart the resource group or recreate the job/session as per the "action to take effect".
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No
Feedback