Product Design Philosophy
WeData is built on the product design principles of MLOps and includes a data science module.
MLOps Principles and Value
MLOps (Machine Learning Operations) is a set of engineering methods that connect the AI team with Business and Ops teams. It establishes a standardized, automated, and continuously improved management system for the full lifecycle of Machine Learning models, enabling organizations to stably, reliably, and efficiently produce high-quality models at scale for business empowerment. The core approach is to achieve cost reduction and efficiency enhancement in large-scale AI development by addressing the following issues:
Model lifecycle lacks unified management
Code assets, data assets, algorithm assets and model assets lack uniform version management and trace ability;
Businesses lack appropriate standards in the ML production to application process;
Long model development and deployment iteration cycle
Algorithmia 2020: 64% of businesses take over one month to deploy a new model, with 18% of companies needing more than 90 days to go live;
Model service is not sustainable
Model iteration and deployment speed cannot keep up with rapid business requirement changes.
From the moment it goes live, the model risks degradation (data drift, effect drift).
Degree of automation is low
Many manual processes, less efficient, high labor cost.
Lack of comprehensive monitoring and alerting mechanism, unable to detect errors prior to occurrence of damage and correct in time.
Cross-team collaboration is difficult
Different tools and workflows across teams.
Silo effect and communication gap between business team, Ops team and AI team are insurmountable.
High potential risk
Technical risk: unstable model performance, vulnerable infrastructure;
Compliance risk: violates government regulations and company policy.
Our Insight and Advantage
Building data science capabilities on a data platform like WeData inherently provides powerful Data Integration, Data Development, and Data Governance abilities, naturally addressing the fragmented nature of traditional data platforms and AI development platforms.
Data development and AI development separation
Big data and AI are two independent systems, making it difficult to implement end-to-end processes such as sample cleaning, storage, analysis, training, and reasoning.
High storage&computing costs
Data needs to flow between two systems.
Big data and AI cannot share CPU and GPU computing resources.
Our Core Philosophy
1. Always advance AI R&D projects with business objectives as traction.
2. AI R&D can be achieved by focusing on data.
3. Drive the full lifecycle via a modular platform, such as data exploration, feature engineering, model training, and online service.
4. Use automated processes to implement continuous training, continuous integration, and continuous delivery.
Feature Overview
The WeData data science module includes four core function modules: Experiment Management, Feature Management, Model Management, and Model Service. It closely collaborates with associated products like Studio, Workflow, Data Quality, and Engine, thereby achieving MLOps capability implementation and realizing end-to-end capability across the whole lifecycle of "Data-Model-Inference."
Core Modules
Module | Core Features |
Experiment Management | Enable the MLflow service in Studio, you can call MLflow-related functions in the experiment to record parameters, metrics, and results for each experiment, then view them in Experiment Management, thereby achieving traceability and reproducibility. And provided AutoML to support no-code development. |
Feature Management | Use the feature processing API provided by WeData in Studio to create, write, read, search, sync, and consume feature tables. You can also view and manage features in Feature Management to implement unified management and consumption of features. |
Model Management | Enable the MLflow service in Studio. You can call MLflow-related functions in the experiment to register a model or perform visual model registration in Experiment Management. It supports viewing key information of the model as well as its association with experiments/runs and services. |
Model Service | Support creating API service from models in Model Management, performing service monitoring and other features, and viewing the relationship between models for easy information tracing. |
Peripheral Modules
Module | Core Features |
Studio | The main workspace for AI development is Studio, where users can edit, debug, and run code, and call the MLflow and feature-management-toolkit to perform CRUD operations on feature tables, model training, and model registration. |
Workflow | The main workspace for automated processes is Studio, where users can debug code and submit it to workflow settings for periodic scheduling to achieve automated and recurring model production. |
Data Quality | Model service inference tables, feature tables, and training data tables can trigger data quality tasks to view quality information such as field analysis, drift analysis, and model metrics. |
Engine | Data science integrates with DLC and EMR as data sources, offline feature storage, and training resources for AI development. |