Scenarios
You need to add large model services to the AI Gateway so that the gateway can proxy requests to the corresponding model providers, enabling unified access, routing, degradation, and key management. The AI Gateway supports adding model services from providers such as Hunyuan, Google Gemini, DeepSeek, Qwen, OpenAI, and so on. This document describes how to add, edit, and delete model services for the AI Gateway.
Operation Steps
Add Model Service
2. On the instance list page, click the "ID" of the gateway instance to be configured to go to its basic information page.
3. In the left sidebar, click Model Management, then click the Model Services tab, and in the service list, click New.
4. In the "New Model Service" window, complete the configuration for the first step, "Basic Information".
|
Service Name | Yes | Enter a service name. The name can contain up to 60 characters, including uppercase and lowercase letters in Chinese and English, digits, and separators ("-", "_"). It cannot start with a digit or a separator, and cannot end with a separator. |
Service type | Yes | Fixed to "AI model service". |
Model provider | Yes | Select a model vendor. Supported vendors include Hunyuan, Google-Gemini, DeepSeek, Qwen, and OpenAI. |
Model protocol | Yes | Select the model protocol you need to use based on the protocols supported by the model vendor. |
Service Address | Yes | Confirm the service address of the model service. |
Model key | Yes | Select a pre-configured API key for this vendor, or click "New Key" to navigate to the key management page and add one. The gateway will use this key to call the corresponding model API. |
Secret usage policy | No | Defines how the keys are used when multiple keys are configured. The default is round-robin, which can balance the load across multiple keys. |
Description | No | Description of this service for subsequent management. |
Note:
The big model capabilities provided by the AI model service are offered by third parties, not directly by the AI gateway. Users shall independently evaluate the service applicability and reliability, ensuring their usage complies with relevant laws and agreements. We shall not be held liable for any consequences arising from violations of regulations.
5. After completing the basic information, click Next to go to the "Select Model Policy" step.
Model Selection Method: This configuration determines how the gateway handles the model (model) parameter in client requests.
Passthrough Request Model
The gateway will ignore the model parameter in client requests and consistently use the model specified in the "Default Model" section below. This mode is suitable for cost control and high availability scenarios, facilitating unified routing and degradation handling.
Default Model: When the "Model Selection Method" is "Specified Model", you must select a specific model name here.
Model Fallback: When it is enabled, if a request to the "Default Model" fails, the gateway can automatically switch (Fallback) to other available models according to predefined rules, ensuring service high availability.
Fallback Rules: After enabling Fallback, you need to select or configure the list of fallback models and switching rules here when the primary model is unavailable.
The gateway will directly use the model parameter from client requests and forward it to the vendor. This mode is suitable for scenarios requiring flexible client-side control over model selection, but ensure that clients pass the correct model name.
Model Parameter Validation: When it is enabled, the gateway will validate whether the model parameter in client requests is within the allowed list.
Allowed Model List: defines the allowlist of model names that the client is allowed to request.
Handling Policy for Validation Failure: defines the policy for handling model validation failures, supporting "return a 404 error" or "fall back to the default model and degrade the service".
6. After the configuration is completed, click Confirm to create the model service.
7. After adding, the newly added service will appear in the service list. Click Service ID/Name to view detailed service information.
Edit Service
On the Model Service list page, locate the target service, click Edit under its operation column to modify the service configuration information, and after modification, click Confirm to save.
Deleting a Service
On the Model Service list page, locate the target service, click Delete under its operation column, and a dependency check will be performed before deletion.
1. The system will pop up a dialog box for you to confirm the deletion and automatically check whether the service is bound to other resources (such as "Model API").
2. Verify results:
If there are no dependencies: the pop-up window will directly display the service ID and name, and click Confirm to delete.
If there are dependencies: the pop-up window will display "Resource Deletion Dependency Check Results" below the service information, and prompt "There are unresolved dependencies", while listing specific dependencies.
3. If there are dependencies, you need to first remove all listed dependencies. After removing the dependencies, click the Recheck operation in the pop-up window, and the system will perform the validation again. When the validation passes and the dependency prompt disappears, click Confirm to finally delete the service. To cancel the deletion, click Cancel.