tencent cloud

Guide on Publishing Online Services Using Custom Images
Download PDF
Last updated: 2025-05-09 17:09:16
Guide on Publishing Online Services Using Custom Images
Last updated: 2025-05-09 17:09:16
Download PDF

Overview

This document introduces two methods for creating TI-ONE custom images, as well as the specification constraints that need to be followed. Through typical cases, it demonstrates how to create custom images and publish online services.

Creating Custom Images Based on tiinfer Framework Basic Images

tiinfer Framework Basic Image Description

The platform provides a basic inference image with the built-in tiinfer framework.
ccr.ccs.tencentyun.com/tione-public-images/ti-cloud-gpu-base-tiinfer:py38-cu111-1.0.0
The basic image is produced based on CentOS and contains the following software packages:
Software or Package
Version
CUDA
11.1.1
python
3.9.13
cos-python-sdk-v5
1.9.14
coscmd
1.8.6.24
numpy
1.23.1
msgpack
1.0.5
opencv-python
4.6.0.66
opencv-contrib-python
4.6.0.66
pandas
1.4.3
Pillow
9.4.0
tiinfer
0.1.1
mosec-tiinfer
0.0.6
Startup command of the basic image: /usr/local/service/ti-cloud-infer/entrypoint.sh
The content of entrypoint.sh is:
#!/bin/bash
source /etc/profile
source /root/.bashrc
export LD_LIBRARY_PATH=/usr/local/python3/lib/python3.8/site-packages/torch/lib:/usr/local/openmpi/lib:/usr/local/nccl/lib:/usr/local/cuda/lib64:/usr/local/python3/lib:/usr/local/python3/lib64:/usr/local/openmpi/lib:/usr/local/gcc/lib:/usr/local/gcc/lib64


MODEL_DIR=/data/model

echo "================== code path ${MODEL_DIR}=========="
cd ${MODEL_DIR}

if [ -f "requirements.txt" ]; then
  echo "============== install python requirements ======================"
  echo "python3 -m pip install -r requirements.txt"
  python3 -m pip install -r requirements.txt
  echo "============== install python requirements done ================="
fi

echo "====================== start serving ============================"
echo "python3 -m tiinfer"
export TI_MODEL_DIR=${MODEL_DIR}
python3 -m tiinfer --timeout 30000
The startup logic is:
1.1 Read the requirements.txt file under the environment variable ${MODEL_DIR} directory, and use pip to install the specified dependency Python package.
1.2 The tiinfer framework will read the files under the ${MODEL_DIR} environment variable. After loading the model, it will start an HTTP service and listen on the port defined by the environment variable ${REST_PORT} .
1.3 When the tiinfer framework is started, the model will be loaded from the model_service.py file.

Customizing Image Specification

1. Add a reference to the basic image in the Dockerfile file. For example:
FROM ccr.ccs.tencentyun.com/tione-public-images/ti-cloud-gpu-base-tiinfer:py38-cu111-1.0.0
2. The custom logic is implemented mainly by modifying the model_service.py file and the entrypoint.sh file.
3. When Cloud File Storage (CFS) or Cloud Object Storage (COS) is used as the model source, the platform will, by default, place the model files, including those in sub-directories, under the source path to the /data/model directory of the service instance. Therefore, custom code and data cannot be placed in the /data/model directory. Otherwise, they will be overwritten by the platform.

Creating an Image

This case study introduces a simple adder implemented based on the basic image of the tiinfer framework, by changing the model_service.py and entrypoint.sh files.
Note: In this case, instead of using the model repository feature provided by the platform to manage the model, the model and inference code are directly encapsulated into the image. Therefore, it is necessary to avoid placing the model and code in the /data/model directory.

Writing Code

There are a total of 3 files:
File
Function
model_service.py
Write an adder model as required by tiinfer.
entrypoint.sh
Launch scripts. You can install more dependency packages.
Dockerfile
Copy the first two files to the image.
1. Content of model_service.py:
from typing import Dict
import tiinfer

class AdderModel(tiinfer.Model):
 def __init__(self, model_dir: str):
     super().__init__(model_dir)

 def load(self) -> bool:
     self.ready = True
     return self.ready

 def preprocess(self, request: Dict) -> Dict:
     return request

 def predict(self, request: Dict) -> Dict:
     return {'result': request['a'] + request['b']}

 def postprocess(self, result: Dict) -> Dict:
     return result
2. Content of entrypoint.sh
#!/bin/bash
source /etc/profile
source /root/.bashrc
export LD_LIBRARY_PATH=/usr/local/python3/lib/python3.8/site-packages/torch/lib:/usr/local/openmpi/lib:/usr/local/nccl/lib:/usr/local/cuda/lib64:/usr/local/python3/lib:/usr/local/python3/lib64:/usr/local/openmpi/lib:/usr/local/gcc/lib:/usr/local/gcc/lib64

MODEL_DIR=/opt/model

echo "================== code path ${MODEL_DIR}=========="
cd ${MODEL_DIR}

if [ -f "requirements.txt" ]; then
  echo "============== install python requirements ======================"
  echo "python3 -m pip install -r requirements.txt"
  python3 -m pip install -r requirements.txt
  echo "============== install python requirements done ================="
fi

echo "====================== start serving ============================"
echo "python3 -m tiinfer"
export TI_MODEL_DIR=${MODEL_DIR}
python3 -m tiinfer --timeout 30000
Note: In line MODEL_DIR=/opt/model of the above code, change the startup directory from the default /data/model to /opt/model to avoid being overwritten by the platform.
3. Content of Dockerfile:
FROM ccr.ccs.tencentyun.com/tione-public-images/ti-cloud-gpu-base-tiinfer:py38-cu111-1.0.0

COPY model_service.py /opt/model/model_service.py
COPY entrypoint.sh ./entrypoint.sh
RUN chmod +x ./entrypoint.sh
Note that the above code copies model_service.py to the /opt/model directory instead of the default /data/model directory to avoid being overwritten by the platform.

Packaging Images

1. Procedure:
Configure the docker environment locally and enable the Tencent Container Registry;
Create a namespace and a personal image repository;
Compile a custom inference image and push it to the personal image repository;
When starting the model service, choose not to use the model file in the instance container column, select the runtime environment to enter the personal image repository list, and select the custom image environment pushed in the previous step;
Configure parameters and start the service.
2. Details:
Run the following command for packaging:
docker build . --tag ccr.ccs.tencentyun.com/YOUR_NAMESPACE/YOUR_IMAGENAME
After packaging is completed, check whether the service is running properly locally as follows:
Execute docker run -d --name myinfer ccr.ccs.tencentyun.com/YOUR_NAMESPACE/YOUT_IMAGENAME to run the service.
Execute docker exec -it myinfer bash to enter the container.
Execute curl http://127.0.0.1:8501/v1/models/m:predict -d '{"a": 1, "b": 2}' in the container to get a correct return result: {"result": 3}
Exit the container, return to the local environment, and upload the image: docker push ccr.ccs.tencentyun.com/YOUR_NAMESPACE/YOUR_IMAGENAME.

Creating Custom Images Based on Other Inference Frameworks

The platform allows you to use other inference frameworks to deploy model online services via custom images.

Customizing Image Specification

1. The service receives requests over the HTTP protocol and supports only the POST method.
2. When CFS or COS is used as the model source, the platform will, by default, place the model files, including those in sub-directories, under the source path to the /data/model directory of the service instance. Therefore, custom code and data cannot be placed in the /data/model directory. Otherwise, they will be overwritten by the platform.
3. The image has been verified locally and can provide services properly.

Uploading Custom Images and Publishing Inference Service

Uploading a Custom Image

1. Log in to Tencent Container Registry.
2. On the mirror repository page, click Create.
3. Upload the image.
Click the shortcut command after the image repository to view the operation commands and upload the image.

Publishing an Online Service

On the TI-ONE Online Service page, click Create Service.

Method 1: Packaging the Model for Use in Images

If you have packaged the model in an image, you can use the image to publish services directly.
Configuration details of service instance:
Model source: Select Mirror.
Models and runtime environments: Select custom.
Select model and runtime environments: Select the previously uploaded image.
Port number: Fill in the port number for providing services.


image.png






Method 2: Mounting the Model Stored in CFS to the Container for Use

If your image only serves as the runtime environment for the service, you can upload the model to CFS and then mount it to the container. The model will be mounted under the /data/model directory, and your service can load the model from this directory.
Configuration details of service instance:
Model source: Select CFS.
Select Model: Select CFS for model storage and fill in the path where the model is stored in the CFS. To upload the model to CFS, please refer to Importing and Deploying Custom Large Language Models.
Models and runtime environments: Select custom.
Select model and runtime environments: Select the previously uploaded image.
Port number: Fill in the port number for providing services.

image.png




Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback