tencent cloud

Resource Functions
Download PDF
Last updated: 2025-12-03 18:30:47
Resource Functions
Last updated: 2025-12-03 18:30:47
Download PDF

Function res_local

Function Definition

Use the res_local function to pull environment variables from advanced configuration in the current data processing task.

Syntax Description

res_local(param, default=None, type="auto")

Parameter Description

Parameter Name
Parameter Description
Parameter Type
Required
Parameter Default Value
Parameter Value Range
param
Field name corresponding to the environment variable in advanced configuration
string
Yes
-
-
default
If the field value does not exist, return the value of this parameter. Default value: None.
string
Yes
None
-
type
Data format for data output.
auto (default): Convert the original value to JSON format. If conversion fails, return the original value.
JSON: Convert the original value to JSON format. If conversion fails, return the parameter value of default.
raw: Return the original value.
string
Yes
auto
-

Example

In the advanced configuration of data processing, add an environment variable field time_session with a value of 30.
Original logs:
{}
Processing rules:
fields_set("time_session", res_local("time_session"))
Processing result:
{"time_session":"30"}

Function res_rds_mysql

Function Definition

Use the res_rds_mysql function to pull data from the cloud MySQL database or SQL execution results. The first time it will pull full data by default. The function can be used with t_table_map() to associate dimension tables.

Prerequisites

MySQL side: If the access account has restricted the host IP address, allow 169.254.%. For details, see MySQL-Create Account.
CLS data processing side: When creating a new processing task, toggle on the external data source switch and configure your MySQL access information.

Syntax Description

res_rds_mysql(alias, database="database name", sql="select name from person_info", refresh_interval=0, base_retry_back_off=1, max_retry_back_off=60, update_time_key=None, use_ssl=False)

Parameter Description

Parameter Name
Parameter Description
Parameter Type
Required
Parameter Default Value
Parameter Value Range
alias
Configured database information alias
string
Yes
-
-
database
Database name.
string
Yes
-
-
sql
SQL statement to retrieve data
string
Yes
-
-
refresh_interval
Fetch interval (unit: second). Default value is 0, which means only fetch once.
number
No
0
-
base_retry_back_off
Failed to pull data, re-pull time interval. Default value is 1, unit: second.
number
No
0
-
max_retry_back_off
The maximum time interval to retry request after failed to pull data. Default value is 60, unit: second, recommended to use the default value.
number
No
60
-
update_time_key
Used for incremental data retrieval. If this parameter is not configured, perform a full update.
string
No
-
-
use_ssl
Whether to use SSL protocol for secure connection
bool
No
False
-

Example

Pulling Data in the Table

Create a new table test in MySQL and add the following data:

Original logs:
[
    {
        "user_id": 1
    },
    {
        "user_id": 3
    }
]
Processing rules:
//On the console, configure the alias of external data MySQL as hm, set the db of mysql to test222, and the table name to test
//Pull all data from MySQL and use the t_table_map function to associate dimension tables

t_table_map(
    res_rds_mysql(alias="hm",database="test222",sql="select * from test"),
    "user_id",
    ["gameid", "game"]
)
Processing result:
[
{
"user_id":"1"
},
{
"game":"wangzhe",
"gameid":"123",
"user_id":"3"
}
]

Sample 2: Pulling Partial Data

Create a new table test in MySQL and add the following data:
id
game_id

game_name

region

game_details
1
10001
Honor of Kings
CN
MOBA
2
10002
League of Legends
NA
PC MOBA
3
10003
Genshin Impact
CN
RPG
4
10004
Black Myth: Wukong
CN
PC Game
5
10005
Diablo
NA
Role play
Original logs:
[
    {
        "id": 1
    },
    {
        "id": 2
    },
    {
        "id": 3
    }
]
Processing rules:
//On the console, configure the alias of external data MySQL as hm, set the db of mysql to test222, and the table name to test
//select * from test where region='CN', pull data with region='CN' from MySQL, use t_table_map function to associate dimension table

t_table_map(
    res_rds_mysql(alias="hm",database="test222",sql="select * from test where region='CN'"),
    "id",
    ["game", "game_details"]
)
Processing result:
[
{
"game_details":"MOBA mobile game"
"game_name":"Honor of Kings"
"id":"1"
},
{
"id":"2"
},
{
"game_details":"open world RPG game"
"game_name":"Genshin Impact"
"id":"3"
}
]

Joining MySQL

Two Data Pull Methods

Supports two data pull methods: one-time pull and incremental retrieval.
One-time pull: Pull all data from the MySQL data source at once. Suitable for scenarios with fixed data volume and no need for follow-up incremental updates, such as static configuration tables and archived data.
Incremental data pull: Requires the user to specify the update_time_key field in the MySQL table (this field must record the timestamp for marking data generation/update time). The data processing task will automatically use the "maximum MySQL data timestamp" from the last pull as the base, only pulling newly-added or updated data with timestamps equal to or greater than this value. Suitable for scenarios requiring continuous sync of MySQL incremental data, such as real-time business tables and dynamic log associated tables.

Data Association Core Logic

Regardless of the pull method, CLS first caches the retrieved MySQL data and then performs Join computation based on the cached data and log data to ensure stability and efficiency in the association process.

Performance Description and Scaling Recommendation

To ensure processing efficiency, pay attention to the following content:
Data volume restriction: Keep the total data volume for Join computation within 1,000,000 entries. Exceeding this limit may cause Join computation delay and impact overall task performance.
Scale-out support: If your MySQL data source has a large volume (requiring long-term stable association of over 1 million entries), you can contact us through Tencent Cloud sales to request resource scale-out to match business data association needs.
If the MySQL database contains large volumes of data, do not directly associate the primary database in the production environment with CLS. Avoid triggering overly complex SQL queries through CLS, as this may impact the stability of the production environment. Recommend using a read-only instance.

Fees

MySQL data amount is measured and billed as amount of data processed.

Data Security

It is recommended to create a specialized database account for CLS to access TencentDB for MySQL, minimizing the corresponding resource and operation permissions. CLS only requires permission to query, with no need for edit or delete permission. For the setting method, please refer to TencentDB for MySQL Account Permissions Modification. Keep the account information safe and do not leak it.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback