Cloud Log Service (CLS) supports remote non-intrusive subscription to MySQL binary logs (binlogs). This log records all operations that modify the MySQL database in the event form. It is suitable for scenarios requiring real-time query, analysis, and audit of incremental database changes. This document introduces how to subscribe to MySQL binlogs.
Prerequisites
A MySQL database has been created with the binlog feature enabled in ROW mode.
A MySQL account with the REPLICATION SLAVE, REPLICATION CLIENT, and SELECT permissions has been created.
2. In the left sidebar, click Log Topic. Then, select the log topic you want to ship and click its name to go to the log topic details page.
3. Select the Collection Configuration tab, and click Add in the MySQL Binlog Subscription Configuration section to go to the subscription task configuration process page.
Step 2: Configure MySQL
Configure the following parameters in the Configure MySQL step:
Parameter
Required
Description
MySQL Type
Yes
MySQL type to which you want to subscribe. Supported types include:
If the MySQL type is TencentDB for MySQL or TDSQL-C for MySQL, you can select the target TencentDB for MySQL instance from the drop-down options.
Note:
Currently, you can only select a TencentDB for MySQL instance in the same region as the log topic. To subscribe to binlogs from a MySQL instance in a different region, set MySQL Type to Self-built MySQL and subscribe to binlogs through the public network.
Access mode
Yes
If the MySQL type is set to Self-built MySQL, you can choose to access your MySQL through the private network address or public network address.
Network
Yes
If the MySQL type is set to Self-built MySQL and the private network address is used as the access method, you need to specify the VPC network of the MySQL instance.
Note:
The VPC network should be in the same region as the log topic.
Network service type
Yes
If the MySQL type is set to Self-built MySQL and the private network address is used as the access method, you need to specify the network service type of the target MySQL:
If your MySQL needs to be accessed through Cloud Load Balancer (CLB), select CLB.
If your MySQL server allows direct access, select CVM.
Private/Public Network Access Address
Yes
If the MySQL type is set to Self-Built MySQL, specify the private network access address or public network access address of MySQL based on the selected access method.
MySQL Port
Yes
If the MySQL type is set to Self-Built MySQL, specify the MySQL port.
Username
Yes
Specify the username for accessing MySQL.
Note:
The following permissions need to be enabled for the account: REPLICATION SLAVE, REPLICATION CLIENT, and SELECT.
Password
Yes
Specify the password for accessing MySQL.
Step 3: Configuring Subscription Rules
In the Configure the subscription rule step, configure the following parameters:
Parameter
Required
Description
Subscription Rule Name
Yes
Specify the name of the current subscription rule.
Start position
Yes
Specify the starting position for binlog subscription. The starting position can be defined using one of the following 3 methods:
Latest position: collects binlogs from the latest position.
Specified position: starts collecting binlogs from the specified position.
Specified GTID: starts collecting binlogs from the specified GTID (transaction ID) position.
Binlog File Name
Yes
If Starting Position is set to the specified position, you need to specify the binlog file name, such as mysql-bin.000005.
Starting Binlog Position
Yes
If Starting Position is set to the specified position, you need to specify the position in the binlog file to start collecting data. For example, to collect the starting index of the Nth log entry in mysql-bin.000005, specify the position as the position+size of the (N-1)th log entry (where position is the starting index of the binlog entry in the binlog file, and size is the size of the binlog entry). To collect data from the beginning of the file, specify the position as 0.
Note:
Ensure that the position exactly matches the starting index of a binlog entry. Otherwise, the subscription may fail.
Starting GTID
Yes
If Starting Position is set to thespecified GTID, you need to specify the starting GTID (transaction ID).
Event Type
Yes
Type of the binlog event that needs to be collected. The following 4 event types are supported:
DDL: data definition language
Insert: insertion
Update: update
Delete: deletion
Event Metadata
No
The following 6 types of binlog-related metadata are supported for selection to be uploaded along with the logs:
log_name: binlog file name
position: starting index of the binlog entry in the binlog file
size: binlog size
server_id: secondary server ID
gtid: transaction ID
task_id: current subscription task ID
Note:
Binlogs for DDL events do not support size metadata.
Timestamp
Yes
The following 2 types of time can be selected as the timestamp of the collected binlog:
Collection time: time when the binlog is collected.
Event time: time when the event corresponding to the binlog occurs.
Database Table Allowlist
No
When this option is enabled, the binlogs of the specified database tables are collected. When it is disabled, the binlogs of all databases and tables are collected. Multiple allowlists can be configured, each requiring the following information:
Database name: name of the database associated with the binlog to be collected.
Database table: name of the database table associated with the binlog to be collected. Multiple names can be specified.
Note:
DDL event binlog collection is not subject to allowlist restrictions.
Database Table Blocklist
No
When this option is enabled, the binlogs of specified databases and tables can be ignored and are not collected. Multiple blocklists can be configured, each requiring the following information:
Database name: name of the database associated with the binlog to be ignored.
Database table: name of the database table associated with the binlog to be ignored. Multiple names can be specified.
Note:
DDL event binlog collection is not subject to blocklist restrictions.
Flatten Event Data
No
If this option is enabled, different events will be split into multiple logs, and operated fields will be tiled in each log. For example, if two database operations are performed, one update to fields a and b, and one update to fields d and e, two logs will be generated with values a:xxx, old_a:xxx, b:xxx, old_b:xxx and d:xxx, old_d:xxx, e:xxx, old_e:xxx respectively.
If this option is disabled, event data will be centrally packaged into two fields, old_data and data, in array+JSON format. For example, if two database operations are performed, one update to fields a and b, and one update to fields d and e, the log value will be old_data:[{a:xxx,b:xxx}, {d:xxx,e:xxx}], data:[{a:xxx,b:xxx}, {d:xxx,e:xxx}].
Preview Binlog
No
Click to obtain the first binlog entry that matches the subscription type from the database and display its content.
Step 4: Index
1. Click Next to go to the Index Configuration page.
2. On the Index Configuration page, configure the following information. For configuration details, see Configuring Indexes.
Note:
Index configuration must be enabled before you can perform searches.
Step 5: Search and Analysis
You have completed the MySQL binlog subscription configuration. Next, you can start using log search and analysis. For details, see Search and Analysis.
Log Field Description
Fixed Fields
Field
Description
db
Database name.
table
Table name.
query
query
type
Event Type
after
When type is set to insert, update, or delete, this field contains the affected fields and result values, wrapped in a JSON format.
before
When type is set to update, this field contains the affected fields and their values before the update, wrapped in a JSON format.
errorCode
When type is set to DDL, this field contains the error code from the execution of the DDL statement.
executionTime
When type is set to DDL, this field contains the time taken to execute the DDL statement.
Metadata Fields
Note:
These fields are uploaded only after being specified in the binlog subscription rules.
Field
Description
log_name
Binlog File Name
position
Starting index of the binlog entry in the binlog file.
size
Binlog size.
server_id
Slave ID
gtid
Transaction ID.
task_id
Subscription rule task ID.
Other Operations
Managing Subscription Tasks
After completing the MySQL binlog subscription task configuration, you can view and manage all created MySQL binlog subscription tasks in the Collection Configuration section of the log topic details page.
