TBase Offline Single Table Read Node Configuration
Parameters
Description
Data Source
Available TBase data source.
Database
Supports selecting or manually entering the database name to be read
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the Data Integration network is connected.
Schema
Supports selection or manual input of the Schema name to be read.
Table
Supports selection or manual input of the table name to be read. Supports reading multiple tables simultaneously. When configured for multiple tables, you need to ensure the schema structure of these tables is consistent.
Split Key
When extracting data from TBase, if you specify splitPk, it means you want to partition the data by the field represented by splitPk. Data synchronization thus will start concurrent tasks to improve efficiency.
It is recommended to use the table primary key as splitPk because it is usually evenly distributed, thus the resulting partitions are less likely to have data hotspots.
Currently, splitPk only supports partitioning integer data, not strings, floating points, or dates. If you specify an unsupported type, the splitPk feature will be ignored, and single-channel synchronization will be used.
If the splitPk value is set to empty, it is considered that partitioning for a single table is not allowed, so single-channel extraction will be used.
Filter Conditios(Optional)
Fill in the corresponding filter statement based on the data type. This statement will serve as the filter condition for the data to be synchronized.
TBase generates SQL based on the specified where condition and extracts data according to this SQL. For example, during testing, you can specify the where condition as limit 10. In actual business scenarios, you usually choose to synchronize the day's data, specifying the where condition as gmt_create > $bizdate.
The where condition can effectively perform incremental business synchronization.
If the where condition is empty, it is considered that the entire table's information will be synchronized.
TBase Offline Single Table Write Node Configuration
Parameters
Description
Data Destination
The TBase data source to write to.
Database
Supports selection or manual input of the database name to write to
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the Data Integration network is connected.
Schema
Supports selection or manual input of the Schema name to be read.
Table
Supports selection or manual input of the table name to write to
If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the Data Integration network is connected.
Whether to Clear Table
Before writing to the TBase data table, you can manually choose whether to clear the data table.
Write Mode
TBase supports two write modes:
append: Append write, an error occurs when there is a primary key conflict.
upsert: Update write, data is updated when there is a primary key conflict.
Batch Submission Size
The number of records submitted in one batch can greatly reduce the number of network interactions between the data synchronization system and TBase, and enhance the overall throughput. If the value is set too high, it may cause the data synchronization running process to encounter OOM exceptions.
Pre-Executed SQL (Optional)
The SQL statement executed before the synchronization task. Fill in the correct SQL syntax according to the data source type, such as clearing the old data in the table before execution (truncate table tablename).
Post-Executed SQL(Optional)
The SQL statement executed after the synchronization task. Fill in the correct SQL syntax according to the data source type, such as adding a timestamp (alter table tablename add colname timestamp DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP).
Data type conversion support
The supported data types and conversion mapping for TBase writing are as follows (when handling TBase, the data types of TBase data sources will be mapped to those of the data processing engine first):
