SQL Server Offline Single Table Read Node Configuration
Parameters
Description
Data Source
Available SQL Server Data Sources.
Database
Supports selecting or manually entering the database name to be read
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the Data Integration network is connected.
Schema
Supports selection or manual input of the Schema name to be read.
Table
Supports selection or manual input of the table name to be read. Only one table can be synchronized per task.
Split Key
Specify the field for data sharding. After specifying, concurrent tasks will be launched for data synchronization. You can use a column in the source data table as the partition key. It is recommended to use the primary key or indexed column as the partition key.
Filter Conditions (Optional)
Fill in the corresponding filter statement based on the data type. This statement will serve as the filter condition for the data to be synchronized.
SQL Server concatenates SQL based on the specified where condition and extracts data accordingly. For example, during testing, the where condition can be specified as limit 10. In actual business scenarios, the data for the current day is usually selected for synchronization, with the where condition specified as gmt_create > $bizdate.
The where condition can effectively perform incremental business synchronization.
If the where condition is empty, it is considered that the entire table's information will be synchronized.
SQL Server Offline Single Table Write Node Configuration
Parameters
Description
Data Destination
SQL Server Data Source to be written to.
Database
Supports selection or manual input of the database name to write to
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the Data Integration network is connected.
Schema
Supports selection or manual input of the Schema name to be read.
Table
Supports selection or manual input of the table name to write to
If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the Data Integration network is connected.
Whether to Clear Table
Before writing to the SQL Server data table, you can manually choose whether to clear the data table.
Write Mode
SQL Server supports two write modes:
append: Append write.
upsert: Data update and write based on the primary key field.
Batch Submission Size
The size of the record batch for one-time submission can greatly reduce the number of network interactions between the data synchronization system and SQL Server, enhancing overall throughput. If the value is set too high, it may cause the data synchronization process to experience OOM exceptions.
Pre-Executed SQL (Optional)
The SQL statement executed before the synchronization task. Fill in the correct SQL syntax according to the data source type, such as clearing the old data in the table before execution (truncate table tablename).
Post-Executed SQL (Optional)
The SQL statement executed after the synchronization task. Fill in the correct SQL syntax according to the data source type, such as adding a timestamp (alter table tablename add colname timestamp DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP).
Data type conversion support
Read
Supported SQL Server Data Types and Conversion Relationships (when processing SQL Server, SQL Server data source types are mapped to data processing engine types):
SQL Server Data Types
Internal Types
bigint, int, smallint, tinyint
Long
float, decimal, real, numeric
Double
char,nchar,ntext,nvarchar,text,varchar,sysname
String
date, datetime, time,timestamp,datetimeoffset
Date
bit
Boolean
binary,varbinary
Bytes
Write
Supported SQL Server Write Data Types and Conversion Relationships:
