DataInLong provides real-time writing capability for Elasticsearch. This document introduces the current support status for Elasticsearch real-time data synchronization.
Supported Editions
Currently, DataInLong supports database-level real-time writing for Elasticsearch. To use the real-time writing capability, the following version restrictions must be followed:
Type
Edition
Elasticsearch
5.x, 6.x, 7.x
Use Limits
Currently, only one real-time full database link is supported from TDSQL-C MySQL to Elasticsearch.
Elasticsearch primary key values default to the _id value, which cannot be modified.
Currently, only the _id of the document is supported as the partition column.
Elasticsearch currently uses dynamic field mapping for table creation. The DDL change strategy does not support automatic column type modification, column deletion, or renaming a column (it will be recognized as adding a new column).
Kerberos authentication is not supported for Elasticsearch.
Real-time full database writing configuration
Supported Data Sources
Currently, the following source types are supported for real-time full database synchronization to the Elasticsearch target end:
There are some differences in the configuration between Elasticsearch versions 6.x and above (inclusive) and below:
Configuration for versions 6.x and above (inclusive) is as follows:
Configuration for versions below 6.x is as follows:
Parameters
Description
Data Destination
Select the target data source to be synchronized.
ES Version
The corresponding version is automatically displayed based on the selected data source
Write Mode
Upsert: Update Write, update all fields of each record (currently only this mode is supported).
Index/Type Matching Strategy
ES index and type name matching rules:
Index versions 6.x and above (inclusive) default to the same name as the source table; type defaults to _doc and cannot be modified
Index versions below 6.x default to the same name as the source database; type defaults to the same name as the source table
Custom Definition: Supports generating target index/type names using a combination of built-in parameters and strings.
Note:
Example: If the source table name is table1, and the mapping rule is ${table_name_di_src}_inlong, the data from table1 will be finally mapped to table1_inlong.
Gradual Value Method
Currently only supports default generation of _id value
Log collection write node configuration
Parameters
Description
Data Source
Select an available Elasticsearch data source from the current project.
Index
Index name in Elasticsearch data source.
type
Automatically identify based on index; Elasticsearch version 7.X defaults type to _doc.
Write Mode
Elasticsearch only supports row-by-row updating, updating all fields of each record.
Primary key value method
Supports three value methods:
Source table primary key: document id uses the primary key of the source table.
Composite Primary Key: document id uses multiple columns of the source table to determine.
No Primary Key: defaults to generating _id value.
Enabling a Route
Does Elasticsearch enable routing partition index data? After enabling the routing feature, you can control which partition to use for storing documents in Elasticsearch.
Advanced Settings (optional)
You can configure parameters according to business needs.
Write Data Type Conversion Supported
Internal Types
JSON Types
CHAR / VARCHAR / STRING
string
BOOLEAN
boolean
BINARY / VARBINARY
string with encoding: base64
DECIMAL
number
TINYINT
number
SMALLINT
number
INT
number
BIGINT
number
FLOAT
number
DOUBLE
number
DATE
string with format: date
TIME
string with format: time
TIMESTAMP
string with format: date-time
TIMESTAMP_WITH_LOCAL_TIME_ZONE
string with format: date-time (with UTC time zone)
