tencent cloud

Tencent Cloud TCHouse-P

Release Notes
Product Introduction
Overview
Use Cases
Node Specification
Cluster Architecture
Purchase Guide
Billing Overview
Purchase Methods
Payment Overdue
Getting Started
Operation Guide
Managing Cluster
Accessing Data Warehouse
Monitoring and Alarming
Access Management
Performance Metrics
TPC-B
Tools and Downloads
Data Ingestion
Importing TencentDB Data Offline with DataX
Syncing Incremental Data from MySQL with DataX
Importing and Exporting COS Data at High Speed with External Table
Syncing EMR Data with External Table
Implementing CDWPG UPSERT with Rule
Data Warehouse Development
Creating Airflow in Cloud
API Documentation
History
Introduction
API Category
Making API Requests
Information Query APIs
Instance APIs
Query APIs
Cluster Management APIs
Cluster Operation APIs
Data Types
Error Codes
Practical Tutorial
Data Warehouse Table Development
Table Distribution Key Selection
Table Storage Format Selection
Table Partition Usage
Extension Usage
Cold Data Backup
Statistics and Space Maintenance
FAQs
Tencent Cloud TCHouse-P Policy
Service Level Agreement
Privacy Policy
Data Processing And Security Agreement
Contact Us
DokumentasiTencent Cloud TCHouse-PData IngestionSyncing Incremental Data from MySQL with DataX

Syncing Incremental Data from MySQL with DataX

PDF
Mode fokus
Ukuran font
Terakhir diperbarui: 2024-11-27 15:36:05
This document describes how to use DataX modified by HashData to incrementally sync data from MySQL to Tencent Cloud TCHouse-P.
Follow the steps below to use DataX to incrementally sync data from MySQL to Tencent Cloud TCHouse-P:
1. Read the MaxTime since the last successful sync from the local file (for the initial sync, you can specify an initial time value as required by your business).
2. Use MaxTime as the LastTime (lower limit of the incremental sync) and CurTime as the upper limit.
3. Modify the datax.json configuration to specify the time interval (WHERE clause in SQL) of the synced table as [LastTime, CurTime).
4. Execute DataX sync. After successful sync, write CurTime to the local file for the next sync.
5. Repeat steps 1–4 for multiple incremental syncs.
A sample datax.json configuration file is as shown below:
{
    "job": {
        "setting": {
            "speed": {
                "channel": 3, 
                "byte": 1048576, 
                "record": 1000
            }, 
            "errorLimit": {
                "record": 2, 
                "percentage": 0.02
            }
        }, 
        "content": [
            {
                "reader": {
                    "name": "mysqlreader", 
                    "parameter": {
                        "username": "******", 
                        "password": "******", 
                        "connection": [
                            {
                                "jdbcUrl": [
                                    "jdbc:mysql://***:***/test?serverTimezone=Asia/Shanghai"
                                ],
                                "querySql": [
                                    "select * from cdw_test_table where updateTime >= '${lastTime}' and updateTime < '${currentTime}'"
                                ]
                            }
                        ]
                    }
                }, 
                "writer": {
                    "name": "gpdbwriter", 
                    "parameter": {
                        "username": "******", 
                        "password": "******", 
                        "column": [
                                 "*"
                        ],
                        "segment_reject_limit": 0, 
                        "copy_queue_size": 2000, 
                        "num_copy_processor": 1, 
                        "num_copy_writer": 1, 
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:postgresql://***:***/***", 
                                "table": [
                                    "ods_cdw_test_table"
                                ]
                            }
                        ]
                    }
                }
            }
        ]
    }
}

Topik sebelumnya: Importing TencentDB Data Offline with DataXTopik selanjutnya: Importing and Exporting COS Data at High Speed with External Table

Bantuan dan Dukungan

Apakah halaman ini membantu?

masukan