- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Access Management
- Preparations
- Data Migration
- Databases Supported by Data Migration
- Cross-Account TencentDB Instance Migration
- Migrating to MySQL
- Migrating to MariaDB
- Migrating to TDSQL-C for MySQL
- Migrating to TDSQL MySQL
- Migrating to PostgreSQL
- Migrating to MongoDB
- Migrating to SQL Server
- Migrating to TencentDB for Redis
- Offline Migration of MySQL Data
- Advanced Migration Operations
- Task Management
- Data Sync
- Databases Supported by Data Sync
- Cross-Account TencentDB Instance Sync
- Synchronize to PostgreSQL
- Sync into Tencent Cloud
- Sync from Tencent Cloud
- Advanced Sync Operations
- Task Management
- Task Status Description
- Viewing Task
- Starting Task
- Retrying Task (MySQL)
- Verifying Task
- Stopping Task
- Pausing Task
- Resuming Task
- Resetting Task
- Deleting Task
- Changing Configuration
- Transition from Pay-as-You-Go to Monthly Subscription
- Terminating/Returning Instance
- Renewing Task
- Deisolating Task
- Deleting Instance
- Viewing Log
- Deleting Task
- Data Subscription (Kafka Edition)
- Databases Supported by Data Subscription
- Creating Data Subscription Task
- Data Consumption
- Creating Consumer Group
- Managing Consumer Group
- Consuming MySQL Data
- Consuming MongoDB Data
- Consuming TDSQL for PostgreSQL Data
- Consuming Subscribed Data with Kafka Client (Protobuf)
- Consuming Subscribed Data with Kafka Client (Avro)
- Consuming Subscribed Data with Kafka Client (JSON)
- Consuming Subscribed Data with Flink Client (Avro)
- Consuming Data with Flink
- Advanced Subscription Operations
- Task Management
- Data Migration (OldDTS)
- Data Subscription (OldDTS)
- Fix for Verification Failure
- Check Item Overview
- Check Items of MySQL/MariaDB/Percona/TDSQL-C for MySQL/TDSQL for MySQL
- MongoDB Check Items
- PostgreSQL Check Items
- TDSQL for PostgreSQL
- Redis/Tendis
- SQL Server
- Database Connection Check
- Source Database Existence Check
- Target Database Existence Check
- Peripheral Check
- Migration Network Check
- Version Check
- Source Instance Permission Check
- Account Conflict Check
- Partial Database Parameter Check
- Source Instance Parameter Check
- Source Instance Type (Master or Replica)
- Parameter Configuration Conflict Check
- Target Database Content Conflict Check
- Target Database Space Check
- Target Database Load Check
- Local Disk Space Check
- Binlog Parameter Check
- Oplog Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Incremental Migration Precondition Check
- Extension/Plugin Compatibility Check
- Source Database Balancer Check
- Source Database Node Role Check
- ShardKey Check
- Warning Item Check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Level-2 Subpartitioned Table Check
- Primary Key Check
- DDL Check for Tables to Be Migrated
- System Database Conflict Check
- Table Structure Check for Source and Target Instances
- InnoDB Table Check
- Migration Object Dependency Check
- Constraint Check
- Cutover Description
- Monitoring and Alarms
- Ops Management
- DTS-DBbridge
- Best Practices
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- (NewDTS) Data Migration APIs
- DescribeMigrationJobs
- DescribeMigrationDetail
- CreateMigrationService
- ModifyMigrationJob
- CreateMigrateCheckJob
- DescribeMigrationCheckJob
- SkipCheckItem
- StartMigrateJob
- StopMigrateJob
- ResumeMigrateJob
- RecoverMigrateJob
- ModifyMigrateName
- ModifyMigrateJobSpec
- IsolateMigrateJob
- DestroyMigrateJob
- DescribeMigrateDBInstances
- CompleteMigrateJob
- PauseMigrateJob
- ContinueMigrateJob
- Data Sync APIs
- ModifySyncJobConfig
- CreateModifyCheckSyncJob
- DescribeModifyCheckSyncJobResult
- StartModifySyncJob
- DescribeSyncJobs
- CreateSyncJob
- ConfigureSyncJob
- CreateCheckSyncJob
- SkipSyncCheckItem
- DescribeCheckSyncJobResult
- StartSyncJob
- StopSyncJob
- ResumeSyncJob
- ResizeSyncJob
- RecoverSyncJob
- IsolateSyncJob
- DestroySyncJob
- PauseSyncJob
- ContinueSyncJob
- Data Consistency Check APIs
- Data Types
- Error Codes
- DTS API 2018-03-30
- SDK Documentation
- FAQs
- Error Handling (NewDTS)
- Error Handling (OldDTS)
- Service Agreement
- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Access Management
- Preparations
- Data Migration
- Databases Supported by Data Migration
- Cross-Account TencentDB Instance Migration
- Migrating to MySQL
- Migrating to MariaDB
- Migrating to TDSQL-C for MySQL
- Migrating to TDSQL MySQL
- Migrating to PostgreSQL
- Migrating to MongoDB
- Migrating to SQL Server
- Migrating to TencentDB for Redis
- Offline Migration of MySQL Data
- Advanced Migration Operations
- Task Management
- Data Sync
- Databases Supported by Data Sync
- Cross-Account TencentDB Instance Sync
- Synchronize to PostgreSQL
- Sync into Tencent Cloud
- Sync from Tencent Cloud
- Advanced Sync Operations
- Task Management
- Task Status Description
- Viewing Task
- Starting Task
- Retrying Task (MySQL)
- Verifying Task
- Stopping Task
- Pausing Task
- Resuming Task
- Resetting Task
- Deleting Task
- Changing Configuration
- Transition from Pay-as-You-Go to Monthly Subscription
- Terminating/Returning Instance
- Renewing Task
- Deisolating Task
- Deleting Instance
- Viewing Log
- Deleting Task
- Data Subscription (Kafka Edition)
- Databases Supported by Data Subscription
- Creating Data Subscription Task
- Data Consumption
- Creating Consumer Group
- Managing Consumer Group
- Consuming MySQL Data
- Consuming MongoDB Data
- Consuming TDSQL for PostgreSQL Data
- Consuming Subscribed Data with Kafka Client (Protobuf)
- Consuming Subscribed Data with Kafka Client (Avro)
- Consuming Subscribed Data with Kafka Client (JSON)
- Consuming Subscribed Data with Flink Client (Avro)
- Consuming Data with Flink
- Advanced Subscription Operations
- Task Management
- Data Migration (OldDTS)
- Data Subscription (OldDTS)
- Fix for Verification Failure
- Check Item Overview
- Check Items of MySQL/MariaDB/Percona/TDSQL-C for MySQL/TDSQL for MySQL
- MongoDB Check Items
- PostgreSQL Check Items
- TDSQL for PostgreSQL
- Redis/Tendis
- SQL Server
- Database Connection Check
- Source Database Existence Check
- Target Database Existence Check
- Peripheral Check
- Migration Network Check
- Version Check
- Source Instance Permission Check
- Account Conflict Check
- Partial Database Parameter Check
- Source Instance Parameter Check
- Source Instance Type (Master or Replica)
- Parameter Configuration Conflict Check
- Target Database Content Conflict Check
- Target Database Space Check
- Target Database Load Check
- Local Disk Space Check
- Binlog Parameter Check
- Oplog Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Incremental Migration Precondition Check
- Extension/Plugin Compatibility Check
- Source Database Balancer Check
- Source Database Node Role Check
- ShardKey Check
- Warning Item Check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Level-2 Subpartitioned Table Check
- Primary Key Check
- DDL Check for Tables to Be Migrated
- System Database Conflict Check
- Table Structure Check for Source and Target Instances
- InnoDB Table Check
- Migration Object Dependency Check
- Constraint Check
- Cutover Description
- Monitoring and Alarms
- Ops Management
- DTS-DBbridge
- Best Practices
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- (NewDTS) Data Migration APIs
- DescribeMigrationJobs
- DescribeMigrationDetail
- CreateMigrationService
- ModifyMigrationJob
- CreateMigrateCheckJob
- DescribeMigrationCheckJob
- SkipCheckItem
- StartMigrateJob
- StopMigrateJob
- ResumeMigrateJob
- RecoverMigrateJob
- ModifyMigrateName
- ModifyMigrateJobSpec
- IsolateMigrateJob
- DestroyMigrateJob
- DescribeMigrateDBInstances
- CompleteMigrateJob
- PauseMigrateJob
- ContinueMigrateJob
- Data Sync APIs
- ModifySyncJobConfig
- CreateModifyCheckSyncJob
- DescribeModifyCheckSyncJobResult
- StartModifySyncJob
- DescribeSyncJobs
- CreateSyncJob
- ConfigureSyncJob
- CreateCheckSyncJob
- SkipSyncCheckItem
- DescribeCheckSyncJobResult
- StartSyncJob
- StopSyncJob
- ResumeSyncJob
- ResizeSyncJob
- RecoverSyncJob
- IsolateSyncJob
- DestroySyncJob
- PauseSyncJob
- ContinueSyncJob
- Data Consistency Check APIs
- Data Types
- Error Codes
- DTS API 2018-03-30
- SDK Documentation
- FAQs
- Error Handling (NewDTS)
- Error Handling (OldDTS)
- Service Agreement
Issue
In data subscription scenarios, if you use your own consumer to consume data, you may encounter the following exceptions:
1. Data cannot be consumed.
2. The consumed data is either lost or duplicated.
3. The consumer delay keeps increasing.
Troubleshooting
1. Data cannot be consumed
If your own consumer fails to consume data, use the demo provided by DTS for consumption test first.
If the demo can normally consume data, you need to check your own consumer.
If the demo also cannot consume data, further troubleshoot as follows:
Check the network environment of the consumer. The consumer must be in the Tencent Cloud private network and in the same region as where the DTS data subscription task is.
Check whether the demo starting parameters are correct, especially the consumer group password.
Check whether the demo version is correct. The required demo version varies by source database type and data format.
Check the number of unconsumed messages on the consumer group management page in the console to see if the subscription task has written data to Kafka.
2. The consumed data is either lost or duplicated
When a data subscription task is restarted, data duplication may occur in the producer, causing the duplication of the consumed data. This scenario rarely occurs. Data duplication or data loss in other scenarios are not supposed to occur.
Generally, data duplication or data loss is caused by exceptions in your own consumer. To troubleshoot, reproduce the problem first. You can choose one of the following two methods to reproduce the problem:
In the console, change the Kafka consumption offset back to the previous offset and consume data again.
Create a consumer group and use it to consume data again. Consumption in different consumer groups does not affect each other.
If the problem can be reproduced, submit a ticket to handle it; otherwise, check whether your own consumer is abnormal.
3. The consumer delay keeps increasing
1. The commit logic of the consumer has been modified.
If the consumer only consumes data but doesn't commit the consumption offset, the offset in Kafka won't be updated. The default commit logic of the DTS demo is that, every time a checkpoint message is consumed, the consumption offset will be committed. The subscription service writes a checkpoint message about every 10 seconds. If you modify the commit rule, the consumer delay may keep increasing. To solve this problem, check the commit rule of your consumer first.
2. The consumption efficiency is too low.
The consumption efficiency may be affected by the network condition, the processing efficiency of the consumer, concurrent consumption, or consumption in multiple partitions. You can create a consumer group and compare the consumption efficiency of the DTS demo with that of your own consumer to find the cause of the low efficiency. You can also check the network condition to improve the data processing speed, or increase the number of consumers to implement concurrent consumption for topics in multiple partitions.
Yes
No
Was this page helpful?