Creating a Data Consistency Check Task
Scenarios
Data consistency check compares table data between the source and target databases through Data Transfer Service (DTS) during data migration. It provides comparison results and inconsistency details to help users quickly verify synchronization results before cutover. The data consistency check task runs independently and does not affect the normal business of the source database or the DTS tasks.
Note:
The consistency check is only for auxiliary data verification. Users are required to perform experiment operations by themselves to ensure that the results meet the migration requirements before the formal migration.
Currently, the following links support data consistency checks:
PostgreSQL > PostgreSQL
Must-Knows and Constraints
1. The data consistency check task may increase the load on the source database instance. Therefore, it is recommended to perform these operations during off-peak hours.
2. The data consistency check is currently supported only when the migration type is set to full + incremental migration.
3. Sampling comparison requires tables to have a primary key or unique key; otherwise, they will be skipped. The row quantity check does not require a primary or unique key. If a table lacks a primary key or unique key and contains over 10,000 rows, a full comparison cannot be performed.
4. If the user chooses to complete or terminate the DTS task before the data consistency check task is finished, the data consistency check task will fail.
Creating a Data Consistency Check Task
Note:
When a separate data check task is created, only the independent check is supported, and the built-in check is not supported.
The independent check operates independently from the DTS task and selects the same data blocks from both the source and target for comparison. Check tasks cannot be initiated once the DTS task stops running.
Independent check tasks can only be initiated when the task is running and in the incremental synchronization phase. Once the task is completed, check tasks cannot be initiated. Based on different migration phases, you can initiate check tasks using different check methods multiple times and in batches.
1. Log in to the DTS console, select Data Migration in the left sidebar, and click View in the Operation column of the task list, or directly click the task ID to go to the task details page.
2. Click Data Consistency Check to go to the data consistency check task list. Click Create Data Consistency Check Task to create a data check task.
Configuration details for the check task are as follows:
Parameter | Description |
Verification Method | Independent Check: The check service operates independently from the DTS task and can compare data outside the migration scope. |
Verification Scope | Check the structures of data objects from the source and target. If you confirm that their structures are consistent, disable the structure check to save time. |
Verification Benchmark | The source is used as the check benchmark by default. |
Verification Type | Full verification: Perform a check on the full data of the selected check objects. Row count check: Compare only the number of rows in the selected check objects. It is not required that the table objects have a primary key during row quantity comparison, and this check can be performed on tables without primary keys. Sample-based verification: Perform a data consistency check on a selected proportion of the check objects. The sampling ratio can be 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, or 90%. |
Check Object | All migration objects: The check scope includes all objects selected for the migration task. Custom: Select specific objects from the selected migration objects for checking. |
Thread Count | The value range is 1–8. Select an appropriate value according to the actual situation. Increasing the number of threads can accelerate the consistency check speed, but it will also increase the load on the source and target databases. Each additional thread approximately increases CPU usage by one core. |
3. After the configuration is completed, click Create and Start Consistency Check Task to initiate the check task.
Viewing the Data Consistency Check Results
Note:
When the first inconsistent table is detected during a check task, the system marks the task comparison result as inconsistent and sets the task status to failed.
1. On the migration task homepage, view the check result (consistent or inconsistent) in the Last Check Result column, and click View More to go to the check details page.
2. Click View to view the check results.
Summary of Data Check Results
The summary of data check results is shown in the table below:
Item | Detail |
Overview | Comparison type: Currently, all are independent checks. Verification Method: Available methods include full check, sampling check, and row quantity check. Structure Verification: This option can be enabled or disabled. Status: status of the current check task, which can be created, waiting to run, running, or completed. Comparison Conclusion: result of running the current check task, which can be inconsistent or consistent. Thread Count: number of threads configured for the current task. |
Estimated Tables | Total number of all tables that need to be checked, which is estimated by the system. |
Checked Tables | Current number of tables that have completed the check for the task. |
Inconsistent Tables | Number of tables with inconsistencies between the source and target among the tables that have completed the check. Go to the Data Validation Details page to view the tables that are inconsistent. |
Skipped table count | Number of skipped tables in the check. |
Details of Data Check Results
PostgreSQL data check tasks display the following types of inconsistencies during the check, each described below:
Data Validation Details
This section displays the details of inconsistencies in the data that has completed the check. Click View in the operation column of the details list to locate specific data, as shown in the figure below:
Table structure verification details
Only tasks with structure verification enabled will display structure verification details. During structure verification, the system lists details of all inconsistent table structures that have been verified.
To view the detailed SQL of structure inconsistencies, click Details in the structure column in the list, as shown in the figure below:
Owner verification details
Data check tasks list the details of OWNER inconsistencies in all tables that have been checked.
Skipped details
Data check tasks list details of all tables that have been skipped during verification.
How It Works
Comparison of row quantity: Check whether the number of rows of the source and target tables is consistent.
Sampling comparison
Tables without a primary key or unique index do not support sampling comparison and will be skipped directly by the system.
For other tables, the system first checks whether the row quantity of the source and target is consistent. If inconsistent, they are marked as inconsistent and skipped, and the system proceeds to check other tables.
When the row quantity of tables is consistent, the system splits a table into multiple chunks based on the sampling ratio, randomly selects rows within each chunk, and calculates the MD5 values of both source and target for comparison. Identical MD5 values indicate data consistency.
Full comparison:
For tables without a primary key or unique index:
If a table contains over 10,000 rows, the system will not perform a data consistency check.
If the number of rows does not exceed 10,000, the system sorts all columns to obtain all table data, calculates the MD5 values for both the source and target, and identical MD5 values indicate data consistency.
Help and Support
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback