Overview
Scenarios
Product Architecture
Instance Types
Compatibility Notes
Performance Optimization of Replica Migration
Traditional Migration Process
1. Data Extraction Phase: Traverse and extract data for specific key ranges from the Leader's LSM-Tree. Data may be scattered across different levels and multiple SST files, involving scanning and filtering item by item, which incurs significant I/O overhead.
2. Data Transmission Phase: Transmit KV data to the target side item by item, resulting in inefficient transmission and poor network utilization.
3. Data Writing Phase: The target first writes external SST files, then ingests them into the LSM-Tree. This requires consolidation with local SST files, generating additional compaction overhead.
4. Replica Deletion Phase: Remove data from the LSM-Tree, which requires traversing underlying SST files and incurs significant I/O overhead.
Optimization Scheme: Alignment of SST File Boundaries
During the underlying data (compaction) process, additional logic is added to ensure alignment between SST file boundaries and the boundaries of the data shard Region.
Implementation Mechanism
During the compaction process in the background, the system invokes a module called the SST partitioner. When compaction detects excessively large files, they are split based on Region boundaries. This ensures that a single SST file contains data from only one data shard, preventing the merging of data from different regions.
As shown in the figure, the L1 level: ranges such as 100-200, 200-300, 300-400 are clearly separated; at the L2/L3 levels: no data spans across Region boundaries.
Optimized Migration Process
1.1 Direct Identification of Files: Without traversing the data, SST files corresponding to the Region are directly identified.
1.2 Whole-File Transfer: Directly transmit compressed SST files to the target side.
1.3 Direct ingest: The target side does not require complex compression and directly inserts the data.
Handling of Technical Details: Before migration, compact unordered L0 data down to the L1 level. Since L0 data originates from memtable flushes and remains relatively limited in volume, the compaction overhead is acceptable.
Improved performance
compaction-related IO overhead decreases.
Improvements in migration and full backup performance.
Improvement in range physical deletion performance.
It provides the technical foundation for the Bulk Load feature, enabling efficient transmission and loading of SST files, thereby significantly enhancing the efficiency of data migration in TDStore.
Solution | Rate | Remarks |
transmitting KV over stream RPC (before optimization) | 22.6M/s | 20G data volume per RepGroup |
Direct sending of local SST (after optimization) | 455.1M/s | |
피드백