Note:
The data in this section was obtained from a 12-hour long-term stability test conducted by the TDStore team in an internal lab environment, using sysbench with a write-intensive workload. The data is intended to illustrate trends and magnitudes and does not constitute a performance commitment for specific hardware or business scenarios.
Benchmark Testing
To facilitate readers in reproducing and interpreting the subsequent data, the load configurations common to all tests are listed first:
Load Model
|
Stress testing tool | sysbench |
Load Type | Random INSERT (Write-Oriented) |
Concurrent Threads | 256 |
Number of Tables | 64 |
Duration per Round | 12 hours (Long-term stability) |
KV Size Characteristics
From the perspective of the underlying TDStore (RocksDB), two KV records are actually generated for each row:
|
Primary Key Index (PK) | 8 B | 9,405 B |
Secondary Index | 26 B | 4 B |
Why is it designed this way?
1. The primary key Value is approximately 9.4KB: far exceeding the common min_blob_size threshold (1KB), it will be entirely separated into Blob files, which fully demonstrates the mitigating effect of KV separation on write amplification.
2. The secondary index Value is extremely small (4 B): far below the threshold, it will not take the KV separation path. This also allows for verifying the impact of enabling KV separation on small Value indexes (actual tests show no significant adverse degradation).
3. 256 threads × 64 tables: This configuration prevents lock contention on a single table from masking differences at the storage engine layer, thereby making Compaction / Blob file management the actual source of the bottleneck.
Note:
Conversion logic for applying to your own business: If the average row length in your business is close to 10KB (such as rich text, JSON details, and wide tables with large fields), the improvement ratios from the measured data below can serve as a reference. The larger the row length, the closer the benefit of KV separation approaches its upper limit.
Standard Server Scenario (CPU Bottleneck)
Test Environment: A high-end single server with ample I/O and CPU as the bottleneck, using sysbench with 512 threads and 16 Compaction threads, running for 12 hours. Under this configuration, the primary focus for KV separation is to observe "whether it introduces additional overhead," with limited theoretical gains.
|
KV separation not enabled | 18,573 | Controlled by the pending_bytes deferred write, the macro-level performance shows a significant decline; write operations pause during Level expansion, resulting in substantial micro-level fluctuations. |
KV separation + forced GC | 21,170 ⬆️ approx. 14% | The macro-level decline is slight (with good number of Blobs control); the micro-level fluctuations are significant due to the impact of L0 accumulation. |
KV separation + dynamic merging of small blobs | 20,918 ⬆️ approx. 13% | The micro-level performance is the most stable; it requires increasing the level0_file_num_compaction_trigger to control the number of Blob files. |
Key Observations:
1. Writes are more stable: After KV separation is enabled, stalls are almost eliminated, resulting in a smoother process compared to the baseline's "Stall → stop → recovery" cycle.
2. The decline is more controllable: The performance decline in the baseline originates from the accumulation of pending_bytes; the decline under KV separation originates from the accumulation of Blob files. The former accelerates with increasing data volume, while the upper limit of the latter is adjustable.
3. Even under CPU bottleneck conditions, KV separation can still achieve a 13~14% throughput improvement, with no observed adverse degradation.
Cloud Disk Scenario (I/O Bottleneck)
Test Environment: Disk read and write bandwidth were each limited to 300 MB/s using cgroups, with sysbench running 128 threads and 8 Compaction threads configured, making it more closely resemble a real cloud deployment environment.
Standard Configuration (Default LSM Shape)
Default configuration: L0 is uncompressed, and L1 is standard size.
|
KV separation not enabled | 4,213 | — |
KV separation (dynamic Blob merging, controlled) | 7,087 | ⬆️ +68% |
KV separation + disabling Blob merging during peak load periods | 9,177 | ⬆️ +118% |
Tuning Configuration (LSM-Tree Widening + L0 Compression)
Optimized configuration: compression_per_level = LZ4:LZ4:ZSTD:..., max_bytes_for_level_base = 1 GB
|
KV separation not enabled | 10,920 | — |
KV separation (dynamic Blob merging) | 11,829 | ⬆️ +8% |
KV separation + disabling Blob merging during peak load periods | 15,702 | ⬆️ +44% |
Key Observations:
1. I/O bottleneck scenarios are the optimal stage for KV separation: Throughput doubles directly under the default configuration. The reason is that Compaction no longer competes for valuable I/O bandwidth for large values, leaving significantly more bandwidth for foreground writes and Raft Log.
2. P95 Tail Latency Improvement: Even with relatively large fluctuations in average QPS, the P95 tail latency is generally better than the baseline after KV separation is enabled.
3. Under high write pressure, "pausing Blob merging" is a better policy: Merging Blobs itself also consumes I/O. Temporarily disabling it during high write pressure and re-enabling it during low-traffic periods is a recommended Ops practice in cloud disk environments.
Measured Conclusions
CPU bottleneck scenario: KV separation performance is approximately flat to a +14% improvement (with a significant stability improvement).
I/O bottleneck scenario: KV separation yields approximately a +44% to +118% improvement (with a significant throughput increase).
The more I/O-constrained the environment and the larger the Value size, the more significant the benefits of KV separation become.
|
Throughput ceiling | Limited by Compaction write amplification | Significantly improved (especially under I/O bottlenecks). |
Write stall (Stall) | Frequent, worsening with data growth | Almost eliminated. |
Micro-level jitter | Large (due to periodic changes in pending_bytes) | Small (affected by the periodic merging of Blobs). |
Macro-level declining trend | Continuously declining | Stabilizing. |
Disk write volume | High (due to repeated writes of large values) | Low (large values are written only once). |
Parameter tuning complexity | Low | Medium (requires balancing GC, compaction, and L0 thresholds). |
Recommended Configuration Summary:
On the business side, you only need to enable enable_blob_files + min_blob_size=1024 + enable_blob_garbage_collection (see Quick Start). Further Tuning by DBA