V18.0.1
Version Release Notes
Data dictionary cache size is configurable.
Added max_dd_cache_size to limit the data dictionary cache capacity.
Bug fixes
Fixed the issue where excessive data migration from the source node to the destination node during capacity balancing caused the original source node to become the migration target in the next cycle, resulting in data shuttling back and forth between the two nodes.
Fixed the issue where triggering a promote-demote job during migration after Learner replicas are introduced would demote replicas scheduled for removal to Learner status, requiring waiting for heartbeat updates before issuing the remove job.
Fixed the issue where migration would get stuck due to the absence of heartbeat reporting from unhealthy or disconnected nodes.
Fixed the issue where the newly added parameter expired-log-receiver-topology-keep-time in V17.0.0 is missing the JSON tag.
Fixed an issue where multi-job tasks may record invalid job IDs, causing tasks to fail to terminate due to inability to locate the job, ultimately resulting in task timeout.
Fixed the issue where hotspot scheduling could not split data objects out from within the Replication Group (RG).
V18.0.0
Version Release Notes
Distributed Transaction
Supports pessimistic lock waiting for SQL query nodes
By querying the performance_schema.data_locks, performance_schema.data_lock_waits, and information_schema.tdstore_part_ctx tables, you can obtain the pessimistic lock information and participant information for the current node's TDStore.
High availability
Enhanced Containerized HA (High Availability) Capabilities
Auxiliary enhancements have been made to the existing HA capabilities, enabling the probe module to detect abnormal states of nodes, clusters, and pods within the cluster within seconds. This triggers alarms to promptly notify Ops personnel of HA anomalies, ensuring the health of user instances.
Memory-based table cache eviction policy
The eviction policy for the memory usage of the open table cache is implemented, corresponding to the variable table_open_cache_max_mem_size.
Enhanced robustness in obtaining GTS (Global Timestamp Service)
Optimized the retry logic for SQLEngine to obtain GTS.
Performance optimization
Migration Task Performance and Stability Optimization
The snapshot data generation method of Install Snapshot was optimized from scan to hard link, resulting in a more than 20-fold improvement in the migration performance of a single RG.
In-depth optimization of the two-phase mechanism for the Raft layer's Install Snapshot has improved the success rate in high-load scenarios.
Refactored Raft Log Retention Policy
In the new policy, if a log synchronization is completed, it is deleted within a short time; if an abnormality occurs in log synchronization, the logs are retained for a longer period to wait for the synchronization process to recover.
Addition of Multiple fsync Policies for Raft Log
Raft Log disk writing supports three fsync policies: 1. Call fsync immediately after writing is completed; 2. Call fsync based on the amount of data written or time intervals; 3. Do not call fsync, leaving it to the operating system to invoke autonomously.
Multi INSERT IGNORE and Multi INSERT with unique indexes in tables support Batch Put optimization (INSERT ON DUPLICATE KEY UPDATE does not support Batch Put optimization).
In older versions, when a table has a unique index or the INSERT statement is INSERT IGNORE, the insertion process does not use Batch Put optimization and instead degenerates to row-by-row insertion. This version adds Batch Check to detect the uniqueness of primary keys and unique indexes, so that Batch Put optimization can be used in these two scenarios. This feature can be controlled via the tdsql_stmt_optim_batch_check variable.
Reduce Partitioned Table Memory Footprint
Excessive partitions consume significant memory; the code has been optimized to achieve a reduction in the memory footprint of partitioned tables.
Refactored Routing Cache
The computational layer's routing cache has been refactored by splitting it into Node cache, ReplicationGroup cache, and Region cache. This reduces unnecessary memory allocation and copying, facilitates data read/write operations and transaction RPC (remote procedure call) coding, and paves the way for future implementation of Region Merge and Replication Group Merge.
Configure the Number of Buckets for Routing Cache
For instances with large data volumes, the number of Nodes, ReplicationGroups, and Regions may be high, leading to increased memory usage by the computational layer's routing cache. This memory consumption is proportional to the number of cache buckets. Therefore, the GLOBAL parameter tdsql_route_cache_buckets has been introduced to adjust the number of routing cache buckets.
Optimized DDL (Data Definition Language) Execution Time
To implement Online Schema Change, DDL operations must perform write fence (DDL barrier) operations at various stages. In this version, write fences are aggregated and batched to accelerate DDL execution.
Reduced Unnecessary System Calls Related to vio
To mitigate the impact of epoll_ctl calls on bthread scheduling efficiency, unnecessary system calls related to vio have been reduced.
Restoring via Backups
The Raft layer supports precise perception of incremental backup positions.
The default retention policy for Raft Logs in older versions preserves logs for 1 hour to ensure continuity of incremental backups. With the Raft layer now supporting accurate perception of incremental backup positions, it provides the data foundation for fine-grained control of the Raft Log retention policy, enabling timely reclamation of disk space occupied by obsolete Raft Logs.
TDBR supports multi level ingest.
The cluster recovery process supports multi-level ingest to mitigate severe SST (Sorted String Table) file bloat during restoration.
TDBR supports closing residual data of unloaded nodes in COS.
For incremental backups, data in COS for unloaded nodes should be closed in a timely manner to prevent potential restoration failures caused by residual data.
Data Import
BulkLoad Liveness Probe Mechanism
Before BulkLoad data import, the MC (Metadata Cache) will disable certain scheduling. When SQLEngine, myloader, or MC are terminated, the MC can automatically resume scheduling without requiring manual intervention.
BulkLoad supports transmitting data in segments.
BulkLoad transmits data using segments to reduce the need for frequent memory allocations.
BulkLoad restricts all SQL within the same transaction to be of the same SQL Statement type.
When data is imported via BulkLoad, all SQL statements within the same transaction are restricted to a single type.
BulkLoad deletes remote SST files during rollback.
During BulkLoad rollback, remote nodes may have already transmitted data. It is essential to ensure that remote SST files can be successfully cleared during rollback.
Query processing
Parallel Query Supports Full Pushdown of Grouping Operations (GROUP BY).
Typically, parallel plans require two-stage grouping operations. Release 18 supports full pushdown of grouping operations for certain queries, eliminating the need for the second-stage grouping. This feature is enabled only when parallel scans use primary key and Unique Key scans.
SELECT COUNT(*) without GROUP BY uses parallel query by default.
Prior to Release 18, queries for total row counts (such as SELECT COUNT(*) without GROUP BY) leveraged storage engine-level parallelism by default, unaffected by parallel query parameters like max_parallel_degree. Starting with Release 18, these queries are implemented using the parallel query framework by default. Refer to the parallel_query_switch parameter documentation for usage details.
Parallel Query supports parallel execution of statements containing scalar subqueries.
In Release 17, queries containing subqueries did not utilize parallel execution plans. Release 18 supports parallel execution of queries containing the following subqueries, while also enabling cross-node execution (MPP, Massively Parallel Processing) for these queries: 1. Uncorrelated scalar subqueries in any position; 2. Parallel execution of correlated subqueries in WHERE clauses.
Parallel Query supports parallel scanning of non-first tables.
In Release 17, for multi-table queries, only the first table supported parallel scanning. If the first table could not be scanned in parallel, the query would fall back to serial execution. Release 18 removes this limitation and supports cross-node execution (MPP). The PARALLEL hint can be used to specify tables for parallel scanning.
Parallel Query supports aggregate functions with DISTINCT.
Release 18 removes the previous restriction that parallel queries could not contain aggregate functions with DISTINCT. If a query contains aggregate functions with DISTINCT, neither the grouping (GROUP BY) nor the sorting operations of that statement can be pushed down to parallel workers for execution.
Operations
Parameters Support SQL Querying and Modification.
MC supports querying system information using SQL from MySQL Client, including: RG (Replication Group) metadata, DataObject metadata, parameters, tasks, and so on. MC parameters can be modified using stored procedures.
tdstore_client_new_console Stability Enhancements
Fixed the issue where the console could not obtain information on deleted Replication Groups.
Optimized the RPC packet size for the get_rg_info command to avoid exceeding the limit and causing disconnections.
Periodically dumping SHOW PROCESSLIST
Periodically collect SHOW PROCESSLIST information to facilitate problem analysis.
Default export of RPC trace information to slow query logs.
Added RPC trace information to facilitate problem analysis.
Bug fixes
Fixed an issue where INSERT transactions could violate the uniqueness constraint of Unique Secondary Index under specific concurrency timing scenarios.
Fixed an issue where hybrid nodes attempted to connect to non-existent nodes.
Fixed an issue where LocalSqlRunner had a memory leak.
Added index length statistics.
Fixed an issue where unlocking a range lock converted it into a pending single-row lock request, thus keeping the corresponding key range locked and impacting availability.
Fixed an issue where SQLEngine reported errors when sending commit requests to the hybrid node of the old RG leader during resource group leader switch and migration.
Fixed an issue where after a transaction updated a non-indexed column of a row just committed by a concurrent transaction, subsequent reads using a secondary index might fail to retrieve the latest modification (which was readable via the primary key).
Fixed an issue where hotspot scheduling randomly initiated leader switch tasks based on read or write loads, and flawed hotspot statistics methodology caused excessive initiation of leader switch tasks.
Parameter Change
|
Modification | | table_definition_cache is used to set the number of cached table definition objects. The table definition cache stores metadata of tables in the database (such as column names, data types, and so on) to enable quick access to this information during query execution. When this limit is reached, the system evicts the least recently used table definitions based on the LRU (Least Recently Used) algorithm to free up space for new table definitions.
|
Modification | | tdsql_max_retry specifies the maximum number of times retries are allowed for failures (such as network/RPC errors).
|
Modification | | table_open_cache is used to set the number of cached open table objects. When a table is opened, the system stores the table's structural information in the table cache for reuse in subsequent queries. The table_open_cache parameter value indicates the maximum number of tables that can be opened simultaneously. When this limit is reached, the system closes the least recently used tables based on the LRU (Least Recently Used) algorithm to free up space for new tables.
|