tencent cloud

Background

Lucene currently supports two compression algorithms for storing document fields data:

• LZ4
• Deflate

LZ4 has a higher compression and decompression speed, while Deflate has a higher compression ratio. They have obvious differences in performance and compression ratio. Based on these two existing compression algorithm, you cannot get a good balance between compression ratio and performance. Lucene uses LZ4 by default.

Optimized Scheme

ES integrates the industry-leading advanced compression algorithm Zstandard (ZSTD) to improve the compression ratio while reducing the performance loss.

Strengths of Zstandard compression algorithm

The Zstandard compression algorithm has the strengths of both LZ4 and Deflate: its performance is equivalent to that of LZ4 (tests with log data show that Zstandard is slightly better than LZ4), while its compression ratio is only slightly lower than that of Deflate.

The following are the comparison results of the three compression algorithms:

Note：

1. Test data: based on a typical log application.
2. Test method: based on Elasticsearch REST High Level Client API.

Directions

Based on REST High Level Client API

When creating an index, add the index.codec configuration item for CreateIndexRequest and set the value to zstandard:

CreateIndexRequest createRequest = new CreateIndexRequest(indexName);
createRequest.settings(Settings.builder()
  .put("index.number_of_shards", shards)
  .put("index.number_of_replicas", replicas)
  .put("index.codec", "zstandard")
);

Based on HTTP request

Similarly, add the index.codec configuration item in settings and set the value to zstandard:

PUT /newIndex 
{
  "settings": {
   "index.codec": "zstandard",
    "index.number_of_shards": 1
  }
} 

Optimization Effect

ZSTD has a 35% higher row storage compression ratio than LZ4 and a performance comparable to that of LZ4.

Supported Versions

6.8.2, 7.5.1, and 7.10.1