jabberwocky238/rocksdb
1
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2026-07-07 14:47:40 +08:00
Code Issues Packages Projects Releases Wiki Activity

Unify LZ4 and LZ4HC compression levels (#14819)

Browse Source 
Summary:
kLZ4Compression and kLZ4HCCompression share the same on-disk format and decompressor, but historically kLZ4Compression only honored negative (acceleration) levels while kLZ4HCCompression only honored positive levels. This unifies them so `compression_opts.level` alone selects the variant: level <= 0 uses LZ4 fast (acceleration = -level) and level >= 1 uses LZ4HC (1..12), regardless of which of the two types is configured.

The configured type now only determines the default compression level (LZ4: acceleration 1, equivalent to level -1; LZ4HC: level 9).

For code simplicity, the recorded per-block type comes from the compression type derived from the level, which could differ from the configured type. To preserve the originally configured choice for debugging/tracking, it is recorded as a `_type=<decimal>` pseudo-option in the rocksdb.compression_options SST table property.

Out-of-range non-default levels are clamped to the nearest effective value (LZ4 acceleration capped at 65537, which also avoids signed overflow negating INT_MIN; LZ4HC level capped at 12). The cost-aware (auto-tune) compressor's LZ4 level grid is changed to negative accelerations so it actually exercises fast LZ4 (positive levels now route to LZ4HC).

Related inclusion: The ZSTD library has a discontinuity at level=0, which maps to level 3, which is more aggressive than levels 1 and 2, which are more aggressive than levels -1, -2, etc. For better friendliness to auto-tuning (etc.), we now map level 0 to be the same as level -1, so that increasing compression level numbers have non-decreasing aggressiveness.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14819

Test Plan:
New unit tests in compression_test.cc:
- UnifiedLZ4LZ4HCLevels: for a representative set of non-default levels, both configured types produce identical output and the same recorded type (selected by the level), levels that clamp to the same effective parameter compress identically, and each round-trips; plus per-type default-level behavior.
- ConfiguredCompressionTypeRecordedInProperties: the `_type=` pseudo- option appears in the SST table property for each configured type.
- ZSTDLevelZeroMapsToMinusOne: level 0 behaves like -1, not like 3.

Reviewed By: joshkang97

Differential Revision: D107536580

Pulled By: pdillinger

fbshipit-source-id: a1281956f70a75d0620cb73d0bfb9ad76c52cca3
This commit is contained in:
Peter Dillinger
2026-06-11 14:10:21 -07:00
committed by meta-codesync[bot]
parent 4026c3cc08
commit 6d4a8144e0
10 changed files with 369 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
include/rocksdb/compression_type.h
+22 -4
View File
@@ -22,6 +22,8 @@ enum CompressionType : unsigned char {
kSnappyCompression = 0x01,
kZlibCompression = 0x02,
kBZip2Compression = 0x03,
// NOTE: LZ4 and LZ4HC should be considered variants of the same compression
// type. See CompressionOptions::level
kLZ4Compression = 0x04,
kLZ4HCCompression = 0x05,
kXpressCompression = 0x06,
@@ -188,10 +190,26 @@ struct CompressionOptions {
// `kDefaultCompressionLevel` values will either favor speed over
// compression ratio or have no effect.
//
// In LZ4 specifically, the absolute value of a negative `level` internally
// configures the `acceleration` parameter. For example, set `level=-10` for
// `acceleration=10`. This negation is necessary to ensure decreasing `level`
// values favor speed over compression ratio.
// LZ4 and LZ4HC share one on-disk format, so they expose a single monotonic
// `level` axis: `level <= 0` selects LZ4 fast with `acceleration = -level`
// (e.g. -10 means acceleration 10, 0 is clamped to minimum acceleration 1),
// and `level >= 1` selects LZ4HC at that level (1..12). The configured type
// (kLZ4Compression vs kLZ4HCCompression) only sets the default `level` and
// the compression type byte recorded per block. Defaults are acceleration 1
// (level -1) for LZ4 and level 9 for LZ4HC. Levels outside the
// algorithm-recognized ranges are clamped to the nearest effective value:
// below -65537 acts like -65537 (max acceleration) and above 12 like 12 (max
// compression effort).
//
// For ZSTD (the other generally recommended compression alongside LZ4),
// `level` follows zstd's own scale, where higher values trade more CPU for a
// better compression ratio. The standard range is 1 (fastest) through 19,
// with 20..22 being "ultra" levels that require substantially more memory to
// compress (and decompress). zstd also supports negative levels (-1 and
// below) that favor speed over ratio, analogous to LZ4 acceleration.
// RocksDB's default (kDefaultCompressionLevel) maps to zstd's default of 3.
// (zstd itself treats a `level` of 0 as "use the default", but prefer
// kDefaultCompressionLevel to request that.)
int level = kDefaultCompressionLevel;
// zlib only: strategy parameter. See https://www.zlib.net/manual.html
include/rocksdb/options.h
+1 -9
View File
@@ -209,15 +209,7 @@ struct ColumnFamilyOptions : public AdvancedColumnFamilyOptions {
// incompressible, the compression implementation will
// efficiently detect that and fall back on no compression.
//
// If you do not set `compression_opts.level`, or set it to
// `CompressionOptions::kDefaultCompressionLevel`, we will attempt to pick the
// default corresponding to `compression` as follows:
//
// - kZSTD: 3
// - kZlibCompression: Z_DEFAULT_COMPRESSION (currently -1)
// - kLZ4HCCompression: 0
// - kLZ4: -1 (i.e., `acceleration=1`; see `CompressionOptions::level` doc)
// - For all others, we do not specify a compression level
// See CompressionOptions for more options.
//
// Dynamically changeable through SetOptions() API
CompressionType compression;
table/block_based/block_based_table_builder.cc
+10
View File
@@ -1123,6 +1123,16 @@ struct BlockBasedTableBuilder::Rep {
props.compression_options =
CompressionOptionsToString(tbo.compression_opts);
// Record the configured compression type as an extra pseudo-option for
// debugging/tracking. The per-block compression type actually recorded can
// differ from this (e.g. LZ4 vs LZ4HC is selected by compression level;
// compression manager can override), so this preserves the originally
// configured choice. Underscore prefix indicates a special pseudo-option.
props.compression_options.append("_type=");
props.compression_options.append(
std::to_string(static_cast<int>(tbo.compression_type)));
props.compression_options.append("; ");
auto* mgr = tbo.moptions.compression_manager.get();
if (mgr == nullptr) {
uses_explicit_compression_manager = false;
tools/db_bench_tool.cc
+2 -1
View File
@@ -3065,7 +3065,8 @@ class Benchmark {
auto compressor = GetBuiltinV2CompressionManager()->GetCompressor(
opts, FLAGS_compression_type_e);
if (compressor &&
compressor->GetPreferredCompressionType() != FLAGS_compression_type_e) {
CanonicalCompressionType(compressor->GetPreferredCompressionType()) !=
CanonicalCompressionType(FLAGS_compression_type_e)) {
// For benchmarking, don't fall back on a different compression type
compressor.reset();
}
unreleased_history/behavior_changes/unify_lz4_lz4hc_levels.md
+1
View File
@@ -0,0 +1 @@
Brought more unity to `kLZ4Compression` and `kLZ4HCCompression` by giving each access to the other's compression levels. Negative (fast) values previously only available to LZ4 and positive (slow) values previously only available to LZ4HC are now available to both, so configuring a non-default `compression_opts.level` now selects the LZ4 compressor variant. This is a behavior change for previously tolerated but dubious configurations such as positive compression level with `kLZ4Compression`. `kLZ4Compression` and `kLZ4HCCompression` keep their effective default levels, equivalent to -1 and 9 respectively.
unreleased_history/behavior_changes/zstd_level_zero.md
+1
View File
@@ -0,0 +1 @@
Removed a discontinuity in `kZSTD` compression levels at `compression_opts.level == 0` by mapping that setting to `level = -1` rather than to `level = 3` as the ZSTD library does internally. This improves the compression auto-tuning landscape.
util/auto_tune_compressor.cc
+7 -7
View File
@@ -13,13 +13,13 @@
#include "util/stop_watch.h"
namespace ROCKSDB_NAMESPACE {
const std::vector<std::vector<int>> CostAwareCompressor::kCompressionLevels{
{0}, // KSnappyCompression
{}, // kZlibCompression
{}, // kBZip2Compression
{1, 4, 9}, // kLZ4Compression
{1, 4, 9}, // klZ4HCCompression
{}, // kXpressCompression
{1, 15, 22} // kZSTD
{0}, // KSnappyCompression
{}, // kZlibCompression
{}, // kBZip2Compression
{-1, -4, -9}, // kLZ4Compression (negative => LZ4 fast acceleration)
{1, 4, 9}, // kLZ4HCCompression
{}, // kXpressCompression
{1, 15, 22} // kZSTD
};
int CompressionRejectionProbabilityPredictor::Predict() const {
util/compression.cc
+51 -24
View File
@@ -355,10 +355,7 @@ std::string ZSTD_FinalizeDictionary(const std::string& samples,
if (samples.empty()) {
return "";
}
if (level == CompressionOptions::kDefaultCompressionLevel) {
// NB: ZSTD_CLEVEL_DEFAULT is historically == 3
level = ZSTD_CLEVEL_DEFAULT;
}
level = SanitizeZSTDCompressionLevel(level);
std::string dict_data(max_dict_bytes, '\0');
size_t dict_len = ZDICT_finalizeDictionary(
dict_data.data(), max_dict_bytes, samples.data(),
@@ -818,12 +815,7 @@ class BuiltinLZ4CompressorV2WithDict : public CompressorWithSimpleDictBase {
LZ4_loadDict(stream, dict_data_.data(),
static_cast<int>(dict_data_.size()));
}
int acceleration;
if (opts_.level < 0) {
acceleration = -opts_.level;
} else {
acceleration = 1;
}
int acceleration = LZ4AccelerationFromLevel(opts_.level);
auto outlen = LZ4_compress_fast_continue(
stream, uncompressed_data.data(), alg_output,
static_cast<int>(uncompressed_data.size()),
@@ -849,6 +841,24 @@ class BuiltinLZ4CompressorV2WithDict : public CompressorWithSimpleDictBase {
*out_compression_type = kNoCompression;
return Status::OK();
}
protected:
// Translates a non-positive `level` to an LZ4 "acceleration" value (=
// -level), clamped to the maximum effective acceleration. The clamp avoids
// signed overflow when negating INT_MIN and reflects that lz4 internally caps
// acceleration at LZ4_ACCELERATION_MAX (currently 65537; not exposed by
// lz4.h).
[[maybe_unused]] static int LZ4AccelerationFromLevel(int level) {
assert(level <= 0 || level == CompressionOptions::kDefaultCompressionLevel);
constexpr int kLZ4MaxAcceleration = 65537; // == LZ4_ACCELERATION_MAX
if (level <= -kLZ4MaxAcceleration) {
return kLZ4MaxAcceleration;
}
if (level >= 0) {
return 1;
}
return -level;
}
};
class BuiltinLZ4CompressorV2NoDict final
@@ -889,12 +899,7 @@ class BuiltinLZ4CompressorV2NoDict final
*out_compression_type = kNoCompression;
return Status::OK();
}
int acceleration;
if (opts_.level < 0) {
acceleration = -opts_.level;
} else {
acceleration = 1;
}
int acceleration = LZ4AccelerationFromLevel(opts_.level);
auto outlen =
LZ4_compress_fast(uncompressed_data.data(), alg_output,
static_cast<int>(uncompressed_data.size()),
@@ -970,6 +975,10 @@ class BuiltinLZ4HCCompressorV2 final : public CompressorWithSimpleDictBase {
int level = opts_.level;
if (level == CompressionOptions::kDefaultCompressionLevel) {
level = 0; // lz4hc.h says any value < 1 will be sanitized to default
} else if (level > LZ4HC_CLEVEL_MAX) {
// Map large positive levels to the maximum effective level. lz4hc clamps
// internally too, but make it explicit for clarity/determinism.
level = LZ4HC_CLEVEL_MAX;
}
ManagedWorkingArea tmp_wa;
@@ -1116,11 +1125,7 @@ class BuiltinZSTDCompressorV2 final : public CompressorBase {
#else // ROCKSDB_ZSTD_CUSTOM_MEM
ZSTD_createCCtx();
#endif // ROCKSDB_ZSTD_CUSTOM_MEM
auto level = opts_.level;
if (level == CompressionOptions::kDefaultCompressionLevel) {
// NB: ZSTD_CLEVEL_DEFAULT is historically == 3
level = ZSTD_CLEVEL_DEFAULT;
}
int level = SanitizeZSTDCompressionLevel(opts_.level);
size_t err = ZSTD_CCtx_setParameter(ctx, ZSTD_c_compressionLevel, level);
if (ZSTD_isError(err)) {
assert(false);
@@ -1766,9 +1771,31 @@ class BuiltinCompressionManagerV2 final : public CompressionManager {
case kBZip2Compression:
return std::make_unique<BuiltinBZip2CompressorV2>(opts);
case kLZ4Compression:
return std::make_unique<BuiltinLZ4CompressorV2NoDict>(opts);
case kLZ4HCCompression:
return std::make_unique<BuiltinLZ4HCCompressorV2>(opts);
case kLZ4HCCompression: {
// LZ4 and LZ4HC share the same wire format and decompressor. It is a
// historical oddity that they are distinct CompressionTypes. To blend
// them together more naturally, we now allow either configured
// compression type to access the same spectrum of compression levels
// supported by LZ4 and LZ4HC compressors. The appropriate compressor
// for the specified level is chosen regardless of which of the two
// types is configured, except that the default levels vary:
// level <= 0 -> LZ4 fast (acceleration = max(1, -level))
// level >= 1 -> LZ4HC (that level, 1..12)
// default level ->
// LZ4 -> LZ4 fast, acceleration 1 (equivalent to level -1)
// LZ4HC -> LZ4HC level 9
bool use_hc;
if (opts.level == CompressionOptions::kDefaultCompressionLevel) {
use_hc = (type == kLZ4HCCompression);
} else {
use_hc = (opts.level >= 1);
}
if (use_hc) {
return std::make_unique<BuiltinLZ4HCCompressorV2>(opts);
} else {
return std::make_unique<BuiltinLZ4CompressorV2NoDict>(opts);
}
}
case kXpressCompression:
return std::make_unique<BuiltinXpressCompressorV2>(opts);
case kZSTD:
util/compression.h
+34 -8
View File
@@ -55,6 +55,26 @@ ZSTD_customMem GetJeZstdAllocationOverrides();
#endif // defined(ZSTD) && defined(ROCKSDB_JEMALLOC) && defined(OS_WIN) &&
// defined(ZSTD_STATIC_LINKING_ONLY)
#ifdef ZSTD
// Translates a configured compression_opts.level into the effective ZSTD
// compression level. Besides resolving the "use default" sentinel, this
// removes a discontinuity at level 0: ZSTD itself treats a requested level of
// 0 as "use the default level" (historically 3), which would make level 0
// more aggressive than levels 1 and 2. Mapping 0 to -1 keeps the level spectrum
// monotonic, which is friendlier to auto-tuning.
inline int SanitizeZSTDCompressionLevel(int level) {
if (level == CompressionOptions::kDefaultCompressionLevel) {
// NB: ZSTD_CLEVEL_DEFAULT is historically == 3
return ZSTD_CLEVEL_DEFAULT;
} else if (level == 0) {
// Avoid library's discontinuity at level 0
return -1;
} else {
return level;
}
}
#endif // ZSTD
// Cached data represents a portion that can be re-used
// If, in the future we have more than one native context to
// cache we can arrange this as a tuple
@@ -227,10 +247,7 @@ struct CompressionDict {
#ifdef ZSTD
zstd_cdict_ = nullptr;
if (!dict_.empty() && type == kZSTD) {
if (level == CompressionOptions::kDefaultCompressionLevel) {
// NB: ZSTD_CLEVEL_DEFAULT is historically == 3
level = ZSTD_CLEVEL_DEFAULT;
}
level = SanitizeZSTDCompressionLevel(level);
// Should be safe (but slower) if below call fails as we'll use the
// raw dictionary to compress.
zstd_cdict_ = ZSTD_createCDict(dict_.data(), dict_.size(), level);
@@ -316,10 +333,7 @@ class CompressionContext : public Compressor::WorkingArea {
#ifdef ZSTD
if (type == kZSTD) {
zstd_ctx_ = CreateZSTDContext();
if (level == CompressionOptions::kDefaultCompressionLevel) {
// NB: ZSTD_CLEVEL_DEFAULT is historically == 3
level = ZSTD_CLEVEL_DEFAULT;
}
level = SanitizeZSTDCompressionLevel(level);
size_t err =
ZSTD_CCtx_setParameter(zstd_ctx_, ZSTD_c_compressionLevel, level);
if (ZSTD_isError(err)) {
@@ -544,6 +558,18 @@ inline bool ZSTD_FinalizeDictionarySupported() {
#endif
}
// Use to check whether compression types are related or unrelated
inline CompressionType CanonicalCompressionType(CompressionType type) {
switch (type) {
// Configuring LZ4 or LZ4HC can result in using the other, depending on
// compression level.
case kLZ4HCCompression:
return kLZ4Compression;
default:
return type;
}
}
// The new compression APIs intentionally make it difficult to generate
// compressed data larger than the original. (It is better to store the
// uncompressed version in that case.) For legacy cases that must store
util/compression_test.cc
+240 -15
View File
@@ -2498,30 +2498,52 @@ TEST_F(DBCompressionTest, GetRecommendedParallelThreads) {
ASSERT_EQ(compressor->GetRecommendedParallelThreads(), 1U);
}
// Custom parallel_threads value is returned, except fast compressors
// (Snappy, LZ4, ZSTD with level < 0) override to 1
// Custom parallel_threads value (8) is returned unless a compressor overrides
// it to 1 for being "fast". Snappy always overrides; Zlib never does. (Use a
// positive level so it does not route ZSTD to an accelerated variant below.)
opts.parallel_threads = 8;
opts.level = 3; // non-negative for ZSTD
for (auto type : {kSnappyCompression, kZlibCompression, kLZ4Compression,
kLZ4HCCompression, kZSTD}) {
opts.level = 3;
for (auto type : {kSnappyCompression, kZlibCompression, kZSTD}) {
if (!mgr->SupportsCompressionType(type)) {
continue;
}
auto compressor = mgr->GetCompressor(opts, type);
ASSERT_NE(compressor, nullptr);
uint32_t expected = 8U;
if (type == kSnappyCompression || type == kLZ4Compression) {
expected = 1U;
}
uint32_t expected = type == kSnappyCompression ? 1U : 8U;
ASSERT_EQ(compressor->GetRecommendedParallelThreads(), expected);
}
// ZSTD with negative level should override to 1
opts.level = -1;
opts.parallel_threads = 8;
auto zstd_compressor = mgr->GetCompressor(opts, kZSTD);
if (zstd_compressor) {
ASSERT_EQ(zstd_compressor->GetRecommendedParallelThreads(), 1U);
// LZ4 family: after unifying LZ4 and LZ4HC, the compression level (not the
// configured type) selects the variant, and that determines parallelism:
// level <= 0 -> LZ4 fast -> overrides to 1
// level >= 1 -> LZ4HC -> no override (parallel allowed)
if (mgr->SupportsCompressionType(kLZ4Compression)) {
for (auto type : {kLZ4Compression, kLZ4HCCompression}) {
for (int level : {-10, -1, 0, 1, 4, 9, 12}) {
SCOPED_TRACE("type=" + std::to_string(static_cast<int>(type)) +
" level=" + std::to_string(level));
opts.level = level;
auto compressor = mgr->GetCompressor(opts, type);
ASSERT_NE(compressor, nullptr);
uint32_t expected = level >= 1 ? 8U : 1U;
ASSERT_EQ(compressor->GetRecommendedParallelThreads(), expected);
}
}
}
// ZSTD: accelerated (negative) levels override to 1. Level 0 is intentionally
// left as a "backdoor" to allow parallel compression even though it otherwise
// behaves like the fast level -1.
if (mgr->SupportsCompressionType(kZSTD)) {
opts.level = -1;
auto compressor = mgr->GetCompressor(opts, kZSTD);
ASSERT_NE(compressor, nullptr);
ASSERT_EQ(compressor->GetRecommendedParallelThreads(), 1U);
opts.level = 0;
compressor = mgr->GetCompressor(opts, kZSTD);
ASSERT_NE(compressor, nullptr);
ASSERT_EQ(compressor->GetRecommendedParallelThreads(), 8U);
}
}
@@ -2600,6 +2622,209 @@ TEST_F(DBCompressionTest, CompressionManagerOverridesParallelThreads) {
}
}
TEST_F(DBCompressionTest, UnifiedLZ4LZ4HCLevels) {
// LZ4 and LZ4HC share the same wire format and decompressor, so the
// compression level alone selects which algorithm runs. A given non-default
// level therefore produces identical output regardless of which of the two
// types is configured, and the recorded compression type follows the
// compressor actually used (LZ4 fast -> kLZ4Compression, LZ4HC ->
// kLZ4HCCompression).
auto mgr = GetBuiltinV2CompressionManager();
if (!mgr->SupportsCompressionType(kLZ4Compression)) {
ROCKSDB_GTEST_SKIP("LZ4 not supported");
return;
}
auto decompressor = mgr->GetDecompressor();
// Highly compressible input so compression is accepted.
std::string input;
for (int i = 0; i < 256; i++) {
input.append("abcdefgh");
}
auto compress_one = [&](CompressionType configured_type, int level,
std::string* out, CompressionType* actual) {
CompressionOptions opts;
opts.level = level;
auto compressor = mgr->GetCompressor(opts, configured_type);
ASSERT_NE(compressor, nullptr);
out->resize(input.size() * 2 + 1024);
size_t out_size = out->size();
CompressionType type_out = kNoCompression;
ASSERT_OK(compressor->CompressBlock(input, out->data(), &out_size,
&type_out, nullptr));
ASSERT_NE(type_out, kNoCompression);
out->resize(out_size);
*actual = type_out;
};
auto round_trip = [&](const std::string& compressed, CompressionType type) {
Decompressor::Args args;
args.compression_type = type;
args.compressed_data = Slice(compressed);
ASSERT_OK(decompressor->ExtractUncompressedSize(args));
std::string uncompressed;
uncompressed.resize(args.uncompressed_size);
ASSERT_OK(decompressor->DecompressBlock(args, uncompressed.data()));
ASSERT_EQ(uncompressed, input);
};
// Non-default levels: the configured type does not matter (kLZ4Compression
// and kLZ4HCCompression produce identical output and the same recorded type,
// which the level itself selects). Levels that map to the same effective
// algorithm parameter after clamping compress identically: LZ4 fast uses
// acceleration 1 for level 0 or -1 and caps acceleration at 65537
// (level <= -65537); LZ4HC caps the level at 12.
std::string accel1; // LZ4 fast, acceleration 1
std::string accel_max; // LZ4 fast, acceleration clamped to 65537
std::string hc_max; // LZ4HC, level clamped to 12
for (int level : {-1, 0, -3, -65537, -65538, -1000000, -2000000000, 1, 4, 9,
12, 13, 100, 1000, 2000000000}) {
SCOPED_TRACE("level=" + std::to_string(level));
std::string out_lz4;
std::string out_lz4hc;
CompressionType actual_lz4;
CompressionType actual_lz4hc;
compress_one(kLZ4Compression, level, &out_lz4, &actual_lz4);
compress_one(kLZ4HCCompression, level, &out_lz4hc, &actual_lz4hc);
// The configured type does not affect the result for a given level.
ASSERT_EQ(actual_lz4, actual_lz4hc);
ASSERT_EQ(out_lz4, out_lz4hc);
// The level sign selects the algorithm (and the recorded type).
ASSERT_EQ(actual_lz4, level >= 1 ? kLZ4HCCompression : kLZ4Compression);
round_trip(out_lz4, actual_lz4);
// Levels that clamp to the same effective parameter compress identically.
if (level == -1) {
accel1 = out_lz4;
} else if (level == 0) {
ASSERT_EQ(out_lz4, accel1);
} else if (level == -65537) {
accel_max = out_lz4;
} else if (level < -65537) {
ASSERT_EQ(out_lz4, accel_max);
} else if (level == 12) {
hc_max = out_lz4;
} else if (level > 12) {
ASSERT_EQ(out_lz4, hc_max);
}
}
// Default level: the configured type selects fast (LZ4) vs HC (LZ4HC).
{
std::string out;
CompressionType actual;
compress_one(kLZ4Compression, CompressionOptions::kDefaultCompressionLevel,
&out, &actual);
ASSERT_EQ(actual, kLZ4Compression);
round_trip(out, actual);
compress_one(kLZ4HCCompression,
CompressionOptions::kDefaultCompressionLevel, &out, &actual);
ASSERT_EQ(actual, kLZ4HCCompression);
round_trip(out, actual);
}
}
TEST_F(DBCompressionTest, ZSTDLevelZeroMapsToMinusOne) {
// ZSTD itself treats a requested compression level of 0 as "use the default
// level" (historically 3). That makes level 0 a discontinuity in the
// otherwise monotonic level spectrum: it would compress more strongly than
// levels -1, -2, etc. RocksDB instead maps level 0 to -1 so the spectrum is
// continuous, which is friendlier to auto-tuning. Verify that level 0
// compresses identically to level -1 and differently from level 3 (the ZSTD
// default).
auto mgr = GetBuiltinV2CompressionManager();
if (!mgr->SupportsCompressionType(kZSTD)) {
ROCKSDB_GTEST_SKIP("ZSTD not supported");
return;
}
auto decompressor = mgr->GetDecompressor();
// Semi-compressible input large enough that nearby ZSTD levels (-1 vs 3)
// produce different output.
Random rnd(301);
std::string input;
for (int i = 0; i < 2000; i++) {
input.append(rnd.RandomString(16));
input.append(8, 'x');
}
auto compress_one = [&](int level, std::string* out) {
CompressionOptions opts;
opts.level = level;
auto compressor = mgr->GetCompressor(opts, kZSTD);
ASSERT_NE(compressor, nullptr);
out->resize(input.size() * 2 + 1024);
size_t out_size = out->size();
CompressionType type_out = kNoCompression;
ASSERT_OK(compressor->CompressBlock(input, out->data(), &out_size,
&type_out, nullptr));
ASSERT_EQ(type_out, kZSTD);
out->resize(out_size);
};
auto round_trip = [&](const std::string& compressed) {
Decompressor::Args args;
args.compression_type = kZSTD;
args.compressed_data = Slice(compressed);
ASSERT_OK(decompressor->ExtractUncompressedSize(args));
std::string uncompressed;
uncompressed.resize(args.uncompressed_size);
ASSERT_OK(decompressor->DecompressBlock(args, uncompressed.data()));
ASSERT_EQ(uncompressed, input);
};
std::string out_level0;
std::string out_level_minus1;
std::string out_level3;
compress_one(0, &out_level0);
compress_one(-1, &out_level_minus1);
compress_one(3, &out_level3);
// Level 0 behaves like level -1, not like the ZSTD default (3).
ASSERT_EQ(out_level0, out_level_minus1);
ASSERT_NE(out_level0, out_level3);
round_trip(out_level0);
round_trip(out_level_minus1);
round_trip(out_level3);
}
TEST_F(DBCompressionTest, ConfiguredCompressionTypeRecordedInProperties) {
// The configured compression type is recorded as a `_type=<decimal>`
// pseudo-option in the SST `rocksdb.compression_options` table property, so
// it can be recovered for debugging even when the per-block recorded type
// differs from the configured type.
if (!CompressionTypeSupported(kLZ4Compression)) {
ROCKSDB_GTEST_SKIP("LZ4 not supported");
return;
}
for (auto type : {kLZ4Compression, kLZ4HCCompression}) {
Options options = CurrentOptions();
options.compression = type;
DestroyAndReopen(options);
Random rnd(301);
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(100)));
}
ASSERT_OK(Flush());
TablePropertiesCollection props;
ASSERT_OK(db_->GetPropertiesOfAllTables(&props));
ASSERT_FALSE(props.empty());
std::string expected = "_type=" + std::to_string(static_cast<int>(type));
for (const auto& kv : props) {
ASSERT_NE(kv.second->compression_options.find(expected),
std::string::npos)
<< "compression_options=" << kv.second->compression_options;
}
}
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();