Compare commits

...

6 Commits

Author SHA1 Message Date
Jay Huh 10dfc6acb8 Update version and HISTORY.md for 11.5.2 release 2026-06-25 14:52:45 -07:00
Jay Huh d46d4b7f82 Revert reverse-comparator handling in range tree lock manager (#14887)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14887

This reverts two changes to the range-lock endpoint comparator and restores the prior, direction-agnostic behavior:
- https://github.com/facebook/rocksdb/pull/14831 ("Fix range lock manager crash with reverse comparator"), which made RangeTreeLockManager::CompareDbtEndpoints reverse-aware, and
- https://github.com/facebook/rocksdb/pull/14880 (a follow-up that only corrected the equal-length / point-range branch).

Why: #14831 taught the endpoint comparator about reverse-ordered column families by flipping the suffix tie-break based on comparator direction. However, callers that use range locking on a reverse-ordered CF already flip the start/end endpoints themselves before calling lock_range() -- for example, MyRocks does this in ha_rocksdb::set_range_lock() (see its "RangeFlagsShouldBeFlippedForRevCF" comment). The two corrections stack and produce left > right again, which:
(a) trips paranoid_invariant(compare(left, right) <= 0) in toku::locktree::try_acquire_lock() and aborts on equality/point locks, and
(b) for range scans, mis-orders the endpoints and produces an incorrect lock range, leading to silent correctness issues under concurrency.
#14880 only addressed (a) (the equal-length / point-range branch), not (b), so it was an incomplete fix at the wrong layer.

Decision: keep CompareDbtEndpoints direction-agnostic and make it an explicit caller requirement to pass a flipped (swapped start/end, with negated infimum/supremum flags) range for range locking to work on a reverse-ordered column family. A function-level comment documenting this contract is added so reverse-awareness is not reintroduced here.

Removes the RangeLockWithReverseComparator unit test added by #14831 (it asserts the reverted direction-flipped ordering). Keeps a regression test that a point range lock [infimum(K), supremum(K)] works on a reverse-comparator column family.

Reviewed By: xingbowang

Differential Revision: D109713989

fbshipit-source-id: 4ea0999c92645e1adc9cd1114d924cad142f2001
2026-06-25 14:26:31 -07:00
Jay Huh a127d1533b Update version and HISTORY.md for 11.5.1 release 2026-06-24 16:25:47 -07:00
Josh Kang 928ebd5b8b Do not allow read only DBs to delete obsolete files (#14881)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14881

Read-only DB instances can share a directory with a live writer, so their view of live files is only a snapshot from the time they opened. After read-only DB open began setting `opened_successfully_` for its intended open-success semantics, that state had an unintended side effect: the common close path treated a successful read-only open like a read-write open and ran obsolete-file cleanup. If the writer created or made files live after the read-only handle opened, that cleanup could use the stale read-only live set and delete files still needed by the writer.

This change records whether a `DBImpl` was opened read-only and skips close-time obsolete-file cleanup for read-only DBs instead of simply keeping `opened_successfully_=false`, which could be confusing and easily mistaken in the future again.

For completeness I also updated other callsites of `opened_successfully_`, but those should not be real bugs as ReadOnly DBs do not run flushes/compactions or write to WALs.

Reviewed By: xingbowang

Differential Revision: D109622445

fbshipit-source-id: d7be4b20fce86ccb218a63ac6f5b707b316aac79
2026-06-24 16:23:13 -07:00
Jay Huh afcf52a996 Fix range lock manager assertion crash on reverse-comparator CF point locks (#14880)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14880

The RocksDB 11.5.0 reverse-comparator range-lock fix (the HISTORY.md entry "Fixed a bug where the range lock manager would crash with an assertion failure when using a reverse comparator column family") made `RangeTreeLockManager::CompareDbtEndpoints` reverse-aware by wrapping the endpoint-suffix tie-break results in an `is_reverse` flip. The length-disparity (different user key) branches need that flip, but the equal-length / equal-user-key branch was flipped too, and that is incorrect.

Range-lock endpoints are encoded as `[suffix_byte][user_key]` where the suffix is `SUFFIX_INFIMUM` (0x0) or `SUFFIX_SUPREMUM` (0x1). When two endpoints share the same user key and differ only in the suffix byte, the suffix is a positional boundary marker for that key, not key content, so the infimum endpoint must sort before the supremum endpoint in any total order, regardless of the column family sort direction. Flipping it for a reverse comparator makes `m_cmp(infimum(K), supremum(K)) > 0`.

MyRocks issues an equality lock (e.g. `UPDATE t1 SET a=123 WHERE a=35`) as the range `[infimum(K), supremum(K)]` over the same user key K. On a reverse (`rev:`) column family — which uses the stock `ReverseBytewiseComparator()`, so `is_reverse` is true — the flipped comparator reports `left > right`, tripping `paranoid_invariant(m_cmp(left_key, right_key) <= 0)` in `toku::locktree::try_acquire_lock` and aborting the server with SIGABRT. This broke many MyRocks `---range_locking` MTR tests (partial_index*, bypass_select_range_sk*, alter_table_online_debug, allow_no_primary_key_with_sk) after RocksDB was bumped to 11.5.0 in D108936924.

Fix: in the equal-length / equal-user-key branch of `CompareDbtEndpoints`, compare the suffix bytes without the `is_reverse` flip so infimum always precedes supremum. The length-disparity branches keep the reverse-aware flip (still covered by `RangeLockWithReverseComparator`).

Note: this lands in `internal_repo_rocksdb/repo`, the RocksDB development tree; it reaches `fbcode/rocksdb/src` (what MyRocks/ZippyDB link) through the RocksDB release sync. Tracked in T276980728.

Reviewed By: xingbowang

Differential Revision: D109591908

fbshipit-source-id: 078b16e6e51f4f365b4a8d539ac5b968a7461729
2026-06-24 16:23:04 -07:00
anand1976 e688e7b235 Update HISTORY.md for 11.5.0 release 2026-06-17 14:24:03 -07:00
26 changed files with 118 additions and 66 deletions
+36
View File
@@ -1,6 +1,42 @@
# Rocksdb Change Log
> NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
## 11.5.2 (06/25/2026)
### Bug Fixes
* Reverted PR14831 that made range_lock_manager aware of reverse-order CF
## 11.5.1 (06/24/2026)
### Bug Fixes
* Fixed a bug where closing a read-only DB instance could delete live SST files created by a concurrent read-write DB sharing the same directory.
## 11.5.0 (06/16/2026)
### New Features
* External table readers that open files through `ExternalTableOptions::fs` now update RocksDB SST/file-read statistics and file IO listener callbacks, making external file IO activity visible in existing read metrics.
* Added `DB::PrepareFileIngestion()`, a two-phase form of `IngestExternalFile`/`IngestExternalFiles`. `PrepareFileIngestion()` performs all of the work that does not require the DB mutex (validating the arguments, reading each external file's metadata, and linking/copying the files into the DB) and returns an opaque `FileIngestionHandle`. `DB::CommitFileIngestionHandle()` (or `DB::CommitFileIngestionHandles()` for several handles at once) then makes the prepared files visible; multiple handles are committed atomically in a single MANIFEST write, or none are. `FileIngestionHandle::Abort()` (or simply destroying the handle) cancels a prepared ingestion and rolls it back. This gives flexibility to the application to Prepare a file separately from when its committed, and may shorten an applications critical path on `IngestExternalFile`.
* Added two stats histograms reporting the per-call latency (in microseconds) of each successful `IngestExternalFile`/`IngestExternalFiles` call, split by phase: `rocksdb.ingest.external.file.prepare.micros` (argument validation and reading/validating/linking the external files, which does not block live writes) and `rocksdb.ingest.external.file.run.micros` (the ingestion performed under the DB mutex while live writes are blocked). Failed ingestions are not recorded.
* Added `DBOptions::use_direct_io_for_compaction_reads` (default false). When enabled, compaction-input SST reads use `O_DIRECT` while user reads remain buffered, avoiding page-cache eviction of hot user-read data by sequential compaction scans. Pair with `use_direct_io_for_flush_and_compaction = true` on write-heavy workloads for direct I/O on both compaction inputs and outputs. Rejected at Open when combined with `allow_mmap_reads = true`. No-op when `use_direct_reads = true` is already set (since all reads are already direct).
### Public API Changes
* `DB::GetPreparedFileInfoForExternalSstIngestion()` can now prepare metadata for a live DB-generated SST file so it can be passed to `IngestExternalFileArg::file_infos`. The input path must exactly match a live table file owned by the source DB, and the returned metadata handle must outlive the ingestion.
* `ExternalSstFileInfo` gained a `prepared_file_info` member produced by `SstFileWriter::Finish`, and `IngestExternalFileArg` gained a `file_infos` field: a vector of borrowed `const PreparedFileInfo*` pointers, parallel to `external_files`. The owning `ExternalSstFileInfo::prepared_file_info` handle must outlive the ingestion. When set, `IngestExternalFiles()` reuses the supplied per-file metadata instead of re-opening and scanning each file to recompute it, avoiding that extra I/O.
* Corrected the public C++ option spelling from `memtable_veirfy_per_key_checksum_on_seek` to `memtable_verify_per_key_checksum_on_seek`. RocksDB continues to accept the old misspelled OPTIONS-file key for compatibility, while newly serialized OPTIONS files use the corrected key.
* Added experimental read-scoped block buffer provider API, configured through `ReadOptions::read_scoped_block_buffer_provider`, for supported block-based table iterator scans and MultiScan reads to use caller-provided read-scoped storage for final data-block contents. When configured, supported provider-backed scan data-block reads bypass the data-block cache. The provider is ignored when mmap reads are enabled. RocksDB may still use ordinary temporary scratch for serialized block bytes, such as when a block may be compressed. Get and MultiGet do not currently provide API guarantees for this provider.
* Added `FileSystem::SyncFile()` and `Env::SyncFile()` public APIs for syncing or fsyncing a file by name without requiring callers to reopen it as writable.
### Behavior Changes
* The default `compression` for column families is now `kLZ4Compression` rather than `kSnappyCompression`. This affects only column families that do not explicitly set `compression`, and only newly-written SST files. The change is fully compatible: existing data remains readable with no migration needed, as RocksDB selects the decompressor per block. LZ4 offers slightly better compression ratios and decompression CPU efficiency vs. Snappy and similar compression CPU, tested across various server CPUs. When support is not compiled in, the fallback path for default compression is LZ4 -> Snappy -> NoCompression.
* Disable parallel compression (ignore `CompressionOptions::parallel_threads`) for fast built-in compressors: Snappy, LZ4 (accelerated, not LZ4HC), and accelerated levels of ZSTD (level < 0). These do not generally benefit from parallel compression. This behavior can be overridden with a custom Compressor from a custom CompressionManager.
* `PosixFileSystem::OptimizeForCompactionTableRead` now delegates to the base `FileSystem::OptimizeForCompactionTableRead` implementation before applying its Linux-only compaction-readahead clamp (added for #12038). The returned `FileOptions::use_direct_reads` therefore reflects `DBOptions::use_direct_reads`, consistent with the base class and with other `FileSystem` implementations. Previously the override ignored the base implementation and keyed both the returned options and the Linux readahead clamp off the incoming `FileOptions` rather than `DBOptions`. At the in-tree call site the incoming `FileOptions::use_direct_reads` already matches `DBOptions::use_direct_reads`, so this is effectively a no-op for existing configurations; it removes a latent inconsistency where a caller passing `FileOptions` whose `use_direct_reads` disagreed with `DBOptions` would have had the global flag silently ignored for compaction-input reads on Linux. (Note: the `#12038` readahead clamp still keys off the global `use_direct_reads`; it is not skipped for the compaction-only `use_direct_io_for_compaction_reads` path, since that flag enables O_DIRECT after this hook runs.)
* Brought more unity to `kLZ4Compression` and `kLZ4HCCompression` by giving each access to the other's compression levels. Negative (fast) values previously only available to LZ4 and positive (slow) values previously only available to LZ4HC are now available to both, so configuring a non-default `compression_opts.level` now selects the LZ4 compressor variant. This is a behavior change for previously tolerated but dubious configurations such as positive compression level with `kLZ4Compression`. `kLZ4Compression` and `kLZ4HCCompression` keep their effective default levels, equivalent to -1 and 9 respectively.
* Removed a discontinuity in `kZSTD` compression levels at `compression_opts.level == 0` by mapping that setting to `level = -1` rather than to `level = 3` as the ZSTD library does internally. This improves the compression auto-tuning landscape.
### Bug Fixes
* Fixed a bug where the range lock manager would crash with an assertion failure when using a reverse comparator column family.
### Performance Improvements
* Reduced commit latency when ingesting many external files by allowing `IngestExternalFileOptions::file_opening_threads` to open table readers for committed ingested files using multiple threads.
* Reduced commit latency for large external file ingestions into the last level by adding `IngestExternalFileOptions::prefetch_lmax_index_and_filter_blocks`, which can skip commit-time index and filter block prefetching for cache-backed table metadata.
## 11.4.0 (06/02/2026)
### Public API Changes
* Added `rocksdb_options_set_memtable_batch_lookup_optimization()` and `rocksdb_options_get_memtable_batch_lookup_optimization()` to the C API, exposing the existing `AdvancedColumnFamilyOptions::memtable_batch_lookup_optimization` field. This allows C API users (and downstream language bindings) to enable the skip-list memtable's batch-lookup optimization for `MultiGet`, which caches the search path between consecutive keys and reduces per-key cost from O(log N) to O(log d) where d is the distance between consecutive keys.
+4 -2
View File
@@ -176,6 +176,7 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
const bool seq_per_batch, const bool batch_per_txn,
bool read_only)
: dbname_(dbname),
read_only_(read_only),
own_info_log_(options.info_log == nullptr),
initial_db_options_(SanitizeOptions(dbname, options, read_only,
&init_logger_creation_s_)),
@@ -672,7 +673,8 @@ void DBImpl::UnregisterBlobDirectWriteColumnFamily() {
Status DBImpl::MaybeWriteWalMarkersToManifestOnClose() {
mutex_.AssertHeld();
if (!mutable_db_options_.optimize_manifest_for_recovery ||
!opened_successfully_ || versions_ == nullptr || logs_.empty()) {
!opened_successfully_ || read_only_ || versions_ == nullptr ||
logs_.empty()) {
return Status::OK();
}
@@ -880,7 +882,7 @@ Status DBImpl::CloseHelper() {
// manifest file), it is not able to identify live files correctly. As a
// result, all "live" files can get deleted by accident. However, corrupted
// manifest is recoverable by RepairDB().
if (opened_successfully_) {
if (opened_successfully_ && !read_only_) {
JobContext job_context(next_job_id_.fetch_add(1));
FindObsoleteFiles(&job_context, true);
+1
View File
@@ -1397,6 +1397,7 @@ class DBImpl : public DB {
protected:
const std::string dbname_;
const bool read_only_;
// TODO(peterd): unify with VersionSet::db_id_
std::string db_id_;
// db_session_id_ is an identifier that gets reset
+1 -1
View File
@@ -3223,7 +3223,7 @@ void DBImpl::ResumeAllCompactions() {
void DBImpl::MaybeScheduleFlushOrCompaction() {
mutex_.AssertHeld();
TEST_SYNC_POINT("DBImpl::MaybeScheduleFlushOrCompaction:Start");
if (!opened_successfully_) {
if (!opened_successfully_ || read_only_) {
// Compaction may introduce data race to DB open
return;
}
+35
View File
@@ -8089,6 +8089,41 @@ INSTANTIATE_TEST_CASE_P(OpenFilesAsync, OpenFilesAsyncTest,
::testing::Values(-1, 10),
::testing::Bool()));
TEST_F(DBTest, ReadOnlyCloseDoesNotDeleteWriterFiles) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.disable_auto_compactions = true;
DestroyAndReopen(options);
ASSERT_OK(Put("before", "value"));
ASSERT_OK(Flush());
std::unique_ptr<DB> read_only_db;
ASSERT_OK(DB::OpenForReadOnly(options, dbname_, &read_only_db));
ASSERT_OK(Put("after", "value"));
ASSERT_OK(Flush());
std::vector<LiveFileMetaData> live_files;
db_->GetLiveFilesMetaData(&live_files);
ASSERT_GE(live_files.size(), 2);
std::string newest_sst_path;
uint64_t newest_file_number = 0;
for (const auto& live_file : live_files) {
if (live_file.file_number > newest_file_number) {
newest_file_number = live_file.file_number;
newest_sst_path = live_file.directory + "/" + live_file.relative_filename;
}
}
ASSERT_FALSE(newest_sst_path.empty());
ASSERT_OK(env_->FileExists(newest_sst_path));
read_only_db.reset();
ASSERT_OK(env_->FileExists(newest_sst_path));
}
// Test mix of races with async file open, reads, compactions
TEST_P(OpenFilesAsyncTest, ConcurrentFileAccess) {
Options options = CurrentOptions();
+1 -1
View File
@@ -13,7 +13,7 @@
// minor or major version number planned for release.
#define ROCKSDB_MAJOR 11
#define ROCKSDB_MINOR 5
#define ROCKSDB_PATCH 0
#define ROCKSDB_PATCH 2
// Make it easy to do conditional compilation based on version checks, i.e.
// #if ROCKSDB_VERSION_GE(4, 5, 6)
@@ -1 +0,0 @@
The default `compression` for column families is now `kLZ4Compression` rather than `kSnappyCompression`. This affects only column families that do not explicitly set `compression`, and only newly-written SST files. The change is fully compatible: existing data remains readable with no migration needed, as RocksDB selects the decompressor per block. LZ4 offers slightly better compression ratios and decompression CPU efficiency vs. Snappy and similar compression CPU, tested across various server CPUs. When support is not compiled in, the fallback path for default compression is LZ4 -> Snappy -> NoCompression.
@@ -1 +0,0 @@
* Disable parallel compression (ignore `CompressionOptions::parallel_threads`) for fast built-in compressors: Snappy, LZ4 (accelerated, not LZ4HC), and accelerated levels of ZSTD (level < 0). These do not generally benefit from parallel compression. This behavior can be overridden with a custom Compressor from a custom CompressionManager.
@@ -1 +0,0 @@
`PosixFileSystem::OptimizeForCompactionTableRead` now delegates to the base `FileSystem::OptimizeForCompactionTableRead` implementation before applying its Linux-only compaction-readahead clamp (added for #12038). The returned `FileOptions::use_direct_reads` therefore reflects `DBOptions::use_direct_reads`, consistent with the base class and with other `FileSystem` implementations. Previously the override ignored the base implementation and keyed both the returned options and the Linux readahead clamp off the incoming `FileOptions` rather than `DBOptions`. At the in-tree call site the incoming `FileOptions::use_direct_reads` already matches `DBOptions::use_direct_reads`, so this is effectively a no-op for existing configurations; it removes a latent inconsistency where a caller passing `FileOptions` whose `use_direct_reads` disagreed with `DBOptions` would have had the global flag silently ignored for compaction-input reads on Linux. (Note: the `#12038` readahead clamp still keys off the global `use_direct_reads`; it is not skipped for the compaction-only `use_direct_io_for_compaction_reads` path, since that flag enables O_DIRECT after this hook runs.)
@@ -1 +0,0 @@
Brought more unity to `kLZ4Compression` and `kLZ4HCCompression` by giving each access to the other's compression levels. Negative (fast) values previously only available to LZ4 and positive (slow) values previously only available to LZ4HC are now available to both, so configuring a non-default `compression_opts.level` now selects the LZ4 compressor variant. This is a behavior change for previously tolerated but dubious configurations such as positive compression level with `kLZ4Compression`. `kLZ4Compression` and `kLZ4HCCompression` keep their effective default levels, equivalent to -1 and 9 respectively.
@@ -1 +0,0 @@
Removed a discontinuity in `kZSTD` compression levels at `compression_opts.level == 0` by mapping that setting to `level = -1` rather than to `level = 3` as the ZSTD library does internally. This improves the compression auto-tuning landscape.
@@ -1 +0,0 @@
Fixed a bug where the range lock manager would crash with an assertion failure when using a reverse comparator column family.
@@ -1 +0,0 @@
External table readers that open files through `ExternalTableOptions::fs` now update RocksDB SST/file-read statistics and file IO listener callbacks, making external file IO activity visible in existing read metrics.
@@ -1 +0,0 @@
Added `DB::PrepareFileIngestion()`, a two-phase form of `IngestExternalFile`/`IngestExternalFiles`. `PrepareFileIngestion()` performs all of the work that does not require the DB mutex (validating the arguments, reading each external file's metadata, and linking/copying the files into the DB) and returns an opaque `FileIngestionHandle`. `DB::CommitFileIngestionHandle()` (or `DB::CommitFileIngestionHandles()` for several handles at once) then makes the prepared files visible; multiple handles are committed atomically in a single MANIFEST write, or none are. `FileIngestionHandle::Abort()` (or simply destroying the handle) cancels a prepared ingestion and rolls it back. This gives flexibility to the application to Prepare a file separately from when its committed, and may shorten an applications critical path on `IngestExternalFile`.
@@ -1 +0,0 @@
Added two stats histograms reporting the per-call latency (in microseconds) of each successful `IngestExternalFile`/`IngestExternalFiles` call, split by phase: `rocksdb.ingest.external.file.prepare.micros` (argument validation and reading/validating/linking the external files, which does not block live writes) and `rocksdb.ingest.external.file.run.micros` (the ingestion performed under the DB mutex while live writes are blocked). Failed ingestions are not recorded.
@@ -1 +0,0 @@
Added `DBOptions::use_direct_io_for_compaction_reads` (default false). When enabled, compaction-input SST reads use `O_DIRECT` while user reads remain buffered, avoiding page-cache eviction of hot user-read data by sequential compaction scans. Pair with `use_direct_io_for_flush_and_compaction = true` on write-heavy workloads for direct I/O on both compaction inputs and outputs. Rejected at Open when combined with `allow_mmap_reads = true`. No-op when `use_direct_reads = true` is already set (since all reads are already direct).
@@ -1 +0,0 @@
Reduced commit latency when ingesting many external files by allowing `IngestExternalFileOptions::file_opening_threads` to open table readers for committed ingested files using multiple threads.
@@ -1 +0,0 @@
Reduced commit latency for large external file ingestions into the last level by adding `IngestExternalFileOptions::prefetch_lmax_index_and_filter_blocks`, which can skip commit-time index and filter block prefetching for cache-backed table metadata.
@@ -1 +0,0 @@
`DB::GetPreparedFileInfoForExternalSstIngestion()` can now prepare metadata for a live DB-generated SST file so it can be passed to `IngestExternalFileArg::file_infos`. The input path must exactly match a live table file owned by the source DB, and the returned metadata handle must outlive the ingestion.
@@ -1 +0,0 @@
`ExternalSstFileInfo` gained a `prepared_file_info` member produced by `SstFileWriter::Finish`, and `IngestExternalFileArg` gained a `file_infos` field: a vector of borrowed `const PreparedFileInfo*` pointers, parallel to `external_files`. The owning `ExternalSstFileInfo::prepared_file_info` handle must outlive the ingestion. When set, `IngestExternalFiles()` reuses the supplied per-file metadata instead of re-opening and scanning each file to recompute it, avoiding that extra I/O.
@@ -1 +0,0 @@
Corrected the public C++ option spelling from `memtable_veirfy_per_key_checksum_on_seek` to `memtable_verify_per_key_checksum_on_seek`. RocksDB continues to accept the old misspelled OPTIONS-file key for compatibility, while newly serialized OPTIONS files use the corrected key.
@@ -1 +0,0 @@
Added experimental read-scoped block buffer provider API, configured through `ReadOptions::read_scoped_block_buffer_provider`, for supported block-based table iterator scans and MultiScan reads to use caller-provided read-scoped storage for final data-block contents. When configured, supported provider-backed scan data-block reads bypass the data-block cache. The provider is ignored when mmap reads are enabled. RocksDB may still use ordinary temporary scratch for serialized block bytes, such as when a block may be compressed. Get and MultiGet do not currently provide API guarantees for this provider.
@@ -1 +0,0 @@
Added `FileSystem::SyncFile()` and `Env::SyncFile()` public APIs for syncing or fsyncing a file by name without requiring callers to reopen it as writable.
@@ -102,9 +102,14 @@ TEST_F(RangeLockingTest, BasicRangeLocking) {
delete txn1;
}
// CompareDbtEndpoints must honor ReverseBytewiseComparator direction
// in length-disparity case.
TEST_F(RangeLockingTest, RangeLockWithReverseComparator) {
// A point-equality range lock passes [infimum(K), supremum(K)]: the same user
// key with the infimum endpoint as the left bound and the supremum endpoint as
// the right bound. The infimum endpoint must sort before the supremum endpoint
// even under a reverse comparator (the suffix marker is positional, not key
// content), otherwise the locktree's left <= right invariant is violated and
// try_acquire_lock aborts. Regression test for a crash on MyRocks equality
// lookups on a reverse ('rev:') column family.
TEST_F(RangeLockingTest, PointRangeLockWithReverseComparator) {
delete db;
db = nullptr;
ASSERT_OK(DestroyDB(dbname, options));
@@ -121,19 +126,12 @@ TEST_F(RangeLockingTest, RangeLockWithReverseComparator) {
auto cf = db->DefaultColumnFamily();
Transaction* txn0 = db->BeginTransaction(write_options, txn_options);
Transaction* txn1 = db->BeginTransaction(write_options, txn_options);
ASSERT_OK(txn0->GetRangeLock(cf, Endpoint("aa"), Endpoint("a")));
auto s = txn1->GetRangeLock(cf, Endpoint("ab"), Endpoint("a"));
ASSERT_TRUE(s.IsTimedOut());
ASSERT_OK(txn1->GetRangeLock(cf, Endpoint("b"), Endpoint("b")));
// [infimum("a"), supremum("a")] : same user key, infimum -> supremum.
ASSERT_OK(txn0->GetRangeLock(cf, Endpoint("a", false), Endpoint("a", true)));
txn0->Rollback();
txn1->Rollback();
delete txn0;
delete txn1;
}
// CompareDbtEndpoints must use CompareWithoutTimestamp for range lock
@@ -39,18 +39,6 @@ RangeLockManagerHandle* NewRangeLockManager(
static const char SUFFIX_INFIMUM = 0x0;
static const char SUFFIX_SUPREMUM = 0x1;
struct RangeTreeCmpContext {
const Comparator* cmp;
bool is_reverse;
};
namespace {
[[nodiscard]] bool IsReverseComparator(const Comparator* cmp) {
return cmp == ReverseBytewiseComparator() ||
cmp == ReverseBytewiseComparatorWithU64Ts();
}
} // namespace
// Convert Endpoint into an internal format used for storing it in locktree
// (DBT structure is used for passing endpoints to locktree and getting back)
void serialize_endpoint(const Endpoint& endp, std::string* buf) {
@@ -213,6 +201,20 @@ void RangeTreeLockManager::UnLock(PessimisticTransaction* txn,
((RangeTreeLockTracker*)range_tracker)->ReleaseLocks(this, txn, all_keys);
}
// Comparator for range-lock endpoints. This is intentionally
// direction-AGNOSTIC: it orders the user-key bytes with the column family's own
// comparator, then breaks ties on the endpoint-type suffix byte (infimum before
// supremum), which is a positional marker independent of the CF's sort
// direction.
//
// CALLER REQUIREMENT: it is the caller's responsibility to pass in a flipped
// (swapped start/end, with negated infimum/supremum flags) range for RangeLock
// to work on a REVERSE-ORDERED column family. The locktree requires every range
// to satisfy left <= right, and this comparator does NOT compensate for reverse
// ordering on the caller's behalf. MyRocks does this flip today in
// ha_rocksdb::set_range_lock() (see its "RangeFlagsShouldBeFlippedForRevCF"
// comment). Do not re-add reverse-awareness here (that double-corrects callers
// like MyRocks that already flip).
int RangeTreeLockManager::CompareDbtEndpoints(void* arg, const DBT* a_key,
const DBT* b_key) {
const char* a = (const char*)a_key->data;
@@ -225,24 +227,26 @@ int RangeTreeLockManager::CompareDbtEndpoints(void* arg, const DBT* a_key,
// Compare the values. The first byte encodes the endpoint type, its value
// is either SUFFIX_INFIMUM or SUFFIX_SUPREMUM.
const auto* ctx = static_cast<const RangeTreeCmpContext*>(arg);
const auto* cmp = ctx->cmp;
const bool is_reverse = ctx->is_reverse;
Comparator* cmp = (Comparator*)arg;
int res = cmp->CompareWithoutTimestamp(
Slice(a + 1, min_len - 1), /*a_has_ts=*/false, Slice(b + 1, min_len - 1),
/*b_has_ts=*/false);
if (!res) {
// The user keys compared equal above, so order by the endpoint-type suffix
// byte (a positional boundary marker: infimum before the key, supremum
// after) -- direction-independent; see the function-level comment.
if (b_len > min_len) {
int r = a[0] == SUFFIX_INFIMUM ? -1 : 1;
return is_reverse ? -r : r;
// a's user key is a prefix of b's; a is shorter.
return a[0] == SUFFIX_INFIMUM ? -1 : 1;
} else if (a_len > min_len) {
int r = b[0] == SUFFIX_INFIMUM ? 1 : -1;
return is_reverse ? -r : r;
// b's user key is a prefix of a's; b is shorter.
return b[0] == SUFFIX_INFIMUM ? 1 : -1;
} else {
// Same user key: infimum sorts before supremum.
if (a[0] < b[0]) {
return is_reverse ? 1 : -1;
return -1;
} else if (a[0] > b[0]) {
return is_reverse ? -1 : 1;
return 1;
} else {
return 0;
}
@@ -365,11 +369,10 @@ RangeLockManagerHandle::Counters RangeTreeLockManager::GetStatus() {
}
std::shared_ptr<toku::locktree> RangeTreeLockManager::MakeLockTreePtr(
toku::locktree* lt, std::unique_ptr<RangeTreeCmpContext> ctx) {
toku::locktree* lt) {
toku::locktree_manager* ltm = &ltm_;
return std::shared_ptr<toku::locktree>(
lt,
[ltm, ctx = std::move(ctx)](toku::locktree* p) { ltm->release_lt(p); });
lt, [ltm](toku::locktree* p) { ltm->release_lt(p); });
}
void RangeTreeLockManager::AddColumnFamily(const ColumnFamilyHandle* cfh) {
@@ -378,18 +381,15 @@ void RangeTreeLockManager::AddColumnFamily(const ColumnFamilyHandle* cfh) {
InstrumentedMutexLock l(&ltree_map_mutex_);
if (ltree_map_.find(column_family_id) == ltree_map_.end()) {
DICTIONARY_ID dict_id = {.dictid = column_family_id};
auto ctx = std::make_unique<RangeTreeCmpContext>(RangeTreeCmpContext{
cfh->GetComparator(), IsReverseComparator(cfh->GetComparator())});
toku::comparator cmp;
cmp.create(CompareDbtEndpoints, ctx.get());
cmp.create(CompareDbtEndpoints, (void*)cfh->GetComparator());
toku::locktree* ltree =
ltm_.get_lt(dict_id, cmp,
/* on_create_extra*/ static_cast<void*>(this));
// This is ok to because get_lt has copied the comparator:
cmp.destroy();
ltree_map_.insert(
{column_family_id, MakeLockTreePtr(ltree, std::move(ctx))});
ltree_map_.insert({column_family_id, MakeLockTreePtr(ltree)});
}
}
@@ -20,8 +20,6 @@ namespace ROCKSDB_NAMESPACE {
typedef DeadlockInfoBufferTempl<RangeDeadlockPath> RangeDeadlockInfoBuffer;
struct RangeTreeCmpContext;
// A Range Lock Manager that uses PerconaFT's locktree library
class RangeTreeLockManager : public RangeLockManagerBase,
public RangeLockManagerHandle {
@@ -118,8 +116,7 @@ class RangeTreeLockManager : public RangeLockManagerBase,
RangeDeadlockInfoBuffer dlock_buffer_;
std::shared_ptr<toku::locktree> MakeLockTreePtr(
toku::locktree* lt, std::unique_ptr<RangeTreeCmpContext> ctx = nullptr);
std::shared_ptr<toku::locktree> MakeLockTreePtr(toku::locktree* lt);
static int CompareDbtEndpoints(void* arg, const DBT* a_key, const DBT* b_key);
// Callbacks