Update for 10.9.1 patch

Fix resumable compaction to prevent resumption at truncated range deletion boundaries (#14184 )
Summary: **Context/Summary:** Truncated range deletion in input files can be output by CompactionIterator with type kMaxValid instead of kTypeRangeDeletion, to satisfy ordering requirement between the truncated range deletion start key and a file's point keys. There was a plan to skip such key in https://github.com/facebook/rocksdb/pull/14122 but blockers remain to fulfill the plan. Resumable compaction is not able to handle resumption from range deletion well at this point and should consider kMaxValid type same as kTypeRangeDeletion for resumption. Previously, it didn't and mistakenly allow resumption from a delete range. That led to an assertion failure, complaining about lacking information to update file boundaries in the presence of range deletion needed during cutting an output file, after the compaction resumes from that delete range and happens to cut the output file shortly after without any point keys in between. ``` frame https://github.com/facebook/rocksdb/issues/9: 0x00007f4f4743bc93 libc.so.6`__GI___assert_fail(assertion="meta.smallest.size() > 0", file="db/compaction/compaction_outputs.cc", line=530, function="rocksdb::Status rocksdb::CompactionOutputs::AddRangeDels(rocksdb::CompactionRangeDelAggregator&, const rocksdb::Slice*, const rocksdb::Slice*, rocksdb::CompactionIterationStats&, bool, const rocksdb::InternalKeyComparator&, rocksdb::SequenceNumber, std::pair<long unsigned int, long unsigned int>, const rocksdb::Slice&, const string&)") at assert.c:101:3 frame https://github.com/facebook/rocksdb/issues/10: 0x00007f4f4808c68c librocksdb.so.10.9`rocksdb::CompactionOutputs::AddRangeDels(this=0x00007f4f0c27e1a0, range_del_agg=0x00007f4f0c21ecc0, comp_start_user_key=0x0000000000000000, comp_end_user_key=0x0000000000000000, range_del_out_stats=0x00007f4f0dffa140, bottommost_level=false, icmp=0x00007f4ef4c93040, earliest_snapshot=13108729, keep_seqno_range=<unavailable>, next_table_min_key=0x00007f4ef4c8f540, full_history_ts_low="") at compaction_outputs.cc:530:7 frame https://github.com/facebook/rocksdb/issues/11: 0x00007f4f480480dd librocksdb.so.10.9`rocksdb::CompactionJob::FinishCompactionOutputFile(this=0x00007f4f0dffb890, input_status=<unavailable>, prev_table_last_internal_key=0x00007f4f0dffa650, next_table_min_key=0x00007f4ef4c8f540, comp_start_user_key=0x0000000000000000, comp_end_user_key=0x0000000000000000, c_iter=0x00007f4ef4c8f400, sub_compact=0x00007f4f0c27e000, outputs=0x00007f4f0c27e1a0) at compaction_job.cc:1917:31 ``` This PR simply prevents MaxValid from being a resumption point like regular range deletion - see commit 842d66eb18ea67e965d6acb1fce12c18eeb778d2 Besides that, the PR also improves the testing, variable naming, logging in resumable compaction codes that were needed to debug this assertion failure - see commit https://github.com/facebook/rocksdb/pull/14184/commits/aecd4e7f971f6dd4df672d9e5f1409fe4747c561. These improvements are covered by existing tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14184 Test Plan: - The stress initially surfaced the error. Using the exact same LSM shapes and files that were used in stress test but in a unit test, I'm able to get a deterministic repro and confirmed the fix resolves the error. This is the repro test https://github.com/hx235/rocksdb/commit/1075936e693c68c960761855900c53f5b894f57a ``` ./compaction_service_test --gtest_filter=ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume # Before fix [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from ResumableCompactionServiceTest [ RUN ] ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume compaction_service_test: db/compaction/compaction_outputs.cc:530: rocksdb::Status rocksdb::CompactionOutputs::AddRangeDels(rocksdb::CompactionRangeDelAggregator&, const rocksdb::Slice*, const rocksdb::Slice*, rocksdb::CompactionIterationStats&, bool, const rocksdb::InternalKeyComparator&, rocksdb::SequenceNumber, std::pair<long unsigned int, long unsigned int>, const rocksdb::Slice&, const string&): Assertion `meta.smallest.size() > 0' failed. Received signal 6 (Aborted) Invoking GDB for stack trace... [New LWP 2621610] [New LWP 2621611] [New LWP 2621612] [New LWP 2621613] [New LWP 2621614] [New LWP 2621630] [New LWP 2621631] # After fix Note: Google Test filter = ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from ResumableCompactionServiceTest [ RUN ] ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume [ OK ] ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume (4722 ms) [----------] 1 test from ResumableCompactionServiceTest (4722 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (4722 ms total) [ PASSED ] 1 test. ``` - Follow-up: I tried a couple time to coerce the truncated range delete from scratch in the unit test but failed doing so. Considering kMaxValid may not be outputted by compaction iterator anymore after https://github.com/facebook/rocksdb/pull/14122/files gets landed again (and obsolete the bug) ADN the simple nature of this fix 842d66eb18ea67e965d6acb1fce12c18eeb778d2 AND the worst case of such fix going wrong is just less resumption, I decided to leave writing a unit test to coerce truncated ranged deletion from scratch a follow-up. Maybe I will draw inspiration from https://github.com/facebook/rocksdb/pull/14122/files. Reviewed By: jaykorean Differential Revision: D88912663 Pulled By: hx235 fbshipit-source-id: 80a01135684c8fea659650faaa00c2dc452c482a
2026-07-07 14:47:40 +08:00 · 2025-12-11 19:04:28 -08:00 · 2025-12-11 18:39:27 -08:00 · 2025-12-04 19:06:26 -08:00 · 2025-12-04 13:30:39 -08:00 · 2025-11-24 13:40:36 -08:00
41 changed files with 298 additions and 308 deletions
@@ -1,6 +1,33 @@
 # Rocksdb Change Log
 > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`

+## 10.9.1 (12/11/2025)
+### Bug Fixes
+* Fix resumable compaction incorrectly allowing resumption from a truncated range deletion that is not well handled currently.
+
+## 10.9.0 (11/21/2025)
+### New Features
+* Added an auto-tuning feature for DB manifest file size that also (by default) improves the safety of existing configurations in case `max_manifest_file_size` is repeatedly exceeded. The new recommendation is to set `max_manifest_file_size` to something small like 1MB and tune `max_manifest_space_amp_pct` as needed to balance write amp and space amp in the manifest. Refer to comments on those options in `DBOptions` for details. Both options are (now) mutable.
+* Added a new API to support option migration for multiple column families
+* Added new option target_file_size_is_upper_bound  that makes most compaction output SST files come close to the target file size without exceeding it, rather than commonly exceeding it by some fraction (current behavior). For now the new behavior is off by default, but we expect to enable it by default in the future.
+* Add a new option allow_trivial_move in CompactionOptions to allow CompactFiles to perform trivial move if possible. By default the flag of allow_trivial_move is false, so it preserve the original behavior.
+
+### Public API Changes
+* To reduce risk of ODR violations or similar, `ROCKSDB_USING_THREAD_STATUS` has been removed from public headers and replaced with static `const bool ThreadStatus::kEnabled`. Some other uses of conditional compilation have been removed from public API headers to reduce risk of ODR violations or other issues.
+
+### Behavior Changes
+* PosixWritableFile now repositions the seek pointer to the new end of file after a call to Truncate.
+* Updated standalone range deletion L0 file compaction behavior to avoid compacting with any newer L0 files (which is expensive and not useful).
+
+### Bug Fixes
+* Fix a bug where compaction with range deletion can persist kTypeMaxValid in MANIFEST as file metadata. kTypeMaxValid is not supposed to be persisted and can change as new value types are introduced. This can cause a forward compatibility issue where older versions of RocksDB don't recognize kTypeMaxValid from newer versions. A new placeholder value type kTypeTruncatedRangeDeletionSentinel is also introduced to replace kTypeMaxValid when reading existing SST files' metadata from MANIFEST. This allows us to strengthen some checks to avoid using kTypeMaxValid in the future.
+* Fixed a bug where `DB::GetSortedWalFiles()` could hang when waiting for a purge operation that found nothing to do (potentially triggered by iterator release, flush, compaction, etc.).
+* Fixed a bug in MultiScan where `max_sequential_skip_in_iterations` could cause the iterator to seek backward to already-unpinned blocks when the same user key spans multiple data blocks, leading to assertion failures or seg fault.
+* Fixed a bug for `WAL_ttl_seconds > 0` use cases where the newest archived WAL files could be incorrectly deleted when the system clock moved backwards.
+
+### Performance Improvements
+* Added optimization that allowed for the asynchronous prefetching of all data outlined in a multiscan iterator. This optimization was applied to the level iterator, which prefetches all data through each of the block-based iterators.
+
 ## 10.8.0 (10/21/2025)
 ### New Features
 * Add kFSPrefetch to FSSupportedOps enum to allow file systems to indicate prefetch support capability, avoiding unnecessary prefetch system calls on file systems that don't support them.
@@ -1480,11 +1480,11 @@ CompactionJob::CreateFileHandlers(SubcompactionState* sub_compact,
  const CompactionFileCloseFunc close_file_func =
      [this, sub_compact, start_user_key, end_user_key](
          const Status& status,
-          const ParsedInternalKey& prev_table_last_internal_key,
+          const ParsedInternalKey& prev_iter_output_internal_key,
          const Slice& next_table_min_key, const CompactionIterator* c_iter,
          CompactionOutputs& outputs) {
        return this->FinishCompactionOutputFile(
-            status, prev_table_last_internal_key, next_table_min_key,
+            status, prev_iter_output_internal_key, next_table_min_key,
            start_user_key, end_user_key, c_iter, sub_compact, outputs);
      };

@@ -1499,8 +1499,8 @@ Status CompactionJob::ProcessKeyValue(
  const uint64_t kRecordStatsEvery = 1000;
  [[maybe_unused]] const std::optional<const Slice> end = sub_compact->end;

-  IterKey last_output_key;
-  ParsedInternalKey last_output_ikey;
+  IterKey prev_iter_output_key;
+  ParsedInternalKey prev_iter_output_internal_key;

  TEST_SYNC_POINT_CALLBACK(
      "CompactionJob::ProcessKeyValueCompaction()::Processing",
@@ -1551,9 +1551,9 @@ Status CompactionJob::ProcessKeyValue(
    // and `close_file_func`.
    // TODO: it would be better to have the compaction file open/close moved
    // into `CompactionOutputs` which has the output file information.
-    status =
-        sub_compact->AddToOutput(*c_iter, use_proximal_output, open_file_func,
-                                 close_file_func, last_output_ikey);
+    status = sub_compact->AddToOutput(*c_iter, use_proximal_output,
+                                      open_file_func, close_file_func,
+                                      prev_iter_output_internal_key);
    if (!status.ok()) {
      break;
    }
@@ -1562,9 +1562,10 @@ Status CompactionJob::ProcessKeyValue(
                             static_cast<void*>(const_cast<std::atomic<bool>*>(
                                 &manual_compaction_canceled_)));

-    last_output_key.SetInternalKey(c_iter->key(), &last_output_ikey);
-    last_output_ikey.sequence = ikey.sequence;
-    last_output_ikey.type = ikey.type;
+    prev_iter_output_key.SetInternalKey(c_iter->key(),
+                                        &prev_iter_output_internal_key);
+    prev_iter_output_internal_key.sequence = ikey.sequence;
+    prev_iter_output_internal_key.type = ikey.type;
    c_iter->Next();

 #ifndef NDEBUG
@@ -1871,7 +1872,7 @@ void CompactionJob::RecordDroppedKeys(

 Status CompactionJob::FinishCompactionOutputFile(
    const Status& input_status,
-    const ParsedInternalKey& prev_table_last_internal_key,
+    const ParsedInternalKey& prev_iter_output_internal_key,
    const Slice& next_table_min_key, const Slice* comp_start_user_key,
    const Slice* comp_end_user_key, const CompactionIterator* c_iter,
    SubcompactionState* sub_compact, CompactionOutputs& outputs) {
@@ -2049,7 +2050,7 @@ Status CompactionJob::FinishCompactionOutputFile(
  }

  if (s.ok() && ShouldUpdateSubcompactionProgress(sub_compact, c_iter,
-                                                  prev_table_last_internal_key,
+                                                  prev_iter_output_internal_key,
                                                  next_table_min_key, meta)) {
    UpdateSubcompactionProgress(c_iter, next_table_min_key, sub_compact);
    s = PersistSubcompactionProgress(sub_compact);
@@ -2060,10 +2061,10 @@ Status CompactionJob::FinishCompactionOutputFile(

 bool CompactionJob::ShouldUpdateSubcompactionProgress(
    const SubcompactionState* sub_compact, const CompactionIterator* c_iter,
-    const ParsedInternalKey& prev_table_last_internal_key,
+    const ParsedInternalKey& prev_iter_output_internal_key,
    const Slice& next_table_min_internal_key, const FileMetaData* meta) const {
  const auto* cfd = sub_compact->compaction->column_family_data();
-  // No need to update when the output will not get persisted
+  // No need to update when the progress will not get persisted
  if (compaction_progress_writer_ == nullptr) {
    return false;
  }
@@ -2087,19 +2088,24 @@ bool CompactionJob::ShouldUpdateSubcompactionProgress(
  }

  // LIMITATION: Compaction progress persistence disabled for file boundaries
-  // contaning range deletions. Range deletions can span file boundaries, making
-  // it difficult (but possible) to ensure adjacent output tables have different
-  // user keys. See the last check for why different users keys of adjacent
-  // output tables are needed
+  // containing range deletions. Range deletions can span file boundaries,
+  // making it difficult to ensure adjacent output tables have different user
+  // keys. See the last check for why different users keys of adjacent output
+  // tables are needed
  const ValueType next_table_min_internal_key_type =
      ExtractValueType(next_table_min_internal_key);
-  const ValueType prev_table_last_internal_key_type =
-      prev_table_last_internal_key.user_key.empty()
+  const ValueType prev_iter_output_internal_key_type =
+      prev_iter_output_internal_key.user_key.empty()
          ? ValueType::kTypeValue
-          : prev_table_last_internal_key.type;
+          : prev_iter_output_internal_key.type;

-  if (next_table_min_internal_key_type == ValueType::kTypeRangeDeletion ||
-      prev_table_last_internal_key_type == ValueType::kTypeRangeDeletion) {
+  // Range deletes truncated to align with file boundaries may be output by the
+  // compaction iterator with `ValueType::kTypeMaxValid` instead of the original
+  // type.
+  if ((next_table_min_internal_key_type == ValueType::kTypeRangeDeletion ||
+       next_table_min_internal_key_type == ValueType::kTypeMaxValid) ||
+      (prev_iter_output_internal_key_type == ValueType::kTypeRangeDeletion ||
+       prev_iter_output_internal_key_type == ValueType::kTypeMaxValid)) {
    return false;
  }

@@ -2109,9 +2115,9 @@ bool CompactionJob::ShouldUpdateSubcompactionProgress(
  const Slice next_table_min_user_key =
      ExtractUserKey(next_table_min_internal_key);
  const Slice prev_table_last_user_key =
-      prev_table_last_internal_key.user_key.empty()
+      prev_iter_output_internal_key.user_key.empty()
          ? Slice()
-          : prev_table_last_internal_key.user_key;
+          : prev_iter_output_internal_key.user_key;

  if (cfd->user_comparator()->EqualWithoutTimestamp(next_table_min_user_key,
                                                    prev_table_last_user_key)) {
@@ -430,7 +430,7 @@ class CompactionJob {

  Status FinishCompactionOutputFile(
      const Status& input_status,
-      const ParsedInternalKey& prev_table_last_internal_key,
+      const ParsedInternalKey& prev_iter_output_internal_key,
      const Slice& next_table_min_key, const Slice* comp_start_user_key,
      const Slice* comp_end_user_key, const CompactionIterator* c_iter,
      SubcompactionState* sub_compact, CompactionOutputs& outputs);
@@ -545,7 +545,7 @@ class CompactionJob {

  bool ShouldUpdateSubcompactionProgress(
      const SubcompactionState* sub_compact, const CompactionIterator* c_iter,
-      const ParsedInternalKey& prev_table_last_internal_key,
+      const ParsedInternalKey& prev_iter_output_internal_key,
      const Slice& next_table_min_internal_key, const FileMetaData* meta) const;

  void UpdateSubcompactionProgress(const CompactionIterator* c_iter,
@@ -2424,6 +2424,7 @@ class ResumableCompactionJobTest : public CompactionJobTestBase {
  bool enable_cancel_ = false;
  std::atomic<int> stop_count_{0};
  std::atomic<bool> cancel_{false};
+  SequenceNumber cancel_before_seqno = kMaxSequenceNumber;

  void SetUp() override {
    CompactionJobTestBase::SetUp();
@@ -2437,7 +2438,9 @@ class ResumableCompactionJobTest : public CompactionJobTestBase {
          if (enable_cancel_) {
            ParsedInternalKey parsed_key;
            if (ParseInternalKey(pair->second, &parsed_key, true).ok()) {
-              if (parsed_key.user_key == kCancelBeforeThisKey) {
+              if (parsed_key.user_key == kCancelBeforeThisKey &&
+                  (cancel_before_seqno == kMaxSequenceNumber ||
+                   parsed_key.sequence == cancel_before_seqno)) {
                cancel_.store(true);
              }
            }
@@ -2665,7 +2668,7 @@ class ResumableCompactionJobTest : public CompactionJobTestBase {
      const std::initializer_list<mock::KVPair>& input_file_2,
      uint64_t last_sequence, const std::vector<uint64_t>& snapshots,
      const std::string& expected_next_key_to_compact,
-      const std::vector<std::string>& expected_input_keys, bool exists_progress,
+      const std::vector<std::string>& expected_input_keys,
      bool cancelled_past_mid_point = false) {
    std::shared_ptr<Statistics> stats = ROCKSDB_NAMESPACE::CreateDBStatistics();

@@ -2701,7 +2704,7 @@ class ResumableCompactionJobTest : public CompactionJobTestBase {

    // Resume compaction
    CompactionProgress compaction_progress;
-    if (exists_progress) {
+    if (expected_next_key_to_compact != "") {
      compaction_progress.push_back(
          ReadAndParseProgress(compaction_progress_file));
    }
@@ -2774,35 +2777,43 @@ TEST_F(ResumableCompactionJobTest, BasicProgressResume) {
      4U /* last_sequence */, {} /* snapshots */,
      kCancelBeforeThisKey /* expected_next_key_to_compact */,
      {"a", "b", "bb", kCancelBeforeThisKey} /* expected_input_keys */,
-      true /* exists_progress */, true /* cancelled_past_mid_point*/);
+      true /* cancelled_past_mid_point */);
 }

 TEST_F(ResumableCompactionJobTest, NoProgressResumeOnSameKey) {
  NewDB();

+  // `cancel_before_seqno` is set to 0U to force cancellation after
+  // `kCancelBeforeThisKey@1` instead of `kCancelBeforeThisKey@2`.
+  // The seqno is 0 because `kCancelBeforeThisKey@1` will have its sequence
+  // number zeroed during compaction while `kCancelBeforeThisKey@2` won't be
+  cancel_before_seqno = 0U;
  RunCancelAndResumeTest(
      {{KeyStr(kCancelBeforeThisKey, 1U, kTypeValue),
        "val1"}} /* input_file_1 */,
-      {{KeyStr(kCancelBeforeThisKey, 2U, kTypeValue),
-        "val2"}} /* input_file_2 */,
-      2U /* last_sequence */, {1U} /* snapshots */,
+      {{KeyStr(kCancelBeforeThisKey, 2U, kTypeValue), "val11"},
+       {KeyStr("d", 3U, kTypeValue), "val2"}} /* input_file_2 */,
+      3U /* last_sequence */, {1U} /* snapshots */,
      "" /* expected_next_key_to_compact */,
-      {kCancelBeforeThisKey, kCancelBeforeThisKey} /* expected_input_keys */,
-      false /* exists_progress */);
+      {kCancelBeforeThisKey, kCancelBeforeThisKey,
+       "d"} /* expected_input_keys */);
 }

 TEST_F(ResumableCompactionJobTest, NoProgressResumeOnDeleteRange) {
  NewDB();

  RunCancelAndResumeTest(
-      {{KeyStr(kCancelBeforeThisKey, 1U, kTypeValue),
-        "val1"}} /* input_file_1 */,
-      {{KeyStr(kCancelBeforeThisKey, 2U, kTypeRangeDeletion),
-        "val2"}} /* input_file_2 */,
-      2U /* last_sequence */, {1U} /* snapshots */,
-      "" /* expected_next_key_to_compact */,
-      {kCancelBeforeThisKey, kCancelBeforeThisKey} /* expected_input_keys */,
-      false /* exists_progress */);
+      {{KeyStr("a", 1U, kTypeValue), "val1"},
+       {KeyStr("b", 2U, kTypeValue), "val2"},
+       {KeyStr(kCancelBeforeThisKey, 3U, kTypeValue),
+        "val3"}} /* input_file_1 */,
+      {{KeyStr(kCancelBeforeThisKey, 4U, kTypeRangeDeletion),
+        "range_deletion_end_key"},
+       {KeyStr("d", 5U, kTypeValue), "val4"}} /* input_file_2 */,
+      5U /* last_sequence */, {3U} /* snapshots */,
+      "b" /* expected_next_key_to_compact */,
+      {"a", "b", kCancelBeforeThisKey, kCancelBeforeThisKey,
+       "d"} /* expected_input_keys */);
 }

 TEST_F(ResumableCompactionJobTest, NoProgressResumeOnMerge) {
@@ -2817,8 +2828,7 @@ TEST_F(ResumableCompactionJobTest, NoProgressResumeOnMerge) {
        "val4"}} /* input_file_2 */,
      4U /* last_sequence */, {} /* snapshots */,
      "bb" /* expected_next_key_to_compact */,
-      {"a", "b", "bb", kCancelBeforeThisKey} /* expected_input_keys */,
-      true /* exists_progress */);
+      {"a", "b", "bb", kCancelBeforeThisKey} /* expected_input_keys */);
 }

 TEST_F(ResumableCompactionJobTest, NoProgressResumeOnSingleDelete) {
@@ -2834,8 +2844,7 @@ TEST_F(ResumableCompactionJobTest, NoProgressResumeOnSingleDelete) {
      5U /* last_sequence */, {3U} /* snapshots */,
      "b" /* expected_next_key_to_compact */,
      {"a", "b", kCancelBeforeThisKey, kCancelBeforeThisKey,
-       "d"} /* expected_input_keys */,
-      true /* exists_progress */);
+       "d"} /* expected_input_keys */);
 }

 TEST_F(ResumableCompactionJobTest, NoProgressResumeOnDeletionAtBottom) {
@@ -2851,8 +2860,7 @@ TEST_F(ResumableCompactionJobTest, NoProgressResumeOnDeletionAtBottom) {
      5U /* last_sequence */, {3U} /* snapshots */,
      "b" /* expected_next_key_to_compact */,
      {"a", "b", kCancelBeforeThisKey, kCancelBeforeThisKey,
-       "d"} /* expected_input_keys */,
-      true /* exists_progress */);
+       "d"} /* expected_input_keys */);
 }
 }  // namespace ROCKSDB_NAMESPACE

@@ -364,7 +364,7 @@ Status CompactionOutputs::AddToOutput(
    const CompactionIterator& c_iter,
    const CompactionFileOpenFunc& open_file_func,
    const CompactionFileCloseFunc& close_file_func,
-    const ParsedInternalKey& prev_table_last_internal_key) {
+    const ParsedInternalKey& prev_iter_output_internal_key) {
  Status s;
  bool is_range_del = c_iter.IsDeleteRangeSentinelKey();
  if (is_range_del && compaction_->bottommost_level()) {
@@ -375,8 +375,8 @@ Status CompactionOutputs::AddToOutput(
  }
  const Slice& key = c_iter.key();
  if (ShouldStopBefore(c_iter) && HasBuilder()) {
-    s = close_file_func(c_iter.InputStatus(), prev_table_last_internal_key, key,
-                        &c_iter, *this);
+    s = close_file_func(c_iter.InputStatus(), prev_iter_output_internal_key,
+                        key, &c_iter, *this);
    if (!s.ok()) {
      return s;
    }
@@ -262,7 +262,7 @@ class CompactionOutputs {
  Status AddToOutput(const CompactionIterator& c_iter,
                     const CompactionFileOpenFunc& open_file_func,
                     const CompactionFileCloseFunc& close_file_func,
-                     const ParsedInternalKey& prev_table_last_internal_key);
+                     const ParsedInternalKey& prev_iter_output_internal_key);

  // Close the current output. `open_file_func` is needed for creating new file
  // for range-dels only output file.
@@ -109,12 +109,12 @@ Status SubcompactionState::AddToOutput(
    const CompactionIterator& iter, bool use_proximal_output,
    const CompactionFileOpenFunc& open_file_func,
    const CompactionFileCloseFunc& close_file_func,
-    const ParsedInternalKey& prev_table_last_internal_key) {
+    const ParsedInternalKey& prev_iter_output_internal_key) {
  // update target output
  current_outputs_ =
      use_proximal_output ? &proximal_level_outputs_ : &compaction_outputs_;
  return current_outputs_->AddToOutput(iter, open_file_func, close_file_func,
-                                       prev_table_last_internal_key);
+                                       prev_iter_output_internal_key);
 }

 }  // namespace ROCKSDB_NAMESPACE
@@ -221,7 +221,7 @@ class SubcompactionState {
  Status AddToOutput(const CompactionIterator& iter, bool use_proximal_output,
                     const CompactionFileOpenFunc& open_file_func,
                     const CompactionFileCloseFunc& close_file_func,
-                     const ParsedInternalKey& prev_table_last_internal_key);
+                     const ParsedInternalKey& prev_iter_output_internal_key);

  // Close all compaction output files, both output_to_proximal_level outputs
  // and normal outputs.
@@ -1102,11 +1102,6 @@ Status DBImplSecondary::InitializeCompactionWorkspace(
    return s;
  }

-  ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                 "Initialized compaction workspace with %zu subcompaction "
-                 "progress to resume",
-                 compaction_progress_.size());
-
  return Status::OK();
 }

@@ -1219,6 +1214,11 @@ Status DBImplSecondary::PrepareCompactionProgressState() {
      return HandleInvalidOrNoCompactionProgress(compaction_progress_file_path,
                                                 scan_result);
    }
+
+    ROCKS_LOG_DEBUG(
+        immutable_db_options_.info_log,
+        "Loaded compaction progress with %zu subcompaction(s) from %s",
+        compaction_progress_.size(), compaction_progress_file_path.c_str());
    return s;
  } else {
    return HandleInvalidOrNoCompactionProgress(
@@ -1740,6 +1740,11 @@ Status DBImplSecondary::FinalizeCompactionProgressWriter(
    return HandleCompactionProgressWriterCreationFailure(
        "" /* temp_file_path */, final_file_path, compaction_progress_writer);
  }
+
+  ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
+                  "Finalized compaction progress writer onto %s",
+                  final_file_path.c_str());
+
  return Status::OK();
 }
 }  // namespace ROCKSDB_NAMESPACE
@@ -3826,133 +3826,88 @@ TEST_F(DBRangeDelTest, RowCache) {
  ASSERT_OK(Put(Key(5), "foo"));
 }

-TEST_F(DBRangeDelTest, FileCutWithTruncatedRangeDelKey) {
-  // Test for a bug that used to generate files with meta.smallest
-  // containing kMaxValid.
-  //
-  // Setup:
-  // - Write Key(2), Key(3) and DeleteRange(Key(1), Key(4))
-  // - Flush to L0
-  // - Use SingleKeySstPartitioner to force each user key into its own file
-  // - Compact files from L0 to L1 will generate files
-  // File[0]:
-  //   smallest=[user_key=key000001, seq=4, type=15],
-  //   largest= [user_key=key000002, seq=72057594037927935, type=15]
-  // File[1]:
-  //   smallest=[user_key=key000002, seq=2, type=1],
-  //   largest= [user_key=key000003, seq=72057594037927935, type=15]
-  // File[2]:
-  //   smallest=[user_key=key000003, seq=3, type=1],
-  //   largest= [user_key=key000004, seq=72057594037927935, type=15]
-  // With range deletions truncated to each files key range.
-  //
-  // - Compact these files again into L2. RocksDB usede to set truncated
-  // range deletion start key to have value type kMaxValid. The range deletion
-  // start key is used in compaction file cutting decision.
-  // - Verify the file boundary keys after compaction have valid boundary keys
-  //
-  // Before the fix:
-  // File[0]:
-  //   smallest=[user_key=key000001, seq=4, type=15],
-  //   largest= [user_key=key000002, seq=72057594037927935, type=15]
-  // File[1]:
-  //   smallest=[user_key=key000002, seq=2, type=26],
-  //   largest= [user_key=key000003, seq=72057594037927935, type=15]
-  // File[2]:
-  //   smallest=[user_key=key000003, seq=3, type=26],
-  //   largest= [user_key=key000004, seq=72057594037927935, type=15]
-  //
-  // After the fix:
-  // File[0]:
-  //   smallest=[user_key=key000001, seq=4, type=15],
-  //   largest= [user_key=key000002, seq=72057594037927935, type=15]
-  // File[1]:
-  //   smallest=[user_key=key000002, seq=2, type=1],
-  //   largest= [user_key=key000003, seq=72057594037927935, type=15]
-  // File[2]:
-  //   smallest=[user_key=key000003, seq=3, type=1],
-  //   largest= [user_key=key000004, seq=72057594037927935, type=15]
+TEST_F(DBRangeDelTest, SeekForPrevTest) {
+  // open db
+  Options options = GetDefaultOptions();
+  options.create_if_missing = true;
+  options.compaction_style = kCompactionStyleUniversal;

-  Options options = CurrentOptions();
-  options.disable_auto_compactions = true;
+  // add SST partitioner, split sst file with prefix length 2
+  options.sst_partitioner_factory = NewSstPartitionerFixedPrefixFactory(2);
+  Reopen(options);

-  // Use partitioner that cuts before every new user key.
-  // Key(x) generates keys of length 9.
-  auto factory = std::shared_ptr<SstPartitionerFactory>(
-      NewSstPartitionerFixedPrefixFactory(10));
-  options.sst_partitioner_factory = factory;
+  // File uses SST partitioner, so it will be split into 3 files
+  // SST file 1: ka1, ka2
+  // SST file 2: kb1
+  // SST file 3: kc1, kc2
+  // Delete range covers from ka2 to kc2, which means record ka2 and kb1, kc1
+  // are covered by the delete range

-  DestroyAndReopen(options);
+  std::vector<std::pair<std::string, std::string>> kv = {{"ka1", "value_1"},
+                                                         {"ka2", "value_2"},
+                                                         {"kb1", "value_3"},
+                                                         {"kc1", "value_4"},
+                                                         {"kc2", "value_5"}};
+  for (auto& p : kv) {
+    ASSERT_OK(Put(p.first, p.second));
+  }

-  Random rnd(301);
-
-  // Create a file in a lower level so the compactions below are not
-  // bottommost compactions. Range deletion start keys are not considered
-  // in bottommost compaction.
-  ASSERT_OK(Put(Key(3), rnd.RandomBinaryString(100)));
  ASSERT_OK(Flush());
-  MoveFilesToLevel(6);
-  ASSERT_EQ(1, NumTableFilesAtLevel(6));
+  // Compact to Lmax, it should have seq 0 now.
+  ASSERT_OK(CompactRange(CompactRangeOptions(), nullptr, nullptr));

-  ASSERT_OK(Put(Key(2), rnd.RandomString(100)));
-  // Snapshots keep point keys alive.
-  ManagedSnapshot snapshot1(db_);
-  ASSERT_OK(Put(Key(3), rnd.RandomString(100)));
-  ManagedSnapshot snapshot2(db_);
-  ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(1),
-                             Key(4)));
-  ASSERT_OK(Flush());
-  ASSERT_EQ(1, NumTableFilesAtLevel(0));
-
-  ColumnFamilyMetaData cf_meta_l0;
-  db_->GetColumnFamilyMetaData(db_->DefaultColumnFamily(), &cf_meta_l0);
-  ASSERT_EQ(1, cf_meta_l0.levels[0].files.size());
-  std::vector<std::string> l0_filenames;
-  for (const auto& sst_file : cf_meta_l0.levels[0].files) {
-    l0_filenames.push_back(sst_file.name);
-  }
-
-  // Compact L0 files to L1
-  CompactionOptions compact_options_l0;
-  ASSERT_OK(db_->CompactFiles(compact_options_l0, l0_filenames, 1));
-  ASSERT_EQ(3, NumTableFilesAtLevel(1));
-
-  // Check L1 file metadata
-  std::vector<std::vector<FileMetaData>> files_l1;
-  dbfull()->TEST_GetFilesMetaData(db_->DefaultColumnFamily(), &files_l1);
-
-  for (const auto& file : files_l1[1]) {
-    ASSERT_LT(ExtractValueType(file.smallest.Encode()), kTypeMaxValid);
-    ASSERT_LT(ExtractValueType(file.largest.Encode()), kTypeMaxValid);
-  }
-
-  // Get file names from level 1
-  ColumnFamilyMetaData cf_meta;
-  db_->GetColumnFamilyMetaData(db_->DefaultColumnFamily(), &cf_meta);
-  std::vector<std::string> input_filenames;
-  for (const auto& sst_file : cf_meta.levels[1].files) {
-    input_filenames.push_back(sst_file.name);
-  }
-
-  // Compact files from L1 to L2
-  CompactionOptions compact_options;
-  ASSERT_OK(db_->CompactFiles(compact_options, input_filenames, 2));
-
-  // Check L2 file metadata
-  std::vector<std::vector<FileMetaData>> files;
-  dbfull()->TEST_GetFilesMetaData(db_->DefaultColumnFamily(), &files);
-
-  for (const auto& file : files[2]) {
-    ASSERT_LT(ExtractValueType(file.smallest.Encode()), kTypeMaxValid);
-    ASSERT_LT(ExtractValueType(file.largest.Encode()), kTypeMaxValid);
-  }
-
-  // // Verify iteration works correctly
-  std::unique_ptr<Iterator> iter{db_->NewIterator(ReadOptions())};
+  // Open an iterator and create a snapshot, so that keys are not deleted
+  // completely by delete range in SST
+  ReadOptions read_opts;
+  read_opts.snapshot = db_->GetSnapshot();
+  std::unique_ptr<Iterator> iter(db_->NewIterator(read_opts));
  iter->SeekToFirst();
+  // iterate all the keys and validate the value
+  for (int i = 0; iter->Valid(); iter->Next()) {
+    ASSERT_EQ(kv[i].first, iter->key().ToString());
+    ASSERT_EQ(kv[i].second, iter->value().ToString());
+    i++;
+  }
+
+  // use delete range to delete the record
+  ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "ka2",
+                             "kc2"));
+  // Flush
+  ASSERT_OK(Flush());
+  // Compact to Lmax
+  ASSERT_OK(CompactRange(CompactRangeOptions(), nullptr, nullptr));
+
+  // Close the iterator and release the snapshot.
  ASSERT_OK(iter->status());
-  ASSERT_FALSE(iter->Valid());
+  iter.reset();
+  db_->ReleaseSnapshot(read_opts.snapshot);
+
+  // create second iterator, seek each key and validate result
+  std::unique_ptr<Iterator> iter2(db_->NewIterator(ReadOptions()));
+  // Validate keys are deleted
+  iter2->SeekToFirst();
+  ASSERT_TRUE(iter2->Valid());
+  ASSERT_EQ("ka1", iter2->key().ToString());
+  iter2->Next();
+  ASSERT_TRUE(iter2->Valid());
+  ASSERT_EQ("kc2", iter2->key().ToString());
+  iter2->Next();
+  ASSERT_FALSE(iter2->Valid());
+
+  // Validate seek for prev result
+  for (auto& p : kv) {
+    iter2->SeekForPrev(p.first);
+    ASSERT_TRUE(iter2->Valid());
+    if (p.first == "kc2") {
+      ASSERT_EQ("kc2", iter2->key().ToString());
+    } else {
+      ASSERT_EQ("ka1", iter2->key().ToString());
+    }
+  }
+  ASSERT_OK(iter2->status());
+  iter2.reset();
 }
+
 }  // namespace ROCKSDB_NAMESPACE

 int main(int argc, char** argv) {
@@ -72,12 +72,6 @@ enum ValueType : unsigned char {
  kTypeColumnFamilyWideColumnEntity = 0x17,     // WAL only
  kTypeValuePreferredSeqno = 0x18,              // Value with a unix write time
  kTypeColumnFamilyValuePreferredSeqno = 0x19,  // WAL only
-  // Placeholder value type for legacy SST files with incorrectly persisted
-  // file boundaries. Prior to the fix, TruncatedRangeDelIterator assigned
-  // kTypeMaxValid to truncated range deletion keys, which was then
-  // incorrectly persisted to SST file metadata. This placeholder value allows
-  // reading such legacy files for without using kTypeMaxValid.
-  kTypeTruncatedRangeDeletionSentinel = 0x1A,
  kTypeMaxValid,    // Should be after the last valid type, only used for
                    // validation
  kMaxValue = 0x7F  // Not used for storing records.
@@ -124,11 +118,10 @@ inline bool IsValueType(ValueType t) {

 // Checks whether a type is from user operation
 // kTypeRangeDeletion is in meta block so this API is separated from above
-// kTypeTruncatedRangeDeletionSentinel is for legacy files with incorrectly
-// persisted file boundaries.
+// kTypeMaxValid can be from keys generated by
+// TruncatedRangeDelIterator::start_key()
 inline bool IsExtendedValueType(ValueType t) {
-  return IsValueType(t) || t == kTypeRangeDeletion ||
-         t == kTypeTruncatedRangeDeletionSentinel;
+  return IsValueType(t) || t == kTypeRangeDeletion || t == kTypeMaxValid;
 }

 // We leave eight bits empty at the bottom so a type and sequence#
@@ -187,7 +180,8 @@ inline size_t InternalKeyEncodingLength(const ParsedInternalKey& key) {
 // Pack a sequence number and a ValueType into a uint64_t
 inline uint64_t PackSequenceAndType(uint64_t seq, ValueType t) {
  assert(seq <= kMaxSequenceNumber);
-  assert(IsExtendedValueType(t));
+  // kTypeMaxValid is used in TruncatedRangeDelIterator, see its constructor.
+  assert(IsExtendedValueType(t) || t == kTypeMaxValid);
  return (seq << 8) | t;
 }

@@ -36,6 +36,7 @@ TruncatedRangeDelIterator::TruncatedRangeDelIterator(
    Status pik_status = ParseInternalKey(smallest->Encode(), &parsed_smallest,
                                         false /* log_err_key */);  // TODO
    pik_status.PermitUncheckedError();
+    parsed_smallest.type = kTypeMaxValid;
    assert(pik_status.ok());
    smallest_ = &parsed_smallest;
  }
@@ -70,6 +71,9 @@ TruncatedRangeDelIterator::TruncatedRangeDelIterator(
      // the truncated end key can cover the largest key in this sstable, reduce
      // its sequence number by 1.
      parsed_largest.sequence -= 1;
+      // This line is not needed for correctness, but it ensures that the
+      // truncated end key is not covering keys from the next SST file.
+      parsed_largest.type = kTypeMaxValid;
    }
    largest_ = &parsed_largest;
  }
@@ -89,9 +89,7 @@ void VerifyIterator(
  for (size_t i = 0; i < expected_range_dels.size(); i++, iter->Next()) {
    ASSERT_TRUE(iter->Valid());
    EXPECT_EQ(0, icmp.Compare(iter->start_key(), expected_range_dels[i].start));
-    EXPECT_EQ(0, icmp.Compare(iter->end_key(), expected_range_dels[i].end))
-        << iter->end_key().DebugString(false, false) << " "
-        << expected_range_dels[i].end.DebugString(false, false);
+    EXPECT_EQ(0, icmp.Compare(iter->end_key(), expected_range_dels[i].end));
    EXPECT_EQ(expected_range_dels[i].seq, iter->seq());
  }
  EXPECT_FALSE(iter->Valid());
@@ -307,28 +305,28 @@ TEST_F(RangeDelAggregatorTest, TruncatedIterPartiallyCutTombstones) {

  VerifyIterator(
      &iter, bytewise_icmp,
-      {{InternalValue("d", 7, kTypeValue), UncutEndpoint("e"), 10},
+      {{InternalValue("d", 7, kTypeMaxValid), UncutEndpoint("e"), 10},
       {InternalValue("e", 8, kTypeRangeDeletion), UncutEndpoint("g"), 8},
       {InternalValue("j", 4, kTypeRangeDeletion),
-        InternalValue("m", 8, kTypeValue), 4}});
+        InternalValue("m", 8, kTypeMaxValid), 4}});

  VerifySeek(
      &iter, bytewise_icmp,
-      {{"d", InternalValue("d", 7, kTypeValue), UncutEndpoint("e"), 10},
+      {{"d", InternalValue("d", 7, kTypeMaxValid), UncutEndpoint("e"), 10},
       {"e", InternalValue("e", 8, kTypeRangeDeletion), UncutEndpoint("g"), 8},
       {"ia", InternalValue("j", 4, kTypeRangeDeletion),
-        InternalValue("m", 8, kTypeValue), 4, false /* invalid */},
+        InternalValue("m", 8, kTypeMaxValid), 4, false /* invalid */},
       {"n", InternalValue("", 0, kTypeRangeDeletion), UncutEndpoint(""), 0,
        true /* invalid */},
-       {"", InternalValue("d", 7, kTypeValue), UncutEndpoint("e"), 10}});
+       {"", InternalValue("d", 7, kTypeMaxValid), UncutEndpoint("e"), 10}});

  VerifySeekForPrev(
      &iter, bytewise_icmp,
-      {{"d", InternalValue("d", 7, kTypeValue), UncutEndpoint("e"), 10},
+      {{"d", InternalValue("d", 7, kTypeMaxValid), UncutEndpoint("e"), 10},
       {"e", InternalValue("e", 8, kTypeRangeDeletion), UncutEndpoint("g"), 8},
       {"ia", InternalValue("e", 8, kTypeRangeDeletion), UncutEndpoint("g"), 8},
       {"n", InternalValue("j", 4, kTypeRangeDeletion),
-        InternalValue("m", 8, kTypeValue), 4, false /* invalid */},
+        InternalValue("m", 8, kTypeMaxValid), 4, false /* invalid */},
       {"", InternalValue("", 0, kTypeRangeDeletion), UncutEndpoint(""), 0,
        true /* invalid */}});
 }
@@ -347,21 +345,23 @@ TEST_F(RangeDelAggregatorTest, TruncatedIterFullyCutTombstones) {
  TruncatedRangeDelIterator iter(std::move(input_iter), &bytewise_icmp,
                                 &smallest, &largest);

-  VerifyIterator(&iter, bytewise_icmp,
-                 {{InternalValue("f", 7, kTypeValue), UncutEndpoint("g"), 8}});
+  VerifyIterator(
+      &iter, bytewise_icmp,
+      {{InternalValue("f", 7, kTypeMaxValid), UncutEndpoint("g"), 8}});

-  VerifySeek(&iter, bytewise_icmp,
-             {{"d", InternalValue("f", 7, kTypeValue), UncutEndpoint("g"), 8},
-              {"f", InternalValue("f", 7, kTypeValue), UncutEndpoint("g"), 8},
-              {"j", InternalValue("", 0, kTypeRangeDeletion), UncutEndpoint(""),
-               0, true /* invalid */}});
+  VerifySeek(
+      &iter, bytewise_icmp,
+      {{"d", InternalValue("f", 7, kTypeMaxValid), UncutEndpoint("g"), 8},
+       {"f", InternalValue("f", 7, kTypeMaxValid), UncutEndpoint("g"), 8},
+       {"j", InternalValue("", 0, kTypeRangeDeletion), UncutEndpoint(""), 0,
+        true /* invalid */}});

  VerifySeekForPrev(
      &iter, bytewise_icmp,
      {{"d", InternalValue("", 0, kTypeRangeDeletion), UncutEndpoint(""), 0,
        true /* invalid */},
-       {"f", InternalValue("f", 7, kTypeValue), UncutEndpoint("g"), 8},
-       {"j", InternalValue("f", 7, kTypeValue), UncutEndpoint("g"), 8}});
+       {"f", InternalValue("f", 7, kTypeMaxValid), UncutEndpoint("g"), 8},
+       {"j", InternalValue("f", 7, kTypeMaxValid), UncutEndpoint("g"), 8}});
 }

 TEST_F(RangeDelAggregatorTest, SingleIterInAggregator) {
@@ -30,7 +30,6 @@ uint64_t PackFileNumberAndPathId(uint64_t number, uint64_t path_id) {
 Status FileMetaData::UpdateBoundaries(const Slice& key, const Slice& value,
                                      SequenceNumber seqno,
                                      ValueType value_type) {
-  assert(value_type < kTypeMaxValid);
  if (value_type == kTypeBlobIndex) {
    BlobIndex blob_index;
    const Status s = blob_index.DecodeFrom(value);
@@ -317,9 +317,6 @@ struct FileMetaData {
  void UpdateBoundariesForRange(const InternalKey& start,
                                const InternalKey& end, SequenceNumber seqno,
                                const InternalKeyComparator& icmp) {
-    assert(ExtractValueType(start.Encode()) < kTypeMaxValid);
-    assert(ExtractValueType(end.Encode()) < kTypeMaxValid);
-
    if (smallest.size() == 0 || icmp.Compare(start, smallest) < 0) {
      smallest = start;
    }
@@ -560,16 +557,19 @@ struct SubcompactionProgress {
      Slice key_slice(next_internal_key_to_compact);
      if (ParseInternalKey(key_slice, &parsed_key, false /* log_err_key */)
              .ok()) {
-        oss << "user_key=\"" << parsed_key.user_key.ToString(false /* hex */)
-            << "\" (hex:" << parsed_key.user_key.ToString(true /* hex */)
-            << ")";
+        oss << "user_key(hex)=" << parsed_key.user_key.ToString(true /* hex */);
        oss << ", seq=";
        if (parsed_key.sequence == kMaxSequenceNumber) {
          oss << "kMaxSequenceNumber";
        } else {
          oss << parsed_key.sequence;
        }
-        oss << ", type=" << static_cast<int>(parsed_key.type);
+        oss << ", type=";
+        if (parsed_key.type == kValueTypeForSeek) {
+          oss << "kValueTypeForSeek";
+        } else {
+          oss << static_cast<int>(parsed_key.type);
+        }
      } else {
        oss << "raw=" << key_slice.ToString(true /* hex */);
      }
@@ -125,7 +125,7 @@ class Compressor {
  // dictionary associated with a returned compressor must be read from
  // GetSerializedDict().
  virtual std::unique_ptr<Compressor> MaybeCloneSpecialized(
-      CacheEntryRole block_type, DictSampleArgs&& dict_samples) {
+      CacheEntryRole block_type, DictSampleArgs&& dict_samples) const {
    // Default implementation: no specialization
    (void)block_type;
    (void)dict_samples;
@@ -138,7 +138,7 @@ class Compressor {
  // A convenience function when a clone is needed and may or may not be
  // specialized.
  std::unique_ptr<Compressor> CloneMaybeSpecialized(
-      CacheEntryRole block_type, DictSampleArgs&& dict_samples) {
+      CacheEntryRole block_type, DictSampleArgs&& dict_samples) const {
    auto clone = MaybeCloneSpecialized(block_type, std::move(dict_samples));
    if (clone == nullptr) {
      clone = Clone();
@@ -496,7 +496,7 @@ class CompressorWrapper : public Compressor {
  // when the wrapped Compressor uses the default implementation of
  // MaybeCloneSpecialized(). This needs to be overridden if not.
  std::unique_ptr<Compressor> MaybeCloneSpecialized(
-      CacheEntryRole block_type, DictSampleArgs&& dict_samples) override {
+      CacheEntryRole block_type, DictSampleArgs&& dict_samples) const override {
    auto clone =
        wrapped_->MaybeCloneSpecialized(block_type, std::move(dict_samples));
    // Assert default no-op MaybeCloneSpecialized()
@@ -25,7 +25,7 @@ namespace ROCKSDB_NAMESPACE {
 // with `Options::compaction_options_fifo.max_table_files_size` > 0 can cause
 // the whole DB to be dropped right after migration if the migrated data is
 // larger than `max_table_files_size`
-Status OptionChangeMigration(std::string& dbname, const Options& old_opts,
+Status OptionChangeMigration(const std::string& dbname, const Options& old_opts,
                             const Options& new_opts);

 // Multi-CF version: Prepares a database with multiple column families to be
@@ -13,7 +13,7 @@
 // minor or major version number planned for release.
 #define ROCKSDB_MAJOR 10
 #define ROCKSDB_MINOR 9
-#define ROCKSDB_PATCH 0
+#define ROCKSDB_PATCH 1

 // Make it easy to do conditional compilation based on version checks, i.e.
 // #if ROCKSDB_VERSION_GE(4, 5, 6)
@@ -191,6 +191,13 @@ class CompactionMergingIterator : public InternalIterator {

    bool operator()(HeapItem* a, HeapItem* b) const {
      int r = comparator_->Compare(a->key(), b->key());
+      // For each file, we assume all range tombstone start keys come before
+      // its file boundary sentinel key (file's meta.largest key).
+      // In the case when meta.smallest = meta.largest and range tombstone start
+      // key is truncated at meta.smallest, the start key will have op_type =
+      // kMaxValid to make it smaller (see TruncatedRangeDelIterator
+      // constructor). The following assertion validates this assumption.
+      assert(a->type == b->type || r != 0);
      return r > 0;
    }

@@ -235,24 +242,8 @@ class CompactionMergingIterator : public InternalIterator {
    return !minHeap_.empty() ? minHeap_.top() : nullptr;
  }

-  // For each file under a LevelIterator, the lifetime of range tombstone
-  // iterator is tied to the point key iterator. So we want scan through
-  // all range tombstone start keys before the file boundary sentinel key
-  // (file's meta.largest). When meta.smallest == meta.largest, the truncated
-  // range del start key may be ordered after meta.largest.
-  // Here we skip the first range deletion start key if it's truncated.
-  // This range deletion start key is redundant for compaction file cutting
-  // decision anyway, since the same point key will be considered for file
-  // cutting too.
-  void InsertNextValidRangeTombstoneAtLevel(size_t level) {
+  void InsertRangeTombstoneAtLevel(size_t level) {
    if (range_tombstone_iters_[level]->Valid()) {
-      if (range_tombstone_iters_[level]->start_key().type !=
-          kTypeRangeDeletion) {
-        range_tombstone_iters_[level]->Next();
-        if (!range_tombstone_iters_[level]->Valid()) {
-          return;
-        }
-      }
      pinned_heap_item_[level].SetTombstoneForCompaction(
          range_tombstone_iters_[level]->start_key());
      minHeap_.push(&pinned_heap_item_[level]);
@@ -271,7 +262,7 @@ void CompactionMergingIterator::SeekToFirst() {
  for (size_t i = 0; i < range_tombstone_iters_.size(); ++i) {
    if (range_tombstone_iters_[i]) {
      range_tombstone_iters_[i]->SeekToFirst();
-      InsertNextValidRangeTombstoneAtLevel(i);
+      InsertRangeTombstoneAtLevel(i);
    }
  }

@@ -299,7 +290,7 @@ void CompactionMergingIterator::Seek(const Slice& target) {
                 0) {
        range_tombstone_iters_[i]->Next();
      }
-      InsertNextValidRangeTombstoneAtLevel(i);
+      InsertRangeTombstoneAtLevel(i);
    }
  }

@@ -366,7 +357,7 @@ void CompactionMergingIterator::FindNextVisibleKey() {
      minHeap_.pop();
    }
    if (range_tombstone_iters_[current->level]) {
-      InsertNextValidRangeTombstoneAtLevel(current->level);
+      InsertRangeTombstoneAtLevel(current->level);
    }
  }
 }
@@ -796,7 +796,7 @@ struct CompressorCustomAlg : public CompressorWrapper {
  }

  std::unique_ptr<Compressor> MaybeCloneSpecialized(
-      CacheEntryRole block_type, DictSampleArgs&& dict_samples) override {
+      CacheEntryRole block_type, DictSampleArgs&& dict_samples) const override {
    auto clone =
        wrapped_->CloneMaybeSpecialized(block_type, std::move(dict_samples));
    return std::make_unique<CompressorCustomAlg>(std::move(clone));
@@ -137,7 +137,7 @@ EOF

 # To check for DB forward compatibility with loading options (old version
 # reading data from new), as well as backward compatibility
-declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb" "9.8.fb" "9.9.fb" "9.10.fb" "9.11.fb" "10.0.fb" "10.1.fb" "10.2.fb" "10.3.fb" "10.4.fb" "10.5.fb" "10.6.fb" "10.7.fb" "10.8.fb")
+declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb" "9.8.fb" "9.9.fb" "9.10.fb" "9.11.fb" "10.0.fb" "10.1.fb" "10.2.fb" "10.3.fb" "10.4.fb" "10.5.fb" "10.6.fb" "10.7.fb" "10.8.fb" "10.9.fb")
 # To check for DB forward compatibility without loading options (in addition
 # to the "with loading options" set), as well as backward compatibility
 declare -a db_forward_no_options_refs=() # N/A at the moment
@@ -1 +0,0 @@
-PosixWritableFile now repositions the seek pointer to the new end of file after a call to Truncate.
@@ -1 +0,0 @@
-* Updated standalone range deletion L0 file compaction behavior to avoid compacting with any newer L0 files (which is expensive and not useful).
@@ -1 +0,0 @@
-* Fix a bug where compaction with range deletion can persist kTypeMaxValid in MANIFEST as file metadata. kTypeMaxValid is not supposed to be persisted and can change as new value types are introduced. This can cause a forward compatibility issue where older versions of RocksDB don't recognize kTypeMaxValid from newer versions. A new placeholder value type kTypeTruncatedRangeDeletionSentinel is also introduced to replace kTypeMaxValid when reading existing SST files' metadata from MANIFEST. This allows us to strengthen some checks to avoid using kTypeMaxValid in the future.
@@ -1 +0,0 @@
-Fixed a bug where `DB::GetSortedWalFiles()` could hang when waiting for a purge operation that found nothing to do (potentially triggered by iterator release, flush, compaction, etc.).
@@ -1 +0,0 @@
-Fixed a bug in MultiScan where `max_sequential_skip_in_iterations` could cause the iterator to seek backward to already-unpinned blocks when the same user key spans multiple data blocks, leading to assertion failures or seg fault.
@@ -1 +0,0 @@
-Fixed a bug for `WAL_ttl_seconds > 0` use cases where the newest archived WAL files could be incorrectly deleted when the system clock moved backwards.
@@ -1 +0,0 @@
-Add a new option allow_trivial_move in CompactionOptions to allow CompactFiles to perform trivial move if possible. By default the flag of allow_trivial_move is false, so it preserve the original behavior.
@@ -1 +0,0 @@
-* Added an auto-tuning feature for DB manifest file size that also (by default) improves the safety of existing configurations in case `max_manifest_file_size` is repeatedly exceeded. The new recommendation is to set `max_manifest_file_size` to something small like 1MB and tune `max_manifest_space_amp_pct` as needed to balance write amp and space amp in the manifest. Refer to comments on those options in `DBOptions` for details. Both options are (now) mutable.
@@ -1 +0,0 @@
-Added a new API to support option migration for multiple column families
@@ -1 +0,0 @@
-Added new option target_file_size_is_upper_bound  that makes most compaction output SST files come close to the target file size without exceeding it, rather than commonly exceeding it by some fraction (current behavior). For now the new behavior is off by default, but we expect to enable it by default in the future.
@@ -1 +0,0 @@
-Added optimization that allowed for the asynchronous prefetching of all data outlined in a multiscan iterator. This optimization was applied to the level iterator, which prefetches all data through each of the block-based iterators.
@@ -1 +0,0 @@
-* To reduce risk of ODR violations or similar, `ROCKSDB_USING_THREAD_STATUS` has been removed from public headers and replaced with static `const bool ThreadStatus::kEnabled`. Some other uses of conditional compilation have been removed from public API headers to reduce risk of ODR violations or other issues.
@@ -63,6 +63,13 @@ std::unique_ptr<Compressor> AutoSkipCompressorWrapper::Clone() const {
  return std::make_unique<AutoSkipCompressorWrapper>(wrapped_->Clone(), opts_);
 }

+std::unique_ptr<Compressor> AutoSkipCompressorWrapper::MaybeCloneSpecialized(
+    CacheEntryRole block_type, DictSampleArgs&& dict_samples) const {
+  auto clone =
+      wrapped_->CloneMaybeSpecialized(block_type, std::move(dict_samples));
+  return std::make_unique<AutoSkipCompressorWrapper>(std::move(clone), opts_);
+}
+
 Status AutoSkipCompressorWrapper::CompressBlock(
    Slice uncompressed_data, char* compressed_output,
    size_t* compressed_output_size, CompressionType* out_compression_type,
@@ -198,7 +205,7 @@ CompressionType CostAwareCompressor::GetPreferredCompressionType() const {
  return kZSTD;
 }
 std::unique_ptr<Compressor> CostAwareCompressor::MaybeCloneSpecialized(
-    CacheEntryRole block_type, DictSampleArgs&& dict_samples) {
+    CacheEntryRole block_type, DictSampleArgs&& dict_samples) const {
  // TODO: full dictionary compression support. Currently this just falls
  // back on a non-multi compressor when asked to use a dictionary.
  auto idx = allcompressors_index_.back();
@@ -65,6 +65,8 @@ class AutoSkipCompressorWrapper : public CompressorWrapper {
                                     const CompressionOptions& opts);

  std::unique_ptr<Compressor> Clone() const override;
+  std::unique_ptr<Compressor> MaybeCloneSpecialized(
+      CacheEntryRole block_type, DictSampleArgs&& dict_samples) const override;
  Status CompressBlock(Slice uncompressed_data, char* compressed_output,
                       size_t* compressed_output_size,
                       CompressionType* out_compression_type,
@@ -156,7 +158,7 @@ class CostAwareCompressor : public Compressor {
  CompressionType GetPreferredCompressionType() const override;
  ManagedWorkingArea ObtainWorkingArea() override;
  std::unique_ptr<Compressor> MaybeCloneSpecialized(
-      CacheEntryRole block_type, DictSampleArgs&& dict_samples) override;
+      CacheEntryRole block_type, DictSampleArgs&& dict_samples) const override;

  Status CompressBlock(Slice uncompressed_data, char* compressed_output,
                       size_t* compressed_output_size,
@@ -162,7 +162,7 @@ class CompressorBase : public Compressor {
  CompressionOptions opts_;
 };

-class BuiltinCompressorV1 : public CompressorBase {
+class BuiltinCompressorV1 final : public CompressorBase {
 public:
  const char* Name() const override { return "BuiltinCompressorV1"; }

@@ -236,7 +236,7 @@ class CompressorWithSimpleDictBase : public CompressorBase {

  std::unique_ptr<Compressor> MaybeCloneSpecialized(
      CacheEntryRole /*block_type*/,
-      DictSampleArgs&& dict_samples) final override {
+      DictSampleArgs&& dict_samples) const final override {
    assert(dict_samples.Verify());
    if (dict_samples.empty()) {
      // Nothing to specialize on
@@ -256,7 +256,7 @@ class CompressorWithSimpleDictBase : public CompressorBase {
 // NOTE: the legacy behavior is to pretend to use dictionary compression when
 // enabled, including storing a dictionary block, but to ignore it. That is
 // matched here.
-class BuiltinSnappyCompressorV2 : public CompressorWithSimpleDictBase {
+class BuiltinSnappyCompressorV2 final : public CompressorWithSimpleDictBase {
 public:
  using CompressorWithSimpleDictBase::CompressorWithSimpleDictBase;

@@ -349,7 +349,7 @@ std::pair<char*, size_t> StartCompressBlockV2(Slice uncompressed_data,
  return {alg_output, alg_max_output_size};
 }

-class BuiltinZlibCompressorV2 : public CompressorWithSimpleDictBase {
+class BuiltinZlibCompressorV2 final : public CompressorWithSimpleDictBase {
 public:
  using CompressorWithSimpleDictBase::CompressorWithSimpleDictBase;

@@ -448,7 +448,7 @@ class BuiltinZlibCompressorV2 : public CompressorWithSimpleDictBase {
  }
 };

-class BuiltinBZip2CompressorV2 : public CompressorWithSimpleDictBase {
+class BuiltinBZip2CompressorV2 final : public CompressorWithSimpleDictBase {
 public:
  using CompressorWithSimpleDictBase::CompressorWithSimpleDictBase;

@@ -624,7 +624,8 @@ class BuiltinLZ4CompressorV2WithDict : public CompressorWithSimpleDictBase {
  }
 };

-class BuiltinLZ4CompressorV2NoDict : public BuiltinLZ4CompressorV2WithDict {
+class BuiltinLZ4CompressorV2NoDict final
+    : public BuiltinLZ4CompressorV2WithDict {
 public:
  BuiltinLZ4CompressorV2NoDict(const CompressionOptions& opts)
      : BuiltinLZ4CompressorV2WithDict(opts, /*dict_data=*/{}) {}
@@ -694,7 +695,7 @@ class BuiltinLZ4CompressorV2NoDict : public BuiltinLZ4CompressorV2WithDict {
  }
 };

-class BuiltinLZ4HCCompressorV2 : public CompressorWithSimpleDictBase {
+class BuiltinLZ4HCCompressorV2 final : public CompressorWithSimpleDictBase {
 public:
  using CompressorWithSimpleDictBase::CompressorWithSimpleDictBase;

@@ -790,7 +791,7 @@ class BuiltinLZ4HCCompressorV2 : public CompressorWithSimpleDictBase {
  }
 };

-class BuiltinXpressCompressorV2 : public CompressorWithSimpleDictBase {
+class BuiltinXpressCompressorV2 final : public CompressorWithSimpleDictBase {
 public:
  using CompressorWithSimpleDictBase::CompressorWithSimpleDictBase;

@@ -840,7 +841,7 @@ class BuiltinXpressCompressorV2 : public CompressorWithSimpleDictBase {
  }
 };

-class BuiltinZSTDCompressorV2 : public CompressorBase {
+class BuiltinZSTDCompressorV2 final : public CompressorBase {
 public:
  explicit BuiltinZSTDCompressorV2(const CompressionOptions& opts,
                                   CompressionDict&& dict = {})
@@ -972,7 +973,8 @@ class BuiltinZSTDCompressorV2 : public CompressorBase {
  }

  std::unique_ptr<Compressor> MaybeCloneSpecialized(
-      CacheEntryRole /*block_type*/, DictSampleArgs&& dict_samples) override {
+      CacheEntryRole /*block_type*/,
+      DictSampleArgs&& dict_samples) const override {
    assert(dict_samples.Verify());
    if (dict_samples.empty()) {
      // Nothing to specialize on
@@ -1009,7 +1011,7 @@ class BuiltinZSTDCompressorV2 : public CompressorBase {

 // NOTE: this implementation is intentionally SIMPLE based on existing code
 // and NOT EFFICIENT because this is an old/deprecated format.
-class BuiltinDecompressorV1 : public Decompressor {
+class BuiltinDecompressorV1 final : public Decompressor {
 public:
  const char* Name() const override { return "BuiltinDecompressorV1"; }

@@ -1061,7 +1063,7 @@ class BuiltinDecompressorV1 : public Decompressor {
  }
 };

-class BuiltinCompressionManagerV1 : public CompressionManager {
+class BuiltinCompressionManagerV1 final : public CompressionManager {
 public:
  BuiltinCompressionManagerV1() = default;
  ~BuiltinCompressionManagerV1() override = default;
@@ -1384,7 +1386,7 @@ class BuiltinDecompressorV2 : public Decompressor {
  }
 };

-class BuiltinDecompressorV2SnappyOnly : public BuiltinDecompressorV2 {
+class BuiltinDecompressorV2SnappyOnly final : public BuiltinDecompressorV2 {
 public:
  const char* Name() const override {
    return "BuiltinDecompressorV2SnappyOnly";
@@ -1412,7 +1414,7 @@ class BuiltinDecompressorV2SnappyOnly : public BuiltinDecompressorV2 {
  }
 };

-class BuiltinDecompressorV2WithDict : public BuiltinDecompressorV2 {
+class BuiltinDecompressorV2WithDict final : public BuiltinDecompressorV2 {
 public:
  explicit BuiltinDecompressorV2WithDict(const Slice& dict) : dict_(dict) {}

@@ -1505,7 +1507,7 @@ class BuiltinDecompressorV2OptimizeZstd : public BuiltinDecompressorV2 {
                           std::unique_ptr<Decompressor>* /*out*/) override;
 };

-class BuiltinDecompressorV2OptimizeZstdWithDict
+class BuiltinDecompressorV2OptimizeZstdWithDict final
    : public BuiltinDecompressorV2OptimizeZstd {
 public:
  explicit BuiltinDecompressorV2OptimizeZstdWithDict(const Slice& dict)
@@ -1569,7 +1571,7 @@ Status BuiltinDecompressorV2OptimizeZstd::MaybeCloneForDict(
      serialized_dict);
  return Status::OK();
 }
-class BuiltinCompressionManagerV2 : public CompressionManager {
+class BuiltinCompressionManagerV2 final : public CompressionManager {
 public:
  BuiltinCompressionManagerV2() = default;
  ~BuiltinCompressionManagerV2() override = default;
@@ -1364,7 +1364,8 @@ TEST_P(DBCompressionTestMaybeParallel, CompressionManagerWrapper) {
    }

    std::unique_ptr<Compressor> MaybeCloneSpecialized(
-        CacheEntryRole block_type, DictSampleArgs&& dict_samples) override {
+        CacheEntryRole block_type,
+        DictSampleArgs&& dict_samples) const override {
      std::unique_ptr<Compressor> result = std::make_unique<MyCompressor>(
          wrapped_->CloneMaybeSpecialized(block_type, std::move(dict_samples)));
      if (block_type == CacheEntryRole::kDataBlock) {
@@ -1991,34 +1992,37 @@ class DBAutoSkip : public DBTestBase {
 };

 TEST_F(DBAutoSkip, AutoSkipCompressionManager) {
-  for (auto type : GetSupportedCompressions()) {
-    if (type == kNoCompression) {
-      continue;
+  for (uint32_t max_dict_bytes : {0, 10000}) {
+    for (auto type : GetSupportedCompressions()) {
+      if (type == kNoCompression) {
+        continue;
+      }
+      options.compression = type;
+      options.bottommost_compression = type;
+      options.compression_opts.max_dict_bytes = max_dict_bytes;
+      DestroyAndReopen(options);
+      const int kValueSize = 20000;
+      // This will set the rejection ratio to 60%
+      CompressionUnfriendlyPut(6, kValueSize);
+      CompressionFriendlyPut(4, kValueSize);
+      // This will verify all the data block compressions are bypassed based on
+      // previous prediction
+      CompressionUnfriendlyPut(6, kValueSize);
+      CompressionFriendlyPut(4, kValueSize);
+      // This will set the rejection ratio to 40%
+      CompressionUnfriendlyPut(4, kValueSize);
+      CompressionFriendlyPut(6, kValueSize);
+      // This will verify all the data block compression are attempted based on
+      // previous prediction
+      // Compression will be rejected for 6 compression unfriendly blocks
+      // Compression will be accepted for 4 compression friendly blocks
+      CompressionUnfriendlyPut(6, kValueSize);
+      CompressionFriendlyPut(4, kValueSize);
+      // Extra block write to ensure that the all above cases are checked
+      CompressionFriendlyPut(6, kValueSize);
+      CompressionFriendlyPut(4, kValueSize);
+      ASSERT_OK(Flush());
    }
-    options.compression = type;
-    options.bottommost_compression = type;
-    DestroyAndReopen(options);
-    const int kValueSize = 20000;
-    // This will set the rejection ratio to 60%
-    CompressionUnfriendlyPut(6, kValueSize);
-    CompressionFriendlyPut(4, kValueSize);
-    // This will verify all the data block compressions are bypassed based on
-    // previous prediction
-    CompressionUnfriendlyPut(6, kValueSize);
-    CompressionFriendlyPut(4, kValueSize);
-    // This will set the rejection ratio to 40%
-    CompressionUnfriendlyPut(4, kValueSize);
-    CompressionFriendlyPut(6, kValueSize);
-    // This will verify all the data block compression are attempted based on
-    // previous prediction
-    // Compression will be rejected for 6 compression unfriendly blocks
-    // Compression will be accepted for 4 compression friendly blocks
-    CompressionUnfriendlyPut(6, kValueSize);
-    CompressionFriendlyPut(4, kValueSize);
-    // Extra block write to ensure that the all above cases are checked
-    CompressionFriendlyPut(6, kValueSize);
-    CompressionFriendlyPut(4, kValueSize);
-    ASSERT_OK(Flush());
  }
 }
 class CostAwareTestFlushBlockPolicy : public FlushBlockPolicy {
@@ -48,7 +48,7 @@ Compressor::ManagedWorkingArea MultiCompressorWrapper::ObtainWorkingArea() {
 }

 std::unique_ptr<Compressor> MultiCompressorWrapper::MaybeCloneSpecialized(
-    CacheEntryRole block_type, DictSampleArgs&& dict_samples) {
+    CacheEntryRole block_type, DictSampleArgs&& dict_samples) const {
  // TODO: full dictionary compression support. Currently this just falls
  // back on a non-multi compressor when asked to use a dictionary.
  return compressors_.back()->MaybeCloneSpecialized(block_type,
@@ -25,7 +25,7 @@ class MultiCompressorWrapper : public Compressor {
  CompressionType GetPreferredCompressionType() const override;
  ManagedWorkingArea ObtainWorkingArea() override;
  std::unique_ptr<Compressor> MaybeCloneSpecialized(
-      CacheEntryRole block_type, DictSampleArgs&& dict_samples) override;
+      CacheEntryRole block_type, DictSampleArgs&& dict_samples) const override;

 protected:
  const CompressionOptions opts_;
@@ -41,8 +41,6 @@ static std::unordered_map<std::string, ValueType> value_type_string_map = {
    {"TypeValuePreferredSeqno", ValueType::kTypeValuePreferredSeqno},
    {"TypeColumnFamilyValuePreferredSeqno",
     ValueType::kTypeColumnFamilyValuePreferredSeqno},
-    {"kTypeTruncatedRangeDeletionSentinel",
-     ValueType::kTypeTruncatedRangeDeletionSentinel},
 };

 std::string KeyVersion::GetTypeName() const {
@@ -341,7 +341,7 @@ Status OptionChangeMigration(
  return s;
 }

-Status OptionChangeMigration(std::string& dbname, const Options& old_opts,
+Status OptionChangeMigration(const std::string& dbname, const Options& old_opts,
                             const Options& new_opts) {
  DBOptions old_db_opts(old_opts);
  DBOptions new_db_opts(new_opts);
Author	SHA1	Message	Date
Hui Xiao	5fbc1cd5bc	Update for 10.9.1 patch	2025-12-11 19:04:28 -08:00
Hui Xiao	60f24d394d	Fix resumable compaction to prevent resumption at truncated range deletion boundaries (#14184 ) Summary: Context/Summary: Truncated range deletion in input files can be output by CompactionIterator with type kMaxValid instead of kTypeRangeDeletion, to satisfy ordering requirement between the truncated range deletion start key and a file's point keys. There was a plan to skip such key in https://github.com/facebook/rocksdb/pull/14122 but blockers remain to fulfill the plan. Resumable compaction is not able to handle resumption from range deletion well at this point and should consider kMaxValid type same as kTypeRangeDeletion for resumption. Previously, it didn't and mistakenly allow resumption from a delete range. That led to an assertion failure, complaining about lacking information to update file boundaries in the presence of range deletion needed during cutting an output file, after the compaction resumes from that delete range and happens to cut the output file shortly after without any point keys in between. ``` frame https://github.com/facebook/rocksdb/issues/9: 0x00007f4f4743bc93 libc.so.6`__GI___assert_fail(assertion="meta.smallest.size() > 0", file="db/compaction/compaction_outputs.cc", line=530, function="rocksdb::Status rocksdb::CompactionOutputs::AddRangeDels(rocksdb::CompactionRangeDelAggregator&, const rocksdb::Slice, const rocksdb::Slice, rocksdb::CompactionIterationStats&, bool, const rocksdb::InternalKeyComparator&, rocksdb::SequenceNumber, std::pair<long unsigned int, long unsigned int>, const rocksdb::Slice&, const string&)") at assert.c:101:3 frame https://github.com/facebook/rocksdb/issues/10: 0x00007f4f4808c68c librocksdb.so.10.9`rocksdb::CompactionOutputs::AddRangeDels(this=0x00007f4f0c27e1a0, range_del_agg=0x00007f4f0c21ecc0, comp_start_user_key=0x0000000000000000, comp_end_user_key=0x0000000000000000, range_del_out_stats=0x00007f4f0dffa140, bottommost_level=false, icmp=0x00007f4ef4c93040, earliest_snapshot=13108729, keep_seqno_range=<unavailable>, next_table_min_key=0x00007f4ef4c8f540, full_history_ts_low="") at compaction_outputs.cc:530:7 frame https://github.com/facebook/rocksdb/issues/11: 0x00007f4f480480dd librocksdb.so.10.9`rocksdb::CompactionJob::FinishCompactionOutputFile(this=0x00007f4f0dffb890, input_status=<unavailable>, prev_table_last_internal_key=0x00007f4f0dffa650, next_table_min_key=0x00007f4ef4c8f540, comp_start_user_key=0x0000000000000000, comp_end_user_key=0x0000000000000000, c_iter=0x00007f4ef4c8f400, sub_compact=0x00007f4f0c27e000, outputs=0x00007f4f0c27e1a0) at compaction_job.cc:1917:31 ``` This PR simply prevents MaxValid from being a resumption point like regular range deletion - see commit 842d66eb18ea67e965d6acb1fce12c18eeb778d2 Besides that, the PR also improves the testing, variable naming, logging in resumable compaction codes that were needed to debug this assertion failure - see commit https://github.com/facebook/rocksdb/pull/14184/commits/aecd4e7f971f6dd4df672d9e5f1409fe4747c561. These improvements are covered by existing tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14184 Test Plan: - The stress initially surfaced the error. Using the exact same LSM shapes and files that were used in stress test but in a unit test, I'm able to get a deterministic repro and confirmed the fix resolves the error. This is the repro test https://github.com/hx235/rocksdb/commit/1075936e693c68c960761855900c53f5b894f57a ``` ./compaction_service_test --gtest_filter=ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume # Before fix [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from ResumableCompactionServiceTest [ RUN ] ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume compaction_service_test: db/compaction/compaction_outputs.cc:530: rocksdb::Status rocksdb::CompactionOutputs::AddRangeDels(rocksdb::CompactionRangeDelAggregator&, const rocksdb::Slice, const rocksdb::Slice, rocksdb::CompactionIterationStats&, bool, const rocksdb::InternalKeyComparator&, rocksdb::SequenceNumber, std::pair<long unsigned int, long unsigned int>, const rocksdb::Slice&, const string&): Assertion `meta.smallest.size() > 0' failed. Received signal 6 (Aborted) Invoking GDB for stack trace... [New LWP 2621610] [New LWP 2621611] [New LWP 2621612] [New LWP 2621613] [New LWP 2621614] [New LWP 2621630] [New LWP 2621631] # After fix Note: Google Test filter = ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from ResumableCompactionServiceTest [ RUN ] ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume [ OK ] ResumableCompactionServiceTest.CompactSpecificFilesFromExistingDBWithCancelAndResume (4722 ms) [----------] 1 test from ResumableCompactionServiceTest (4722 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (4722 ms total) [ PASSED ] 1 test. ``` - Follow-up: I tried a couple time to coerce the truncated range delete from scratch in the unit test but failed doing so. Considering kMaxValid may not be outputted by compaction iterator anymore after https://github.com/facebook/rocksdb/pull/14122/files gets landed again (and obsolete the bug) ADN the simple nature of this fix 842d66eb18ea67e965d6acb1fce12c18eeb778d2 AND the worst case of such fix going wrong is just less resumption, I decided to leave writing a unit test to coerce truncated ranged deletion from scratch a follow-up. Maybe I will draw inspiration from https://github.com/facebook/rocksdb/pull/14122/files. Reviewed By: jaykorean Differential Revision: D88912663 Pulled By: hx235 fbshipit-source-id: 80a01135684c8fea659650faaa00c2dc452c482a	2025-12-11 18:39:27 -08:00
Xingbo Wang	268a719a30	Fix missing const for arg of OptionChangeMigration (#14173 ) Summary: Fix missing const for arg of OptionChangeMigration We switched from std::string to std::string & for API OptionChangeMigration, which caused const qualifier to be lost at call site, which causes compilation failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14173 Test Plan: Unit test Reviewed By: pdillinger Differential Revision: D88431457 Pulled By: xingbowang fbshipit-source-id: a705f3b80cc5ff56dab73aa6a31c940798d8df45	2025-12-04 19:06:26 -08:00
Xingbo Wang	9791103976	Revert #14122 "Fix a bug where compaction ..." (#14170 ) Summary: Revert "Fix a bug where compaction with range deletion can persist kTypeMaxValid in file metadata (https://github.com/facebook/rocksdb/issues/14122)" Add a new unit test to capture the situation found by stress test This reverts commit `8c7c8b8dab`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14170 Test Plan: Unit Test Reviewed By: anand1976 Differential Revision: D88395956 Pulled By: xingbowang fbshipit-source-id: 226649dc79a86010ad326ffb2eae35109dc96bc4	2025-12-04 13:30:39 -08:00
Peter Dillinger	4982688c21	Fix AutoSkipCompressorWrapper with new logic (#14150 ) Summary: ... from https://github.com/facebook/rocksdb/issues/14140. The assertion in the default implementation of CompressorWrapper::MaybeCloneSpecialized() could fail because this wrapper wasn't overriding it when it should. (See the NOTE on that implementation.) Because this release already has a breaking modification to the Compressor API (adding Clone()), I took this opportunity to add 'const' to MaybeCloneSpecialized(). Also marked some compression classes as 'final' that could be marked as such. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14150 Test Plan: unit test expanded to cover this case (verified failing before). Audited the rest of our CompressorWrappers. Reviewed By: archang19 Differential Revision: D87793987 Pulled By: pdillinger fbshipit-source-id: 61c4469b84e4a47451a9942df09277faeeccfe63	2025-11-24 13:40:36 -08:00
Xingbo Wang	ef5a37aca8	Cut 10.9.0 branch. Summary: Cut 10.9.0 branch. Test Plan: Reviewers: Subscribers: Tasks: Tags:	2025-11-24 12:53:27 -08:00
				`@@ -1 +0,0 @@`
				`PosixWritableFile now repositions the seek pointer to the new end of file after a call to Truncate.`
				`@@ -1 +0,0 @@`
				`* Updated standalone range deletion L0 file compaction behavior to avoid compacting with any newer L0 files (which is expensive and not useful).`
				`@@ -1 +0,0 @@`
				* Fix a bug where compaction with range deletion can persist kTypeMaxValid in MANIFEST as file metadata. kTypeMaxValid is not supposed to be persisted and can change as new value types are introduced. This can cause a forward compatibility issue where older versions of RocksDB don't recognize kTypeMaxValid from newer versions. A new placeholder value type kTypeTruncatedRangeDeletionSentinel is also introduced to replace kTypeMaxValid when reading existing SST files' metadata from MANIFEST. This allows us to strengthen some checks to avoid using kTypeMaxValid in the future.
				`@@ -1 +0,0 @@`
				Fixed a bug where `DB::GetSortedWalFiles()` could hang when waiting for a purge operation that found nothing to do (potentially triggered by iterator release, flush, compaction, etc.).
				`@@ -1 +0,0 @@`
				Fixed a bug in MultiScan where `max_sequential_skip_in_iterations` could cause the iterator to seek backward to already-unpinned blocks when the same user key spans multiple data blocks, leading to assertion failures or seg fault.
				`@@ -1 +0,0 @@`
				Fixed a bug for `WAL_ttl_seconds > 0` use cases where the newest archived WAL files could be incorrectly deleted when the system clock moved backwards.
				`@@ -1 +0,0 @@`
				`Add a new option allow_trivial_move in CompactionOptions to allow CompactFiles to perform trivial move if possible. By default the flag of allow_trivial_move is false, so it preserve the original behavior.`
				`@@ -1 +0,0 @@`
				* Added an auto-tuning feature for DB manifest file size that also (by default) improves the safety of existing configurations in case `max_manifest_file_size` is repeatedly exceeded. The new recommendation is to set `max_manifest_file_size` to something small like 1MB and tune `max_manifest_space_amp_pct` as needed to balance write amp and space amp in the manifest. Refer to comments on those options in `DBOptions` for details. Both options are (now) mutable.
				`@@ -1 +0,0 @@`
				`Added a new API to support option migration for multiple column families`