Add a FS flag to detect and correct corruption (#12408)

Summary:
Add a flag in `IOOptions` to request the file system to make best efforts to detect data corruption and reconstruct the data if possible. This will be used by RocksDB to retry a read if the previous read returns corrupt data (checksum mismatch). Add a new op to `FSSupportedOps` that, if supported, will trigger this behavior in RocksDB.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/12408

Reviewed By: akankshamahajan15

Differential Revision: D54564218

Pulled By: anand1976

fbshipit-source-id: bc401dcd22a7d320bf63b5183c41714acdce39f5

.github/actions/windows-build-steps/action.yml

@@ -15,30 +15,40 @@ runs:
       SNAPPY_INCLUDE: ${{ github.workspace }}/thirdparty/snappy-1.1.8;${{ github.workspace }}/thirdparty/snappy-1.1.8/build
       SNAPPY_LIB_DEBUG: ${{ github.workspace }}/thirdparty/snappy-1.1.8/build/Debug/snappy.lib
     run: |-
+      # NOTE: if ... Exit $LASTEXITCODE lines needed to exit and report failure
       echo ===================== Install Dependencies =====================
       choco install liberica8jdk -y
+      if(!$?) { Exit $LASTEXITCODE }
       mkdir $Env:THIRDPARTY_HOME
       cd $Env:THIRDPARTY_HOME
       echo "Building Snappy dependency..."
       curl -Lo snappy-1.1.8.zip https://github.com/google/snappy/archive/refs/tags/1.1.8.zip
+      if(!$?) { Exit $LASTEXITCODE }
       unzip -q snappy-1.1.8.zip
+      if(!$?) { Exit $LASTEXITCODE }
       cd snappy-1.1.8
       mkdir build
       cd build
       & cmake -G "$Env:CMAKE_GENERATOR" ..
+      if(!$?) { Exit $LASTEXITCODE }
       msbuild Snappy.sln -maxCpuCount -property:Configuration=Debug -property:Platform=x64
+      if(!$?) { Exit $LASTEXITCODE }
       echo ======================== Build RocksDB =========================
       cd ${{ github.workspace }}
       $env:Path = $env:JAVA_HOME + ";" + $env:Path
       mkdir build
       cd build
       & cmake -G "$Env:CMAKE_GENERATOR" -DCMAKE_BUILD_TYPE=Debug -DOPTDBG=1 -DPORTABLE="$Env:CMAKE_PORTABLE" -DSNAPPY=1 -DJNI=1 ..
+      if(!$?) { Exit $LASTEXITCODE }
       cd ..
       echo "Building with VS version: $Env:CMAKE_GENERATOR"
       msbuild build/rocksdb.sln -maxCpuCount -property:Configuration=Debug -property:Platform=x64
+      if(!$?) { Exit $LASTEXITCODE }
       echo ========================= Test RocksDB =========================
       build_tools\run_ci_db_test.ps1 -SuiteRun arena_test,db_basic_test,db_test,db_test2,db_merge_operand_test,bloom_test,c_test,coding_test,crc32c_test,dynamic_bloom_test,env_basic_test,env_test,hash_test,random_test -Concurrency 16
+      if(!$?) { Exit $LASTEXITCODE }
       echo ======================== Test RocksJava ========================
       cd build\java
       & ctest -C Debug -j 16
+      if(!$?) { Exit $LASTEXITCODE }
     shell: pwsh

HISTORY.md

@@ -1,6 +1,37 @@
 # Rocksdb Change Log
 > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
 
+## 9.0.0 (02/16/2024)
+### New Features
+* Provide support for FSBuffer for point lookups. Also added support for scans and compactions that don't go through prefetching.
+* *Make `SstFileWriter` create SST files without persisting user defined timestamps when the `Option.persist_user_defined_timestamps` flag is set to false.
+* Add support for user-defined timestamps in APIs `DeleteFilesInRanges` and `GetPropertiesOfTablesInRange`.
+* Mark wal\_compression feature as production-ready. Currently only compatible with ZSTD compression.
+
+### Public API Changes
+* Allow setting Stderr logger via C API
+* Declare one Get and one MultiGet variant as pure virtual, and make all the other variants non-overridable. The methods required to be implemented by derived classes of DB allow returning timestamps. It is up to the implementation to check and return an error if timestamps are not supported. The non-batched MultiGet APIs are reimplemented in terms of batched MultiGet, so callers might see a performance improvement.
+* Exposed mode option to Rate Limiter via c api.
+* Removed deprecated option `access_hint_on_compaction_start`
+* Removed deprecated option `ColumnFamilyOptions::check_flush_compaction_key_order`
+* *Remove the default `WritableFile::GetFileSize` and `FSWritableFile::GetFileSize` implementation that returns 0 and make it pure virtual, so that subclasses are enforced to explicitly provide an implementation.
+* Removed deprecated option `ColumnFamilyOptions::level_compaction_dynamic_file_size`
+* *Removed tickers with typos "rocksdb.error.handler.bg.errro.count", "rocksdb.error.handler.bg.io.errro.count", "rocksdb.error.handler.bg.retryable.io.errro.count".
+* Remove the force mode for `EnableFileDeletions` API because it is unsafe with no known legitimate use.
+* Removed deprecated option `ColumnFamilyOptions::ignore_max_compaction_bytes_for_input`
+* `sst_dump --command=check` now compares the number of records in a table with `num_entries` in table property, and reports corruption if there is a mismatch. API `SstFileDumper::ReadSequential()` is updated to optionally do this verification. (#12322)
+
+### Behavior Changes
+* format\_version=6 is the new default setting in BlockBasedTableOptions, for more robust data integrity checking. DBs and SST files written with this setting cannot be read by RocksDB versions before 8.6.0.
+* Compactions can be scheduled in parallel in an additional scenario: multiple files are marked for compaction within a single column family
+* For leveled compaction, RocksDB will try to do intra-L0 compaction if the total L0 size is small compared to Lbase (#12214). Users with atomic_flush=true are more likely to see the impact of this change.
+
+### Bug Fixes
+* Fixed a data race in `DBImpl::RenameTempFileToOptionsFile`.
+* Fix some perf context statistics error in write steps. which include missing write_memtable_time in unordered_write. missing write_memtable_time in PipelineWrite when Writer stat is STATE_PARALLEL_MEMTABLE_WRITER. missing write_delay_time when calling DelayWrite in WriteImplWALOnly function.
+* Fixed a bug that can, under rare circumstances, cause MultiGet to return an incorrect result for a duplicate key in a MultiGet batch.
+* Fix a bug where older data of an ingested key can be returned for read when universal compaction is used
+
 ## 8.11.0 (01/19/2024)
 ### New Features
 * Add new statistics: `rocksdb.sst.write.micros` measures time of each write to SST file; `rocksdb.file.write.{flush|compaction|db.open}.micros` measure time of each write to SST table (currently only block-based table format) and blob file for flush, compaction and db open.

include/rocksdb/file_system.h

@@ -81,7 +81,14 @@ enum class IOType : uint8_t {
 
 // enum representing various operations supported by underlying FileSystem.
 // These need to be set in SupportedOps API for RocksDB to use them.
-enum FSSupportedOps { kAsyncIO, kFSBuffer };
+enum FSSupportedOps {
+  kAsyncIO,   // Supports async reads
+  kFSBuffer,  // Supports handing off the file system allocated read buffer
+              // to the caller of Read/MultiRead
+  kVerifyAndReconstructRead,  // Supports a higher level of data integrity. See
+                              // the verify_and_reconstruct_read flag in
+                              // IOOptions.
+};
 
 // Per-request options that can be passed down to the FileSystem
 // implementation. These are hints and are not necessarily guaranteed to be
@@ -120,6 +127,18 @@ struct IOOptions {
   // directories and list only files in GetChildren API.
   bool do_not_recurse;
 
+  // Setting this flag indicates a corruption was detected by a previous read,
+  // so the caller wants to re-read the data with much stronger data integrity
+  // checking and correction, i.e requests the file system to reconstruct the
+  // data from redundant copies and verify checksums, if available, in order
+  // to have a better chance of success. It is expected that this will have a
+  // much higher overhead than a normal read.
+  // This is a hint. At a minimum, the file system should implement this flag in
+  // FSRandomAccessFile::Read and FSSequentialFile::Read
+  // NOTE: The file system must support kVerifyAndReconstructRead in
+  // FSSupportedOps, otherwise this feature will not be used.
+  bool verify_and_reconstruct_read;
+
   // EXPERIMENTAL
   Env::IOActivity io_activity = Env::IOActivity::kUnknown;
 

java/CMakeLists.txt

@@ -794,20 +794,30 @@ if(NOT MINGW)
   )
 endif()
 
-set(ROCKSDB_JAVADOC_JAR rocksdb-${CMAKE_PROJECT_VERSION}-javadoc.jar)
-add_custom_target(rocksdbjava_javadocs_jar ALL
-        COMMAND ${Java_JAVADOC_EXECUTABLE} -d ${CMAKE_CURRENT_BINARY_DIR}/javadoc -sourcepath ${PROJECT_SOURCE_DIR}/java/src/main/java/ -subpackages org
-        COMMAND ${Java_JAR_EXECUTABLE} -v -c -f ${CMAKE_CURRENT_BINARY_DIR}/${ROCKSDB_JAVADOC_JAR} -C ${CMAKE_CURRENT_BINARY_DIR}/javadoc .
-        BYPRODUCTS ${CMAKE_CURRENT_BINARY_DIR}/${ROCKSDB_JAVADOC_JAR}
-        WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
+# Javadoc Jar
+set(ROCKSDB_JAVADOC_JAR rocksdbjni-${CMAKE_PROJECT_VERSION}-javadoc.jar)
+create_javadoc(rocksdb
+        PACKAGES org.rocksdb org.rocksdb.util
+        SOURCEPATH "${PROJECT_SOURCE_DIR}/java/src/main/java"
+        WINDOWTITLE "RocksDB Java API JavaDoc"
+        AUTHOR FALSE
+        USE FALSE
+        VERSION TRUE
+)
+add_custom_target(rocksdb_javadocs_jar ALL
+  COMMAND ${Java_JAR_EXECUTABLE} cvf ${CMAKE_CURRENT_BINARY_DIR}/${ROCKSDB_JAVADOC_JAR} -C ${CMAKE_CURRENT_BINARY_DIR}/javadoc/rocksdb .
+  BYPRODUCTS ${CMAKE_CURRENT_BINARY_DIR}/${ROCKSDB_JAVADOC_JAR}
+  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
+  DEPENDS rocksdb_javadoc
 )
 
-set(ROCKSDB_SOURCES_JAR rocksdb-${CMAKE_PROJECT_VERSION}-sources.jar)
-add_custom_target(rocksdbjava_sources_jar
-        ${Java_JAR_EXECUTABLE} -v -c -f ${CMAKE_CURRENT_BINARY_DIR}/${ROCKSDB_SOURCES_JAR} -C ${PROJECT_SOURCE_DIR}/java/src/main/java/ .
+# Sources Jar
+set(ROCKSDB_SOURCES_JAR rocksdbjni-${CMAKE_PROJECT_VERSION}-sources.jar)
+add_custom_target(rocksdb_sources_jar ALL
+        ${Java_JAR_EXECUTABLE} cvf ${CMAKE_CURRENT_BINARY_DIR}/${ROCKSDB_SOURCES_JAR} -C ${PROJECT_SOURCE_DIR}/java/src/main/java/ .
         BYPRODUCTS ${CMAKE_CURRENT_BINARY_DIR}/${ROCKSDB_SOURCES_JAR}
         WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
-        )
+)
 
 set(bitness 32)
 if(CMAKE_SIZEOF_VOID_P EQUAL 8)

unreleased_history/behavior_changes/format6.md

@@ -1 +0,0 @@
-format\_version=6 is the new default setting in BlockBasedTableOptions, for more robust data integrity checking. DBs and SST files written with this setting cannot be read by RocksDB versions before 8.6.0.

unreleased_history/behavior_changes/marking_based_speedup.md

@@ -1 +0,0 @@
-Compactions can be scheduled in parallel in an additional scenario: multiple files are marked for compaction within a single column family

unreleased_history/behavior_changes/more-intra-l0-compaction.md

@@ -1 +0,0 @@
-* For leveled compaction, RocksDB will try to do intra-L0 compaction if the total L0 size is small compared to Lbase (#12214). Users with atomic_flush=true are more likely to see the impact of this change.

unreleased_history/bug_fixes/fix_rename_temp_file_to_options_file_race.md

@@ -1 +0,0 @@
-Fixed a data race in `DBImpl::RenameTempFileToOptionsFile`.

unreleased_history/bug_fixes/fix_some_write_perf_statistics.md

@@ -1 +0,0 @@
-Fix some perf context statistics error in write steps. which include missing write_memtable_time in unordered_write. missing write_memtable_time in PipelineWrite when Writer stat is STATE_PARALLEL_MEMTABLE_WRITER. missing write_delay_time when calling DelayWrite in WriteImplWALOnly function.

unreleased_history/bug_fixes/mget_duplicates.md

@@ -1 +0,0 @@
-Fixed a bug that can, under rare circumstances, cause MultiGet to return an incorrect result for a duplicate key in a MultiGet batch.

unreleased_history/bug_fixes/new_ingested_data_with_old_seqno.md

@@ -1 +0,0 @@
-Fix a bug where older data of an ingested key can be returned for read when universal compaction is used

unreleased_history/new_features/fsbuffer.md

@@ -1 +0,0 @@
-Provide support for FSBuffer for point lookups. Also added support for scans and compactions that don't go through prefetching.

unreleased_history/new_features/sst_file_writer_not_persist_timestamp.md

@@ -1 +0,0 @@
-*Make `SstFileWriter` create SST files without persisting user defined timestamps when the `Option.persist_user_defined_timestamps` flag is set to false.

unreleased_history/new_features/udt_support_for_some_apis.md

@@ -1 +0,0 @@
-Add support for user-defined timestamps in APIs `DeleteFilesInRanges` and `GetPropertiesOfTablesInRange`.

unreleased_history/new_features/wal_compression_production.md

@@ -1 +0,0 @@
-Mark wal\_compression feature as production-ready. Currently only compatible with ZSTD compression.

unreleased_history/public_api_changes/allow_setting_stderr_logger_via_c_api.md

@@ -1 +0,0 @@
-Allow setting Stderr logger via C API

unreleased_history/public_api_changes/consolidate_get_mget.md

@@ -1 +0,0 @@
-Declare one Get and one MultiGet variant as pure virtual, and make all the other variants non-overridable. The methods required to be implemented by derived classes of DB allow returning timestamps. It is up to the implementation to check and return an error if timestamps are not supported. The non-batched MultiGet APIs are reimplemented in terms of batched MultiGet, so callers might see a performance improvement.

unreleased_history/public_api_changes/expose_mode_option_rate_limiter_via_c_api.md

@@ -1 +0,0 @@
-Exposed mode option to Rate Limiter via c api.

unreleased_history/public_api_changes/remove_access_hint_on_compaction_start

@@ -1 +0,0 @@
-Removed deprecated option `access_hint_on_compaction_start`

unreleased_history/public_api_changes/remove_check_flush_compaction_key_order.md

@@ -1 +0,0 @@
-Removed deprecated option `ColumnFamilyOptions::check_flush_compaction_key_order`

unreleased_history/public_api_changes/remove_default_get_file_size.md

@@ -1 +0,0 @@
-*Remove the default `WritableFile::GetFileSize` and `FSWritableFile::GetFileSize` implementation that returns 0 and make it pure virtual, so that subclasses are enforced to explicitly provide an implementation.

unreleased_history/public_api_changes/remove_dynamic_level_size.md

@@ -1 +0,0 @@
-Removed deprecated option `ColumnFamilyOptions::level_compaction_dynamic_file_size`

unreleased_history/public_api_changes/remove_error_handler_typo_tickers.md

@@ -1 +0,0 @@
-*Removed tickers with typos "rocksdb.error.handler.bg.errro.count", "rocksdb.error.handler.bg.io.errro.count", "rocksdb.error.handler.bg.retryable.io.errro.count".

unreleased_history/public_api_changes/remove_force_file_deletion.md

@@ -1 +0,0 @@
-Remove the force mode for `EnableFileDeletions` API because it is unsafe with no known legitimate use.

unreleased_history/public_api_changes/remove_ignore_max_compaction_bytes_for_input.md

@@ -1 +0,0 @@
-Removed deprecated option `ColumnFamilyOptions::ignore_max_compaction_bytes_for_input`

unreleased_history/public_api_changes/sst-dump-verify-num-entries.md

@@ -1 +0,0 @@
-* `sst_dump --command=check` now compares the number of records in a table with `num_entries` in table property, and reports corruption if there is a mismatch. API `SstFileDumper::ReadSequential()` is updated to optionally do this verification. (#12322)