Integrated BlobDB for backup/restore support (#8129 )

Summary: Add support for blob files for backup/restore like table files. Since DB session ID is currently not supported for blob files (there is no place to store it in the header), so for blob files uses the kLegacyCrc32cAndFileSize naming scheme even if share_files_with_checksum_naming is set to kUseDbSessionId. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8129 Test Plan: Add new test units Reviewed By: ltamasi Differential Revision: D27408510 Pulled By: akankshamahajan15 fbshipit-source-id: b27434d189a639ef3e6ad165c61a143a2daaf06e
Fix read-only DB writing to filesystem with write_dbid_to_manifest (#8164 )
2026-07-07 22:55:23 +08:00 · 2021-04-07 13:38:54 -07:00 · 2021-04-07 10:26:47 -07:00 · 2021-04-06 23:31:51 -07:00 · 2021-04-06 16:02:04 -07:00 · 2021-04-06 14:37:53 -07:00
177 changed files with 3841 additions and 2067 deletions
@@ -178,7 +178,7 @@ jobs:
    steps:
      - pre-steps
      - install-gflags
-      - run: ASSERT_STATUS_CHECKED=1 TEST_UINT128_COMPAT=1 ROCKSDB_MODIFY_NPHASH=1 LIB_MODE=shared OPT="-DROCKSDB_NAMESPACE=alternative_rocksdb_ns" make V=1 -j32 all check_some | .circleci/cat_ignore_eagain
+      - run: ASSERT_STATUS_CHECKED=1 TEST_UINT128_COMPAT=1 ROCKSDB_MODIFY_NPHASH=1 LIB_MODE=shared OPT="-DROCKSDB_NAMESPACE=alternative_rocksdb_ns" make V=1 -j32 check | .circleci/cat_ignore_eagain
      - post-steps

  build-linux-release:
@@ -583,7 +583,6 @@ set(SOURCES
        db/builder.cc
        db/c.cc
        db/column_family.cc
-        db/compacted_db_impl.cc
        db/compaction/compaction.cc
        db/compaction/compaction_iterator.cc
        db/compaction/compaction_picker.cc
@@ -594,6 +593,7 @@ set(SOURCES
        db/compaction/sst_partitioner.cc
        db/convenience.cc
        db/db_filesnapshot.cc
+        db/db_impl/compacted_db_impl.cc
        db/db_impl/db_impl.cc
        db/db_impl/db_impl_write.cc
        db/db_impl/db_impl_compaction_flush.cc
@@ -651,6 +651,7 @@ set(SOURCES
        env/env_hdfs.cc
        env/file_system.cc
        env/file_system_tracer.cc
+        env/fs_remap.cc
        env/mock_env.cc
        file/delete_scheduler.cc
        file/file_prefetch_buffer.cc
@@ -1,4 +1,28 @@
 # Rocksdb Change Log
+## Unreleased
+### Behavior Changes
+* `ColumnFamilyOptions::sample_for_compression` now takes effect for creation of all block-based tables. Previously it only took effect for block-based tables created by flush.
+* `CompactFiles()` can no longer compact files from lower level to up level, which has the risk to corrupt DB (details: #8063). The validation is also added to all compactions.
+* Fixed some cases in which DB::OpenForReadOnly() could write to the filesystem. If you want a Logger with a read-only DB, you must now set DBOptions::info_log yourself, such as using CreateLoggerFromOptions().
+
+### Bug Fixes
+* Use thread-safe `strerror_r()` to get error messages.
+* Fixed a potential hang in shutdown for a DB whose `Env` has high-pri thread pool disabled (`Env::GetBackgroundThreads(Env::Priority::HIGH) == 0`)
+* Made BackupEngine thread-safe and added documentation comments to clarify what is safe for multiple BackupEngine objects accessing the same backup directory.
+* Fixed crash (divide by zero) when compression dictionary is applied to a file containing only range tombstones.
+* Fixed a backward iteration bug with partitioned filter enabled: not including the prefix of the last key of the previous filter partition in current filter partition can cause wrong iteration result.
+
+### Performance Improvements
+* On ARM platform, use `yield` instead of `wfe` to relax cpu to gain better performance.
+
+### Public API change
+* Added `TableProperties::slow_compression_estimated_data_size` and `TableProperties::fast_compression_estimated_data_size`. When `ColumnFamilyOptions::sample_for_compression > 0`, they estimate what `TableProperties::data_size` would have been if the "fast" or "slow" (see `ColumnFamilyOptions::sample_for_compression` API doc for definitions) compression had been used instead.
+* Update DB::StartIOTrace and remove Env object from the arguments as its redundant and DB already has Env object that is passed down to IOTracer::StartIOTrace
+* For new integrated BlobDB, add support for blob files for backup/restore like table files. Because of current limitations, blob files always use the kLegacyCrc32cAndFileSize naming scheme, and incremental backups must read and checksum all blob files in a DB, even for files that are already backed up.
+
+### New Features
+* Added the ability to open BackupEngine backups as read-only DBs, using BackupInfo::name_for_open and env_for_open provided by BackupEngine::GetBackupInfo() with include_file_details=true.
+
 ## 6.19.0 (03/21/2021)
 ### Bug Fixes
 * Fixed the truncation error found in APIs/tools when dumping block-based SST files in a human-readable format. After fix, the block-based table can be fully dumped as a readable file.
@@ -136,6 +160,7 @@
 * The settings of the DBOptions and ColumnFamilyOptions are now managed by Configurable objects (see New Features).  The same convenience methods to configure these options still exist but the backend implementation has been unified under a common implementation.

 ### New Features
+
 * Methods to configure serialize, and compare -- such as TableFactory -- are exposed directly through the Configurable base class (from which these objects inherit).  This change will allow for better and more thorough configuration management and retrieval in the future.  The options for a Configurable object can be set via the ConfigureFromMap, ConfigureFromString, or ConfigureOption method.  The serialized version of the options of an object can be retrieved via the GetOptionString, ToString, or GetOption methods.  The list of options supported by an object can be obtained via the GetOptionNames method.  The "raw" object (such as the BlockBasedTableOption) for an option may be retrieved via the GetOptions method.  Configurable options can be compared via the AreEquivalent method.  The settings within a Configurable object may be validated via the ValidateOptions method.  The object may be intialized (at which point only mutable options may be updated) via the PrepareOptions method.
 * Introduce options.check_flush_compaction_key_order with default value to be true. With this option, during flush and compaction, key order will be checked when writing to each SST file. If the order is violated, the flush or compaction will fail.
 * Added is_full_compaction to CompactionJobStats, so that the information is available through the EventListener interface.
@@ -43,6 +43,8 @@ to build a portable binary, add `PORTABLE=1` before your make commands, like thi
      command line flags processing. You can compile rocksdb library even
      if you don't have gflags installed.

+* `make check` will also check code formatting, which requires [clang-format](https://clang.llvm.org/docs/ClangFormat.html)
+
 * If you wish to build the RocksJava static target, then cmake is required for building Snappy.

 ## Supported platforms
@@ -94,12 +96,21 @@ to build a portable binary, add `PORTABLE=1` before your make commands, like thi
              sudo yum install libasan

    * Install zstandard:
+        * With [EPEL](https://fedoraproject.org/wiki/EPEL):

-             wget https://github.com/facebook/zstd/archive/v1.1.3.tar.gz
-             mv v1.1.3.tar.gz zstd-1.1.3.tar.gz
-             tar zxvf zstd-1.1.3.tar.gz
-             cd zstd-1.1.3
-             make && sudo make install
+              sudo yum install libzstd-devel
+
+        * With CentOS 8:
+
+              sudo dnf install libzstd-devel
+
+        * From source:
+
+              wget https://github.com/facebook/zstd/archive/v1.1.3.tar.gz
+              mv v1.1.3.tar.gz zstd-1.1.3.tar.gz
+              tar zxvf zstd-1.1.3.tar.gz
+              cd zstd-1.1.3
+              make && sudo make install

 * **OS X**:
    * Install latest C++ compiler that supports C++ 11:
@@ -259,6 +259,8 @@ AM_SHARE = $(AM_V_CCLD) $(CXX) $(PLATFORM_SHARED_LDFLAGS)$@ -L. $(patsubst lib%.
 # Export some common variables that might have been passed as Make variables
 # instead of environment variables.
 dummy := $(shell (export ROCKSDB_ROOT="$(CURDIR)"; \
+                  export CXXFLAGS="$(EXTRA_CXXFLAGS)"; \
+                  export LDFLAGS="$(EXTRA_LDFLAGS)"; \
                  export COMPILE_WITH_ASAN="$(COMPILE_WITH_ASAN)"; \
                  export COMPILE_WITH_TSAN="$(COMPILE_WITH_TSAN)"; \
                  export COMPILE_WITH_UBSAN="$(COMPILE_WITH_UBSAN)"; \
@@ -501,6 +503,12 @@ ifeq ($(USE_FOLLY_DISTRIBUTED_MUTEX),1)
  LIB_OBJECTS += $(patsubst %.cpp, $(OBJ_DIR)/%.o, $(FOLLY_SOURCES))
 endif

+# range_tree is not compatible with non GNU libc on ppc64
+# see https://jira.percona.com/browse/PS-7559
+ifneq ($(PPC_LIBC_IS_GNU),0)
+  LIB_OBJECTS += $(patsubst %.cc, $(OBJ_DIR)/%.o, $(RANGE_TREE_SOURCES))
+endif
+
 GTEST = $(OBJ_DIR)/$(GTEST_DIR)/gtest/gtest-all.o
 TESTUTIL = $(OBJ_DIR)/test_util/testutil.o
 TESTHARNESS = $(OBJ_DIR)/test_util/testharness.o $(TESTUTIL) $(GTEST)
@@ -515,7 +523,8 @@ TOOL_OBJECTS = $(patsubst %.cc, $(OBJ_DIR)/%.o, $(TOOL_LIB_SOURCES))
 ANALYZE_OBJECTS = $(patsubst %.cc, $(OBJ_DIR)/%.o, $(ANALYZER_LIB_SOURCES))
 STRESS_OBJECTS =  $(patsubst %.cc, $(OBJ_DIR)/%.o, $(STRESS_LIB_SOURCES))

-ALL_SOURCES  = $(LIB_SOURCES) $(TEST_LIB_SOURCES) $(MOCK_LIB_SOURCES) $(GTEST_DIR)/gtest/gtest-all.cc
+# Exclude build_version.cc -- a generated source file -- from all sources.  Not needed for dependencies
+ALL_SOURCES  = $(filter-out util/build_version.cc, $(LIB_SOURCES)) $(TEST_LIB_SOURCES) $(MOCK_LIB_SOURCES) $(GTEST_DIR)/gtest/gtest-all.cc
 ALL_SOURCES += $(TOOL_LIB_SOURCES) $(BENCH_LIB_SOURCES) $(ANALYZER_LIB_SOURCES) $(STRESS_LIB_SOURCES)
 ALL_SOURCES += $(TEST_MAIN_SOURCES) $(TOOL_MAIN_SOURCES) $(BENCH_MAIN_SOURCES)

@@ -527,236 +536,34 @@ ifeq ($(USE_FOLLY_DISTRIBUTED_MUTEX),1)
 	ALL_SOURCES += third-party/folly/folly/synchronization/test/DistributedMutexTest.cc
 endif

-PARALLEL_TEST = \
-	backupable_db_test \
-	db_bloom_filter_test \
-	db_compaction_filter_test \
-	db_compaction_test \
-	db_merge_operator_test \
-	db_sst_test \
-	db_test \
-	db_test2 \
-	db_universal_compaction_test \
-	db_wal_test \
-	column_family_test \
-	external_sst_file_test \
-	import_column_family_test \
-	fault_injection_test \
-	file_reader_writer_test \
-	inlineskiplist_test \
-	manual_compaction_test \
-	persistent_cache_test \
-	table_test \
-	transaction_test \
-	point_lock_manager_test \
-	range_locking_test \
-	write_prepared_transaction_test \
-	write_unprepared_transaction_test \
-
-ifeq ($(USE_FOLLY_DISTRIBUTED_MUTEX),1)
-	TESTS += folly_synchronization_distributed_mutex_test
-	PARALLEL_TEST += folly_synchronization_distributed_mutex_test
-	TESTS_PASSING_ASC = folly_synchronization_distributed_mutex_test
-endif
-
 # options_settable_test doesn't pass with UBSAN as we use hack in the test
 ifdef COMPILE_WITH_UBSAN
        TESTS := $(shell echo $(TESTS) | sed 's/\boptions_settable_test\b//g')
 endif
 ifdef ASSERT_STATUS_CHECKED
-# This is a new check for which we will add support incrementally. This
-# list can be removed once support is fully added.
-	TESTS_PASSING_ASC = \
-		arena_test \
-		autovector_test \
-		cache_test \
-		lru_cache_test \
-		blob_file_addition_test \
-		blob_file_builder_test \
-		blob_file_cache_test \
-		blob_file_garbage_test \
-		blob_file_reader_test \
-		bloom_test \
-		cassandra_format_test \
-		cassandra_functional_test \
-		cassandra_row_merge_test \
-		cassandra_serialize_test \
-		cleanable_test \
-		checkpoint_test \
-		coding_test \
-		crc32c_test \
-		dbformat_test \
-		db_basic_test \
-		compact_files_test \
-		compaction_picker_test \
-		comparator_db_test \
-		db_encryption_test \
-		db_iter_test \
-		db_iter_stress_test \
-		db_log_iter_test \
-		db_bloom_filter_test \
-		db_blob_basic_test \
-		db_blob_compaction_test \
-		db_blob_corruption_test \
-		db_blob_index_test \
-		db_block_cache_test \
-		db_compaction_test \
-		db_compaction_filter_test \
-		db_dynamic_level_test \
-		db_flush_test \
-		db_inplace_update_test \
-		db_io_failure_test \
-		db_iterator_test \
-		db_kv_checksum_test \
-		db_logical_block_size_cache_test \
-		db_memtable_test \
-		db_merge_operand_test \
-		db_merge_operator_test \
-		db_wal_test \
-		db_with_timestamp_basic_test \
-		db_with_timestamp_compaction_test \
-		db_write_test \
-		db_options_test \
-		db_properties_test \
-		db_range_del_test \
-		db_secondary_test \
-		deletefile_test \
-		external_sst_file_test \
-		options_file_test \
-		db_sst_test \
-		db_statistics_test \
-		db_table_properties_test \
-		db_tailing_iter_test \
-		fault_injection_test \
-		listener_test \
-		log_test \
-		manual_compaction_test \
-		obsolete_files_test \
-		perf_context_test \
-		periodic_work_scheduler_test \
-		perf_context_test \
-		version_set_test \
-		wal_manager_test \
-		defer_test \
-		filename_test \
-		dynamic_bloom_test \
-		env_basic_test \
-		env_test \
-		env_logger_test \
-		event_logger_test \
-		error_handler_fs_test \
-		external_sst_file_basic_test \
-		auto_roll_logger_test \
-		file_indexer_test \
-		delete_scheduler_test \
-		flush_job_test \
-		hash_table_test \
-		hash_test \
-		heap_test \
-		histogram_test \
-		inlineskiplist_test \
-		io_posix_test \
-		iostats_context_test \
-		ldb_cmd_test \
-		memkind_kmem_allocator_test \
-		merge_test \
-		merger_test \
-		mock_env_test \
-		object_registry_test \
-		optimistic_transaction_test \
-		prefix_test \
-		plain_table_db_test \
-		repair_test \
-		configurable_test \
-		customizable_test \
-		options_settable_test \
-		options_test \
-		point_lock_manager_test \
-		random_access_file_reader_test \
-		random_test \
-		range_del_aggregator_test \
-		sst_file_reader_test \
-		range_tombstone_fragmenter_test \
-		repeatable_thread_test \
-		ribbon_test \
-		skiplist_test \
-		slice_test \
-		slice_transform_test \
-		sst_dump_test \
-		statistics_test \
-		stats_history_test \
-		stringappend_test \
-		thread_local_test \
-		trace_analyzer_test \
-		transaction_test \
-		env_timed_test \
-		filelock_test \
-		timer_queue_test \
-		timer_test \
-		options_util_test \
-		persistent_cache_test \
-		util_merge_operators_test \
-		block_cache_trace_analyzer_test \
-		block_cache_tracer_test \
-		cache_simulator_test \
-		sim_cache_test \
-		version_builder_test \
-		version_edit_test \
-		work_queue_test \
-		write_buffer_manager_test \
-		write_controller_test \
-		write_prepared_transaction_test \
-		write_unprepared_transaction_test \
-		compaction_iterator_test \
-		compaction_job_test \
-		compaction_job_stats_test \
-		io_tracer_test \
-		io_tracer_parser_test \
-		prefetch_test \
-		merge_helper_test \
-		memtable_list_test \
-		flush_job_test \
-		block_based_filter_block_test \
-		block_fetcher_test \
-		block_test \
-		data_block_hash_index_test \
-		full_filter_block_test \
-		partitioned_filter_block_test \
-		column_family_test \
-		file_reader_writer_test \
-		rate_limiter_test \
-		corruption_test \
-		reduce_levels_test \
-		thread_list_test \
-		compact_on_deletion_collector_test \
-		db_universal_compaction_test \
-		import_column_family_test \
-		option_change_migration_test \
-		cuckoo_table_builder_test \
-		cuckoo_table_db_test \
-		cuckoo_table_reader_test \
-		memory_test \
-		table_test \
-		backupable_db_test \
-		blob_db_test \
-		ttl_test \
-		write_batch_test \
-		write_batch_with_index_test \
+	# TODO: finish fixing all tests to pass this check
+	TESTS_FAILING_ASC = \
+		db_test \
+		db_test2 \
+		range_locking_test \
+		testutil_test \

-ifeq ($(USE_FOLLY_DISTRIBUTED_MUTEX),1)
-TESTS_PASSING_ASC += folly_synchronization_distributed_mutex_test
+	# Since we have very few ASC exclusions left, excluding them from
+	# the build is the most convenient way to exclude them from testing
+	TESTS := $(filter-out $(TESTS_FAILING_ASC),$(TESTS))
 endif

-	# Enable building all unit tests, but use check_some to run only tests
-	# known to pass ASC (ASSERT_STATUS_CHECKED)
-	ROCKSDBTESTS_SUBSET ?= $(TESTS_PASSING_ASC)
-	# Alternate: only build unit tests known to pass ASC, and run them
-	# with make check
-	#TESTS := $(filter $(TESTS_PASSING_ASC),$(TESTS))
-	#PARALLEL_TEST := $(filter $(TESTS_PASSING_ASC),$(PARALLEL_TEST))
-else
-	ROCKSDBTESTS_SUBSET ?= $(TESTS)
-endif
+ROCKSDBTESTS_SUBSET ?= $(TESTS)
+
+# env_test - suspicious use of test::TmpDir
+# deletefile_test - serial because it generates giant temporary files in
+#   its various tests. Parallel can fill up your /dev/shm
+NON_PARALLEL_TEST = \
+	env_test \
+	deletefile_test \
+
+PARALLEL_TEST = $(filter-out $(NON_PARALLEL_TEST), $(TESTS))
+
 # Not necessarily well thought out or up-to-date, but matches old list
 TESTS_PLATFORM_DEPENDENT := \
 	db_basic_test \
@@ -929,7 +736,8 @@ endif  # PLATFORM_SHARED_EXT
 	analyze tools tools_lib \
 	blackbox_crash_test_with_atomic_flush whitebox_crash_test_with_atomic_flush  \
 	blackbox_crash_test_with_txn whitebox_crash_test_with_txn \
-	blackbox_crash_test_with_best_efforts_recovery
+	blackbox_crash_test_with_best_efforts_recovery \
+	blackbox_crash_test_with_ts whitebox_crash_test_with_ts


 all: $(LIBRARY) $(BENCHMARKS) tools tools_lib test_libs $(TESTS)
@@ -1167,6 +975,8 @@ crash_test_with_txn: whitebox_crash_test_with_txn blackbox_crash_test_with_txn

 crash_test_with_best_efforts_recovery: blackbox_crash_test_with_best_efforts_recovery

+crash_test_with_ts: whitebox_crash_test_with_ts blackbox_crash_test_with_ts
+
 blackbox_crash_test: db_stress
 	$(PYTHON) -u tools/db_crashtest.py --simple blackbox $(CRASH_TEST_EXT_ARGS)
 	$(PYTHON) -u tools/db_crashtest.py blackbox $(CRASH_TEST_EXT_ARGS)
@@ -1180,6 +990,9 @@ blackbox_crash_test_with_txn: db_stress
 blackbox_crash_test_with_best_efforts_recovery: db_stress
 	$(PYTHON) -u tools/db_crashtest.py --test_best_efforts_recovery blackbox $(CRASH_TEST_EXT_ARGS)

+blackbox_crash_test_with_ts: db_stress
+	$(PYTHON) -u tools/db_crashtest.py --enable_ts blackbox $(CRASH_TEST_EXT_ARGS)
+
 ifeq ($(CRASH_TEST_KILL_ODD),)
  CRASH_TEST_KILL_ODD=888887
 endif
@@ -1198,6 +1011,10 @@ whitebox_crash_test_with_txn: db_stress
 	$(PYTHON) -u tools/db_crashtest.py --txn whitebox --random_kill_odd \
      $(CRASH_TEST_KILL_ODD) $(CRASH_TEST_EXT_ARGS)

+whitebox_crash_test_with_ts: db_stress
+	$(PYTHON) -u tools/db_crashtest.py --enable_ts whitebox --random_kill_odd \
+      $(CRASH_TEST_KILL_ODD) $(CRASH_TEST_EXT_ARGS)
+
 asan_check: clean
 	COMPILE_WITH_ASAN=1 $(MAKE) check -j32
 	$(MAKE) clean
@@ -1343,8 +1160,9 @@ analyze_incremental:
 		$(MAKE) dbg

 CLEAN_FILES += unity.cc
-unity.cc: Makefile
+unity.cc: Makefile util/build_version.cc.in
 	rm -f $@ $@-t
+	$(AM_V_at)$(gen_build_version) > util/build_version.cc
 	for source_file in $(LIB_SOURCES); do \
 		echo "#include \"$$source_file\"" >> $@-t; \
 	done
@@ -1474,7 +1292,7 @@ memtablerep_bench: $(OBJ_DIR)/memtable/memtablerep_bench.o $(LIBRARY)
 filter_bench: $(OBJ_DIR)/util/filter_bench.o $(LIBRARY)
 	$(AM_LINK)

-db_stress: $(OBJ_DIR)/db_stress_tool/db_stress.o $(STRESS_LIBRARY) $(TOOLS_LIBRARY) $(LIBRARY)
+db_stress: $(OBJ_DIR)/db_stress_tool/db_stress.o $(STRESS_LIBRARY) $(TOOLS_LIBRARY) $(TESTUTIL) $(LIBRARY)
 	$(AM_LINK)

 write_stress: $(OBJ_DIR)/tools/write_stress.o $(LIBRARY)
@@ -2183,8 +2001,8 @@ SNAPPY_DOWNLOAD_BASE ?= https://github.com/google/snappy/archive
 LZ4_VER ?= 1.9.3
 LZ4_SHA256 ?= 030644df4611007ff7dc962d981f390361e6c97a34e5cbc393ddfbe019ffe2c1
 LZ4_DOWNLOAD_BASE ?= https://github.com/lz4/lz4/archive
-ZSTD_VER ?= 1.4.7
-ZSTD_SHA256 ?= 085500c8d0b9c83afbc1dc0d8b4889336ad019eba930c5d6a9c6c86c20c769c8
+ZSTD_VER ?= 1.4.9
+ZSTD_SHA256 ?= acf714d98e3db7b876e5b540cbf6dee298f60eb3c0723104f6d3f065cd60d6a8
 ZSTD_DOWNLOAD_BASE ?= https://github.com/facebook/zstd/archive
 CURL_SSL_OPTS ?= --tlsv1

@@ -2484,12 +2302,14 @@ endif
 # ---------------------------------------------------------------------------
 #  	Source files dependencies detection
 # ---------------------------------------------------------------------------
-
+# If skip dependencies is ON, skip including the dep files
+ifneq ($(SKIP_DEPENDS), 1)
 DEPFILES = $(patsubst %.cc, $(OBJ_DIR)/%.cc.d, $(ALL_SOURCES))
 DEPFILES+ = $(patsubst %.c, $(OBJ_DIR)/%.c.d, $(LIB_SOURCES_C) $(TEST_MAIN_SOURCES_C))
 ifeq ($(USE_FOLLY_DISTRIBUTED_MUTEX),1)
  DEPFILES +=$(patsubst %.cpp, $(OBJ_DIR)/%.cpp.d, $(FOLLY_SOURCES))
 endif
+endif

 # Add proper dependency support so changing a .h file forces a .cc file to
 # rebuild.
@@ -2529,28 +2349,9 @@ endif
 build_subset_tests: $(ROCKSDBTESTS_SUBSET)
 	$(AM_V_GEN)if [ -n "$${ROCKSDBTESTS_SUBSET_TESTS_TO_FILE}" ]; then echo "$(ROCKSDBTESTS_SUBSET)" > "$${ROCKSDBTESTS_SUBSET_TESTS_TO_FILE}"; else echo "$(ROCKSDBTESTS_SUBSET)"; fi

-# if the make goal is either "clean" or "format", we shouldn't
-# try to import the *.d files.
-# TODO(kailiu) The unfamiliarity of Make's conditions leads to the ugly
-# working solution.
-ifneq ($(MAKECMDGOALS),clean)
-ifneq ($(MAKECMDGOALS),format)
-ifneq ($(MAKECMDGOALS),check-format)
-ifneq ($(MAKECMDGOALS),check-buck-targets)
-ifneq ($(MAKECMDGOALS),jclean)
-ifneq ($(MAKECMDGOALS),jtest)
-ifneq ($(MAKECMDGOALS),rocksdbjavastatic)
-ifneq ($(MAKECMDGOALS),rocksdbjavastatic_deps)
-ifneq ($(MAKECMDGOALS),package)
-ifneq ($(MAKECMDGOALS),analyze)
+# Remove the rules for which dependencies should not be generated and see if any are left.
+#If so, include the dependencies; if not, do not include the dependency files
+ROCKS_DEP_RULES=$(filter-out clean format check-format check-buck-targets jclean jtest package analyze tags rocksdbjavastatic% unity.% unity_test, $(MAKECMDGOALS))
+ifneq ("$(ROCKS_DEP_RULES)", "")
 -include $(DEPFILES)
 endif
-endif
-endif
-endif
-endif
-endif
-endif
-endif
-endif
-endif
@@ -69,25 +69,25 @@ def get_cc_files(repo_path):
    return cc_files


-# Get parallel tests from Makefile
-def get_parallel_tests(repo_path):
+# Get non_parallel tests from Makefile
+def get_non_parallel_tests(repo_path):
    Makefile = repo_path + "/Makefile"

    s = set({})

-    found_parallel_tests = False
+    found_non_parallel_tests = False
    for line in open(Makefile):
        line = line.strip()
-        if line.startswith("PARALLEL_TEST ="):
-            found_parallel_tests = True
-        elif found_parallel_tests:
+        if line.startswith("NON_PARALLEL_TEST ="):
+            found_non_parallel_tests = True
+        elif found_non_parallel_tests:
            if line.endswith("\\"):
                # remove the trailing \
                line = line[:-1]
                line = line.strip()
                s.add(line)
            else:
-                # we consumed all the parallel tests
+                # we consumed all the non_parallel tests
                break

    return s
@@ -123,10 +123,10 @@ def generate_targets(repo_path, deps_map):
    src_mk = parse_src_mk(repo_path)
    # get all .cc files
    cc_files = get_cc_files(repo_path)
-    # get parallel tests from Makefile
-    parallel_tests = get_parallel_tests(repo_path)
+    # get non_parallel tests from Makefile
+    non_parallel_tests = get_non_parallel_tests(repo_path)

-    if src_mk is None or cc_files is None or parallel_tests is None:
+    if src_mk is None or cc_files is None or non_parallel_tests is None:
        return False

    extra_argv = ""
@@ -141,11 +141,15 @@ def generate_targets(repo_path, deps_map):
    TARGETS.add_library(
        "rocksdb_lib",
        src_mk["LIB_SOURCES"] +
+        # always add range_tree, it's only excluded on ppc64, which we don't use internally
+        src_mk["RANGE_TREE_SOURCES"] +
        src_mk["TOOL_LIB_SOURCES"])
    # rocksdb_whole_archive_lib
    TARGETS.add_library(
        "rocksdb_whole_archive_lib",
        src_mk["LIB_SOURCES"] +
+        # always add range_tree, it's only excluded on ppc64, which we don't use internally
+        src_mk["RANGE_TREE_SOURCES"] +
        src_mk["TOOL_LIB_SOURCES"],
        deps=None,
        headers=None,
@@ -207,7 +211,7 @@ def generate_targets(repo_path, deps_map):
            TARGETS.register_test(
                test_target_name,
                test_src,
-                test in parallel_tests,
+                test not in non_parallel_tests,
                json.dumps(deps['extra_deps']),
                json.dumps(deps['extra_compiler_flags']))

@@ -87,6 +87,7 @@ cpp_binary(
    os_preprocessor_flags = ROCKSDB_OS_PREPROCESSOR_FLAGS,
    compiler_flags = ROCKSDB_COMPILER_FLAGS,
    preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
+    include_paths = ROCKSDB_INCLUDE_PATHS,
    deps = [":rocksdb_test_lib"],
 ) if not is_opt_mode else None

@@ -94,10 +94,12 @@ ROCKSDB_PREPROCESSOR_FLAGS = [

    # Added missing flags from output of build_detect_platform
    "-DROCKSDB_BACKTRACE",
+]

-    # Directories with files for #include
-    "-I" + REPO_PATH + "include/",
-    "-I" + REPO_PATH,
+# Directories with files for #include
+ROCKSDB_INCLUDE_PATHS = [
+    "",
+    "include",
 ]

 ROCKSDB_ARCH_PREPROCESSOR_FLAGS = {{
@@ -145,6 +147,7 @@ cpp_library(
    os_deps = ROCKSDB_OS_DEPS,
    os_preprocessor_flags = ROCKSDB_OS_PREPROCESSOR_FLAGS,
    preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
+    include_paths = ROCKSDB_INCLUDE_PATHS,
    deps = [{deps}],
    external_deps = ROCKSDB_EXTERNAL_DEPS{extra_external_deps},
    link_whole = {link_whole},
@@ -161,6 +164,7 @@ cpp_library(
    os_deps = ROCKSDB_OS_DEPS,
    os_preprocessor_flags = ROCKSDB_OS_PREPROCESSOR_FLAGS,
    preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
+    include_paths = ROCKSDB_INCLUDE_PATHS,
    deps = ROCKSDB_LIB_DEPS,
    external_deps = ROCKSDB_EXTERNAL_DEPS,
 )
@@ -173,6 +177,7 @@ cpp_binary(
    arch_preprocessor_flags = ROCKSDB_ARCH_PREPROCESSOR_FLAGS,
    compiler_flags = ROCKSDB_COMPILER_FLAGS,
    preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
+    include_paths = ROCKSDB_INCLUDE_PATHS,
    deps = [{deps}],
    external_deps = ROCKSDB_EXTERNAL_DEPS,
 )
@@ -203,6 +208,7 @@ ROCKS_TESTS = [
        os_preprocessor_flags = ROCKSDB_OS_PREPROCESSOR_FLAGS,
        compiler_flags = ROCKSDB_COMPILER_FLAGS + extra_compiler_flags,
        preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
+        include_paths = ROCKSDB_INCLUDE_PATHS,
        deps = [":rocksdb_test_lib"] + extra_deps,
        external_deps = ROCKSDB_EXTERNAL_DEPS + [
            ("googletest", None, "gtest"),
@@ -177,7 +177,7 @@ case "$TARGET_OS" in
        PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt -ldl"
        if test $ROCKSDB_USE_IO_URING; then
            # check for liburing
-            $CXX $CFLAGS -x c++ - -luring -o /dev/null 2>/dev/null  <<EOF
+            $CXX $PLATFORM_CXXFLAGS -x c++ - -luring -o /dev/null 2>/dev/null  <<EOF
              #include <liburing.h>
              int main() {
                struct io_uring ring;
@@ -288,7 +288,7 @@ if [ "$CROSS_COMPILE" = "true" -o "$FBCODE_BUILD" = "true" ]; then
 else
    if ! test $ROCKSDB_DISABLE_FALLOCATE; then
        # Test whether fallocate is available
-        $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <fcntl.h>
          #include <linux/falloc.h>
          int main() {
@@ -304,7 +304,7 @@ EOF
    if ! test $ROCKSDB_DISABLE_SNAPPY; then
        # Test whether Snappy library is installed
        # http://code.google.com/p/snappy/
-        $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <snappy.h>
          int main() {}
 EOF
@@ -319,7 +319,7 @@ EOF
        # Test whether gflags library is installed
        # http://gflags.github.io/gflags/
        # check if the namespace is gflags
-        if $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
+        if $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
          #include <gflags/gflags.h>
          using namespace GFLAGS_NAMESPACE;
          int main() {}
@@ -328,7 +328,7 @@ EOF
          COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=1"
          PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
        # check if namespace is gflags
-        elif $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
+        elif $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
            #include <gflags/gflags.h>
            using namespace gflags;
            int main() {}
@@ -337,7 +337,7 @@ EOF
          COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=1 -DGFLAGS_NAMESPACE=gflags"
          PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
        # check if namespace is google
-        elif $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
+        elif $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
            #include <gflags/gflags.h>
            using namespace google;
            int main() {}
@@ -350,7 +350,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_ZLIB; then
        # Test whether zlib library is installed
-        $CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <zlib.h>
          int main() {}
 EOF
@@ -363,7 +363,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_BZIP; then
        # Test whether bzip library is installed
-        $CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <bzlib.h>
          int main() {}
 EOF
@@ -376,7 +376,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_LZ4; then
        # Test whether lz4 library is installed
-        $CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <lz4.h>
          #include <lz4hc.h>
          int main() {}
@@ -390,7 +390,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_ZSTD; then
        # Test whether zstd library is installed
-        $CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <zstd.h>
          int main() {}
 EOF
@@ -403,7 +403,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_NUMA; then
        # Test whether numa is available
-        $CXX $CFLAGS -x c++ - -o /dev/null -lnuma 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null -lnuma 2>/dev/null  <<EOF
          #include <numa.h>
          #include <numaif.h>
          int main() {}
@@ -417,7 +417,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_TBB; then
        # Test whether tbb is available
-        $CXX $CFLAGS $LDFLAGS -x c++ - -o /dev/null -ltbb 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS $LDFLAGS -x c++ - -o /dev/null -ltbb 2>/dev/null  <<EOF
          #include <tbb/tbb.h>
          int main() {}
 EOF
@@ -430,7 +430,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_JEMALLOC; then
        # Test whether jemalloc is available
-        if echo 'int main() {}' | $CXX $CFLAGS -x c++ - -o /dev/null -ljemalloc \
+        if echo 'int main() {}' | $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null -ljemalloc \
          2>/dev/null; then
            # This will enable some preprocessor identifiers in the Makefile
            JEMALLOC=1
@@ -451,7 +451,7 @@ EOF
    fi
    if ! test $JEMALLOC && ! test $ROCKSDB_DISABLE_TCMALLOC; then
        # jemalloc is not available. Let's try tcmalloc
-        if echo 'int main() {}' | $CXX $CFLAGS -x c++ - -o /dev/null \
+        if echo 'int main() {}' | $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null \
          -ltcmalloc 2>/dev/null; then
            PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -ltcmalloc"
            JAVA_LDFLAGS="$JAVA_LDFLAGS -ltcmalloc"
@@ -460,7 +460,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_MALLOC_USABLE_SIZE; then
        # Test whether malloc_usable_size is available
-        $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <malloc.h>
          int main() {
            size_t res = malloc_usable_size(0);
@@ -475,7 +475,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_MEMKIND; then
        # Test whether memkind library is installed
-        $CXX $CFLAGS $COMMON_FLAGS -lmemkind -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -lmemkind -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <memkind.h>
          int main() {
            memkind_malloc(MEMKIND_DAX_KMEM, 1024);
@@ -491,7 +491,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_PTHREAD_MUTEX_ADAPTIVE_NP; then
        # Test whether PTHREAD_MUTEX_ADAPTIVE_NP mutex type is available
-        $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <pthread.h>
          int main() {
            int x = PTHREAD_MUTEX_ADAPTIVE_NP;
@@ -506,7 +506,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_BACKTRACE; then
        # Test whether backtrace is available
-        $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <execinfo.h>
          int main() {
            void* frames[1];
@@ -518,7 +518,7 @@ EOF
            COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_BACKTRACE"
        else
            # Test whether execinfo library is installed
-            $CXX $CFLAGS -lexecinfo -x c++ - -o /dev/null 2>/dev/null  <<EOF
+            $CXX $PLATFORM_CXXFLAGS -lexecinfo -x c++ - -o /dev/null 2>/dev/null  <<EOF
              #include <execinfo.h>
              int main() {
                void* frames[1];
@@ -535,7 +535,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_PG; then
        # Test if -pg is supported
-        $CXX $CFLAGS -pg -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -pg -x c++ - -o /dev/null 2>/dev/null  <<EOF
          int main() {
            return 0;
          }
@@ -547,7 +547,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_SYNC_FILE_RANGE; then
        # Test whether sync_file_range is supported for compatibility with an old glibc
-        $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <fcntl.h>
          int main() {
            int fd = open("/dev/null", 0);
@@ -561,7 +561,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_SCHED_GETCPU; then
        # Test whether sched_getcpu is supported
-        $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <sched.h>
          int main() {
            int cpuid = sched_getcpu();
@@ -575,7 +575,7 @@ EOF

    if ! test $ROCKSDB_DISABLE_AUXV_GETAUXVAL; then
        # Test whether getauxval is supported
-        $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+        $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
          #include <sys/auxv.h>
          int main() {
            uint64_t auxv = getauxval(AT_HWCAP);
@@ -603,7 +603,7 @@ fi
 # -Wshorten-64-to-32 breaks compilation on FreeBSD i386
 if ! [ "$TARGET_OS" = FreeBSD -a "$TARGET_ARCHITECTURE" = i386 ]; then
  # Test whether -Wshorten-64-to-32 is available
-  $CXX $CFLAGS -x c++ - -o /dev/null -Wshorten-64-to-32 2>/dev/null  <<EOF
+  $CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null -Wshorten-64-to-32 2>/dev/null  <<EOF
    int main() {}
 EOF
  if [ "$?" = 0 ]; then
@@ -668,6 +668,23 @@ else
  fi
 fi

+if test -n "`echo $TARGET_ARCHITECTURE | grep ^ppc64`"; then
+  # check for GNU libc on ppc64
+  $CXX -x c++ - -o /dev/null 2>/dev/null <<EOF
+    #include <stdio.h>
+    #include <stdlib.h>
+    #include <gnu/libc-version.h>
+
+    int main(int argc, char *argv[]) {
+      printf("GNU libc version: %s\n", gnu_get_libc_version());
+      return 0;
+    }
+EOF
+  if [ "$?" != 0 ]; then
+      PPC_LIBC_IS_GNU=0
+  fi
+fi
+
 if test "$TRY_SSE_ETC"; then
  # The USE_SSE flag now means "attempt to compile with widely-available
  # Intel architecture extensions utilized by specific optimizations in the
@@ -861,3 +878,6 @@ echo "LUA_PATH=$LUA_PATH" >> "$OUTPUT"
 if test -n "$USE_FOLLY_DISTRIBUTED_MUTEX"; then
  echo "USE_FOLLY_DISTRIBUTED_MUTEX=$USE_FOLLY_DISTRIBUTED_MUTEX" >> "$OUTPUT"
 fi
+if test -n "$PPC_LIBC_IS_GNU"; then
+  echo "PPC_LIBC_IS_GNU=$PPC_LIBC_IS_GNU" >> "$OUTPUT"
+fi
@@ -136,9 +136,11 @@ then
  FORMAT_UPSTREAM_MERGE_BASE="$(git merge-base "$FORMAT_UPSTREAM" HEAD)"
  # Get the differences
  diffs=$(git diff -U0 "$FORMAT_UPSTREAM_MERGE_BASE" | $CLANG_FORMAT_DIFF -p 1)
+  echo "Checking format of changes not yet in $FORMAT_UPSTREAM..."
 else
  # Check the format of uncommitted lines,
  diffs=$(git diff -U0 HEAD | $CLANG_FORMAT_DIFF -p 1)
+  echo "Checking format of uncommitted changes..."
 fi

 if [ -z "$diffs" ]
@@ -548,6 +548,36 @@ STRESS_CRASH_TEST_WITH_TXN_COMMANDS="[
    }
 ]"

+#
+# RocksDB stress/crash test with timestamp
+#
+STRESS_CRASH_TEST_WITH_TS_COMMANDS="[
+    {
+        'name':'Rocksdb Stress and Crash Test with ts',
+        'oncall':'$ONCALL',
+        'executeLocal': 'true',
+        'timeout': 86400,
+        'steps': [
+            $CLEANUP_ENV,
+            {
+                'name':'Build and run RocksDB debug stress tests',
+                'shell':'cd $WORKING_DIR; $SHM $DEBUG $NON_TSAN_CRASH make J=1 db_stress || $CONTRUN_NAME=db_stress $TASK_CREATION_TOOL',
+                'user':'root',
+                $PARSER
+            },
+            {
+                'name':'Build and run RocksDB debug crash tests with ts',
+                'timeout': 86400,
+                'shell':'cd $WORKING_DIR; $SHM $DEBUG $NON_TSAN_CRASH make J=1 crash_test_with_ts || $CONTRUN_NAME=crash_test_with_ts $TASK_CREATION_TOOL',
+                'user':'root',
+                $PARSER
+            },
+            $UPLOAD_DB_DIR,
+        ],
+        $REPORT
+    }
+]"
+
 # RocksDB write stress test.
 # We run on disk device on purpose (i.e. no $SHM)
 # because we want to add some randomness to fsync commands
@@ -1220,6 +1250,9 @@ case $1 in
  stress_crash_with_txn)
    echo $STRESS_CRASH_TEST_WITH_TXN_COMMANDS
    ;;
+  stress_crash_with_ts)
+    echo $STRESS_CRASH_TEST_WITH_TS_COMMANDS
+    ;;
  write_stress)
    echo $WRITE_STRESS_COMMANDS
    ;;
@@ -239,7 +239,7 @@ class ALIGN_AS(CACHE_LINE_SIZE) LRUCacheShard final : public CacheShard {
  //  not threadsafe
  size_t TEST_GetLRUSize();

-  //  Retrives high pri pool ratio
+  //  Retrieves high pri pool ratio
  double GetHighPriPoolRatio();

 private:
@@ -328,7 +328,7 @@ class LRUCache

  //  Retrieves number of elements in LRU, for unit test purpose only
  size_t TEST_GetLRUSize();
-  //  Retrives high pri pool ratio
+  //  Retrieves high pri pool ratio
  double GetHighPriPoolRatio();

 private:
@@ -30,7 +30,7 @@ class LRUCacheTest : public testing::Test {
    DeleteCache();
    cache_ = reinterpret_cast<LRUCacheShard*>(
        port::cacheline_aligned_alloc(sizeof(LRUCacheShard)));
-    new (cache_) LRUCacheShard(capacity, false /*strict_capcity_limit*/,
+    new (cache_) LRUCacheShard(capacity, false /*strict_capacity_limit*/,
                               high_pri_pool_ratio, use_adaptive_mutex,
                               kDontChargeCacheMetadata);
  }
@@ -223,7 +223,7 @@ TEST_F(DBBlobBasicTest, GenerateIOTracing) {
    std::unique_ptr<TraceWriter> trace_writer;
    ASSERT_OK(
        NewFileTraceWriter(env_, EnvOptions(), trace_file, &trace_writer));
-    ASSERT_OK(db_->StartIOTrace(env_, TraceOptions(), std::move(trace_writer)));
+    ASSERT_OK(db_->StartIOTrace(TraceOptions(), std::move(trace_writer)));

    constexpr char key[] = "key";
    constexpr char blob_value[] = "blob_value";
@@ -236,7 +236,7 @@ TEST_F(DBBlobBasicTest, GenerateIOTracing) {
    ASSERT_OK(env_->FileExists(trace_file));
  }
  {
-    // Parse trace file to check file opertions related to blob files are
+    // Parse trace file to check file operations related to blob files are
    // recorded.
    std::unique_ptr<TraceReader> trace_reader;
    ASSERT_OK(
@@ -52,8 +52,8 @@ TableBuilder* NewTableBuilder(
        int_tbl_prop_collector_factories,
    uint32_t column_family_id, const std::string& column_family_name,
    WritableFileWriter* file, const CompressionType compression_type,
-    uint64_t sample_for_compression, const CompressionOptions& compression_opts,
-    int level, const bool skip_filters, const uint64_t creation_time,
+    const CompressionOptions& compression_opts, int level,
+    const bool skip_filters, const uint64_t creation_time,
    const uint64_t oldest_key_time, const uint64_t target_file_size,
    const uint64_t file_creation_time, const std::string& db_id,
    const std::string& db_session_id) {
@@ -63,10 +63,10 @@ TableBuilder* NewTableBuilder(
  return ioptions.table_factory->NewTableBuilder(
      TableBuilderOptions(ioptions, moptions, internal_comparator,
                          int_tbl_prop_collector_factories, compression_type,
-                          sample_for_compression, compression_opts,
-                          skip_filters, column_family_name, level,
-                          creation_time, oldest_key_time, target_file_size,
-                          file_creation_time, db_id, db_session_id),
+                          compression_opts, skip_filters, column_family_name,
+                          level, creation_time, oldest_key_time,
+                          target_file_size, file_creation_time, db_id,
+                          db_session_id),
      column_family_id, file);
 }

@@ -85,11 +85,10 @@ Status BuildTable(
    std::vector<SequenceNumber> snapshots,
    SequenceNumber earliest_write_conflict_snapshot,
    SnapshotChecker* snapshot_checker, const CompressionType compression,
-    uint64_t sample_for_compression, const CompressionOptions& compression_opts,
-    bool paranoid_file_checks, InternalStats* internal_stats,
-    TableFileCreationReason reason, IOStatus* io_status,
-    const std::shared_ptr<IOTracer>& io_tracer, EventLogger* event_logger,
-    int job_id, const Env::IOPriority io_priority,
+    const CompressionOptions& compression_opts, bool paranoid_file_checks,
+    InternalStats* internal_stats, TableFileCreationReason reason,
+    IOStatus* io_status, const std::shared_ptr<IOTracer>& io_tracer,
+    EventLogger* event_logger, int job_id, const Env::IOPriority io_priority,
    TableProperties* table_properties, int level, const uint64_t creation_time,
    const uint64_t oldest_key_time, Env::WriteLifeTimeHint write_hint,
    const uint64_t file_creation_time, const std::string& db_id,
@@ -163,9 +162,8 @@ Status BuildTable(
      builder = NewTableBuilder(
          ioptions, mutable_cf_options, internal_comparator,
          int_tbl_prop_collector_factories, column_family_id,
-          column_family_name, file_writer.get(), compression,
-          sample_for_compression, compression_opts, level,
-          false /* skip_filters */, creation_time, oldest_key_time,
+          column_family_name, file_writer.get(), compression, compression_opts,
+          level, false /* skip_filters */, creation_time, oldest_key_time,
          0 /*target_file_size*/, file_creation_time, db_id, db_session_id);
    }

@@ -50,7 +50,6 @@ TableBuilder* NewTableBuilder(
        int_tbl_prop_collector_factories,
    uint32_t column_family_id, const std::string& column_family_name,
    WritableFileWriter* file, const CompressionType compression_type,
-    const uint64_t sample_for_compression,
    const CompressionOptions& compression_opts, int level,
    const bool skip_filters = false, const uint64_t creation_time = 0,
    const uint64_t oldest_key_time = 0, const uint64_t target_file_size = 0,
@@ -80,7 +79,6 @@ extern Status BuildTable(
    std::vector<SequenceNumber> snapshots,
    SequenceNumber earliest_write_conflict_snapshot,
    SnapshotChecker* snapshot_checker, const CompressionType compression,
-    const uint64_t sample_for_compression,
    const CompressionOptions& compression_opts, bool paranoid_file_checks,
    InternalStats* internal_stats, TableFileCreationReason reason,
    IOStatus* io_status, const std::shared_ptr<IOTracer>& io_tracer,
@@ -4790,7 +4790,10 @@ void rocksdb_transaction_destroy(rocksdb_transaction_t* txn) {

 const rocksdb_snapshot_t* rocksdb_transaction_get_snapshot(
    rocksdb_transaction_t* txn) {
-  rocksdb_snapshot_t* result = new rocksdb_snapshot_t;
+  // This will be freed later on using free, so use malloc here to avoid a
+  // mismatch
+  rocksdb_snapshot_t* result =
+      (rocksdb_snapshot_t*)malloc(sizeof(rocksdb_snapshot_t));
  result->rep = txn->rep->GetSnapshot();
  return result;
 }
@@ -253,7 +253,7 @@ extern Status CheckCFPathsSupported(const DBOptions& db_options,

 extern ColumnFamilyOptions SanitizeOptions(const ImmutableDBOptions& db_options,
                                           const ColumnFamilyOptions& src);
-// Wrap user defined table proproties collector factories `from cf_options`
+// Wrap user defined table properties collector factories `from cf_options`
 // into internal ones in int_tbl_prop_collector_factories. Add a system internal
 // one too.
 extern void GetIntTblPropCollectorFactory(
@@ -441,7 +441,7 @@ class ColumnFamilyData {
  // Get SuperVersion stored in thread local storage. If it does not exist,
  // get a reference from a current SuperVersion.
  SuperVersion* GetThreadLocalSuperVersion(DBImpl* db);
-  // Try to return SuperVersion back to thread local storage. Retrun true on
+  // Try to return SuperVersion back to thread local storage. Return true on
  // success and false on failure. It fails when the thread local storage
  // contains anything other than SuperVersion::kSVInUse flag.
  bool ReturnThreadLocalSuperVersion(SuperVersion* sv);
@@ -118,6 +118,78 @@ TEST_F(CompactFilesTest, L0ConflictsFiles) {
  delete db;
 }

+TEST_F(CompactFilesTest, MultipleLevel) {
+  Options options;
+  options.create_if_missing = true;
+  options.level_compaction_dynamic_level_bytes = true;
+  options.num_levels = 6;
+  // Add listener
+  FlushedFileCollector* collector = new FlushedFileCollector();
+  options.listeners.emplace_back(collector);
+
+  DB* db = nullptr;
+  DestroyDB(db_name_, options);
+  Status s = DB::Open(options, db_name_, &db);
+  ASSERT_OK(s);
+  ASSERT_NE(db, nullptr);
+
+  // create couple files in L0, L3, L4 and L5
+  for (int i = 5; i > 2; --i) {
+    collector->ClearFlushedFiles();
+    ASSERT_OK(db->Put(WriteOptions(), ToString(i), ""));
+    ASSERT_OK(db->Flush(FlushOptions()));
+    auto l0_files = collector->GetFlushedFiles();
+    ASSERT_OK(db->CompactFiles(CompactionOptions(), l0_files, i));
+
+    std::string prop;
+    ASSERT_TRUE(
+        db->GetProperty("rocksdb.num-files-at-level" + ToString(i), &prop));
+    ASSERT_EQ("1", prop);
+  }
+  ASSERT_OK(db->Put(WriteOptions(), ToString(0), ""));
+  ASSERT_OK(db->Flush(FlushOptions()));
+
+  ColumnFamilyMetaData meta;
+  db->GetColumnFamilyMetaData(&meta);
+  // Compact files except the file in L3
+  std::vector<std::string> files;
+  for (int i = 0; i < 6; ++i) {
+    if (i == 3) continue;
+    for (auto& file : meta.levels[i].files) {
+      files.push_back(file.db_path + "/" + file.name);
+    }
+  }
+
+  SyncPoint::GetInstance()->LoadDependency({
+      {"CompactionJob::Run():Start", "CompactFilesTest.MultipleLevel:0"},
+      {"CompactFilesTest.MultipleLevel:1", "CompactFilesImpl:3"},
+  });
+  SyncPoint::GetInstance()->EnableProcessing();
+
+  std::thread thread([&] {
+    TEST_SYNC_POINT("CompactFilesTest.MultipleLevel:0");
+    ASSERT_OK(db->Put(WriteOptions(), "bar", "v2"));
+    ASSERT_OK(db->Put(WriteOptions(), "foo", "v2"));
+    ASSERT_OK(db->Flush(FlushOptions()));
+    TEST_SYNC_POINT("CompactFilesTest.MultipleLevel:1");
+  });
+
+  // Compaction cannot move up the data to higher level
+  // here we have input file from level 5, so the output level has to be >= 5
+  for (int invalid_output_level = 0; invalid_output_level < 5;
+       invalid_output_level++) {
+    s = db->CompactFiles(CompactionOptions(), files, invalid_output_level);
+    std::cout << s.ToString() << std::endl;
+    ASSERT_TRUE(s.IsInvalidArgument());
+  }
+
+  ASSERT_OK(db->CompactFiles(CompactionOptions(), files, 5));
+  SyncPoint::GetInstance()->DisableProcessing();
+  thread.join();
+
+  delete db;
+}
+
 TEST_F(CompactFilesTest, ObsoleteFiles) {
  Options options;
  // to trigger compaction more easily
@@ -519,7 +519,7 @@ uint64_t Compaction::OutputFilePreallocationSize() const {

  // Over-estimate slightly so we don't end up just barely crossing
  // the threshold
-  // No point to prellocate more than 1GB.
+  // No point to preallocate more than 1GB.
  return std::min(uint64_t{1073741824},
                  preallocation_size + (preallocation_size / 10));
 }
@@ -341,7 +341,7 @@ class Compaction {
  const uint32_t output_path_id_;
  CompressionType output_compression_;
  CompressionOptions output_compression_opts_;
-  // If true, then the comaction can be done by simply deleting input files.
+  // If true, then the compaction can be done by simply deleting input files.
  const bool deletion_compaction_;

  // Compaction input files organized by level. Constant after construction
@@ -135,7 +135,7 @@ CompactionIterator::CompactionIterator(
 }

 CompactionIterator::~CompactionIterator() {
-  // input_ Iteartor lifetime is longer than pinned_iters_mgr_ lifetime
+  // input_ Iterator lifetime is longer than pinned_iters_mgr_ lifetime
  input_->SetPinnedItersMgr(nullptr);
 }

@@ -38,7 +38,7 @@ class NoMergingMergeOp : public MergeOperator {

 // Compaction filter that gets stuck when it sees a particular key,
 // then gets unstuck when told to.
-// Always returns Decition::kRemove.
+// Always returns Decision::kRemove.
 class StallingFilter : public CompactionFilter {
 public:
  Decision FilterV2(int /*level*/, const Slice& key, ValueType /*type*/,
@@ -189,7 +189,7 @@ class FakeCompaction : public CompactionIterator::CompactionProxy {
  bool is_allow_ingest_behind = false;
 };

-// A simplifed snapshot checker which assumes each snapshot has a global
+// A simplified snapshot checker which assumes each snapshot has a global
 // last visible sequence.
 class TestSnapshotChecker : public SnapshotChecker {
 public:
@@ -711,7 +711,7 @@ TEST_P(CompactionIteratorTest, ZeroOutSequenceAtBottomLevel) {
  RunTest({test::KeyStr("a", 1, kTypeValue), test::KeyStr("b", 2, kTypeValue)},
          {"v1", "v2"},
          {test::KeyStr("a", 0, kTypeValue), test::KeyStr("b", 2, kTypeValue)},
-          {"v1", "v2"}, kMaxSequenceNumber /*last_commited_seq*/,
+          {"v1", "v2"}, kMaxSequenceNumber /*last_committed_seq*/,
          nullptr /*merge_operator*/, nullptr /*compaction_filter*/,
          true /*bottommost_level*/);
 }
@@ -720,15 +720,14 @@ TEST_P(CompactionIteratorTest, ZeroOutSequenceAtBottomLevel) {
 // permanently.
 TEST_P(CompactionIteratorTest, RemoveDeletionAtBottomLevel) {
  AddSnapshot(1);
-  RunTest({test::KeyStr("a", 1, kTypeDeletion),
-           test::KeyStr("b", 3, kTypeDeletion),
-           test::KeyStr("b", 1, kTypeValue)},
-          {"", "", ""},
-          {test::KeyStr("b", 3, kTypeDeletion),
-           test::KeyStr("b", 0, kTypeValue)},
-          {"", ""},
-          kMaxSequenceNumber /*last_commited_seq*/, nullptr /*merge_operator*/,
-          nullptr /*compaction_filter*/, true /*bottommost_level*/);
+  RunTest(
+      {test::KeyStr("a", 1, kTypeDeletion), test::KeyStr("b", 3, kTypeDeletion),
+       test::KeyStr("b", 1, kTypeValue)},
+      {"", "", ""},
+      {test::KeyStr("b", 3, kTypeDeletion), test::KeyStr("b", 0, kTypeValue)},
+      {"", ""}, kMaxSequenceNumber /*last_committed_seq*/,
+      nullptr /*merge_operator*/, nullptr /*compaction_filter*/,
+      true /*bottommost_level*/);
 }

 // In bottommost level, single deletions earlier than earliest snapshot can be
@@ -738,7 +737,7 @@ TEST_P(CompactionIteratorTest, RemoveSingleDeletionAtBottomLevel) {
  RunTest({test::KeyStr("a", 1, kTypeSingleDeletion),
           test::KeyStr("b", 2, kTypeSingleDeletion)},
          {"", ""}, {test::KeyStr("b", 2, kTypeSingleDeletion)}, {""},
-          kMaxSequenceNumber /*last_commited_seq*/, nullptr /*merge_operator*/,
+          kMaxSequenceNumber /*last_committed_seq*/, nullptr /*merge_operator*/,
          nullptr /*compaction_filter*/, true /*bottommost_level*/);
 }

@@ -895,7 +894,7 @@ TEST_F(CompactionIteratorWithSnapshotCheckerTest,
          {"v1", "v2", "v3"},
          {test::KeyStr("a", 0, kTypeValue), test::KeyStr("b", 2, kTypeValue),
           test::KeyStr("c", 3, kTypeValue)},
-          {"v1", "v2", "v3"}, kMaxSequenceNumber /*last_commited_seq*/,
+          {"v1", "v2", "v3"}, kMaxSequenceNumber /*last_committed_seq*/,
          nullptr /*merge_operator*/, nullptr /*compaction_filter*/,
          true /*bottommost_level*/);
 }
@@ -906,9 +905,7 @@ TEST_F(CompactionIteratorWithSnapshotCheckerTest,
  RunTest(
      {test::KeyStr("a", 1, kTypeDeletion), test::KeyStr("b", 2, kTypeDeletion),
       test::KeyStr("c", 3, kTypeDeletion)},
-      {"", "", ""},
-      {},
-      {"", ""}, kMaxSequenceNumber /*last_commited_seq*/,
+      {"", "", ""}, {}, {"", ""}, kMaxSequenceNumber /*last_committed_seq*/,
      nullptr /*merge_operator*/, nullptr /*compaction_filter*/,
      true /*bottommost_level*/);
 }
@@ -916,15 +913,14 @@ TEST_F(CompactionIteratorWithSnapshotCheckerTest,
 TEST_F(CompactionIteratorWithSnapshotCheckerTest,
       NotRemoveDeletionIfValuePresentToEarlierSnapshot) {
  AddSnapshot(2,1);
-  RunTest(
-      {test::KeyStr("a", 4, kTypeDeletion), test::KeyStr("a", 1, kTypeValue),
-          test::KeyStr("b", 3, kTypeValue)},
-      {"", "", ""},
-      {test::KeyStr("a", 4, kTypeDeletion), test::KeyStr("a", 0, kTypeValue),
-            test::KeyStr("b", 3, kTypeValue)},
-      {"", "", ""}, kMaxSequenceNumber /*last_commited_seq*/,
-      nullptr /*merge_operator*/, nullptr /*compaction_filter*/,
-      true /*bottommost_level*/);
+  RunTest({test::KeyStr("a", 4, kTypeDeletion),
+           test::KeyStr("a", 1, kTypeValue), test::KeyStr("b", 3, kTypeValue)},
+          {"", "", ""},
+          {test::KeyStr("a", 4, kTypeDeletion),
+           test::KeyStr("a", 0, kTypeValue), test::KeyStr("b", 3, kTypeValue)},
+          {"", "", ""}, kMaxSequenceNumber /*last_committed_seq*/,
+          nullptr /*merge_operator*/, nullptr /*compaction_filter*/,
+          true /*bottommost_level*/);
 }

 TEST_F(CompactionIteratorWithSnapshotCheckerTest,
@@ -936,7 +932,7 @@ TEST_F(CompactionIteratorWithSnapshotCheckerTest,
          {"", "", ""},
          {test::KeyStr("b", 2, kTypeSingleDeletion),
           test::KeyStr("c", 3, kTypeSingleDeletion)},
-          {"", ""}, kMaxSequenceNumber /*last_commited_seq*/,
+          {"", ""}, kMaxSequenceNumber /*last_committed_seq*/,
          nullptr /*merge_operator*/, nullptr /*compaction_filter*/,
          true /*bottommost_level*/);
 }
@@ -986,8 +982,8 @@ TEST_F(CompactionIteratorWithSnapshotCheckerTest,
 }

 // Compaction filter should keep uncommitted key as-is, and
-//   * Convert the latest velue to deletion, and/or
-//   * if latest value is a merge, apply filter to all suequent merges.
+//   * Convert the latest value to deletion, and/or
+//   * if latest value is a merge, apply filter to all subsequent merges.

 TEST_F(CompactionIteratorWithSnapshotCheckerTest, CompactionFilter_Value) {
  std::unique_ptr<CompactionFilter> compaction_filter(
@@ -150,7 +150,7 @@ struct CompactionJob::SubcompactionState {
      // This subcompaction's output could be empty if compaction was aborted
      // before this subcompaction had a chance to generate any output files.
      // When subcompactions are executed sequentially this is more likely and
-      // will be particulalry likely for the later subcompactions to be empty.
+      // will be particularly likely for the later subcompactions to be empty.
      // Once they are run in parallel however it should be much rarer.
      return nullptr;
    } else {
@@ -410,7 +410,7 @@ void CompactionJob::Prepare() {
  AutoThreadOperationStageUpdater stage_updater(
      ThreadStatus::STAGE_COMPACTION_PREPARE);

-  // Generate file_levels_ for compaction berfore making Iterator
+  // Generate file_levels_ for compaction before making Iterator
  auto* c = compact_->compaction;
  assert(c->column_family_data() != nullptr);
  assert(c->column_family_data()->current()->storage_info()->NumLevelFiles(
@@ -1770,7 +1770,6 @@ Status CompactionJob::OpenCompactionOutputFile(
      cfd->internal_comparator(), cfd->int_tbl_prop_collector_factories(),
      cfd->GetID(), cfd->GetName(), sub_compact->outfile.get(),
      sub_compact->compaction->output_compression(),
-      0 /*sample_for_compression */,
      sub_compact->compaction->output_compression_opts(),
      sub_compact->compaction->output_level(), skip_filters,
      oldest_ancester_time, 0 /* oldest_key_time */,
@@ -1004,6 +1004,7 @@ Status CompactionPicker::SanitizeCompactionInputFiles(
  // any currently-existing files.
  for (auto file_num : *input_files) {
    bool found = false;
+    int input_file_level = -1;
    for (const auto& level_meta : cf_meta.levels) {
      for (const auto& file_meta : level_meta.files) {
        if (file_num == TableFileNameToNumber(file_meta.name)) {
@@ -1013,6 +1014,7 @@ Status CompactionPicker::SanitizeCompactionInputFiles(
                                   " is already being compacted.");
          }
          found = true;
+          input_file_level = level_meta.level;
          break;
        }
      }
@@ -1025,6 +1027,13 @@ Status CompactionPicker::SanitizeCompactionInputFiles(
          "Specified compaction input file " + MakeTableFileName("", file_num) +
          " does not exist in column family " + cf_meta.name + ".");
    }
+    if (input_file_level > output_level) {
+      return Status::InvalidArgument(
+          "Cannot compact file to up level, input file: " +
+          MakeTableFileName("", file_num) + " level " +
+          ToString(input_file_level) + " > output level " +
+          ToString(output_level));
+    }
  }

  return Status::OK();
@@ -650,7 +650,7 @@ TEST_F(CompactionPickerTest, UniversalPeriodicCompaction3) {

 TEST_F(CompactionPickerTest, UniversalPeriodicCompaction4) {
  // The case where universal periodic compaction couldn't form
-  // a compaction that inlcudes any file marked for periodic compaction.
+  // a compaction that includes any file marked for periodic compaction.
  // Right now we form the compaction anyway if it is more than one
  // sorted run. Just put the case here to validate that it doesn't
  // crash.
@@ -800,7 +800,7 @@ TEST_F(CompactionPickerTest, CompactionPriMinOverlapping2) {
  Add(2, 6U, "150", "175",
      60000000U);  // Overlaps with file 26, 27, total size 521M
  Add(2, 7U, "176", "200", 60000000U);  // Overlaps with file 27, 28, total size
-                                        // 520M, the smalelst overlapping
+                                        // 520M, the smallest overlapping
  Add(2, 8U, "201", "300",
      60000000U);  // Overlaps with file 28, 29, total size 521M

@@ -1228,7 +1228,7 @@ TEST_F(CompactionPickerTest, NotScheduleL1IfL0WithHigherPri1) {
  Add(0, 32U, "001", "400", 1000000000U, 0, 0);
  Add(0, 33U, "001", "400", 1000000000U, 0, 0);

-  // L1 total size 2GB, score 2.2. If one file being comapcted, score 1.1.
+  // L1 total size 2GB, score 2.2. If one file being compacted, score 1.1.
  Add(1, 4U, "050", "300", 1000000000U, 0, 0);
  file_map_[4u].first->being_compacted = true;
  Add(1, 5U, "301", "350", 1000000000U, 0, 0);
@@ -1261,7 +1261,7 @@ TEST_F(CompactionPickerTest, NotScheduleL1IfL0WithHigherPri2) {
  Add(0, 32U, "001", "400", 1000000000U, 0, 0);
  Add(0, 33U, "001", "400", 1000000000U, 0, 0);

-  // L1 total size 2GB, score 2.2. If one file being comapcted, score 1.1.
+  // L1 total size 2GB, score 2.2. If one file being compacted, score 1.1.
  Add(1, 4U, "050", "300", 1000000000U, 0, 0);
  Add(1, 5U, "301", "350", 1000000000U, 0, 0);

@@ -733,7 +733,7 @@ Compaction* UniversalCompactionBuilder::PickCompactionToReduceSortedRuns(
 }

 // Look at overall size amplification. If size amplification
-// exceeeds the configured value, then do a compaction
+// exceeds the configured value, then do a compaction
 // of the candidate files all the way upto the earliest
 // base file (overrides configured values of file-size ratios,
 // min_merge_width and max_merge_width).
@@ -7,6 +7,9 @@
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

+#include <iomanip>
+#include <sstream>
+
 #include "db/db_test_util.h"
 #include "options/options_helper.h"
 #include "port/stack_trace.h"
@@ -2117,6 +2120,54 @@ TEST_F(DBBloomFilterTest, DynamicBloomFilterOptions) {
  }
 }

+TEST_F(DBBloomFilterTest, SeekForPrevWithPartitionedFilters) {
+  Options options = CurrentOptions();
+  constexpr size_t kNumKeys = 10000;
+  static_assert(kNumKeys <= 10000, "kNumKeys have to be <= 10000");
+  options.memtable_factory.reset(new SpecialSkipListFactory(kNumKeys + 10));
+  options.create_if_missing = true;
+  constexpr size_t kPrefixLength = 4;
+  options.prefix_extractor.reset(NewFixedPrefixTransform(kPrefixLength));
+  options.compression = kNoCompression;
+  BlockBasedTableOptions bbto;
+  bbto.filter_policy.reset(NewBloomFilterPolicy(50));
+  bbto.index_shortening =
+      BlockBasedTableOptions::IndexShorteningMode::kNoShortening;
+  bbto.block_size = 128;
+  bbto.metadata_block_size = 128;
+  bbto.partition_filters = true;
+  bbto.index_type = BlockBasedTableOptions::IndexType::kTwoLevelIndexSearch;
+  options.table_factory.reset(NewBlockBasedTableFactory(bbto));
+  DestroyAndReopen(options);
+
+  const std::string value(64, '\0');
+
+  WriteOptions write_opts;
+  write_opts.disableWAL = true;
+  for (size_t i = 0; i < kNumKeys; ++i) {
+    std::ostringstream oss;
+    oss << std::setfill('0') << std::setw(4) << std::fixed << i;
+    ASSERT_OK(db_->Put(write_opts, oss.str(), value));
+  }
+  ASSERT_OK(Flush());
+
+  ReadOptions read_opts;
+  // Use legacy, implicit prefix seek
+  read_opts.total_order_seek = false;
+  read_opts.auto_prefix_mode = false;
+  std::unique_ptr<Iterator> it(db_->NewIterator(read_opts));
+  for (size_t i = 0; i < kNumKeys; ++i) {
+    // Seek with a key after each one added but with same prefix. One will
+    // surely cross a partition boundary.
+    std::ostringstream oss;
+    oss << std::setfill('0') << std::setw(4) << std::fixed << i << "a";
+    it->SeekForPrev(oss.str());
+    ASSERT_OK(it->status());
+    ASSERT_TRUE(it->Valid());
+  }
+  it.reset();
+}
+
 #endif  // ROCKSDB_LITE

 }  // namespace ROCKSDB_NAMESPACE
@@ -166,6 +166,66 @@ TEST_F(DBFlushTest, FlushInLowPriThreadPool) {
  ASSERT_EQ(1, num_compactions);
 }

+// Test when flush job is submitted to low priority thread pool and when DB is
+// closed in the meanwhile, CloseHelper doesn't hang.
+TEST_F(DBFlushTest, CloseDBWhenFlushInLowPri) {
+  Options options = CurrentOptions();
+  options.max_background_flushes = 1;
+  options.max_total_wal_size = 8192;
+
+  DestroyAndReopen(options);
+  CreateColumnFamilies({"cf1", "cf2"}, options);
+
+  env_->SetBackgroundThreads(0, Env::HIGH);
+  env_->SetBackgroundThreads(1, Env::LOW);
+  test::SleepingBackgroundTask sleeping_task_low;
+  int num_flushes = 0;
+
+  SyncPoint::GetInstance()->SetCallBack("DBImpl::BGWorkFlush",
+                                        [&](void* /*arg*/) { ++num_flushes; });
+
+  int num_low_flush_unscheduled = 0;
+  SyncPoint::GetInstance()->SetCallBack(
+      "DBImpl::UnscheduleLowFlushCallback", [&](void* /*arg*/) {
+        num_low_flush_unscheduled++;
+        // There should be one flush job in low pool that needs to be
+        // unscheduled
+        ASSERT_EQ(num_low_flush_unscheduled, 1);
+      });
+
+  int num_high_flush_unscheduled = 0;
+  SyncPoint::GetInstance()->SetCallBack(
+      "DBImpl::UnscheduleHighFlushCallback", [&](void* /*arg*/) {
+        num_high_flush_unscheduled++;
+        // There should be no flush job in high pool
+        ASSERT_EQ(num_high_flush_unscheduled, 0);
+      });
+
+  SyncPoint::GetInstance()->EnableProcessing();
+
+  ASSERT_OK(Put(0, "key1", DummyString(8192)));
+  // Block thread so that flush cannot be run and can be removed from the queue
+  // when called Unschedule.
+  env_->Schedule(&test::SleepingBackgroundTask::DoSleepTask, &sleeping_task_low,
+                 Env::Priority::LOW);
+  sleeping_task_low.WaitUntilSleeping();
+
+  // Trigger flush and flush job will be scheduled to LOW priority thread.
+  ASSERT_OK(Put(0, "key2", DummyString(8192)));
+
+  // Close DB and flush job in low priority queue will be removed without
+  // running.
+  Close();
+  sleeping_task_low.WakeUp();
+  sleeping_task_low.WaitUntilDone();
+  ASSERT_EQ(0, num_flushes);
+
+  TryReopenWithColumnFamilies({"default", "cf1", "cf2"}, options);
+  ASSERT_OK(Put(0, "key3", DummyString(8192)));
+  ASSERT_OK(Flush(0));
+  ASSERT_EQ(1, num_flushes);
+}
+
 TEST_F(DBFlushTest, ManualFlushWithMinWriteBufferNumberToMerge) {
  Options options = CurrentOptions();
  options.write_buffer_size = 100;
@@ -4,7 +4,7 @@
 //  (found in the LICENSE.Apache file in the root directory).

 #ifndef ROCKSDB_LITE
-#include "db/compacted_db_impl.h"
+#include "db/db_impl/compacted_db_impl.h"

 #include "db/db_impl/db_impl.h"
 #include "db/version_set.h"
@@ -17,11 +17,13 @@ extern void MarkKeyMayExist(void* arg);
 extern bool SaveValue(void* arg, const ParsedInternalKey& parsed_key,
                      const Slice& v, bool hit_and_return);

-CompactedDBImpl::CompactedDBImpl(
-  const DBOptions& options, const std::string& dbname)
-  : DBImpl(options, dbname), cfd_(nullptr), version_(nullptr),
-    user_comparator_(nullptr) {
-}
+CompactedDBImpl::CompactedDBImpl(const DBOptions& options,
+                                 const std::string& dbname)
+    : DBImpl(options, dbname, /*seq_per_batch*/ false, +/*batch_per_txn*/ true,
+             /*read_only*/ true),
+      cfd_(nullptr),
+      version_(nullptr),
+      user_comparator_(nullptr) {}

 CompactedDBImpl::~CompactedDBImpl() {
 }
@@ -78,6 +80,7 @@ std::vector<Status> CompactedDBImpl::MultiGet(const ReadOptions& options,
                             nullptr, nullptr, nullptr, true, nullptr, nullptr);
      LookupKey lkey(keys[idx], kMaxSequenceNumber);
      Status s = r->Get(options, lkey.internal_key(), &get_context, nullptr);
+      assert(static_cast<size_t>(idx) < statuses.size());
      if (!s.ok() && !s.IsNotFound()) {
        statuses[idx] = s;
      } else {
@@ -18,7 +18,7 @@ class CompactedDBImpl : public DBImpl {
  CompactedDBImpl(const CompactedDBImpl&) = delete;
  void operator=(const CompactedDBImpl&) = delete;

-  virtual ~CompactedDBImpl();
+  ~CompactedDBImpl() override;

  static Status Open(const Options& options, const std::string& dbname,
                     DB** dbptr);
@@ -146,10 +146,11 @@ void DumpSupportInfo(Logger* logger) {
 }  // namespace

 DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
-               const bool seq_per_batch, const bool batch_per_txn)
+               const bool seq_per_batch, const bool batch_per_txn,
+               bool read_only)
    : dbname_(dbname),
      own_info_log_(options.info_log == nullptr),
-      initial_db_options_(SanitizeOptions(dbname, options)),
+      initial_db_options_(SanitizeOptions(dbname, options, read_only)),
      env_(initial_db_options_.env),
      io_tracer_(std::make_shared<IOTracer>()),
      immutable_db_options_(initial_db_options_),
@@ -522,15 +523,11 @@ Status DBImpl::CloseHelper() {
  // marker. After this we do a variant of the waiting and unschedule work
  // (to consider: moving all the waiting into CancelAllBackgroundWork(true))
  CancelAllBackgroundWork(false);
-  int bottom_compactions_unscheduled =
-      env_->UnSchedule(this, Env::Priority::BOTTOM);
-  int compactions_unscheduled = env_->UnSchedule(this, Env::Priority::LOW);
-  int flushes_unscheduled = env_->UnSchedule(this, Env::Priority::HIGH);
-  Status ret = Status::OK();
  mutex_.Lock();
-  bg_bottom_compaction_scheduled_ -= bottom_compactions_unscheduled;
-  bg_compaction_scheduled_ -= compactions_unscheduled;
-  bg_flush_scheduled_ -= flushes_unscheduled;
+  env_->UnSchedule(this, Env::Priority::BOTTOM);
+  env_->UnSchedule(this, Env::Priority::LOW);
+  env_->UnSchedule(this, Env::Priority::HIGH);
+  Status ret = Status::OK();

  // Wait for background work to finish
  while (bg_bottom_compaction_scheduled_ || bg_compaction_scheduled_ ||
@@ -3149,7 +3146,7 @@ SystemClock* DBImpl::GetSystemClock() const {

 #ifndef ROCKSDB_LITE

-Status DBImpl::StartIOTrace(Env* /*env*/, const TraceOptions& trace_options,
+Status DBImpl::StartIOTrace(const TraceOptions& trace_options,
                            std::unique_ptr<TraceWriter>&& trace_writer) {
  assert(trace_writer != nullptr);
  return io_tracer_->StartIOTrace(GetSystemClock(), trace_options,
@@ -129,7 +129,8 @@ class Directories {
 class DBImpl : public DB {
 public:
  DBImpl(const DBOptions& options, const std::string& dbname,
-         const bool seq_per_batch = false, const bool batch_per_txn = true);
+         const bool seq_per_batch = false, const bool batch_per_txn = true,
+         bool read_only = false);
  // No copying allowed
  DBImpl(const DBImpl&) = delete;
  void operator=(const DBImpl&) = delete;
@@ -469,7 +470,7 @@ class DBImpl : public DB {
  Status EndBlockCacheTrace() override;

  using DB::StartIOTrace;
-  Status StartIOTrace(Env* env, const TraceOptions& options,
+  Status StartIOTrace(const TraceOptions& options,
                      std::unique_ptr<TraceWriter>&& trace_writer) override;

  using DB::EndIOTrace;
@@ -1236,7 +1237,7 @@ class DBImpl : public DB {
  virtual bool OwnTablesAndLogs() const { return true; }

  // Set DB identity file, and write DB ID to manifest if necessary.
-  Status SetDBId();
+  Status SetDBId(bool read_only);

  // REQUIRES: db mutex held when calling this function, but the db mutex can
  // be released and re-acquired. Db mutex will be held when the function
@@ -1308,6 +1309,7 @@ class DBImpl : public DB {

  struct LogFileNumberSize {
    explicit LogFileNumberSize(uint64_t _number) : number(_number) {}
+    LogFileNumberSize() {}
    void AddSize(uint64_t new_size) { size += new_size; }
    uint64_t number;
    uint64_t size = 0;
@@ -1413,6 +1415,7 @@ class DBImpl : public DB {
    DBImpl* db;
    // background compaction takes ownership of `prepicked_compaction`.
    PrepickedCompaction* prepicked_compaction;
+    Env::Priority compaction_pri_;
  };

  // Initialize the built-in column family for persistent stats. Depending on
@@ -1507,6 +1510,12 @@ class DBImpl : public DB {
  Status WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
                                     MemTable* mem, VersionEdit* edit);

+  // Get the size of a log file and, if truncate is true, truncate the
+  // log file to its actual size, thereby freeing preallocated space.
+  // Return success even if truncate fails
+  Status GetLogSizeAndMaybeTruncate(uint64_t wal_number, bool truncate,
+                                    LogFileNumberSize* log);
+
  // Restore alive_log_files_ and total_log_size_ after recovery.
  // It needs to run only when there's no flush during recovery
  // (e.g. avoid_flush_during_recovery=true). May also trigger flush
@@ -2223,9 +2232,11 @@ class DBImpl : public DB {
  BlobFileCompletionCallback blob_callback_;
 };

-extern Options SanitizeOptions(const std::string& db, const Options& src);
+extern Options SanitizeOptions(const std::string& db, const Options& src,
+                               bool read_only = false);

-extern DBOptions SanitizeOptions(const std::string& db, const DBOptions& src);
+extern DBOptions SanitizeOptions(const std::string& db, const DBOptions& src,
+                                 bool read_only = false);

 extern CompressionType GetCompressionFlush(
    const ImmutableCFOptions& ioptions,
@@ -260,18 +260,16 @@ Status DBImpl::FlushMemTableToOutputFile(
      // be pessimistic and try write to a new MANIFEST.
      // TODO: distinguish between MANIFEST write and CURRENT renaming
      if (!versions_->io_status().ok()) {
-        if (total_log_size_ > 0) {
-          // If the WAL is empty, we use different error reason
-          error_handler_.SetBGError(io_s,
-                                    BackgroundErrorReason::kManifestWrite);
-        } else {
-          error_handler_.SetBGError(io_s,
-                                    BackgroundErrorReason::kManifestWriteNoWAL);
-        }
-      } else if (total_log_size_ > 0 || !log_io_s.ok()) {
-        error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush);
+        // If WAL sync is successful (either WAL size is 0 or there is no IO
+        // error), all the Manifest write will be map to soft error.
+        // TODO: kManifestWriteNoWAL and kFlushNoWAL are misleading. Refactor is
+        // needed.
+        error_handler_.SetBGError(io_s,
+                                  BackgroundErrorReason::kManifestWriteNoWAL);
      } else {
-        // If the WAL is empty, we use different error reason
+        // If WAL sync is successful (either WAL size is 0 or there is no IO
+        // error), all the other SST file write errors will be set as
+        // kFlushNoWAL.
        error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlushNoWAL);
      }
    } else {
@@ -687,18 +685,16 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
      // be pessimistic and try write to a new MANIFEST.
      // TODO: distinguish between MANIFEST write and CURRENT renaming
      if (!versions_->io_status().ok()) {
-        if (total_log_size_ > 0) {
-          // If the WAL is empty, we use different error reason
-          error_handler_.SetBGError(io_s,
-                                    BackgroundErrorReason::kManifestWrite);
-        } else {
-          error_handler_.SetBGError(io_s,
-                                    BackgroundErrorReason::kManifestWriteNoWAL);
-        }
-      } else if (total_log_size_ > 0) {
-        error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush);
+        // If WAL sync is successful (either WAL size is 0 or there is no IO
+        // error), all the Manifest write will be map to soft error.
+        // TODO: kManifestWriteNoWAL and kFlushNoWAL are misleading. Refactor
+        // is needed.
+        error_handler_.SetBGError(io_s,
+                                  BackgroundErrorReason::kManifestWriteNoWAL);
      } else {
-        // If the WAL is empty, we use different error reason
+        // If WAL sync is successful (either WAL size is 0 or there is no IO
+        // error), all the other SST file write errors will be set as
+        // kFlushNoWAL.
        error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlushNoWAL);
      }
    } else {
@@ -1745,6 +1741,7 @@ Status DBImpl::RunManualCompaction(
      }
      ca = new CompactionArg;
      ca->db = this;
+      ca->compaction_pri_ = Env::Priority::LOW;
      ca->prepicked_compaction = new PrepickedCompaction;
      ca->prepicked_compaction->manual_compaction_state = &manual;
      ca->prepicked_compaction->compaction = compaction;
@@ -2276,6 +2273,7 @@ void DBImpl::MaybeScheduleFlushOrCompaction() {
         unscheduled_compactions_ > 0) {
    CompactionArg* ca = new CompactionArg;
    ca->db = this;
+    ca->compaction_pri_ = Env::Priority::LOW;
    ca->prepicked_compaction = nullptr;
    bg_compaction_scheduled_++;
    unscheduled_compactions_--;
@@ -2463,7 +2461,16 @@ void DBImpl::BGWorkPurge(void* db) {
 }

 void DBImpl::UnscheduleCompactionCallback(void* arg) {
-  CompactionArg ca = *(reinterpret_cast<CompactionArg*>(arg));
+  CompactionArg* ca_ptr = reinterpret_cast<CompactionArg*>(arg);
+  Env::Priority compaction_pri = ca_ptr->compaction_pri_;
+  if (Env::Priority::BOTTOM == compaction_pri) {
+    // Decrement bg_bottom_compaction_scheduled_ if priority is BOTTOM
+    ca_ptr->db->bg_bottom_compaction_scheduled_--;
+  } else if (Env::Priority::LOW == compaction_pri) {
+    // Decrement bg_compaction_scheduled_ if priority is LOW
+    ca_ptr->db->bg_compaction_scheduled_--;
+  }
+  CompactionArg ca = *(ca_ptr);
  delete reinterpret_cast<CompactionArg*>(arg);
  if (ca.prepicked_compaction != nullptr) {
    if (ca.prepicked_compaction->compaction != nullptr) {
@@ -2475,6 +2482,14 @@ void DBImpl::UnscheduleCompactionCallback(void* arg) {
 }

 void DBImpl::UnscheduleFlushCallback(void* arg) {
+  // Decrement bg_flush_scheduled_ in flush callback
+  reinterpret_cast<FlushThreadArg*>(arg)->db_->bg_flush_scheduled_--;
+  Env::Priority flush_pri = reinterpret_cast<FlushThreadArg*>(arg)->thread_pri_;
+  if (Env::Priority::LOW == flush_pri) {
+    TEST_SYNC_POINT("DBImpl::UnscheduleLowFlushCallback");
+  } else if (Env::Priority::HIGH == flush_pri) {
+    TEST_SYNC_POINT("DBImpl::UnscheduleHighFlushCallback");
+  }
  delete reinterpret_cast<FlushThreadArg*>(arg);
  TEST_SYNC_POINT("DBImpl::UnscheduleFlushCallback");
 }
@@ -3077,6 +3092,7 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
    TEST_SYNC_POINT("DBImpl::BackgroundCompaction:ForwardToBottomPriPool");
    CompactionArg* ca = new CompactionArg;
    ca->db = this;
+    ca->compaction_pri_ = Env::Priority::BOTTOM;
    ca->prepicked_compaction = new PrepickedCompaction;
    ca->prepicked_compaction->compaction = c.release();
    ca->prepicked_compaction->manual_compaction_state = nullptr;
@@ -854,7 +854,7 @@ uint64_t PrecomputeMinLogNumberToKeep2PC(
  return min_log_number_to_keep;
 }

-Status DBImpl::SetDBId() {
+Status DBImpl::SetDBId(bool read_only) {
  Status s;
  // Happens when immutable_db_options_.write_dbid_to_manifest is set to true
  // the very first time.
@@ -865,9 +865,15 @@ Status DBImpl::SetDBId() {
    // it is no longer available then at this point DB ID is not in Identity
    // file or Manifest.
    if (s.IsNotFound()) {
-      s = SetIdentityFile(env_, dbname_);
-      if (!s.ok()) {
-        return s;
+      // Create a new DB ID, saving to file only if allowed
+      if (read_only) {
+        db_id_ = env_->GenerateUniqueId();
+        return Status::OK();
+      } else {
+        s = SetIdentityFile(env_, dbname_);
+        if (!s.ok()) {
+          return s;
+        }
      }
    } else if (!s.ok()) {
      assert(s.IsIOError());
@@ -884,7 +890,7 @@ Status DBImpl::SetDBId() {
                                 mutable_cf_options, &edit, &mutex_, nullptr,
                                 /* new_descriptor_log */ false);
    }
-  } else {
+  } else if (!read_only) {
    s = SetIdentityFile(env_, dbname_, db_id_);
  }
  return s;
@@ -24,15 +24,17 @@
 #include "util/rate_limiter.h"

 namespace ROCKSDB_NAMESPACE {
-Options SanitizeOptions(const std::string& dbname, const Options& src) {
-  auto db_options = SanitizeOptions(dbname, DBOptions(src));
+Options SanitizeOptions(const std::string& dbname, const Options& src,
+                        bool read_only) {
+  auto db_options = SanitizeOptions(dbname, DBOptions(src), read_only);
  ImmutableDBOptions immutable_db_options(db_options);
  auto cf_options =
      SanitizeOptions(immutable_db_options, ColumnFamilyOptions(src));
  return Options(db_options, cf_options);
 }

-DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src) {
+DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src,
+                          bool read_only) {
  DBOptions result(src);

  if (result.env == nullptr) {
@@ -50,7 +52,7 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src) {
                             &result.max_open_files);
  }

-  if (result.info_log == nullptr) {
+  if (result.info_log == nullptr && !read_only) {
    Status s = CreateLoggerFromOptions(dbname, result, &result.info_log);
    if (!s.ok()) {
      // No place suitable for logging
@@ -488,7 +490,7 @@ Status DBImpl::Recover(
  if (!s.ok()) {
    return s;
  }
-  s = SetDBId();
+  s = SetDBId(read_only);
  if (s.ok() && !read_only) {
    s = DeleteUnreferencedSstFiles();
  }
@@ -1229,8 +1231,16 @@ Status DBImpl::RecoverLogFiles(const std::vector<uint64_t>& wal_numbers,
    }
  }

-  if (status.ok() && data_seen && !flushed) {
-    status = RestoreAliveLogFiles(wal_numbers);
+  if (status.ok()) {
+    if (data_seen && !flushed) {
+      status = RestoreAliveLogFiles(wal_numbers);
+    } else {
+      // If there's no data in the WAL, or we flushed all the data, still
+      // truncate the log file. If the process goes into a crash loop before
+      // the file is deleted, the preallocated space will never get freed.
+      GetLogSizeAndMaybeTruncate(wal_numbers.back(), true, nullptr)
+          .PermitUncheckedError();
+    }
  }

  event_logger_.Log() << "job" << job_id << "event"
@@ -1239,6 +1249,40 @@ Status DBImpl::RecoverLogFiles(const std::vector<uint64_t>& wal_numbers,
  return status;
 }

+Status DBImpl::GetLogSizeAndMaybeTruncate(uint64_t wal_number, bool truncate,
+                                          LogFileNumberSize* log_ptr) {
+  LogFileNumberSize log(wal_number);
+  std::string fname = LogFileName(immutable_db_options_.wal_dir, wal_number);
+  Status s;
+  // This gets the appear size of the wals, not including preallocated space.
+  s = env_->GetFileSize(fname, &log.size);
+  if (s.ok() && truncate) {
+    std::unique_ptr<FSWritableFile> last_log;
+    Status truncate_status = fs_->ReopenWritableFile(
+        fname,
+        fs_->OptimizeForLogWrite(
+            file_options_,
+            BuildDBOptions(immutable_db_options_, mutable_db_options_)),
+        &last_log, nullptr);
+    if (truncate_status.ok()) {
+      truncate_status = last_log->Truncate(log.size, IOOptions(), nullptr);
+    }
+    if (truncate_status.ok()) {
+      truncate_status = last_log->Close(IOOptions(), nullptr);
+    }
+    // Not a critical error if fail to truncate.
+    if (!truncate_status.ok()) {
+      ROCKS_LOG_WARN(immutable_db_options_.info_log,
+                     "Failed to truncate log #%" PRIu64 ": %s", wal_number,
+                     truncate_status.ToString().c_str());
+    }
+  }
+  if (log_ptr) {
+    *log_ptr = log;
+  }
+  return s;
+}
+
 Status DBImpl::RestoreAliveLogFiles(const std::vector<uint64_t>& wal_numbers) {
  if (wal_numbers.empty()) {
    return Status::OK();
@@ -1254,39 +1298,17 @@ Status DBImpl::RestoreAliveLogFiles(const std::vector<uint64_t>& wal_numbers) {
  total_log_size_ = 0;
  log_empty_ = false;
  for (auto wal_number : wal_numbers) {
-    LogFileNumberSize log(wal_number);
-    std::string fname = LogFileName(immutable_db_options_.wal_dir, wal_number);
-    // This gets the appear size of the wals, not including preallocated space.
-    s = env_->GetFileSize(fname, &log.size);
+    // We preallocate space for wals, but then after a crash and restart, those
+    // preallocated space are not needed anymore. It is likely only the last
+    // log has such preallocated space, so we only truncate for the last log.
+    LogFileNumberSize log;
+    s = GetLogSizeAndMaybeTruncate(
+        wal_number, /*truncate=*/(wal_number == wal_numbers.back()), &log);
    if (!s.ok()) {
      break;
    }
    total_log_size_ += log.size;
    alive_log_files_.push_back(log);
-    // We preallocate space for wals, but then after a crash and restart, those
-    // preallocated space are not needed anymore. It is likely only the last
-    // log has such preallocated space, so we only truncate for the last log.
-    if (wal_number == wal_numbers.back()) {
-      std::unique_ptr<FSWritableFile> last_log;
-      Status truncate_status = fs_->ReopenWritableFile(
-          fname,
-          fs_->OptimizeForLogWrite(
-              file_options_,
-              BuildDBOptions(immutable_db_options_, mutable_db_options_)),
-          &last_log, nullptr);
-      if (truncate_status.ok()) {
-        truncate_status = last_log->Truncate(log.size, IOOptions(), nullptr);
-      }
-      if (truncate_status.ok()) {
-        truncate_status = last_log->Close(IOOptions(), nullptr);
-      }
-      // Not a critical error if fail to truncate.
-      if (!truncate_status.ok()) {
-        ROCKS_LOG_WARN(immutable_db_options_.info_log,
-                       "Failed to truncate log #%" PRIu64 ": %s", wal_number,
-                       truncate_status.ToString().c_str());
-      }
-    }
  }
  if (two_write_queues_) {
    log_write_mutex_.Unlock();
@@ -1358,7 +1380,6 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
          cfd->GetID(), cfd->GetName(), snapshot_seqs,
          earliest_write_conflict_snapshot, snapshot_checker,
          GetCompressionFlush(*cfd->ioptions(), mutable_cf_options),
-          mutable_cf_options.sample_for_compression,
          mutable_cf_options.compression_opts, paranoid_file_checks,
          cfd->internal_stats(), TableFileCreationReason::kRecovery, &io_s,
          io_tracer_, &event_logger_, job_id, Env::IO_HIGH,
@@ -6,7 +6,7 @@
 #include "db/db_impl/db_impl_readonly.h"

 #include "db/arena_wrapped_db_iter.h"
-#include "db/compacted_db_impl.h"
+#include "db/db_impl/compacted_db_impl.h"
 #include "db/db_impl/db_impl.h"
 #include "db/db_iter.h"
 #include "db/merge_context.h"
@@ -19,7 +19,8 @@ namespace ROCKSDB_NAMESPACE {

 DBImplReadOnly::DBImplReadOnly(const DBOptions& db_options,
                               const std::string& dbname)
-    : DBImpl(db_options, dbname) {
+    : DBImpl(db_options, dbname, /*seq_per_batch*/ false,
+             /*batch_per_txn*/ true, /*read_only*/ true) {
  ROCKS_LOG_INFO(immutable_db_options_.info_log,
                 "Opening the db in read only mode");
  LogFlush(immutable_db_options_.info_log);
@@ -131,8 +132,8 @@ Status DBImplReadOnly::NewIterators(
 }

 namespace {
-// Return OK if dbname exists in the file system
-// or create_if_missing is false
+// Return OK if dbname exists in the file system or create it if
+// create_if_missing
 Status OpenForReadOnlyCheckExistence(const DBOptions& db_options,
                                     const std::string& dbname) {
  Status s;
@@ -143,6 +144,9 @@ Status OpenForReadOnlyCheckExistence(const DBOptions& db_options,
    uint64_t manifest_file_number;
    s = VersionSet::GetCurrentManifestPath(dbname, fs.get(), &manifest_path,
                                           &manifest_file_number);
+  } else {
+    // Historic behavior that doesn't necessarily make sense
+    s = db_options.env->CreateDirIfMissing(dbname);
  }
  return s;
 }
@@ -150,7 +154,6 @@ Status OpenForReadOnlyCheckExistence(const DBOptions& db_options,

 Status DB::OpenForReadOnly(const Options& options, const std::string& dbname,
                           DB** dbptr, bool /*error_if_wal_file_exists*/) {
-  // If dbname does not exist in the file system, should not do anything
  Status s = OpenForReadOnlyCheckExistence(options, dbname);
  if (!s.ok()) {
    return s;
@@ -1343,7 +1343,7 @@ void DBIter::Seek(const Slice& target) {
  // we need to find out the next key that is visible to the user.
  ClearSavedValue();
  if (prefix_same_as_start_) {
-    // The case where the iterator needs to be invalidated if it has exausted
+    // The case where the iterator needs to be invalidated if it has exhausted
    // keys within the same prefix of the seek key.
    assert(prefix_extractor_ != nullptr);
    Slice target_prefix = prefix_extractor_->Transform(target);
@@ -1418,7 +1418,7 @@ void DBIter::SeekForPrev(const Slice& target) {
  // backward direction.
  ClearSavedValue();
  if (prefix_same_as_start_) {
-    // The case where the iterator needs to be invalidated if it has exausted
+    // The case where the iterator needs to be invalidated if it has exhausted
    // keys within the same prefix of the seek key.
    assert(prefix_extractor_ != nullptr);
    Slice target_prefix = prefix_extractor_->Transform(target);
@@ -235,7 +235,7 @@ class DBIter final : public Iterator {
  // If `skipping_saved_key` is true, the function will keep iterating until it
  // finds a user key that is larger than `saved_key_`.
  // If `prefix` is not null, the iterator needs to stop when all keys for the
-  // prefix are exhausted and the interator is set to invalid.
+  // prefix are exhausted and the iterator is set to invalid.
  bool FindNextUserEntry(bool skipping_saved_key, const Slice* prefix);
  // Internal implementation of FindNextUserEntry().
  bool FindNextUserEntryInternal(bool skipping_saved_key, const Slice* prefix);
@@ -1175,6 +1175,61 @@ class CountingDeleteTabPropCollectorFactory
  }
 };

+class BlockCountingTablePropertiesCollector : public TablePropertiesCollector {
+ public:
+  static const std::string kNumSampledBlocksPropertyName;
+
+  const char* Name() const override {
+    return "BlockCountingTablePropertiesCollector";
+  }
+
+  Status Finish(UserCollectedProperties* properties) override {
+    (*properties)[kNumSampledBlocksPropertyName] =
+        ToString(num_sampled_blocks_);
+    return Status::OK();
+  }
+
+  Status AddUserKey(const Slice& /*user_key*/, const Slice& /*value*/,
+                    EntryType /*type*/, SequenceNumber /*seq*/,
+                    uint64_t /*file_size*/) override {
+    return Status::OK();
+  }
+
+  void BlockAdd(uint64_t /* block_raw_bytes */,
+                uint64_t block_compressed_bytes_fast,
+                uint64_t block_compressed_bytes_slow) override {
+    if (block_compressed_bytes_fast > 0 || block_compressed_bytes_slow > 0) {
+      num_sampled_blocks_++;
+    }
+  }
+
+  UserCollectedProperties GetReadableProperties() const override {
+    return UserCollectedProperties{
+        {kNumSampledBlocksPropertyName, ToString(num_sampled_blocks_)},
+    };
+  }
+
+ private:
+  uint32_t num_sampled_blocks_ = 0;
+};
+
+const std::string
+    BlockCountingTablePropertiesCollector::kNumSampledBlocksPropertyName =
+        "NumSampledBlocks";
+
+class BlockCountingTablePropertiesCollectorFactory
+    : public TablePropertiesCollectorFactory {
+ public:
+  const char* Name() const override {
+    return "BlockCountingTablePropertiesCollectorFactory";
+  }
+
+  TablePropertiesCollector* CreateTablePropertiesCollector(
+      TablePropertiesCollectorFactory::Context /* context */) override {
+    return new BlockCountingTablePropertiesCollector();
+  }
+};
+
 #ifndef ROCKSDB_LITE
 TEST_F(DBPropertiesTest, GetUserDefinedTableProperties) {
  Options options = CurrentOptions();
@@ -1413,6 +1468,132 @@ TEST_F(DBPropertiesTest, NeedCompactHintPersistentTest) {
  }
 }

+// Excluded from RocksDB lite tests due to `GetPropertiesOfAllTables()` usage.
+TEST_F(DBPropertiesTest, BlockAddForCompressionSampling) {
+  // Sampled compression requires at least one of the following four types.
+  if (!Snappy_Supported() && !Zlib_Supported() && !LZ4_Supported() &&
+      !ZSTD_Supported()) {
+    return;
+  }
+
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  options.table_properties_collector_factories.emplace_back(
+      std::make_shared<BlockCountingTablePropertiesCollectorFactory>());
+
+  for (bool sample_for_compression : {false, true}) {
+    // For simplicity/determinism, sample 100% when enabled, or 0% when disabled
+    options.sample_for_compression = sample_for_compression ? 1 : 0;
+
+    DestroyAndReopen(options);
+
+    // Setup the following LSM:
+    //
+    // L0_0 ["a", "b"]
+    // L1_0 ["a", "b"]
+    //
+    // L0_0 was created by flush. L1_0 was created by compaction. Each file
+    // contains one data block.
+    for (int i = 0; i < 3; ++i) {
+      ASSERT_OK(Put("a", "val"));
+      ASSERT_OK(Put("b", "val"));
+      ASSERT_OK(Flush());
+      if (i == 1) {
+        ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
+      }
+    }
+
+    // A `BlockAdd()` should have been seen for files generated by flush or
+    // compaction when `sample_for_compression` is enabled.
+    TablePropertiesCollection file_to_props;
+    ASSERT_OK(db_->GetPropertiesOfAllTables(&file_to_props));
+    ASSERT_EQ(2, file_to_props.size());
+    for (const auto& file_and_props : file_to_props) {
+      auto& user_props = file_and_props.second->user_collected_properties;
+      ASSERT_TRUE(user_props.find(BlockCountingTablePropertiesCollector::
+                                      kNumSampledBlocksPropertyName) !=
+                  user_props.end());
+      ASSERT_EQ(user_props.at(BlockCountingTablePropertiesCollector::
+                                  kNumSampledBlocksPropertyName),
+                ToString(sample_for_compression ? 1 : 0));
+    }
+  }
+}
+
+class CompressionSamplingDBPropertiesTest
+    : public DBPropertiesTest,
+      public ::testing::WithParamInterface<bool> {
+ public:
+  CompressionSamplingDBPropertiesTest() : fast_(GetParam()) {}
+
+ protected:
+  const bool fast_;
+};
+
+INSTANTIATE_TEST_CASE_P(CompressionSamplingDBPropertiesTest,
+                        CompressionSamplingDBPropertiesTest, ::testing::Bool());
+
+// Excluded from RocksDB lite tests due to `GetPropertiesOfAllTables()` usage.
+TEST_P(CompressionSamplingDBPropertiesTest,
+       EstimateDataSizeWithCompressionSampling) {
+  Options options = CurrentOptions();
+  if (fast_) {
+    // One of the following light compression libraries must be present.
+    if (LZ4_Supported()) {
+      options.compression = kLZ4Compression;
+    } else if (Snappy_Supported()) {
+      options.compression = kSnappyCompression;
+    } else {
+      return;
+    }
+  } else {
+    // One of the following heavy compression libraries must be present.
+    if (ZSTD_Supported()) {
+      options.compression = kZSTD;
+    } else if (Zlib_Supported()) {
+      options.compression = kZlibCompression;
+    } else {
+      return;
+    }
+  }
+  options.disable_auto_compactions = true;
+  // For simplicity/determinism, sample 100%.
+  options.sample_for_compression = 1;
+  Reopen(options);
+
+  // Setup the following LSM:
+  //
+  // L0_0 ["a", "b"]
+  // L1_0 ["a", "b"]
+  //
+  // L0_0 was created by flush. L1_0 was created by compaction. Each file
+  // contains one data block. The value consists of compressible data so the
+  // data block should be stored compressed.
+  std::string val(1024, 'a');
+  for (int i = 0; i < 3; ++i) {
+    ASSERT_OK(Put("a", val));
+    ASSERT_OK(Put("b", val));
+    ASSERT_OK(Flush());
+    if (i == 1) {
+      ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
+    }
+  }
+
+  TablePropertiesCollection file_to_props;
+  ASSERT_OK(db_->GetPropertiesOfAllTables(&file_to_props));
+  ASSERT_EQ(2, file_to_props.size());
+  for (const auto& file_and_props : file_to_props) {
+    ASSERT_GT(file_and_props.second->data_size, 0);
+    if (fast_) {
+      ASSERT_EQ(file_and_props.second->data_size,
+                file_and_props.second->fast_compression_estimated_data_size);
+    } else {
+      ASSERT_EQ(file_and_props.second->data_size,
+                file_and_props.second->slow_compression_estimated_data_size);
+    }
+  }
+}
+
 TEST_F(DBPropertiesTest, EstimateNumKeysUnderflow) {
  Options options = CurrentOptions();
  Reopen(options);
@@ -73,6 +73,15 @@ TEST_F(DBRangeDelTest, FlushOutputHasOnlyRangeTombstones) {
  } while (ChangeOptions(kRangeDelSkipConfigs));
 }

+TEST_F(DBRangeDelTest, DictionaryCompressionWithOnlyRangeTombstones) {
+  Options opts = CurrentOptions();
+  opts.compression_opts.max_dict_bytes = 16384;
+  Reopen(opts);
+  ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "dr1",
+                             "dr2"));
+  ASSERT_OK(db_->Flush(FlushOptions()));
+}
+
 TEST_F(DBRangeDelTest, CompactionOutputHasOnlyRangeTombstone) {
  do {
    Options opts = CurrentOptions();
@@ -751,10 +751,11 @@ TEST_F(DBSSTTest, RateLimitedWALDelete) {
 }

 class DBWALTestWithParam
-    : public DBSSTTest,
+    : public DBTestBase,
      public testing::WithParamInterface<std::tuple<std::string, bool>> {
 public:
-  DBWALTestWithParam() {
+  explicit DBWALTestWithParam()
+      : DBTestBase("/db_wal_test_with_params", /*env_do_fsync=*/true) {
    wal_dir_ = std::get<0>(GetParam());
    wal_dir_same_as_dbname_ = std::get<1>(GetParam());
  }
@@ -1088,6 +1089,12 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowedWithBlobFiles) {
  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
      "BuildTable::AfterDeleteFile",
      [&](void* /*arg*/) { delete_blob_file = true; });
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
+      {
+          "BuildTable::AfterDeleteFile",
+          "DBSSTTest::DBWithMaxSpaceAllowedWithBlobFiles:1",
+      },
+  });

  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();

@@ -1095,6 +1102,8 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowedWithBlobFiles) {
  // This flush will fail
  ASSERT_NOK(Flush());
  ASSERT_TRUE(max_allowed_space_reached);
+
+  TEST_SYNC_POINT("DBSSTTest::DBWithMaxSpaceAllowedWithBlobFiles:1");
  ASSERT_TRUE(delete_blob_file);
  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
 }
@@ -4116,7 +4116,7 @@ TEST_F(DBTest2, TraceWithFilter) {

  // Open another db, replay, and verify the data
  std::string value;
-  std::string dbname2 = test::TmpDir(env_) + "/db_replay";
+  std::string dbname2 = test::PerThreadDBPath(env_, "db_replay");
  ASSERT_OK(DestroyDB(dbname2, options));

  // Using a different name than db2, to pacify infer's use-after-lifetime
@@ -4167,7 +4167,7 @@ TEST_F(DBTest2, TraceWithFilter) {
  ASSERT_OK(DestroyDB(dbname2, options));

  // Set up a new db.
-  std::string dbname3 = test::TmpDir(env_) + "/db_not_trace_read";
+  std::string dbname3 = test::PerThreadDBPath(env_, "db_not_trace_read");
  ASSERT_OK(DestroyDB(dbname3, options));

  DB* db3_init = nullptr;
@@ -4584,7 +4584,7 @@ TEST_F(DBTest2, MultiDBParallelOpenTest) {
  Options options = CurrentOptions();
  std::vector<std::string> dbnames;
  for (int i = 0; i < kNumDbs; ++i) {
-    dbnames.emplace_back(test::TmpDir(env_) + "/db" + ToString(i));
+    dbnames.emplace_back(test::PerThreadDBPath(env_, "db" + ToString(i)));
    ASSERT_OK(DestroyDB(dbnames.back(), options));
  }

@@ -24,13 +24,37 @@ class DBWALTestBase : public DBTestBase {

 #if defined(ROCKSDB_PLATFORM_POSIX)
 public:
+#if defined(ROCKSDB_FALLOCATE_PRESENT)
+  bool IsFallocateSupported() {
+    // Test fallocate support of running file system.
+    // Skip this test if fallocate is not supported.
+    std::string fname_test_fallocate = dbname_ + "/preallocate_testfile";
+    int fd = -1;
+    do {
+      fd = open(fname_test_fallocate.c_str(), O_CREAT | O_RDWR | O_TRUNC, 0644);
+    } while (fd < 0 && errno == EINTR);
+    assert(fd > 0);
+    int alloc_status = fallocate(fd, 0, 0, 1);
+    int err_number = errno;
+    close(fd);
+    assert(env_->DeleteFile(fname_test_fallocate) == Status::OK());
+    if (err_number == ENOSYS || err_number == EOPNOTSUPP) {
+      fprintf(stderr, "Skipped preallocated space check: %s\n",
+              errnoStr(err_number).c_str());
+      return false;
+    }
+    assert(alloc_status == 0);
+    return true;
+  }
+#endif  // ROCKSDB_FALLOCATE_PRESENT
+
  uint64_t GetAllocatedFileSize(std::string file_name) {
    struct stat sbuf;
    int err = stat(file_name.c_str(), &sbuf);
    assert(err == 0);
    return sbuf.st_blocks * 512;
  }
-#endif
+#endif  // ROCKSDB_PLATFORM_POSIX
 };

 class DBWALTest : public DBWALTestBase {
@@ -1849,23 +1873,9 @@ TEST_F(DBWALTest, TruncateLastLogAfterRecoverWithoutFlush) {
    ROCKSDB_GTEST_SKIP("Test requires non-mem environment");
    return;
  }
-  // Test fallocate support of running file system.
-  // Skip this test if fallocate is not supported.
-  std::string fname_test_fallocate = dbname_ + "/preallocate_testfile";
-  int fd = -1;
-  do {
-    fd = open(fname_test_fallocate.c_str(), O_CREAT | O_RDWR | O_TRUNC, 0644);
-  } while (fd < 0 && errno == EINTR);
-  ASSERT_GT(fd, 0);
-  int alloc_status = fallocate(fd, 0, 0, 1);
-  int err_number = errno;
-  close(fd);
-  ASSERT_OK(options.env->DeleteFile(fname_test_fallocate));
-  if (err_number == ENOSYS || err_number == EOPNOTSUPP) {
-    fprintf(stderr, "Skipped preallocated space check: %s\n", strerror(err_number));
+  if (!IsFallocateSupported()) {
    return;
  }
-  ASSERT_EQ(0, alloc_status);

  DestroyAndReopen(options);
  size_t preallocated_size =
@@ -1888,6 +1898,120 @@ TEST_F(DBWALTest, TruncateLastLogAfterRecoverWithoutFlush) {
  ASSERT_LT(GetAllocatedFileSize(dbname_ + file_before->PathName()),
            preallocated_size);
 }
+// Tests that we will truncate the preallocated space of the last log from
+// previous.
+TEST_F(DBWALTest, TruncateLastLogAfterRecoverWithFlush) {
+  constexpr size_t kKB = 1024;
+  Options options = CurrentOptions();
+  options.env = env_;
+  options.avoid_flush_during_recovery = false;
+  options.avoid_flush_during_shutdown = true;
+  if (mem_env_) {
+    ROCKSDB_GTEST_SKIP("Test requires non-mem environment");
+    return;
+  }
+  if (!IsFallocateSupported()) {
+    return;
+  }
+
+  DestroyAndReopen(options);
+  size_t preallocated_size =
+      dbfull()->TEST_GetWalPreallocateBlockSize(options.write_buffer_size);
+  ASSERT_OK(Put("foo", "v1"));
+  VectorLogPtr log_files_before;
+  ASSERT_OK(dbfull()->GetSortedWalFiles(log_files_before));
+  ASSERT_EQ(1, log_files_before.size());
+  auto& file_before = log_files_before[0];
+  ASSERT_LT(file_before->SizeFileBytes(), 1 * kKB);
+  ASSERT_GE(GetAllocatedFileSize(dbname_ + file_before->PathName()),
+            preallocated_size);
+  // The log file has preallocated space.
+  Close();
+
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
+      {{"DBImpl::PurgeObsoleteFiles:Begin",
+        "DBWALTest::TruncateLastLogAfterRecoverWithFlush:AfterRecover"},
+       {"DBWALTest::TruncateLastLogAfterRecoverWithFlush:AfterTruncate",
+        "DBImpl::DeleteObsoleteFileImpl::BeforeDeletion"}});
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
+  port::Thread reopen_thread([&]() { Reopen(options); });
+
+  TEST_SYNC_POINT(
+      "DBWALTest::TruncateLastLogAfterRecoverWithFlush:AfterRecover");
+  // After the flush during Open, the log file should get deleted.  However,
+  // if  the process is in a crash loop, the log file may not get
+  // deleted and thte preallocated space will keep accumulating. So we need
+  // to ensure it gets trtuncated.
+  EXPECT_LT(GetAllocatedFileSize(dbname_ + file_before->PathName()),
+            preallocated_size);
+  TEST_SYNC_POINT(
+      "DBWALTest::TruncateLastLogAfterRecoverWithFlush:AfterTruncate");
+  reopen_thread.join();
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
+}
+
+TEST_F(DBWALTest, TruncateLastLogAfterRecoverWALEmpty) {
+  Options options = CurrentOptions();
+  options.env = env_;
+  options.avoid_flush_during_recovery = false;
+  if (mem_env_ || encrypted_env_) {
+    ROCKSDB_GTEST_SKIP("Test requires non-mem/non-encrypted  environment");
+    return;
+  }
+  if (!IsFallocateSupported()) {
+    return;
+  }
+
+  DestroyAndReopen(options);
+  size_t preallocated_size =
+      dbfull()->TEST_GetWalPreallocateBlockSize(options.write_buffer_size);
+  Close();
+  std::vector<std::string> filenames;
+  std::string last_log;
+  uint64_t last_log_num = 0;
+  ASSERT_OK(env_->GetChildren(dbname_, &filenames));
+  for (auto fname : filenames) {
+    uint64_t number;
+    FileType type;
+    if (ParseFileName(fname, &number, &type, nullptr)) {
+      if (type == kWalFile && number > last_log_num) {
+        last_log = fname;
+      }
+    }
+  }
+  ASSERT_NE(last_log, "");
+  last_log = dbname_ + '/' + last_log;
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
+      {{"DBImpl::PurgeObsoleteFiles:Begin",
+        "DBWALTest::TruncateLastLogAfterRecoverWithFlush:AfterRecover"},
+       {"DBWALTest::TruncateLastLogAfterRecoverWithFlush:AfterTruncate",
+        "DBImpl::DeleteObsoleteFileImpl::BeforeDeletion"}});
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
+      "PosixWritableFile::Close",
+      [](void* arg) { *(reinterpret_cast<size_t*>(arg)) = 0; });
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
+  // Preallocate space for the empty log file. This could happen if WAL data
+  // was buffered in memory and the process crashed.
+  std::unique_ptr<WritableFile> log_file;
+  ASSERT_OK(env_->ReopenWritableFile(last_log, &log_file, EnvOptions()));
+  log_file->SetPreallocationBlockSize(preallocated_size);
+  log_file->PrepareWrite(0, 4096);
+  log_file.reset();
+
+  ASSERT_GE(GetAllocatedFileSize(last_log), preallocated_size);
+
+  port::Thread reopen_thread([&]() { Reopen(options); });
+
+  TEST_SYNC_POINT(
+      "DBWALTest::TruncateLastLogAfterRecoverWithFlush:AfterRecover");
+  // The preallocated space should be truncated.
+  EXPECT_LT(GetAllocatedFileSize(last_log), preallocated_size);
+  TEST_SYNC_POINT(
+      "DBWALTest::TruncateLastLogAfterRecoverWithFlush:AfterTruncate");
+  reopen_thread.join();
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
+}
 #endif  // ROCKSDB_FALLOCATE_PRESENT
 #endif  // ROCKSDB_PLATFORM_POSIX

@@ -616,7 +616,7 @@ class IterKey {
  void EnlargeBuffer(size_t key_size);
 };

-// Convert from a SliceTranform of user keys, to a SliceTransform of
+// Convert from a SliceTransform of user keys, to a SliceTransform of
 // user keys.
 class InternalKeySliceTransform : public SliceTransform {
 public:
@@ -103,7 +103,7 @@ class ErrorHandler {
    bool auto_recovery_;
    bool recovery_in_prog_;
    // A flag to indicate that for the soft error, we should not allow any
-    // backrgound work execpt the work is from recovery.
+    // background work except the work is from recovery.
    bool soft_error_no_bg_work_;

    // Used to store the context for recover, such as flush reason.
@@ -216,7 +216,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWriteRetryableError) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
  s = dbfull()->Resume();
@@ -242,7 +242,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWriteRetryableError) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
  s = dbfull()->Resume();
@@ -256,7 +256,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWriteRetryableError) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
  s = dbfull()->Resume();
@@ -292,7 +292,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWriteFileScopeError) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
  s = dbfull()->Resume();
@@ -306,7 +306,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWriteFileScopeError) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
  s = dbfull()->Resume();
@@ -320,7 +320,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWriteFileScopeError) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
  s = dbfull()->Resume();
@@ -340,7 +340,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWriteFileScopeError) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
  s = dbfull()->Resume();
@@ -649,7 +649,7 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteRetryableError) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->ClearAllCallBacks();
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
@@ -695,7 +695,7 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteFileScopeError) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->ClearAllCallBacks();
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
@@ -1698,7 +1698,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
 // to soft error and trigger auto resume. During auto resume, SwitchMemtable
 // is disabled to avoid small SST tables. Write can still be applied before
 // the bg error is cleaned unless the memtable is full.
-TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover1) {
+TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableErrorAutoRecover1) {
  // Activate the FS before the first resume
  std::shared_ptr<ErrorHandlerFSListener> listener(
      new ErrorHandlerFSListener());
@@ -1744,14 +1744,14 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover1) {
                   ERROR_HANDLER_BG_RETRYABLE_IO_ERROR_COUNT));
  ASSERT_EQ(1, options.statistics->getAndResetTickerCount(
                   ERROR_HANDLER_AUTORESUME_COUNT));
-  ASSERT_EQ(2, options.statistics->getAndResetTickerCount(
+  ASSERT_LE(0, options.statistics->getAndResetTickerCount(
                   ERROR_HANDLER_AUTORESUME_RETRY_TOTAL_COUNT));
-  ASSERT_EQ(0, options.statistics->getAndResetTickerCount(
+  ASSERT_LE(0, options.statistics->getAndResetTickerCount(
                   ERROR_HANDLER_AUTORESUME_SUCCESS_COUNT));
  HistogramData autoresume_retry;
  options.statistics->histogramData(ERROR_HANDLER_AUTORESUME_RETRY_COUNT,
                                    &autoresume_retry);
-  ASSERT_EQ(autoresume_retry.max, 2);
+  ASSERT_GE(autoresume_retry.max, 0);
  ASSERT_OK(Put(Key(2), "val2", wo));
  s = Flush();
  // Since auto resume fails, the bg error is not cleand, flush will
@@ -1768,7 +1768,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover1) {
  Destroy(options);
 }

-TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover2) {
+TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableErrorAutoRecover2) {
  // Activate the FS before the first resume
  std::shared_ptr<ErrorHandlerFSListener> listener(
      new ErrorHandlerFSListener());
@@ -1810,14 +1810,14 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover2) {
                   ERROR_HANDLER_BG_RETRYABLE_IO_ERROR_COUNT));
  ASSERT_EQ(1, options.statistics->getAndResetTickerCount(
                   ERROR_HANDLER_AUTORESUME_COUNT));
-  ASSERT_EQ(1, options.statistics->getAndResetTickerCount(
+  ASSERT_LE(0, options.statistics->getAndResetTickerCount(
                   ERROR_HANDLER_AUTORESUME_RETRY_TOTAL_COUNT));
-  ASSERT_EQ(1, options.statistics->getAndResetTickerCount(
+  ASSERT_LE(0, options.statistics->getAndResetTickerCount(
                   ERROR_HANDLER_AUTORESUME_SUCCESS_COUNT));
  HistogramData autoresume_retry;
  options.statistics->histogramData(ERROR_HANDLER_AUTORESUME_RETRY_COUNT,
                                    &autoresume_retry);
-  ASSERT_EQ(autoresume_retry.max, 1);
+  ASSERT_GE(autoresume_retry.max, 0);
  ASSERT_OK(Put(Key(2), "val2", wo));
  s = Flush();
  // Since auto resume is successful, the bg error is cleaned, flush will
@@ -1827,56 +1827,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover2) {
  Destroy(options);
 }

-TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover1) {
-  // Fail the first resume and make the second resume successful
-  std::shared_ptr<ErrorHandlerFSListener> listener(
-      new ErrorHandlerFSListener());
-  Options options = GetDefaultOptions();
-  options.env = fault_env_.get();
-  options.create_if_missing = true;
-  options.listeners.emplace_back(listener);
-  options.max_bgerror_resume_count = 2;
-  options.bgerror_resume_retry_interval = 100000;  // 0.1 second
-  Status s;
-
-  listener->EnableAutoRecovery(false);
-  DestroyAndReopen(options);
-
-  IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
-  error_msg.SetRetryable(true);
-
-  ASSERT_OK(Put(Key(1), "val1"));
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
-      {{"RecoverFromRetryableBGIOError:BeforeWait0",
-        "FLushWritRetryableeErrorAutoRecover1:0"},
-       {"FLushWritRetryableeErrorAutoRecover1:1",
-        "RecoverFromRetryableBGIOError:BeforeWait1"},
-       {"RecoverFromRetryableBGIOError:RecoverSuccess",
-        "FLushWritRetryableeErrorAutoRecover1:2"}});
-  SyncPoint::GetInstance()->SetCallBack(
-      "BuildTable:BeforeFinishBuildTable",
-      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
-  SyncPoint::GetInstance()->EnableProcessing();
-  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover1:0");
-  fault_fs_->SetFilesystemActive(true);
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover1:1");
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover1:2");
-  SyncPoint::GetInstance()->DisableProcessing();
-
-  ASSERT_EQ("val1", Get(Key(1)));
-  Reopen(options);
-  ASSERT_EQ("val1", Get(Key(1)));
-  ASSERT_OK(Put(Key(2), "val2"));
-  ASSERT_OK(Flush());
-  ASSERT_EQ("val2", Get(Key(2)));
-
-  Destroy(options);
-}
-
-TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover2) {
+TEST_F(DBErrorHandlingFSTest, FLushWritRetryableErrorAutoRecover1) {
  // Activate the FS before the first resume
  std::shared_ptr<ErrorHandlerFSListener> listener(
      new ErrorHandlerFSListener());
@@ -1901,7 +1852,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover2) {

  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  SyncPoint::GetInstance()->DisableProcessing();
  fault_fs_->SetFilesystemActive(true);
  ASSERT_EQ(listener->WaitForRecovery(5000000), true);
@@ -1916,7 +1867,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover2) {
  Destroy(options);
 }

-TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover3) {
+TEST_F(DBErrorHandlingFSTest, FLushWritRetryableErrorAutoRecover2) {
  // Fail all the resume and let user to resume
  std::shared_ptr<ErrorHandlerFSListener> listener(
      new ErrorHandlerFSListener());
@@ -1936,18 +1887,18 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover3) {

  ASSERT_OK(Put(Key(1), "val1"));
  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
-      {{"FLushWritRetryableeErrorAutoRecover3:0",
+      {{"FLushWritRetryableeErrorAutoRecover2:0",
        "RecoverFromRetryableBGIOError:BeforeStart"},
       {"RecoverFromRetryableBGIOError:LoopOut",
-        "FLushWritRetryableeErrorAutoRecover3:1"}});
+        "FLushWritRetryableeErrorAutoRecover2:1"}});
  SyncPoint::GetInstance()->SetCallBack(
      "BuildTable:BeforeFinishBuildTable",
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover3:0");
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover3:1");
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
+  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover2:0");
+  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover2:1");
  fault_fs_->SetFilesystemActive(true);
  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
  SyncPoint::GetInstance()->DisableProcessing();
@@ -1965,173 +1916,6 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover3) {
  Destroy(options);
 }

-TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover4) {
-  // Fail the first resume and does not do resume second time because
-  // the IO error severity is Fatal Error and not Retryable.
-  std::shared_ptr<ErrorHandlerFSListener> listener(
-      new ErrorHandlerFSListener());
-  Options options = GetDefaultOptions();
-  options.env = fault_env_.get();
-  options.create_if_missing = true;
-  options.listeners.emplace_back(listener);
-  options.max_bgerror_resume_count = 2;
-  options.bgerror_resume_retry_interval = 10;  // 0.1 second
-  Status s;
-
-  listener->EnableAutoRecovery(false);
-  DestroyAndReopen(options);
-
-  IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
-  error_msg.SetRetryable(true);
-  IOStatus nr_msg = IOStatus::IOError("No Retryable Fatal IO Error");
-  nr_msg.SetRetryable(false);
-
-  ASSERT_OK(Put(Key(1), "val1"));
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
-      {{"RecoverFromRetryableBGIOError:BeforeStart",
-        "FLushWritRetryableeErrorAutoRecover4:0"},
-       {"FLushWritRetryableeErrorAutoRecover4:2",
-        "RecoverFromRetryableBGIOError:RecoverFail0"}});
-  SyncPoint::GetInstance()->SetCallBack(
-      "BuildTable:BeforeFinishBuildTable",
-      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
-  SyncPoint::GetInstance()->SetCallBack(
-      "RecoverFromRetryableBGIOError:BeforeResume1",
-      [&](void*) { fault_fs_->SetFilesystemActive(false, nr_msg); });
-
-  SyncPoint::GetInstance()->EnableProcessing();
-  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover4:0");
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover4:2");
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
-  SyncPoint::GetInstance()->DisableProcessing();
-  fault_fs_->SetFilesystemActive(true);
-  // Even the FS is recoverd, due to the Fatal Error in bg_error_ the resume
-  // and flush will all fail.
-  ASSERT_EQ("val1", Get(Key(1)));
-  ASSERT_NOK(dbfull()->Resume());
-  ASSERT_EQ("val1", Get(Key(1)));
-  ASSERT_OK(Put(Key(2), "val2"));
-  ASSERT_NOK(Flush());
-  ASSERT_EQ("NOT_FOUND", Get(Key(2)));
-
-  Reopen(options);
-  ASSERT_EQ("val1", Get(Key(1)));
-  ASSERT_OK(Put(Key(2), "val2"));
-  ASSERT_OK(Flush());
-  ASSERT_EQ("val2", Get(Key(2)));
-
-  Destroy(options);
-}
-
-TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover5) {
-  // During the resume, call DB->CLose, make sure the resume thread exist
-  // before close continues. Due to the shutdown, the resume is not successful
-  // and the FS does not become active, so close status is still IO error
-  std::shared_ptr<ErrorHandlerFSListener> listener(
-      new ErrorHandlerFSListener());
-  Options options = GetDefaultOptions();
-  options.env = fault_env_.get();
-  options.create_if_missing = true;
-  options.listeners.emplace_back(listener);
-  options.max_bgerror_resume_count = 2;
-  options.bgerror_resume_retry_interval = 10;  // 0.1 second
-  Status s;
-
-  listener->EnableAutoRecovery(false);
-  DestroyAndReopen(options);
-
-  IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
-  error_msg.SetRetryable(true);
-
-  ASSERT_OK(Put(Key(1), "val1"));
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
-      {{"RecoverFromRetryableBGIOError:BeforeStart",
-        "FLushWritRetryableeErrorAutoRecover5:0"}});
-  SyncPoint::GetInstance()->SetCallBack(
-      "BuildTable:BeforeFinishBuildTable",
-      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
-  SyncPoint::GetInstance()->EnableProcessing();
-  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover5:0");
-  // The first resume will cause recovery_error and its severity is the
-  // Fatal error
-  s = dbfull()->Close();
-  ASSERT_NOK(s);
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
-  SyncPoint::GetInstance()->DisableProcessing();
-  fault_fs_->SetFilesystemActive(true);
-
-  Reopen(options);
-  ASSERT_NE("val1", Get(Key(1)));
-  ASSERT_OK(Put(Key(2), "val2"));
-  s = Flush();
-  ASSERT_OK(s);
-  ASSERT_EQ("val2", Get(Key(2)));
-
-  Destroy(options);
-}
-
-TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover6) {
-  // During the resume, call DB->CLose, make sure the resume thread exist
-  // before close continues. Due to the shutdown, the resume is not successful
-  // and the FS does not become active, so close status is still IO error
-  std::shared_ptr<ErrorHandlerFSListener> listener(
-      new ErrorHandlerFSListener());
-  Options options = GetDefaultOptions();
-  options.env = fault_env_.get();
-  options.create_if_missing = true;
-  options.listeners.emplace_back(listener);
-  options.max_bgerror_resume_count = 2;
-  options.bgerror_resume_retry_interval = 10;  // 0.1 second
-  Status s;
-
-  listener->EnableAutoRecovery(false);
-  DestroyAndReopen(options);
-
-  IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
-  error_msg.SetRetryable(true);
-
-  ASSERT_OK(Put(Key(1), "val1"));
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
-      {{"FLushWritRetryableeErrorAutoRecover6:0",
-        "RecoverFromRetryableBGIOError:BeforeStart"},
-       {"RecoverFromRetryableBGIOError:BeforeWait0",
-        "FLushWritRetryableeErrorAutoRecover6:1"},
-       {"FLushWritRetryableeErrorAutoRecover6:2",
-        "RecoverFromRetryableBGIOError:BeforeWait1"},
-       {"RecoverFromRetryableBGIOError:AfterWait0",
-        "FLushWritRetryableeErrorAutoRecover6:3"}});
-  SyncPoint::GetInstance()->SetCallBack(
-      "BuildTable:BeforeFinishBuildTable",
-      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
-  SyncPoint::GetInstance()->EnableProcessing();
-  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover6:0");
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover6:1");
-  fault_fs_->SetFilesystemActive(true);
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover6:2");
-  TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover6:3");
-  // The first resume will cause recovery_error and its severity is the
-  // Fatal error
-  s = dbfull()->Close();
-  ASSERT_OK(s);
-  SyncPoint::GetInstance()->DisableProcessing();
-
-  Reopen(options);
-  ASSERT_EQ("val1", Get(Key(1)));
-  ASSERT_OK(Put(Key(2), "val2"));
-  s = Flush();
-  ASSERT_OK(s);
-  ASSERT_EQ("val2", Get(Key(2)));
-
-  Destroy(options);
-}
-
 TEST_F(DBErrorHandlingFSTest, ManifestWriteRetryableErrorAutoRecover) {
  // Fail the first resume and let the second resume be successful
  std::shared_ptr<ErrorHandlerFSListener> listener(
@@ -2168,7 +1952,7 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteRetryableErrorAutoRecover) {
      [&](void*) { fault_fs_->SetFilesystemActive(false, error_msg); });
  SyncPoint::GetInstance()->EnableProcessing();
  s = Flush();
-  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
+  ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
  TEST_SYNC_POINT("ManifestWriteRetryableErrorAutoRecover:0");
  fault_fs_->SetFilesystemActive(true);
  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
@@ -125,8 +125,12 @@ void EventHelpers::LogAndNotifyTableFileCreationFinished(
              << table_properties.compression_options << "creation_time"
              << table_properties.creation_time << "oldest_key_time"
              << table_properties.oldest_key_time << "file_creation_time"
-              << table_properties.file_creation_time << "db_id"
-              << table_properties.db_id << "db_session_id"
+              << table_properties.file_creation_time
+              << "slow_compression_estimated_data_size"
+              << table_properties.slow_compression_estimated_data_size
+              << "fast_compression_estimated_data_size"
+              << table_properties.fast_compression_estimated_data_size
+              << "db_id" << table_properties.db_id << "db_session_id"
              << table_properties.db_session_id;

      // user collected properties
@@ -40,16 +40,25 @@ Status ExternalSstFileIngestionJob::Prepare(
    if (!status.ok()) {
      return status;
    }
-    files_to_ingest_.push_back(file_to_ingest);
-  }

-  for (const IngestedFileInfo& f : files_to_ingest_) {
-    if (f.cf_id !=
+    if (file_to_ingest.cf_id !=
            TablePropertiesCollectorFactory::Context::kUnknownColumnFamily &&
-        f.cf_id != cfd_->GetID()) {
+        file_to_ingest.cf_id != cfd_->GetID()) {
      return Status::InvalidArgument(
          "External file column family id don't match");
    }
+
+    if (file_to_ingest.num_entries == 0 &&
+        file_to_ingest.num_range_deletions == 0) {
+      return Status::InvalidArgument("File contain no entries");
+    }
+
+    if (!file_to_ingest.smallest_internal_key.Valid() ||
+        !file_to_ingest.largest_internal_key.Valid()) {
+      return Status::Corruption("Generated table have corrupted keys");
+    }
+
+    files_to_ingest_.emplace_back(std::move(file_to_ingest));
  }

  const Comparator* ucmp = cfd_->internal_comparator().user_comparator();
@@ -83,16 +92,6 @@ Status ExternalSstFileIngestionJob::Prepare(
    return Status::NotSupported("Files have overlapping ranges");
  }

-  for (IngestedFileInfo& f : files_to_ingest_) {
-    if (f.num_entries == 0 && f.num_range_deletions == 0) {
-      return Status::InvalidArgument("File contain no entries");
-    }
-
-    if (!f.smallest_internal_key.Valid() || !f.largest_internal_key.Valid()) {
-      return Status::Corruption("Generated table have corrupted keys");
-    }
-  }
-
  // Copy/Move external files into DB
  std::unordered_set<size_t> ingestion_path_ids;
  for (IngestedFileInfo& f : files_to_ingest_) {
@@ -16,6 +16,7 @@
 #include "rocksdb/sst_file_writer.h"
 #include "test_util/testutil.h"
 #include "util/random.h"
+#include "util/thread_guard.h"
 #include "utilities/fault_injection_env.h"

 namespace ROCKSDB_NAMESPACE {
@@ -1305,38 +1306,38 @@ TEST_F(ExternalSSTFileTest, PickedLevelBug) {

  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();

-  // While writing the MANIFEST start a thread that will ask for compaction
  Status bg_compact_status;
-  ROCKSDB_NAMESPACE::port::Thread bg_compact([&]() {
-    bg_compact_status =
-        db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
-  });
-  TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevelBug:2");
-
-  // Start a thread that will ingest a new file
  Status bg_addfile_status;
-  ROCKSDB_NAMESPACE::port::Thread bg_addfile([&]() {
-    file_keys = {1, 2, 3};
-    bg_addfile_status = GenerateAndAddExternalFile(options, file_keys, 1);
-  });

-  // Wait for AddFile to start picking levels and writing MANIFEST
-  TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevelBug:0");
+  {
+    // While writing the MANIFEST start a thread that will ask for compaction
+    ThreadGuard bg_compact(port::Thread([&]() {
+      bg_compact_status =
+          db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
+    }));
+    TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevelBug:2");

-  TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevelBug:3");
+    // Start a thread that will ingest a new file
+    ThreadGuard bg_addfile(port::Thread([&]() {
+      file_keys = {1, 2, 3};
+      bg_addfile_status = GenerateAndAddExternalFile(options, file_keys, 1);
+    }));

-  // We need to verify that no compactions can run while AddFile is
-  // ingesting the files into the levels it find suitable. So we will
-  // wait for 2 seconds to give a chance for compactions to run during
-  // this period, and then make sure that no compactions where able to run
-  env_->SleepForMicroseconds(1000000 * 2);
-  ASSERT_FALSE(bg_compact_started.load());
+    // Wait for AddFile to start picking levels and writing MANIFEST
+    TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevelBug:0");

-  // Hold AddFile from finishing writing the MANIFEST
-  TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevelBug:1");
+    TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevelBug:3");

-  bg_addfile.join();
-  bg_compact.join();
+    // We need to verify that no compactions can run while AddFile is
+    // ingesting the files into the levels it find suitable. So we will
+    // wait for 2 seconds to give a chance for compactions to run during
+    // this period, and then make sure that no compactions where able to run
+    env_->SleepForMicroseconds(1000000 * 2);
+    ASSERT_FALSE(bg_compact_started.load());
+
+    // Hold AddFile from finishing writing the MANIFEST
+    TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevelBug:1");
+  }

  ASSERT_OK(bg_addfile_status);
  ASSERT_OK(bg_compact_status);
@@ -411,8 +411,7 @@ Status FlushJob::WriteLevel0Table() {
          cfd_->internal_comparator(), cfd_->int_tbl_prop_collector_factories(),
          cfd_->GetID(), cfd_->GetName(), existing_snapshots_,
          earliest_write_conflict_snapshot_, snapshot_checker_,
-          output_compression_, mutable_cf_options_.sample_for_compression,
-          mutable_cf_options_.compression_opts,
+          output_compression_, mutable_cf_options_.compression_opts,
          mutable_cf_options_.paranoid_file_checks, cfd_->internal_stats(),
          TableFileCreationReason::kFlush, &io_s, io_tracer_, event_logger_,
          job_context_->job_id, Env::IO_HIGH, &table_properties_, 0 /* level */,
@@ -426,7 +426,7 @@ void ForwardIterator::SeekInternal(const Slice& internal_key,
      if (seek_to_first) {
        l0_iters_[i]->SeekToFirst();
      } else {
-        // If the target key passes over the larget key, we are sure Next()
+        // If the target key passes over the largest key, we are sure Next()
        // won't go over this file.
        if (user_comparator_->Compare(target_user_key,
                                      l0[i]->largest.user_key()) > 0) {
@@ -72,7 +72,7 @@ using MultiGetRange = MultiGetContext::Range;
 // Note:  Many of the methods in this class have comments indicating that
 // external synchronization is required as these methods are not thread-safe.
 // It is up to higher layers of code to decide how to prevent concurrent
-// invokation of these methods.  This is usually done by acquiring either
+// invocation of these methods.  This is usually done by acquiring either
 // the db mutex or the single writer thread.
 //
 // Some of these methods are documented to only require external
@@ -139,7 +139,7 @@ class MemTable {
  // operations on the same MemTable (unless this Memtable is immutable).
  size_t ApproximateMemoryUsage();

-  // As a cheap version of `ApproximateMemoryUsage()`, this function doens't
+  // As a cheap version of `ApproximateMemoryUsage()`, this function doesn't
  // require external synchronization. The value may be less accurate though
  size_t ApproximateMemoryUsageFast() const {
    return approximate_memory_usage_.load(std::memory_order_relaxed);
@@ -533,7 +533,7 @@ class MemTable {
  SequenceNumber atomic_flush_seqno_;

  // keep track of memory usage in table_, arena_, and range_del_table_.
-  // Gets refrshed inside `ApproximateMemoryUsage()` or `ShouldFlushNow`
+  // Gets refreshed inside `ApproximateMemoryUsage()` or `ShouldFlushNow`
  std::atomic<uint64_t> approximate_memory_usage_;

 #ifndef ROCKSDB_LITE
@@ -521,7 +521,7 @@ void MemTableList::Add(MemTable* m, autovector<MemTable*>* to_delete) {
  InstallNewVersion();
  // this method is used to move mutable memtable into an immutable list.
  // since mutable memtable is already refcounted by the DBImpl,
-  // and when moving to the imutable list we don't unref it,
+  // and when moving to the immutable list we don't unref it,
  // we don't have to ref the memtable here. we just take over the
  // reference from the DBImpl.
  current_->Add(m, to_delete);
@@ -43,12 +43,12 @@ class TruncatedRangeDelIterator {

  void InternalNext();

-  // Seeks to the tombstone with the highest viisble sequence number that covers
+  // Seeks to the tombstone with the highest visible sequence number that covers
  // target (a user key). If no such tombstone exists, the position will be at
  // the earliest tombstone that ends after target.
  void Seek(const Slice& target);

-  // Seeks to the tombstone with the highest viisble sequence number that covers
+  // Seeks to the tombstone with the highest visible sequence number that covers
  // target (a user key). If no such tombstone exists, the position will be at
  // the latest tombstone that starts before target.
  void SeekForPrev(const Slice& target);
@@ -447,13 +447,12 @@ class Repairer {
          nullptr /* blob_file_additions */, cfd->internal_comparator(),
          cfd->int_tbl_prop_collector_factories(), cfd->GetID(), cfd->GetName(),
          {}, kMaxSequenceNumber, snapshot_checker, kNoCompression,
-          0 /* sample_for_compression */, CompressionOptions(), false,
-          nullptr /* internal_stats */, TableFileCreationReason::kRecovery,
-          &io_s, nullptr /*IOTracer*/, nullptr /* event_logger */,
-          0 /* job_id */, Env::IO_HIGH, nullptr /* table_properties */,
-          -1 /* level */, current_time, 0 /* oldest_key_time */, write_hint,
-          0 /* file_creation_time */, "DB Repairer" /* db_id */,
-          db_session_id_);
+          CompressionOptions(), false, nullptr /* internal_stats */,
+          TableFileCreationReason::kRecovery, &io_s, nullptr /*IOTracer*/,
+          nullptr /* event_logger */, 0 /* job_id */, Env::IO_HIGH,
+          nullptr /* table_properties */, -1 /* level */, current_time,
+          0 /* oldest_key_time */, write_hint, 0 /* file_creation_time */,
+          "DB Repairer" /* db_id */, db_session_id_);
      ROCKS_LOG_INFO(db_options_.info_log,
                     "Log #%" PRIu64 ": %d ops saved to Table #%" PRIu64 " %s",
                     log, counter, meta.fd.GetNumber(),
@@ -23,7 +23,7 @@ class SnapshotImpl : public Snapshot {
  SequenceNumber number_;  // const after creation
  // It indicates the smallest uncommitted data at the time the snapshot was
  // taken. This is currently used by WritePrepared transactions to limit the
-  // scope of queries to IsInSnpashot.
+  // scope of queries to IsInSnapshot.
  SequenceNumber min_uncommitted_ = kMinUnCommittedSeq;

  virtual SequenceNumber GetSequenceNumber() const override { return number_; }
@@ -183,7 +183,7 @@ class TableCache {

  Cache* get_cache() const { return cache_; }

-  // Capacity of the backing Cache that indicates inifinite TableCache capacity.
+  // Capacity of the backing Cache that indicates infinite TableCache capacity.
  // For example when max_open_files is -1 we set the backing Cache to this.
  static const int kInfiniteCapacity = 0x400000;

@@ -43,10 +43,10 @@ Status UserKeyTablePropertiesCollector::InternalAdd(const Slice& key,
 }

 void UserKeyTablePropertiesCollector::BlockAdd(
-    uint64_t bLockRawBytes, uint64_t blockCompressedBytesFast,
-    uint64_t blockCompressedBytesSlow) {
-  return collector_->BlockAdd(bLockRawBytes, blockCompressedBytesFast,
-                              blockCompressedBytesSlow);
+    uint64_t block_raw_bytes, uint64_t block_compressed_bytes_fast,
+    uint64_t block_compressed_bytes_slow) {
+  return collector_->BlockAdd(block_raw_bytes, block_compressed_bytes_fast,
+                              block_compressed_bytes_slow);
 }

 Status UserKeyTablePropertiesCollector::Finish(
@@ -27,9 +27,9 @@ class IntTblPropCollector {
  virtual Status InternalAdd(const Slice& key, const Slice& value,
                             uint64_t file_size) = 0;

-  virtual void BlockAdd(uint64_t blockRawBytes,
-                        uint64_t blockCompressedBytesFast,
-                        uint64_t blockCompressedBytesSlow) = 0;
+  virtual void BlockAdd(uint64_t block_raw_bytes,
+                        uint64_t block_compressed_bytes_fast,
+                        uint64_t block_compressed_bytes_slow) = 0;

  virtual UserCollectedProperties GetReadableProperties() const = 0;

@@ -64,9 +64,9 @@ class UserKeyTablePropertiesCollector : public IntTblPropCollector {
  virtual Status InternalAdd(const Slice& key, const Slice& value,
                             uint64_t file_size) override;

-  virtual void BlockAdd(uint64_t blockRawBytes,
-                        uint64_t blockCompressedBytesFast,
-                        uint64_t blockCompressedBytesSlow) override;
+  virtual void BlockAdd(uint64_t block_raw_bytes,
+                        uint64_t block_compressed_bytes_fast,
+                        uint64_t block_compressed_bytes_slow) override;

  virtual Status Finish(UserCollectedProperties* properties) override;

@@ -55,8 +55,7 @@ void MakeBuilder(const Options& options, const ImmutableCFOptions& ioptions,
  builder->reset(NewTableBuilder(
      ioptions, moptions, internal_comparator, int_tbl_prop_collector_factories,
      kTestColumnFamilyId, kTestColumnFamilyName, writable->get(),
-      options.compression, options.sample_for_compression,
-      options.compression_opts, unknown_level));
+      options.compression, options.compression_opts, unknown_level));
 }
 }  // namespace

@@ -176,9 +175,9 @@ class RegularKeysStartWithAInternal : public IntTblPropCollector {
    return Status::OK();
  }

-  void BlockAdd(uint64_t /* blockRawBytes */,
-                uint64_t /* blockCompressedBytesFast */,
-                uint64_t /* blockCompressedBytesSlow */) override {
+  void BlockAdd(uint64_t /* block_raw_bytes */,
+                uint64_t /* block_compressed_bytes_fast */,
+                uint64_t /* block_compressed_bytes_slow */) override {
    // Nothing to do.
    return;
  }
@@ -74,7 +74,7 @@ enum NewFileCustomTag : uint32_t {
  kNeedCompaction = 2,
  // Since Manifest is not entirely forward-compatible, we currently encode
  // kMinLogNumberToKeep as part of NewFile as a hack. This should be removed
-  // when manifest becomes forward-comptabile.
+  // when manifest becomes forward-compatible.
  kMinLogNumberToKeepHack = 3,
  kOldestBlobFileNumber = 4,
  kOldestAncesterTime = 5,
@@ -195,7 +195,7 @@ struct FileMetaData {

  // The file could be the compaction output from other SST files, which could
  // in turn be outputs for compact older SST files. We track the memtable
-  // flush timestamp for the oldest SST file that eventaully contribute data
+  // flush timestamp for the oldest SST file that eventually contribute data
  // to this file. 0 means the information is not available.
  uint64_t oldest_ancester_time = kUnknownOldestAncesterTime;

@@ -408,7 +408,7 @@ class FilePickerMultiGet {
  int GetCurrentLevel() const { return curr_level_; }

  // Iterates through files in the current level until it finds a file that
-  // contains atleast one key from the MultiGet batch
+  // contains at least one key from the MultiGet batch
  bool GetNextFileInLevelWithKeys(MultiGetRange* next_file_range,
                                  size_t* file_index, FdWithKeyRange** fd,
                                  bool* is_last_key_in_file) {
@@ -2786,7 +2786,7 @@ struct Fsize {
  FileMetaData* file;
 };

-// Compator that is used to sort files based on their size
+// Comparator that is used to sort files based on their size
 // In normal mode: descending size
 bool CompareCompensatedSizeDescending(const Fsize& first, const Fsize& second) {
  return (first.file->compensated_file_size >
@@ -3206,7 +3206,7 @@ void VersionStorageInfo::GetCleanInputsWithinInterval(
 // specified range. From that file, iterate backwards and
 // forwards to find all overlapping files.
 // if within_range is set, then only store the maximum clean inputs
-// within range [begin, end]. "clean" means there is a boudnary
+// within range [begin, end]. "clean" means there is a boundary
 // between the files in "*inputs" and the surrounding files
 void VersionStorageInfo::GetOverlappingInputsRangeBinarySearch(
    int level, const InternalKey* begin, const InternalKey* end,
@@ -3517,7 +3517,7 @@ void VersionStorageInfo::CalculateBaseBytes(const ImmutableCFOptions& ioptions,
        //   1. the L0 size is larger than level size base, or
        //   2. number of L0 files reaches twice the L0->L1 compaction trigger
        // We don't do this otherwise to keep the LSM-tree structure stable
-        // unless the L0 compation is backlogged.
+        // unless the L0 compaction is backlogged.
        base_level_size = l0_size;
        if (base_level_ == num_levels_ - 1) {
          level_multiplier_ = 1.0;
@@ -4354,7 +4354,7 @@ Status VersionSet::ProcessManifestWrites(
  return s;
 }

-// 'datas' is gramatically incorrect. We still use this notation to indicate
+// 'datas' is grammatically incorrect. We still use this notation to indicate
 // that this variable represents a collection of column_family_data.
 Status VersionSet::LogAndApply(
    const autovector<ColumnFamilyData*>& column_family_datas,
@@ -4796,7 +4796,7 @@ Status VersionSet::TryRecoverFromOneManifest(
 Status VersionSet::ListColumnFamilies(std::vector<std::string>* column_families,
                                      const std::string& dbname,
                                      FileSystem* fs) {
-  // these are just for performance reasons, not correcntes,
+  // these are just for performance reasons, not correctness,
  // so we're fine using the defaults
  FileOptions soptions;
  // Read "CURRENT" file, which contains a pointer to the current manifest file
@@ -5499,20 +5499,6 @@ bool VersionSet::VerifyCompactionFileConsistency(Compaction* c) {
        "[%s] compaction output being applied to a different base version from"
        " input version",
        c->column_family_data()->GetName().c_str());
-
-    if (vstorage->compaction_style_ == kCompactionStyleLevel &&
-        c->start_level() == 0 && c->num_input_levels() > 2U) {
-      // We are doing a L0->base_level compaction. The assumption is if
-      // base level is not L1, levels from L1 to base_level - 1 is empty.
-      // This is ensured by having one compaction from L0 going on at the
-      // same time in level-based compaction. So that during the time, no
-      // compaction/flush can put files to those levels.
-      for (int l = c->start_level() + 1; l < c->output_level(); l++) {
-        if (vstorage->NumLevelFiles(l) != 0) {
-          return false;
-        }
-      }
-    }
  }

  for (size_t input = 0; input < c->num_input_levels(); ++input) {
@@ -2781,7 +2781,7 @@ class VersionSetTestMissingFiles : public VersionSetTestBase,
          TableBuilderOptions(
              immutable_cf_options_, mutable_cf_options_, *internal_comparator_,
              &int_tbl_prop_collector_factories, kNoCompression,
-              /*_sample_for_compression=*/0, CompressionOptions(),
+              CompressionOptions(),
              /*_skip_filters=*/false, info.column_family, info.level),
          TablePropertiesCollectorFactory::Context::kUnknownColumnFamily,
          fwriter.get()));
@@ -2793,11 +2793,9 @@ class VersionSetTestMissingFiles : public VersionSetTestBase,
      s = fs_->GetFileSize(fname, IOOptions(), &file_size, nullptr);
      ASSERT_OK(s);
      ASSERT_NE(0, file_size);
-      FileMetaData meta;
-      meta = FileMetaData(file_num, /*file_path_id=*/0, file_size, ikey, ikey,
-                          0, 0, false, 0, 0, 0, kUnknownFileChecksum,
-                          kUnknownFileChecksumFuncName);
-      file_metas->emplace_back(meta);
+      file_metas->emplace_back(file_num, /*file_path_id=*/0, file_size, ikey,
+                               ikey, 0, 0, false, 0, 0, 0, kUnknownFileChecksum,
+                               kUnknownFileChecksumFuncName);
    }
  }

@@ -30,7 +30,7 @@ enum ROCKSDB_NAMESPACE::ChecksumType checksum_type_e =
    ROCKSDB_NAMESPACE::kCRC32c;
 enum RepFactory FLAGS_rep_factory = kSkipList;
 std::vector<double> sum_probs(100001);
-int64_t zipf_sum_size = 100000;
+constexpr int64_t zipf_sum_size = 100000;

 namespace ROCKSDB_NAMESPACE {

@@ -233,6 +233,15 @@ size_t GenerateValue(uint32_t rand, char* v, size_t max_sz) {
  return value_sz;  // the size of the value set.
 }

+std::string NowNanosStr() {
+  uint64_t t = db_stress_env->NowNanos();
+  std::string ret;
+  PutFixed64(&ret, t);
+  return ret;
+}
+
+std::string GenerateTimestampForRead() { return NowNanosStr(); }
+
 namespace {

 class MyXXH64Checksum : public FileChecksumGenerator {
@@ -260,9 +260,11 @@ DECLARE_bool(enable_compaction_filter);
 DECLARE_bool(paranoid_file_checks);
 DECLARE_uint64(batch_protection_bytes_per_key);

-const long KB = 1024;
-const int kRandomValueMaxFactor = 3;
-const int kValueMaxLen = 100;
+DECLARE_uint64(user_timestamp_size);
+
+constexpr long KB = 1024;
+constexpr int kRandomValueMaxFactor = 3;
+constexpr int kValueMaxLen = 100;

 // wrapped posix or hdfs environment
 extern ROCKSDB_NAMESPACE::Env* db_stress_env;
@@ -561,6 +563,9 @@ extern StressTest* CreateNonBatchedOpsStressTest();
 extern void InitializeHotKeyGenerator(double alpha);
 extern int64_t GetOneHotKeyID(double rand_seed, int64_t max_key);

+extern std::string GenerateTimestampForRead();
+extern std::string NowNanosStr();
+
 std::shared_ptr<FileChecksumGenFactory> GetFileChecksumImpl(
    const std::string& name);
 }  // namespace ROCKSDB_NAMESPACE
@@ -804,4 +804,8 @@ DEFINE_string(file_checksum_impl, "none",
 DEFINE_int32(write_fault_one_in, 0,
             "On non-zero, enables fault injection on write");

+DEFINE_uint64(user_timestamp_size, 0,
+              "Number of bytes for a user-defined timestamp. Currently, only "
+              "8-byte is supported");
+
 #endif  // GFLAGS
@@ -418,6 +418,8 @@ struct ThreadState {
    std::string value;
    // optional state of all keys in the db
    std::vector<bool>* key_vec;
+
+    std::string timestamp;
  };
  std::queue<std::pair<uint64_t, SnapshotState>> snapshot_queue;

@@ -208,7 +208,7 @@ bool StressTest::BuildOptionsTable() {
    options_tbl.emplace("enable_blob_files",
                        std::vector<std::string>{"false", "true"});
    options_tbl.emplace("min_blob_size",
-                        std::vector<std::string>{"0", "16", "256"});
+                        std::vector<std::string>{"0", "8", "16"});
    options_tbl.emplace("blob_file_size",
                        std::vector<std::string>{"1M", "16M", "256M", "1G"});
    options_tbl.emplace("blob_compression_type", GetBlobCompressionTags());
@@ -317,6 +317,11 @@ Status StressTest::AssertSame(DB* db, ColumnFamilyHandle* cf,
  }
  ReadOptions ropt;
  ropt.snapshot = snap_state.snapshot;
+  Slice ts;
+  if (!snap_state.timestamp.empty()) {
+    ts = snap_state.timestamp;
+    ropt.timestamp = &ts;
+  }
  PinnableSlice exp_v(&snap_state.value);
  exp_v.PinSelf();
  PinnableSlice v;
@@ -422,6 +427,13 @@ void StressTest::PreloadDbAndReopenAsReadOnly(int64_t number_of_keys,
        }
      } else {
        if (!FLAGS_use_txn) {
+          std::string ts_str;
+          Slice ts;
+          if (FLAGS_user_timestamp_size > 0) {
+            ts_str = NowNanosStr();
+            ts = ts_str;
+            write_opts.timestamp = &ts;
+          }
          s = db_->Put(write_opts, cfh, key, v);
        } else {
 #ifndef ROCKSDB_LITE
@@ -564,10 +576,9 @@ void StressTest::OperateDb(ThreadState* thread) {
  if (FLAGS_write_fault_one_in) {
    IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
    error_msg.SetRetryable(true);
-    std::vector<FileType> types;
-    types.push_back(FileType::kTableFile);
-    types.push_back(FileType::kDescriptorFile);
-    types.push_back(FileType::kCurrentFile);
+    std::vector<FileType> types = {FileType::kTableFile,
+                                   FileType::kDescriptorFile,
+                                   FileType::kCurrentFile};
    fault_fs_guard->SetRandomWriteError(
        thread->shared->GetSeed(), FLAGS_write_fault_one_in, error_msg, types);
  }
@@ -766,6 +777,20 @@ void StressTest::OperateDb(ThreadState* thread) {
        }
      }

+      // Assign timestamps if necessary.
+      std::string read_ts_str;
+      std::string write_ts_str;
+      Slice read_ts;
+      Slice write_ts;
+      if (ShouldAcquireMutexOnKey() && FLAGS_user_timestamp_size > 0) {
+        read_ts_str = GenerateTimestampForRead();
+        read_ts = read_ts_str;
+        read_opts.timestamp = &read_ts;
+        write_ts_str = NowNanosStr();
+        write_ts = write_ts_str;
+        write_opts.timestamp = &write_ts;
+      }
+
      int prob_op = thread->rand.Uniform(100);
      // Reset this in case we pick something other than a read op. We don't
      // want to use a stale value when deciding at the beginning of the loop
@@ -856,8 +881,16 @@ std::vector<std::string> StressTest::GetWhiteBoxKeys(ThreadState* thread,
  std::vector<std::string> boundaries;
  for (const LevelMetaData& lmd : cfmd.levels) {
    for (const SstFileMetaData& sfmd : lmd.files) {
-      boundaries.push_back(sfmd.smallestkey);
-      boundaries.push_back(sfmd.largestkey);
+      // If FLAGS_user_timestamp_size > 0, then both smallestkey and largestkey
+      // have timestamps.
+      const auto& skey = sfmd.smallestkey;
+      const auto& lkey = sfmd.largestkey;
+      assert(skey.size() >= FLAGS_user_timestamp_size);
+      assert(lkey.size() >= FLAGS_user_timestamp_size);
+      boundaries.push_back(
+          skey.substr(0, skey.size() - FLAGS_user_timestamp_size));
+      boundaries.push_back(
+          lkey.substr(0, lkey.size() - FLAGS_user_timestamp_size));
    }
  }
  if (boundaries.empty()) {
@@ -1007,6 +1040,7 @@ Status StressTest::TestIterate(ThreadState* thread,
    // iterators with the same set-up, and it doesn't hurt to check them
    // to be equal.
    ReadOptions cmp_ro;
+    cmp_ro.timestamp = readoptionscopy.timestamp;
    cmp_ro.snapshot = snapshot;
    cmp_ro.total_order_seek = true;
    ColumnFamilyHandle* cmp_cfh =
@@ -1126,21 +1160,25 @@ void StressTest::VerifyIterator(ThreadState* thread,
    *diverged = true;
    return;
  } else if (op == kLastOpSeek && ro.iterate_lower_bound != nullptr &&
-             (options_.comparator->Compare(*ro.iterate_lower_bound, seek_key) >=
-                  0 ||
+             (options_.comparator->CompareWithoutTimestamp(
+                  *ro.iterate_lower_bound, /*a_has_ts=*/false, seek_key,
+                  /*b_has_ts=*/false) >= 0 ||
              (ro.iterate_upper_bound != nullptr &&
-               options_.comparator->Compare(*ro.iterate_lower_bound,
-                                            *ro.iterate_upper_bound) >= 0))) {
+               options_.comparator->CompareWithoutTimestamp(
+                   *ro.iterate_lower_bound, /*a_has_ts=*/false,
+                   *ro.iterate_upper_bound, /*b_has_ts*/ false) >= 0))) {
    // Lower bound behavior is not well defined if it is larger than
    // seek key or upper bound. Disable the check for now.
    *diverged = true;
    return;
  } else if (op == kLastOpSeekForPrev && ro.iterate_upper_bound != nullptr &&
-             (options_.comparator->Compare(*ro.iterate_upper_bound, seek_key) <=
-                  0 ||
+             (options_.comparator->CompareWithoutTimestamp(
+                  *ro.iterate_upper_bound, /*a_has_ts=*/false, seek_key,
+                  /*b_has_ts=*/false) <= 0 ||
              (ro.iterate_lower_bound != nullptr &&
-               options_.comparator->Compare(*ro.iterate_lower_bound,
-                                            *ro.iterate_upper_bound) >= 0))) {
+               options_.comparator->CompareWithoutTimestamp(
+                   *ro.iterate_lower_bound, /*a_has_ts=*/false,
+                   *ro.iterate_upper_bound, /*b_has_ts=*/false) >= 0))) {
    // Uppder bound behavior is not well defined if it is smaller than
    // seek key or lower bound. Disable the check for now.
    *diverged = true;
@@ -1209,9 +1247,13 @@ void StressTest::VerifyIterator(ThreadState* thread,
      if ((iter->Valid() && iter->key() != cmp_iter->key()) ||
          (!iter->Valid() &&
           (ro.iterate_upper_bound == nullptr ||
-            cmp->Compare(total_order_key, *ro.iterate_upper_bound) < 0) &&
+            cmp->CompareWithoutTimestamp(total_order_key, /*a_has_ts=*/false,
+                                         *ro.iterate_upper_bound,
+                                         /*b_has_ts=*/false) < 0) &&
           (ro.iterate_lower_bound == nullptr ||
-            cmp->Compare(total_order_key, *ro.iterate_lower_bound) > 0))) {
+            cmp->CompareWithoutTimestamp(total_order_key, /*a_has_ts=*/false,
+                                         *ro.iterate_lower_bound,
+                                         /*b_has_ts=*/false) > 0))) {
        fprintf(stderr,
                "Iterator diverged from control iterator which"
                " has value %s %s\n",
@@ -1326,8 +1368,13 @@ Status StressTest::TestBackupRestore(
    }
  }
  std::vector<BackupInfo> backup_info;
+  // If inplace_not_restore, we verify the backup by opening it as a
+  // read-only DB. If !inplace_not_restore, we restore it to a temporary
+  // directory for verification.
+  bool inplace_not_restore = thread->rand.OneIn(3);
  if (s.ok()) {
-    backup_engine->GetBackupInfo(&backup_info);
+    backup_engine->GetBackupInfo(&backup_info,
+                                 /*include_file_details*/ inplace_not_restore);
    if (backup_info.empty()) {
      s = Status::NotFound("no backups found");
      from = "BackupEngine::GetBackupInfo";
@@ -1343,8 +1390,8 @@ Status StressTest::TestBackupRestore(
  }
  const bool allow_persistent = thread->tid == 0;  // not too many
  bool from_latest = false;
-  if (s.ok()) {
-    int count = static_cast<int>(backup_info.size());
+  int count = static_cast<int>(backup_info.size());
+  if (s.ok() && !inplace_not_restore) {
    if (count > 1) {
      s = backup_engine->RestoreDBFromBackup(
          RestoreOptions(), backup_info[thread->rand.Uniform(count)].backup_id,
@@ -1362,7 +1409,9 @@ Status StressTest::TestBackupRestore(
      }
    }
  }
-  if (s.ok()) {
+  if (s.ok() && !inplace_not_restore) {
+    // Purge early if restoring, to ensure the restored directory doesn't
+    // have some secret dependency on the backup directory.
    uint32_t to_keep = 0;
    if (allow_persistent) {
      // allow one thread to keep up to 2 backups
@@ -1390,10 +1439,21 @@ Status StressTest::TestBackupRestore(
    for (auto name : column_family_names_) {
      cf_descriptors.emplace_back(name, ColumnFamilyOptions(restore_options));
    }
-    s = DB::Open(DBOptions(restore_options), restore_dir, cf_descriptors,
-                 &restored_cf_handles, &restored_db);
-    if (!s.ok()) {
-      from = "DB::Open in backup/restore";
+    if (inplace_not_restore) {
+      BackupInfo& info = backup_info[thread->rand.Uniform(count)];
+      restore_options.env = info.env_for_open.get();
+      s = DB::OpenForReadOnly(DBOptions(restore_options), info.name_for_open,
+                              cf_descriptors, &restored_cf_handles,
+                              &restored_db);
+      if (!s.ok()) {
+        from = "DB::OpenForReadOnly in backup/restore";
+      }
+    } else {
+      s = DB::Open(DBOptions(restore_options), restore_dir, cf_descriptors,
+                   &restored_cf_handles, &restored_db);
+      if (!s.ok()) {
+        from = "DB::Open in backup/restore";
+      }
    }
  }
  // Note the column families chosen by `rand_column_families` cannot be
@@ -1407,8 +1467,16 @@ Status StressTest::TestBackupRestore(
    std::string key_str = Key(rand_keys[0]);
    Slice key = key_str;
    std::string restored_value;
+    ReadOptions read_opts;
+    std::string ts_str;
+    Slice ts;
+    if (FLAGS_user_timestamp_size > 0) {
+      ts_str = GenerateTimestampForRead();
+      ts = ts_str;
+      read_opts.timestamp = &ts;
+    }
    Status get_status = restored_db->Get(
-        ReadOptions(), restored_cf_handles[rand_column_families[i]], key,
+        read_opts, restored_cf_handles[rand_column_families[i]], key,
        &restored_value);
    bool exists = thread->shared->Exists(rand_column_families[i], rand_keys[0]);
    if (get_status.ok()) {
@@ -1426,10 +1494,6 @@ Status StressTest::TestBackupRestore(
      }
    }
  }
-  if (backup_engine != nullptr) {
-    delete backup_engine;
-    backup_engine = nullptr;
-  }
  if (restored_db != nullptr) {
    for (auto* cf_handle : restored_cf_handles) {
      restored_db->DestroyColumnFamilyHandle(cf_handle);
@@ -1437,6 +1501,22 @@ Status StressTest::TestBackupRestore(
    delete restored_db;
    restored_db = nullptr;
  }
+  if (s.ok() && inplace_not_restore) {
+    // Purge late if inplace open read-only
+    uint32_t to_keep = 0;
+    if (allow_persistent) {
+      // allow one thread to keep up to 2 backups
+      to_keep = thread->rand.Uniform(3);
+    }
+    s = backup_engine->PurgeOldBackups(to_keep);
+    if (!s.ok()) {
+      from = "BackupEngine::PurgeOldBackups";
+    }
+  }
+  if (backup_engine != nullptr) {
+    delete backup_engine;
+    backup_engine = nullptr;
+  }
  if (s.ok()) {
    // Preserve directories on failure, or allowed persistent backup
    if (!allow_persistent) {
@@ -1739,6 +1819,7 @@ void StressTest::TestAcquireSnapshot(ThreadState* thread,
                                     const std::string& keystr, uint64_t i) {
  Slice key = keystr;
  ColumnFamilyHandle* column_family = column_families_[rand_column_family];
+  ReadOptions ropt;
 #ifndef ROCKSDB_LITE
  auto db_impl = static_cast_with_check<DBImpl>(db_->GetRootDB());
  const bool ww_snapshot = thread->rand.OneIn(10);
@@ -1748,8 +1829,19 @@ void StressTest::TestAcquireSnapshot(ThreadState* thread,
 #else
  const Snapshot* snapshot = db_->GetSnapshot();
 #endif  // !ROCKSDB_LITE
-  ReadOptions ropt;
  ropt.snapshot = snapshot;
+
+  // Ideally, we want snapshot taking and timestamp generation to be atomic
+  // here, so that the snapshot corresponds to the timestamp. However, it is
+  // not possible with current GetSnapshot() API.
+  std::string ts_str;
+  Slice ts;
+  if (FLAGS_user_timestamp_size > 0) {
+    ts_str = GenerateTimestampForRead();
+    ts = ts_str;
+    ropt.timestamp = &ts;
+  }
+
  std::string value_at;
  // When taking a snapshot, we also read a key from that snapshot. We
  // will later read the same key before releasing the snapshot and
@@ -1771,10 +1863,14 @@ void StressTest::TestAcquireSnapshot(ThreadState* thread,
    }
  }

-  ThreadState::SnapshotState snap_state = {
-      snapshot, rand_column_family, column_family->GetName(),
-      keystr,   status_at,          value_at,
-      key_vec};
+  ThreadState::SnapshotState snap_state = {snapshot,
+                                           rand_column_family,
+                                           column_family->GetName(),
+                                           keystr,
+                                           status_at,
+                                           value_at,
+                                           key_vec,
+                                           ts_str};
  uint64_t hold_for = FLAGS_snapshot_hold_ops;
  if (FLAGS_long_running_snapshots) {
    // Hold 10% of snapshots for 10x more
@@ -1879,6 +1975,13 @@ uint32_t StressTest::GetRangeHash(ThreadState* thread, const Snapshot* snapshot,
  ReadOptions ro;
  ro.snapshot = snapshot;
  ro.total_order_seek = true;
+  std::string ts_str;
+  Slice ts;
+  if (FLAGS_user_timestamp_size > 0) {
+    ts_str = GenerateTimestampForRead();
+    ts = ts_str;
+    ro.timestamp = &ts;
+  }
  std::unique_ptr<Iterator> it(db_->NewIterator(ro, column_family));
  for (it->Seek(start_key);
       it->Valid() && options_.comparator->Compare(it->key(), end_key) <= 0;
@@ -2004,6 +2107,8 @@ void StressTest::PrintEnv() const {
  fprintf(stdout, "Sync fault injection      : %d\n", FLAGS_sync_fault_injection);
  fprintf(stdout, "Best efforts recovery     : %d\n",
          static_cast<int>(FLAGS_best_efforts_recovery));
+  fprintf(stdout, "User timestamp size bytes : %d\n",
+          static_cast<int>(FLAGS_user_timestamp_size));

  fprintf(stdout, "------------------------------------------------\n");
 }
@@ -2247,6 +2352,11 @@ void StressTest::Open() {
  fprintf(stdout, "DB path: [%s]\n", FLAGS_db.c_str());

  Status s;
+
+  if (FLAGS_user_timestamp_size > 0) {
+    CheckAndSetOptionsForUserTimestamp();
+  }
+
  if (FLAGS_ttl == -1) {
    std::vector<std::string> existing_column_families;
    s = DB::ListColumnFamilies(DBOptions(options_), FLAGS_db,
@@ -2498,5 +2608,72 @@ void StressTest::Reopen(ThreadState* thread) {
          clock_->TimeToString(now / 1000000).c_str(), num_times_reopened_);
  Open();
 }
+
+void StressTest::CheckAndSetOptionsForUserTimestamp() {
+  assert(FLAGS_user_timestamp_size > 0);
+  const Comparator* const cmp = test::ComparatorWithU64Ts();
+  assert(cmp);
+  if (FLAGS_user_timestamp_size != cmp->timestamp_size()) {
+    fprintf(stderr,
+            "Only -user_timestamp_size=%d is supported in stress test.\n",
+            static_cast<int>(cmp->timestamp_size()));
+    exit(1);
+  }
+  if (FLAGS_nooverwritepercent > 0) {
+    fprintf(stderr,
+            "-nooverwritepercent must be 0 because SingleDelete must be "
+            "disabled.\n");
+    exit(1);
+  }
+  if (FLAGS_use_merge || FLAGS_use_full_merge_v1) {
+    fprintf(stderr, "Merge does not support timestamp yet.\n");
+    exit(1);
+  }
+  if (FLAGS_delrangepercent > 0) {
+    fprintf(stderr, "DeleteRange does not support timestamp yet.\n");
+    exit(1);
+  }
+  if (FLAGS_use_txn) {
+    fprintf(stderr, "TransactionDB does not support timestamp yet.\n");
+    exit(1);
+  }
+  if (FLAGS_read_only) {
+    fprintf(stderr, "When opened as read-only, timestamp not supported.\n");
+    exit(1);
+  }
+  if (FLAGS_test_secondary || FLAGS_secondary_catch_up_one_in > 0 ||
+      FLAGS_continuous_verification_interval > 0) {
+    fprintf(stderr, "Secondary instance does not support timestamp.\n");
+    exit(1);
+  }
+  if (FLAGS_checkpoint_one_in > 0) {
+    fprintf(stderr,
+            "-checkpoint_one_in=%d requires "
+            "DBImplReadOnly, which is not supported with timestamp\n",
+            FLAGS_checkpoint_one_in);
+    exit(1);
+  }
+#ifndef ROCKSDB_LITE
+  if (FLAGS_enable_blob_files || FLAGS_use_blob_db) {
+    fprintf(stderr, "BlobDB not supported with timestamp.\n");
+    exit(1);
+  }
+#endif  // !ROCKSDB_LITE
+  if (FLAGS_enable_compaction_filter) {
+    fprintf(stderr, "CompactionFilter not supported with timestamp.\n");
+    exit(1);
+  }
+  if (FLAGS_test_cf_consistency || FLAGS_test_batches_snapshots) {
+    fprintf(stderr,
+            "Due to per-key ts-seq ordering constraint, only the (default) "
+            "non-batched test is supported with timestamp.\n");
+    exit(1);
+  }
+  if (FLAGS_ingest_external_file_one_in > 0) {
+    fprintf(stderr, "Bulk loading may not support timestamp yet.\n");
+    exit(1);
+  }
+  options_.comparator = cmp;
+}
 }  // namespace ROCKSDB_NAMESPACE
 #endif  // GFLAGS
@@ -211,6 +211,8 @@ class StressTest {

  void Reopen(ThreadState* thread);

+  void CheckAndSetOptionsForUserTimestamp();
+
  std::shared_ptr<Cache> cache_;
  std::shared_ptr<Cache> compressed_cache_;
  std::shared_ptr<const FilterPolicy> filter_policy_;
@@ -22,6 +22,13 @@ class NonBatchedOpsStressTest : public StressTest {

  void VerifyDb(ThreadState* thread) const override {
    ReadOptions options(FLAGS_verify_checksum, true);
+    std::string ts_str;
+    Slice ts;
+    if (FLAGS_user_timestamp_size > 0) {
+      ts_str = GenerateTimestampForRead();
+      ts = ts_str;
+      options.timestamp = &ts;
+    }
    auto shared = thread->shared;
    const int64_t max_key = shared->GetMaxKey();
    const int64_t keys_per_thread = max_key / shared->GetNumThreads();
@@ -477,6 +484,8 @@ class NonBatchedOpsStressTest : public StressTest {
    int64_t max_key = shared->GetMaxKey();
    int64_t rand_key = rand_keys[0];
    int rand_column_family = rand_column_families[0];
+    std::string write_ts_str;
+    Slice write_ts;
    while (!shared->AllowsOverwrite(rand_key) &&
           (FLAGS_use_merge || shared->Exists(rand_column_family, rand_key))) {
      lock.reset();
@@ -484,6 +493,11 @@ class NonBatchedOpsStressTest : public StressTest {
      rand_column_family = thread->rand.Next() % FLAGS_column_families;
      lock.reset(
          new MutexLock(shared->GetMutexForKey(rand_column_family, rand_key)));
+      if (FLAGS_user_timestamp_size > 0) {
+        write_ts_str = NowNanosStr();
+        write_ts = write_ts_str;
+        write_opts.timestamp = &write_ts;
+      }
    }

    std::string key_str = Key(rand_key);
@@ -559,6 +573,8 @@ class NonBatchedOpsStressTest : public StressTest {
    // OPERATION delete
    // If the chosen key does not allow overwrite and it does not exist,
    // choose another key.
+    std::string write_ts_str;
+    Slice write_ts;
    while (!shared->AllowsOverwrite(rand_key) &&
           !shared->Exists(rand_column_family, rand_key)) {
      lock.reset();
@@ -566,6 +582,11 @@ class NonBatchedOpsStressTest : public StressTest {
      rand_column_family = thread->rand.Next() % FLAGS_column_families;
      lock.reset(
          new MutexLock(shared->GetMutexForKey(rand_column_family, rand_key)));
+      if (FLAGS_user_timestamp_size > 0) {
+        write_ts_str = NowNanosStr();
+        write_ts = write_ts_str;
+        write_opts.timestamp = &write_ts;
+      }
    }

    std::string key_str = Key(rand_key);
@@ -70,7 +70,7 @@ GEM
      jekyll-theme-time-machine (= 0.1.1)
      jekyll-titles-from-headings (= 0.5.3)
      jemoji (= 0.12.0)
-      kramdown (= 2.3.0)
+      kramdown (= 2.3.1)
      kramdown-parser-gfm (= 1.1.0)
      liquid (= 4.0.3)
      mercenary (~> 0.3)
@@ -196,7 +196,7 @@ GEM
      gemoji (~> 3.0)
      html-pipeline (~> 2.2)
      jekyll (>= 3.0, < 5.0)
-    kramdown (2.3.0)
+    kramdown (2.3.1)
      rexml
    kramdown-parser-gfm (1.1.0)
      kramdown (~> 2.0)
@@ -218,7 +218,12 @@ class CompositeEnv : public Env {
    return file_system_->OptimizeForCompactionTableRead(
        FileOptions(env_options), db_options);
  }
-
+  EnvOptions OptimizeForBlobFileRead(
+      const EnvOptions& env_options,
+      const ImmutableDBOptions& db_options) const override {
+    return file_system_->OptimizeForBlobFileRead(FileOptions(env_options),
+                                                 db_options);
+  }
  // This seems to clash with a macro on Windows, so #undef it here
 #ifdef GetFreeSpace
 #undef GetFreeSpace
@@ -536,6 +536,11 @@ class LegacyFileSystemWrapper : public FileSystem {
      const ImmutableDBOptions& db_options) const override {
    return target_->OptimizeForCompactionTableRead(file_options, db_options);
  }
+  FileOptions OptimizeForBlobFileRead(
+      const FileOptions& file_options,
+      const ImmutableDBOptions& db_options) const override {
+    return target_->OptimizeForBlobFileRead(file_options, db_options);
+  }

 #ifdef GetFreeSpace
 #undef GetFreeSpace
@@ -997,6 +1002,12 @@ EnvOptions Env::OptimizeForCompactionTableRead(
  optimized_env_options.use_direct_reads = db_options.use_direct_reads;
  return optimized_env_options;
 }
+EnvOptions Env::OptimizeForBlobFileRead(
+    const EnvOptions& env_options, const ImmutableDBOptions& db_options) const {
+  EnvOptions optimized_env_options(env_options);
+  optimized_env_options.use_direct_reads = db_options.use_direct_reads;
+  return optimized_env_options;
+}

 EnvOptions::EnvOptions(const DBOptions& options) {
  AssignEnvOptions(this, options);
@@ -7,26 +7,21 @@

 #include "env/env_chroot.h"

-#include <errno.h>
-#include <stdlib.h>
-#include <string.h>
-#include <unistd.h>
-
-#include <string>
-#include <utility>
-#include <vector>
+#include <errno.h>   // errno
+#include <stdlib.h>  // realpath, free
+#include <unistd.h>  // geteuid

 #include "env/composite_env_wrapper.h"
-#include "rocksdb/file_system.h"
-#include "rocksdb/status.h"
+#include "env/fs_remap.h"
+#include "util/string_util.h"  // errnoStr

 namespace ROCKSDB_NAMESPACE {
 namespace {
-class ChrootFileSystem : public FileSystemWrapper {
+class ChrootFileSystem : public RemapFileSystem {
 public:
  ChrootFileSystem(const std::shared_ptr<FileSystem>& base,
                   const std::string& chroot_dir)
-      : FileSystemWrapper(base) {
+      : RemapFileSystem(base) {
 #if defined(OS_AIX)
    char resolvedName[PATH_MAX];
    char* real_chroot_dir = realpath(chroot_dir.c_str(), resolvedName);
@@ -42,245 +37,6 @@ class ChrootFileSystem : public FileSystemWrapper {
  }

  const char* Name() const override { return "ChrootFS"; }
-  Status RegisterDbPaths(const std::vector<std::string>& paths) override {
-    std::vector<std::string> encoded_paths;
-    encoded_paths.reserve(paths.size());
-    for (auto& path : paths) {
-      auto status_and_enc_path = EncodePathWithNewBasename(path);
-      if (!status_and_enc_path.first.ok()) {
-        return status_and_enc_path.first;
-      }
-      encoded_paths.emplace_back(status_and_enc_path.second);
-    }
-    return FileSystemWrapper::RegisterDbPaths(encoded_paths);
-  }
-
-  Status UnregisterDbPaths(const std::vector<std::string>& paths) override {
-    std::vector<std::string> encoded_paths;
-    encoded_paths.reserve(paths.size());
-    for (auto& path : paths) {
-      auto status_and_enc_path = EncodePathWithNewBasename(path);
-      if (!status_and_enc_path.first.ok()) {
-        return status_and_enc_path.first;
-      }
-      encoded_paths.emplace_back(status_and_enc_path.second);
-    }
-    return FileSystemWrapper::UnregisterDbPaths(encoded_paths);
-  }
-
-  IOStatus NewSequentialFile(const std::string& fname,
-                             const FileOptions& options,
-                             std::unique_ptr<FSSequentialFile>* result,
-                             IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::NewSequentialFile(status_and_enc_path.second,
-                                                options, result, dbg);
-  }
-
-  IOStatus NewRandomAccessFile(const std::string& fname,
-                               const FileOptions& options,
-                               std::unique_ptr<FSRandomAccessFile>* result,
-                               IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::NewRandomAccessFile(status_and_enc_path.second,
-                                                  options, result, dbg);
-  }
-
-  IOStatus NewWritableFile(const std::string& fname, const FileOptions& options,
-                           std::unique_ptr<FSWritableFile>* result,
-                           IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::NewWritableFile(status_and_enc_path.second,
-                                              options, result, dbg);
-  }
-
-  IOStatus ReuseWritableFile(const std::string& fname,
-                             const std::string& old_fname,
-                             const FileOptions& options,
-                             std::unique_ptr<FSWritableFile>* result,
-                             IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    auto status_and_old_enc_path = EncodePath(old_fname);
-    if (!status_and_old_enc_path.first.ok()) {
-      return status_and_old_enc_path.first;
-    }
-    return FileSystemWrapper::ReuseWritableFile(status_and_old_enc_path.second,
-                                                status_and_old_enc_path.second,
-                                                options, result, dbg);
-  }
-
-  IOStatus NewRandomRWFile(const std::string& fname, const FileOptions& options,
-                           std::unique_ptr<FSRandomRWFile>* result,
-                           IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::NewRandomRWFile(status_and_enc_path.second,
-                                              options, result, dbg);
-  }
-
-  IOStatus NewDirectory(const std::string& dir, const IOOptions& options,
-                        std::unique_ptr<FSDirectory>* result,
-                        IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(dir);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::NewDirectory(status_and_enc_path.second, options,
-                                           result, dbg);
-  }
-
-  IOStatus FileExists(const std::string& fname, const IOOptions& options,
-                      IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::FileExists(status_and_enc_path.second, options,
-                                         dbg);
-  }
-
-  IOStatus GetChildren(const std::string& dir, const IOOptions& options,
-                       std::vector<std::string>* result,
-                       IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePath(dir);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::GetChildren(status_and_enc_path.second, options,
-                                          result, dbg);
-  }
-
-  IOStatus GetChildrenFileAttributes(const std::string& dir,
-                                     const IOOptions& options,
-                                     std::vector<FileAttributes>* result,
-                                     IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePath(dir);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::GetChildrenFileAttributes(
-        status_and_enc_path.second, options, result, dbg);
-  }
-
-  IOStatus DeleteFile(const std::string& fname, const IOOptions& options,
-                      IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePath(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::DeleteFile(status_and_enc_path.second, options,
-                                         dbg);
-  }
-
-  IOStatus CreateDir(const std::string& dirname, const IOOptions& options,
-                     IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(dirname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::CreateDir(status_and_enc_path.second, options,
-                                        dbg);
-  }
-
-  IOStatus CreateDirIfMissing(const std::string& dirname,
-                              const IOOptions& options,
-                              IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(dirname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::CreateDirIfMissing(status_and_enc_path.second,
-                                                 options, dbg);
-  }
-
-  IOStatus DeleteDir(const std::string& dirname, const IOOptions& options,
-                     IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePath(dirname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::DeleteDir(status_and_enc_path.second, options,
-                                        dbg);
-  }
-
-  IOStatus GetFileSize(const std::string& fname, const IOOptions& options,
-                       uint64_t* file_size, IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePath(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::GetFileSize(status_and_enc_path.second, options,
-                                          file_size, dbg);
-  }
-
-  IOStatus GetFileModificationTime(const std::string& fname,
-                                   const IOOptions& options,
-                                   uint64_t* file_mtime,
-                                   IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePath(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::GetFileModificationTime(
-        status_and_enc_path.second, options, file_mtime, dbg);
-  }
-
-  IOStatus RenameFile(const std::string& src, const std::string& dest,
-                      const IOOptions& options, IODebugContext* dbg) override {
-    auto status_and_src_enc_path = EncodePath(src);
-    if (!status_and_src_enc_path.first.ok()) {
-      return status_and_src_enc_path.first;
-    }
-    auto status_and_dest_enc_path = EncodePathWithNewBasename(dest);
-    if (!status_and_dest_enc_path.first.ok()) {
-      return status_and_dest_enc_path.first;
-    }
-    return FileSystemWrapper::RenameFile(status_and_src_enc_path.second,
-                                         status_and_dest_enc_path.second,
-                                         options, dbg);
-  }
-
-  IOStatus LinkFile(const std::string& src, const std::string& dest,
-                    const IOOptions& options, IODebugContext* dbg) override {
-    auto status_and_src_enc_path = EncodePath(src);
-    if (!status_and_src_enc_path.first.ok()) {
-      return status_and_src_enc_path.first;
-    }
-    auto status_and_dest_enc_path = EncodePathWithNewBasename(dest);
-    if (!status_and_dest_enc_path.first.ok()) {
-      return status_and_dest_enc_path.first;
-    }
-    return FileSystemWrapper::LinkFile(status_and_src_enc_path.second,
-                                       status_and_dest_enc_path.second, options,
-                                       dbg);
-  }
-
-  IOStatus LockFile(const std::string& fname, const IOOptions& options,
-                    FileLock** lock, IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    // FileLock subclasses may store path (e.g., PosixFileLock stores it). We
-    // can skip stripping the chroot directory from this path because callers
-    // shouldn't use it.
-    return FileSystemWrapper::LockFile(status_and_enc_path.second, options,
-                                       lock, dbg);
-  }

  IOStatus GetTestDirectory(const IOOptions& options, std::string* path,
                            IODebugContext* dbg) override {
@@ -294,33 +50,12 @@ class ChrootFileSystem : public FileSystemWrapper {
    return CreateDirIfMissing(*path, options, dbg);
  }

-  IOStatus NewLogger(const std::string& fname, const IOOptions& options,
-                     std::shared_ptr<Logger>* result,
-                     IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePathWithNewBasename(fname);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::NewLogger(status_and_enc_path.second, options,
-                                        result, dbg);
-  }
-
-  IOStatus GetAbsolutePath(const std::string& db_path, const IOOptions& options,
-                           std::string* output_path,
-                           IODebugContext* dbg) override {
-    auto status_and_enc_path = EncodePath(db_path);
-    if (!status_and_enc_path.first.ok()) {
-      return status_and_enc_path.first;
-    }
-    return FileSystemWrapper::GetAbsolutePath(status_and_enc_path.second,
-                                              options, output_path, dbg);
-  }
-
- private:
+ protected:
  // Returns status and expanded absolute path including the chroot directory.
  // Checks whether the provided path breaks out of the chroot. If it returns
  // non-OK status, the returned path should not be used.
-  std::pair<IOStatus, std::string> EncodePath(const std::string& path) {
+  std::pair<IOStatus, std::string> EncodePath(
+      const std::string& path) override {
    if (path.empty() || path[0] != '/') {
      return {IOStatus::InvalidArgument(path, "Not an absolute path"), ""};
    }
@@ -333,7 +68,7 @@ class ChrootFileSystem : public FileSystemWrapper {
    char* normalized_path = realpath(res.second.c_str(), nullptr);
 #endif
    if (normalized_path == nullptr) {
-      res.first = IOStatus::NotFound(res.second, strerror(errno));
+      res.first = IOStatus::NotFound(res.second, errnoStr(errno).c_str());
    } else if (strlen(normalized_path) < chroot_dir_.size() ||
               strncmp(normalized_path, chroot_dir_.c_str(),
                       chroot_dir_.size()) != 0) {
@@ -351,7 +86,7 @@ class ChrootFileSystem : public FileSystemWrapper {
  // Similar to EncodePath() except assumes the basename in the path hasn't been
  // created yet.
  std::pair<IOStatus, std::string> EncodePathWithNewBasename(
-      const std::string& path) {
+      const std::string& path) override {
    if (path.empty() || path[0] != '/') {
      return {IOStatus::InvalidArgument(path, "Not an absolute path"), ""};
    }
@@ -370,6 +105,7 @@ class ChrootFileSystem : public FileSystemWrapper {
    return status_and_enc_path;
  }

+ private:
  std::string chroot_dir_;
 };
 }  // namespace
@@ -15,6 +15,9 @@ namespace ROCKSDB_NAMESPACE {

 // Returns an Env that translates paths such that the root directory appears to
 // be chroot_dir. chroot_dir should refer to an existing directory.
+//
+// This class has not been fully analyzed for providing strong security
+// guarantees.
 Env* NewChrootEnv(Env* base_env, const std::string& chroot_dir);

 }  // namespace ROCKSDB_NAMESPACE
@@ -37,10 +37,10 @@ namespace {
 // Log error message
 static Status IOError(const std::string& context, int err_number) {
  return (err_number == ENOSPC)
-             ? Status::NoSpace(context, strerror(err_number))
+             ? Status::NoSpace(context, errnoStr(err_number).c_str())
             : (err_number == ENOENT)
-                   ? Status::PathNotFound(context, strerror(err_number))
-                   : Status::IOError(context, strerror(err_number));
+                   ? Status::PathNotFound(context, errnoStr(err_number).c_str())
+                   : Status::IOError(context, errnoStr(err_number).c_str());
 }

 // assume that there is one global logger for now. It is not thread-safe,
@@ -213,6 +213,8 @@ class HdfsWritableFile: public WritableFile {
    }
  }

+  using WritableFile::Append;
+
  // If the file was successfully created, then this returns true.
  // Otherwise returns false.
  bool isValid() {
@@ -315,7 +315,7 @@ class PosixEnv : public CompositeEnv {
    int ret = gethostname(name, static_cast<size_t>(len));
    if (ret < 0) {
      if (errno == EFAULT || errno == EINVAL) {
-        return Status::InvalidArgument(strerror(errno));
+        return Status::InvalidArgument(errnoStr(errno).c_str());
      } else {
        return IOError("GetHostName", name, errno);
      }
@@ -915,7 +915,7 @@ class IoctlFriendlyTmpdir {
      } else {
        // mkdtemp failed: diagnose it, but don't give up.
        fprintf(stderr, "mkdtemp(%s/...) failed: %s\n", d.c_str(),
-                strerror(errno));
+                errnoStr(errno).c_str());
      }
    }

@@ -1040,7 +1040,8 @@ TEST_P(EnvPosixTestWithParam, AllocateTest) {
    int err_number = 0;
    if (alloc_status != 0) {
      err_number = errno;
-      fprintf(stderr, "Warning: fallocate() fails, %s\n", strerror(err_number));
+      fprintf(stderr, "Warning: fallocate() fails, %s\n",
+              errnoStr(err_number).c_str());
    }
    close(fd);
    ASSERT_OK(env_->DeleteFile(fname_test_fallocate));
@@ -83,6 +83,14 @@ FileOptions FileSystem::OptimizeForCompactionTableRead(
  return optimized_file_options;
 }

+FileOptions FileSystem::OptimizeForBlobFileRead(
+    const FileOptions& file_options,
+    const ImmutableDBOptions& db_options) const {
+  FileOptions optimized_file_options(file_options);
+  optimized_file_options.use_direct_reads = db_options.use_direct_reads;
+  return optimized_file_options;
+}
+
 IOStatus WriteStringToFile(FileSystem* fs, const Slice& data,
                           const std::string& fname, bool should_sync) {
  std::unique_ptr<FSWritableFile> file;
@@ -20,7 +20,7 @@ IOStatus FileSystemTracingWrapper::NewSequentialFile(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          fname.substr(fname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -34,7 +34,7 @@ IOStatus FileSystemTracingWrapper::NewRandomAccessFile(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          fname.substr(fname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -48,7 +48,7 @@ IOStatus FileSystemTracingWrapper::NewWritableFile(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          fname.substr(fname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -62,7 +62,7 @@ IOStatus FileSystemTracingWrapper::ReopenWritableFile(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          fname.substr(fname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -78,7 +78,7 @@ IOStatus FileSystemTracingWrapper::ReuseWritableFile(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          fname.substr(fname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -92,7 +92,7 @@ IOStatus FileSystemTracingWrapper::NewRandomRWFile(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          fname.substr(fname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -106,7 +106,7 @@ IOStatus FileSystemTracingWrapper::NewDirectory(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          name.substr(name.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -121,7 +121,7 @@ IOStatus FileSystemTracingWrapper::GetChildren(const std::string& dir,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          dir.substr(dir.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -135,7 +135,7 @@ IOStatus FileSystemTracingWrapper::DeleteFile(const std::string& fname,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          fname.substr(fname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -149,7 +149,7 @@ IOStatus FileSystemTracingWrapper::CreateDir(const std::string& dirname,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          dirname.substr(dirname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -162,7 +162,7 @@ IOStatus FileSystemTracingWrapper::CreateDirIfMissing(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          dirname.substr(dirname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -176,7 +176,7 @@ IOStatus FileSystemTracingWrapper::DeleteDir(const std::string& dirname,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          dirname.substr(dirname.find_last_of("/\\") + 1));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -193,7 +193,7 @@ IOStatus FileSystemTracingWrapper::GetFileSize(const std::string& fname,
  IOTraceRecord io_record(
      clock_->NowNanos(), TraceType::kIOTracer, io_op_data, __func__, elapsed,
      s.ToString(), fname.substr(fname.find_last_of("/\\") + 1), *file_size);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -210,7 +210,7 @@ IOStatus FileSystemTracingWrapper::Truncate(const std::string& fname,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(),
                          fname.substr(fname.find_last_of("/\\") + 1), size);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -227,7 +227,7 @@ IOStatus FSSequentialFileTracingWrapper::Read(size_t n,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_,
                          result->size(), 0 /*Offset*/);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -243,7 +243,7 @@ IOStatus FSSequentialFileTracingWrapper::InvalidateCache(size_t offset,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_, length,
                          offset);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, nullptr /*dbg*/);
  return s;
 }

@@ -261,7 +261,7 @@ IOStatus FSSequentialFileTracingWrapper::PositionedRead(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_,
                          result->size(), offset);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -279,7 +279,7 @@ IOStatus FSRandomAccessFileTracingWrapper::Read(uint64_t offset, size_t n,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_, n,
                          offset);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -299,7 +299,7 @@ IOStatus FSRandomAccessFileTracingWrapper::MultiRead(FSReadRequest* reqs,
    IOTraceRecord io_record(
        clock_->NowNanos(), TraceType::kIOTracer, io_op_data, __func__, latency,
        reqs[i].status.ToString(), file_name_, reqs[i].len, reqs[i].offset);
-    io_tracer_->WriteIOOp(io_record);
+    io_tracer_->WriteIOOp(io_record, dbg);
  }
  return s;
 }
@@ -317,7 +317,7 @@ IOStatus FSRandomAccessFileTracingWrapper::Prefetch(uint64_t offset, size_t n,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_, n,
                          offset);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -333,7 +333,7 @@ IOStatus FSRandomAccessFileTracingWrapper::InvalidateCache(size_t offset,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_, length,
                          static_cast<uint64_t>(offset));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, nullptr /*dbg*/);
  return s;
 }

@@ -349,7 +349,7 @@ IOStatus FSWritableFileTracingWrapper::Append(const Slice& data,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_,
                          data.size(), 0 /*Offset*/);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -366,7 +366,7 @@ IOStatus FSWritableFileTracingWrapper::PositionedAppend(
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_,
                          data.size(), offset);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -382,7 +382,7 @@ IOStatus FSWritableFileTracingWrapper::Truncate(uint64_t size,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_, size,
                          0 /*Offset*/);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -395,7 +395,7 @@ IOStatus FSWritableFileTracingWrapper::Close(const IOOptions& options,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          file_name_);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -409,7 +409,7 @@ uint64_t FSWritableFileTracingWrapper::GetFileSize(const IOOptions& options,
  io_op_data |= (1 << IOTraceOp::kIOFileSize);
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, "OK", file_name_, file_size);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return file_size;
 }

@@ -425,7 +425,7 @@ IOStatus FSWritableFileTracingWrapper::InvalidateCache(size_t offset,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_, length,
                          static_cast<uint64_t>(offset));
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, nullptr /*dbg*/);
  return s;
 }

@@ -442,7 +442,7 @@ IOStatus FSRandomRWFileTracingWrapper::Write(uint64_t offset, const Slice& data,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_,
                          data.size(), offset);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -460,7 +460,7 @@ IOStatus FSRandomRWFileTracingWrapper::Read(uint64_t offset, size_t n,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer, io_op_data,
                          __func__, elapsed, s.ToString(), file_name_, n,
                          offset);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -473,7 +473,7 @@ IOStatus FSRandomRWFileTracingWrapper::Flush(const IOOptions& options,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          file_name_);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -486,7 +486,7 @@ IOStatus FSRandomRWFileTracingWrapper::Close(const IOOptions& options,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          file_name_);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -499,7 +499,7 @@ IOStatus FSRandomRWFileTracingWrapper::Sync(const IOOptions& options,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          file_name_);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }

@@ -512,7 +512,7 @@ IOStatus FSRandomRWFileTracingWrapper::Fsync(const IOOptions& options,
  IOTraceRecord io_record(clock_->NowNanos(), TraceType::kIOTracer,
                          0 /*io_op_data*/, __func__, elapsed, s.ToString(),
                          file_name_);
-  io_tracer_->WriteIOOp(io_record);
+  io_tracer_->WriteIOOp(io_record, dbg);
  return s;
 }
 }  // namespace ROCKSDB_NAMESPACE
@@ -553,24 +553,35 @@ class PosixFileSystem : public FileSystem {
  IOStatus NewLogger(const std::string& fname, const IOOptions& /*opts*/,
                   std::shared_ptr<Logger>* result,
                   IODebugContext* /*dbg*/) override {
-    FILE* f;
+    FILE* f = nullptr;
+    int fd;
    {
      IOSTATS_TIMER_GUARD(open_nanos);
-      f = fopen(fname.c_str(),
-                "w"
+      fd = open(fname.c_str(),
+                cloexec_flags(O_WRONLY | O_CREAT | O_TRUNC, nullptr),
+                GetDBFileMode(allow_non_owner_access_));
+      if (fd != -1) {
+        f = fdopen(fd,
+                   "w"
 #ifdef __GLIBC_PREREQ
 #if __GLIBC_PREREQ(2, 7)
-                "e"  // glibc extension to enable O_CLOEXEC
+                   "e"  // glibc extension to enable O_CLOEXEC
 #endif
 #endif
-      );
+        );
+      }
    }
-    if (f == nullptr) {
+    if (fd == -1) {
      result->reset();
      return status_to_io_status(
-              IOError("when fopen a file for new logger", fname, errno));
+          IOError("when open a file for new logger", fname, errno));
+    }
+    if (f == nullptr) {
+      close(fd);
+      result->reset();
+      return status_to_io_status(
+          IOError("when fdopen a file for new logger", fname, errno));
    } else {
-      int fd = fileno(f);
 #ifdef ROCKSDB_FALLOCATE_PRESENT
      fallocate(fd, FALLOC_FL_KEEP_SIZE, 0, 4 * 1024);
 #endif
@@ -620,9 +631,10 @@ class PosixFileSystem : public FileSystem {
      }
    }

-    const auto pre_read_errno = errno;  // errno may be modified by readdir
+    // reset errno before calling readdir()
+    errno = 0;
    struct dirent* entry;
-    while ((entry = readdir(d)) != nullptr && errno == pre_read_errno) {
+    while ((entry = readdir(d)) != nullptr) {
      // filter out '.' and '..' directory entries
      // which appear only on some platforms
      const bool ignore =
@@ -631,19 +643,20 @@ class PosixFileSystem : public FileSystem {
      if (!ignore) {
        result->push_back(entry->d_name);
      }
+      errno = 0;  // reset errno if readdir() success
    }

    // always attempt to close the dir
    const auto pre_close_errno = errno;  // errno may be modified by closedir
    const int close_result = closedir(d);

-    if (pre_close_errno != pre_read_errno) {
-      // error occured during readdir
+    if (pre_close_errno != 0) {
+      // error occurred during readdir
      return IOError("While readdir", dir, pre_close_errno);
    }

    if (close_result != 0) {
-      // error occured during closedir
+      // error occurred during closedir
      return IOError("While closedir", dir, errno);
    }

@@ -862,7 +875,7 @@ class PosixFileSystem : public FileSystem {
    char the_path[256];
    char* ret = getcwd(the_path, 256);
    if (ret == nullptr) {
-      return IOStatus::IOError(strerror(errno));
+      return IOStatus::IOError(errnoStr(errno).c_str());
    }

    *output_path = ret;
@@ -0,0 +1,104 @@
+//  Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
+//  This source code is licensed under both the GPLv2 (found in the
+//  COPYING file in the root directory) and Apache 2.0 License
+//  (found in the LICENSE.Apache file in the root directory).
+
+#pragma once
+
+#ifndef ROCKSDB_LITE
+
+#include "rocksdb/file_system.h"
+
+namespace ROCKSDB_NAMESPACE {
+
+// A FileSystem wrapper that only allows read-only operation.
+//
+// This class has not been fully analyzed for providing strong security
+// guarantees.
+class ReadOnlyFileSystem : public FileSystemWrapper {
+  static inline IOStatus FailReadOnly() {
+    IOStatus s = IOStatus::IOError("Attempted write to ReadOnlyFileSystem");
+    assert(s.GetRetryable() == false);
+    return s;
+  }
+
+ public:
+  explicit ReadOnlyFileSystem(const std::shared_ptr<FileSystem>& base)
+      : FileSystemWrapper(base) {}
+
+  IOStatus NewWritableFile(const std::string& /*fname*/,
+                           const FileOptions& /*options*/,
+                           std::unique_ptr<FSWritableFile>* /*result*/,
+                           IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus ReuseWritableFile(const std::string& /*fname*/,
+                             const std::string& /*old_fname*/,
+                             const FileOptions& /*options*/,
+                             std::unique_ptr<FSWritableFile>* /*result*/,
+                             IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus NewRandomRWFile(const std::string& /*fname*/,
+                           const FileOptions& /*options*/,
+                           std::unique_ptr<FSRandomRWFile>* /*result*/,
+                           IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus NewDirectory(const std::string& /*dir*/,
+                        const IOOptions& /*options*/,
+                        std::unique_ptr<FSDirectory>* /*result*/,
+                        IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus DeleteFile(const std::string& /*fname*/,
+                      const IOOptions& /*options*/,
+                      IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus CreateDir(const std::string& /*dirname*/,
+                     const IOOptions& /*options*/,
+                     IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus CreateDirIfMissing(const std::string& dirname,
+                              const IOOptions& options,
+                              IODebugContext* dbg) override {
+    // Allow if dir already exists
+    bool is_dir = false;
+    IOStatus s = IsDirectory(dirname, options, &is_dir, dbg);
+    if (s.ok() && is_dir) {
+      return s;
+    } else {
+      return FailReadOnly();
+    }
+  }
+  IOStatus DeleteDir(const std::string& /*dirname*/,
+                     const IOOptions& /*options*/,
+                     IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus RenameFile(const std::string& /*src*/, const std::string& /*dest*/,
+                      const IOOptions& /*options*/,
+                      IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus LinkFile(const std::string& /*src*/, const std::string& /*dest*/,
+                    const IOOptions& /*options*/,
+                    IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus LockFile(const std::string& /*fname*/, const IOOptions& /*options*/,
+                    FileLock** /*lock*/, IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+  IOStatus NewLogger(const std::string& /*fname*/, const IOOptions& /*options*/,
+                     std::shared_ptr<Logger>* /*result*/,
+                     IODebugContext* /*dbg*/) override {
+    return FailReadOnly();
+  }
+};
+
+}  // namespace ROCKSDB_NAMESPACE
+
+#endif  // ROCKSDB_LITE
@@ -0,0 +1,306 @@
+//  Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
+//  This source code is licensed under both the GPLv2 (found in the
+//  COPYING file in the root directory) and Apache 2.0 License
+//  (found in the LICENSE.Apache file in the root directory).
+
+#ifndef ROCKSDB_LITE
+
+#include "env/fs_remap.h"
+
+namespace ROCKSDB_NAMESPACE {
+
+RemapFileSystem::RemapFileSystem(const std::shared_ptr<FileSystem>& base)
+    : FileSystemWrapper(base) {}
+
+std::pair<IOStatus, std::string> RemapFileSystem::EncodePathWithNewBasename(
+    const std::string& path) {
+  // No difference by default
+  return EncodePath(path);
+}
+
+Status RemapFileSystem::RegisterDbPaths(const std::vector<std::string>& paths) {
+  std::vector<std::string> encoded_paths;
+  encoded_paths.reserve(paths.size());
+  for (auto& path : paths) {
+    auto status_and_enc_path = EncodePathWithNewBasename(path);
+    if (!status_and_enc_path.first.ok()) {
+      return status_and_enc_path.first;
+    }
+    encoded_paths.emplace_back(status_and_enc_path.second);
+  }
+  return FileSystemWrapper::RegisterDbPaths(encoded_paths);
+}
+
+Status RemapFileSystem::UnregisterDbPaths(
+    const std::vector<std::string>& paths) {
+  std::vector<std::string> encoded_paths;
+  encoded_paths.reserve(paths.size());
+  for (auto& path : paths) {
+    auto status_and_enc_path = EncodePathWithNewBasename(path);
+    if (!status_and_enc_path.first.ok()) {
+      return status_and_enc_path.first;
+    }
+    encoded_paths.emplace_back(status_and_enc_path.second);
+  }
+  return FileSystemWrapper::UnregisterDbPaths(encoded_paths);
+}
+
+IOStatus RemapFileSystem::NewSequentialFile(
+    const std::string& fname, const FileOptions& options,
+    std::unique_ptr<FSSequentialFile>* result, IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::NewSequentialFile(status_and_enc_path.second,
+                                              options, result, dbg);
+}
+
+IOStatus RemapFileSystem::NewRandomAccessFile(
+    const std::string& fname, const FileOptions& options,
+    std::unique_ptr<FSRandomAccessFile>* result, IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::NewRandomAccessFile(status_and_enc_path.second,
+                                                options, result, dbg);
+}
+
+IOStatus RemapFileSystem::NewWritableFile(
+    const std::string& fname, const FileOptions& options,
+    std::unique_ptr<FSWritableFile>* result, IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::NewWritableFile(status_and_enc_path.second, options,
+                                            result, dbg);
+}
+
+IOStatus RemapFileSystem::ReuseWritableFile(
+    const std::string& fname, const std::string& old_fname,
+    const FileOptions& options, std::unique_ptr<FSWritableFile>* result,
+    IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  auto status_and_old_enc_path = EncodePath(old_fname);
+  if (!status_and_old_enc_path.first.ok()) {
+    return status_and_old_enc_path.first;
+  }
+  return FileSystemWrapper::ReuseWritableFile(status_and_old_enc_path.second,
+                                              status_and_old_enc_path.second,
+                                              options, result, dbg);
+}
+
+IOStatus RemapFileSystem::NewRandomRWFile(
+    const std::string& fname, const FileOptions& options,
+    std::unique_ptr<FSRandomRWFile>* result, IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::NewRandomRWFile(status_and_enc_path.second, options,
+                                            result, dbg);
+}
+
+IOStatus RemapFileSystem::NewDirectory(const std::string& dir,
+                                       const IOOptions& options,
+                                       std::unique_ptr<FSDirectory>* result,
+                                       IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(dir);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::NewDirectory(status_and_enc_path.second, options,
+                                         result, dbg);
+}
+
+IOStatus RemapFileSystem::FileExists(const std::string& fname,
+                                     const IOOptions& options,
+                                     IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::FileExists(status_and_enc_path.second, options,
+                                       dbg);
+}
+
+IOStatus RemapFileSystem::GetChildren(const std::string& dir,
+                                      const IOOptions& options,
+                                      std::vector<std::string>* result,
+                                      IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePath(dir);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::GetChildren(status_and_enc_path.second, options,
+                                        result, dbg);
+}
+
+IOStatus RemapFileSystem::GetChildrenFileAttributes(
+    const std::string& dir, const IOOptions& options,
+    std::vector<FileAttributes>* result, IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePath(dir);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::GetChildrenFileAttributes(
+      status_and_enc_path.second, options, result, dbg);
+}
+
+IOStatus RemapFileSystem::DeleteFile(const std::string& fname,
+                                     const IOOptions& options,
+                                     IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePath(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::DeleteFile(status_and_enc_path.second, options,
+                                       dbg);
+}
+
+IOStatus RemapFileSystem::CreateDir(const std::string& dirname,
+                                    const IOOptions& options,
+                                    IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(dirname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::CreateDir(status_and_enc_path.second, options, dbg);
+}
+
+IOStatus RemapFileSystem::CreateDirIfMissing(const std::string& dirname,
+                                             const IOOptions& options,
+                                             IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(dirname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::CreateDirIfMissing(status_and_enc_path.second,
+                                               options, dbg);
+}
+
+IOStatus RemapFileSystem::DeleteDir(const std::string& dirname,
+                                    const IOOptions& options,
+                                    IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePath(dirname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::DeleteDir(status_and_enc_path.second, options, dbg);
+}
+
+IOStatus RemapFileSystem::GetFileSize(const std::string& fname,
+                                      const IOOptions& options,
+                                      uint64_t* file_size,
+                                      IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePath(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::GetFileSize(status_and_enc_path.second, options,
+                                        file_size, dbg);
+}
+
+IOStatus RemapFileSystem::GetFileModificationTime(const std::string& fname,
+                                                  const IOOptions& options,
+                                                  uint64_t* file_mtime,
+                                                  IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePath(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::GetFileModificationTime(status_and_enc_path.second,
+                                                    options, file_mtime, dbg);
+}
+
+IOStatus RemapFileSystem::IsDirectory(const std::string& path,
+                                      const IOOptions& options, bool* is_dir,
+                                      IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePath(path);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::IsDirectory(status_and_enc_path.second, options,
+                                        is_dir, dbg);
+}
+
+IOStatus RemapFileSystem::RenameFile(const std::string& src,
+                                     const std::string& dest,
+                                     const IOOptions& options,
+                                     IODebugContext* dbg) {
+  auto status_and_src_enc_path = EncodePath(src);
+  if (!status_and_src_enc_path.first.ok()) {
+    return status_and_src_enc_path.first;
+  }
+  auto status_and_dest_enc_path = EncodePathWithNewBasename(dest);
+  if (!status_and_dest_enc_path.first.ok()) {
+    return status_and_dest_enc_path.first;
+  }
+  return FileSystemWrapper::RenameFile(status_and_src_enc_path.second,
+                                       status_and_dest_enc_path.second, options,
+                                       dbg);
+}
+
+IOStatus RemapFileSystem::LinkFile(const std::string& src,
+                                   const std::string& dest,
+                                   const IOOptions& options,
+                                   IODebugContext* dbg) {
+  auto status_and_src_enc_path = EncodePath(src);
+  if (!status_and_src_enc_path.first.ok()) {
+    return status_and_src_enc_path.first;
+  }
+  auto status_and_dest_enc_path = EncodePathWithNewBasename(dest);
+  if (!status_and_dest_enc_path.first.ok()) {
+    return status_and_dest_enc_path.first;
+  }
+  return FileSystemWrapper::LinkFile(status_and_src_enc_path.second,
+                                     status_and_dest_enc_path.second, options,
+                                     dbg);
+}
+
+IOStatus RemapFileSystem::LockFile(const std::string& fname,
+                                   const IOOptions& options, FileLock** lock,
+                                   IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  // FileLock subclasses may store path (e.g., PosixFileLock stores it). We
+  // can skip stripping the chroot directory from this path because callers
+  // shouldn't use it.
+  return FileSystemWrapper::LockFile(status_and_enc_path.second, options, lock,
+                                     dbg);
+}
+
+IOStatus RemapFileSystem::NewLogger(const std::string& fname,
+                                    const IOOptions& options,
+                                    std::shared_ptr<Logger>* result,
+                                    IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePathWithNewBasename(fname);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::NewLogger(status_and_enc_path.second, options,
+                                      result, dbg);
+}
+
+IOStatus RemapFileSystem::GetAbsolutePath(const std::string& db_path,
+                                          const IOOptions& options,
+                                          std::string* output_path,
+                                          IODebugContext* dbg) {
+  auto status_and_enc_path = EncodePath(db_path);
+  if (!status_and_enc_path.first.ok()) {
+    return status_and_enc_path.first;
+  }
+  return FileSystemWrapper::GetAbsolutePath(status_and_enc_path.second, options,
+                                            output_path, dbg);
+}
+
+}  // namespace ROCKSDB_NAMESPACE
+
+#endif  // ROCKSDB_LITE
@@ -0,0 +1,131 @@
+//  Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
+//  This source code is licensed under both the GPLv2 (found in the
+//  COPYING file in the root directory) and Apache 2.0 License
+//  (found in the LICENSE.Apache file in the root directory).
+
+#pragma once
+
+#ifndef ROCKSDB_LITE
+
+#include <utility>
+
+#include "rocksdb/file_system.h"
+
+namespace ROCKSDB_NAMESPACE {
+
+// An abstract FileSystem wrapper that creates a view of an existing
+// FileSystem by remapping names in some way.
+//
+// This class has not been fully analyzed for providing strong security
+// guarantees.
+class RemapFileSystem : public FileSystemWrapper {
+ public:
+  explicit RemapFileSystem(const std::shared_ptr<FileSystem>& base);
+
+ protected:
+  // Returns status and mapped-to path in the wrapped filesystem.
+  // If it returns non-OK status, the returned path should not be used.
+  virtual std::pair<IOStatus, std::string> EncodePath(
+      const std::string& path) = 0;
+
+  // Similar to EncodePath() except used in cases in which it is OK for
+  // no file or directory on 'path' to already exist, such as if the
+  // operation would create one. However, the parent of 'path' is expected
+  // to exist for the operation to succeed.
+  // Default implementation: call EncodePath
+  virtual std::pair<IOStatus, std::string> EncodePathWithNewBasename(
+      const std::string& path);
+
+ public:
+  // Left abstract:
+  // const char* Name() const override { ... }
+
+  Status RegisterDbPaths(const std::vector<std::string>& paths) override;
+
+  Status UnregisterDbPaths(const std::vector<std::string>& paths) override;
+
+  IOStatus NewSequentialFile(const std::string& fname,
+                             const FileOptions& options,
+                             std::unique_ptr<FSSequentialFile>* result,
+                             IODebugContext* dbg) override;
+
+  IOStatus NewRandomAccessFile(const std::string& fname,
+                               const FileOptions& options,
+                               std::unique_ptr<FSRandomAccessFile>* result,
+                               IODebugContext* dbg) override;
+
+  IOStatus NewWritableFile(const std::string& fname, const FileOptions& options,
+                           std::unique_ptr<FSWritableFile>* result,
+                           IODebugContext* dbg) override;
+
+  IOStatus ReuseWritableFile(const std::string& fname,
+                             const std::string& old_fname,
+                             const FileOptions& options,
+                             std::unique_ptr<FSWritableFile>* result,
+                             IODebugContext* dbg) override;
+
+  IOStatus NewRandomRWFile(const std::string& fname, const FileOptions& options,
+                           std::unique_ptr<FSRandomRWFile>* result,
+                           IODebugContext* dbg) override;
+
+  IOStatus NewDirectory(const std::string& dir, const IOOptions& options,
+                        std::unique_ptr<FSDirectory>* result,
+                        IODebugContext* dbg) override;
+
+  IOStatus FileExists(const std::string& fname, const IOOptions& options,
+                      IODebugContext* dbg) override;
+
+  IOStatus GetChildren(const std::string& dir, const IOOptions& options,
+                       std::vector<std::string>* result,
+                       IODebugContext* dbg) override;
+
+  IOStatus GetChildrenFileAttributes(const std::string& dir,
+                                     const IOOptions& options,
+                                     std::vector<FileAttributes>* result,
+                                     IODebugContext* dbg) override;
+
+  IOStatus DeleteFile(const std::string& fname, const IOOptions& options,
+                      IODebugContext* dbg) override;
+
+  IOStatus CreateDir(const std::string& dirname, const IOOptions& options,
+                     IODebugContext* dbg) override;
+
+  IOStatus CreateDirIfMissing(const std::string& dirname,
+                              const IOOptions& options,
+                              IODebugContext* dbg) override;
+
+  IOStatus DeleteDir(const std::string& dirname, const IOOptions& options,
+                     IODebugContext* dbg) override;
+
+  IOStatus GetFileSize(const std::string& fname, const IOOptions& options,
+                       uint64_t* file_size, IODebugContext* dbg) override;
+
+  IOStatus GetFileModificationTime(const std::string& fname,
+                                   const IOOptions& options,
+                                   uint64_t* file_mtime,
+                                   IODebugContext* dbg) override;
+
+  IOStatus IsDirectory(const std::string& path, const IOOptions& options,
+                       bool* is_dir, IODebugContext* dbg) override;
+
+  IOStatus RenameFile(const std::string& src, const std::string& dest,
+                      const IOOptions& options, IODebugContext* dbg) override;
+
+  IOStatus LinkFile(const std::string& src, const std::string& dest,
+                    const IOOptions& options, IODebugContext* dbg) override;
+
+  IOStatus LockFile(const std::string& fname, const IOOptions& options,
+                    FileLock** lock, IODebugContext* dbg) override;
+
+  IOStatus NewLogger(const std::string& fname, const IOOptions& options,
+                     std::shared_ptr<Logger>* result,
+                     IODebugContext* dbg) override;
+
+  IOStatus GetAbsolutePath(const std::string& db_path, const IOOptions& options,
+                           std::string* output_path,
+                           IODebugContext* dbg) override;
+};
+
+}  // namespace ROCKSDB_NAMESPACE
+
+#endif  // ROCKSDB_LITE
@@ -58,7 +58,7 @@ IOStatus IOError(const std::string& context, const std::string& file_name,
  switch (err_number) {
    case ENOSPC: {
      IOStatus s = IOStatus::NoSpace(IOErrorMsg(context, file_name),
-                                     strerror(err_number));
+                                     errnoStr(err_number).c_str());
      s.SetRetryable(true);
      return s;
    }
@@ -66,10 +66,10 @@ IOStatus IOError(const std::string& context, const std::string& file_name,
      return IOStatus::IOError(IOStatus::kStaleFile);
    case ENOENT:
      return IOStatus::PathNotFound(IOErrorMsg(context, file_name),
-                                    strerror(err_number));
+                                    errnoStr(err_number).c_str());
    default:
      return IOStatus::IOError(IOErrorMsg(context, file_name),
-                               strerror(err_number));
+                               errnoStr(err_number).c_str());
  }
 }

@@ -927,7 +927,7 @@ IOStatus PosixMmapFile::MapNewRegion() {
    }
    if (alloc_status != 0) {
      return IOStatus::IOError("Error allocating space to file : " + filename_ +
-                               "Error : " + strerror(alloc_status));
+                               "Error : " + errnoStr(alloc_status).c_str());
    }
  }

@@ -1213,6 +1213,7 @@ IOStatus PosixWritableFile::Close(const IOOptions& /*opts*/,
  size_t block_size;
  size_t last_allocated_block;
  GetPreallocationStatus(&block_size, &last_allocated_block);
+  TEST_SYNC_POINT_CALLBACK("PosixWritableFile::Close", &last_allocated_block);
  if (last_allocated_block > 0) {
    // trim the extra space preallocated at the end of the file
    // NOTE(ljin): we probably don't want to surface failure as an IOError,
@@ -724,11 +724,12 @@ IOStatus MockFileSystem::ReopenWritableFile(
  MemFile* file = nullptr;
  if (file_map_.find(fn) == file_map_.end()) {
    file = new MemFile(env_, fn, false);
+    // Only take a reference when we create the file objectt
+    file->Ref();
    file_map_[fn] = file;
  } else {
    file = file_map_[fn];
  }
-  file->Ref();
  if (file_opts.use_direct_writes && !supports_direct_io_) {
    return IOStatus::NotSupported("Direct I/O Not Supported");
  } else {
@@ -22,26 +22,26 @@
 #include "util/rate_limiter.h"

 namespace ROCKSDB_NAMESPACE {
-Status RandomAccessFileReader::Create(
+IOStatus RandomAccessFileReader::Create(
    const std::shared_ptr<FileSystem>& fs, const std::string& fname,
    const FileOptions& file_opts,
    std::unique_ptr<RandomAccessFileReader>* reader, IODebugContext* dbg) {
  std::unique_ptr<FSRandomAccessFile> file;
-  Status s = fs->NewRandomAccessFile(fname, file_opts, &file, dbg);
-  if (s.ok()) {
+  IOStatus io_s = fs->NewRandomAccessFile(fname, file_opts, &file, dbg);
+  if (io_s.ok()) {
    reader->reset(new RandomAccessFileReader(std::move(file), fname));
  }
-  return s;
+  return io_s;
 }

-Status RandomAccessFileReader::Read(const IOOptions& opts, uint64_t offset,
-                                    size_t n, Slice* result, char* scratch,
-                                    AlignedBuf* aligned_buf,
-                                    bool for_compaction) const {
+IOStatus RandomAccessFileReader::Read(const IOOptions& opts, uint64_t offset,
+                                      size_t n, Slice* result, char* scratch,
+                                      AlignedBuf* aligned_buf,
+                                      bool for_compaction) const {
  (void)aligned_buf;

  TEST_SYNC_POINT_CALLBACK("RandomAccessFileReader::Read", nullptr);
-  Status s;
+  IOStatus io_s;
  uint64_t elapsed = 0;
  {
    StopWatch sw(clock_, stats_, hist_type_,
@@ -86,22 +86,22 @@ Status RandomAccessFileReader::Read(const IOOptions& opts, uint64_t offset,
          // one iteration of this loop, so we don't need to check and adjust
          // the opts.timeout before calling file_->Read
          assert(!opts.timeout.count() || allowed == read_size);
-          s = file_->Read(aligned_offset + buf.CurrentSize(), allowed, opts,
-                          &tmp, buf.Destination(), nullptr);
+          io_s = file_->Read(aligned_offset + buf.CurrentSize(), allowed, opts,
+                             &tmp, buf.Destination(), nullptr);
        }
        if (ShouldNotifyListeners()) {
          auto finish_ts = FileOperationInfo::FinishNow();
          NotifyOnFileReadFinish(orig_offset, tmp.size(), start_ts, finish_ts,
-                                 s);
+                                 io_s);
        }

        buf.Size(buf.CurrentSize() + tmp.size());
-        if (!s.ok() || tmp.size() < allowed) {
+        if (!io_s.ok() || tmp.size() < allowed) {
          break;
        }
      }
      size_t res_len = 0;
-      if (s.ok() && offset_advance < buf.CurrentSize()) {
+      if (io_s.ok() && offset_advance < buf.CurrentSize()) {
        res_len = std::min(buf.CurrentSize() - offset_advance, n);
        if (aligned_buf == nullptr) {
          buf.Read(scratch, offset_advance, res_len);
@@ -146,14 +146,14 @@ Status RandomAccessFileReader::Read(const IOOptions& opts, uint64_t offset,
          // one iteration of this loop, so we don't need to check and adjust
          // the opts.timeout before calling file_->Read
          assert(!opts.timeout.count() || allowed == n);
-          s = file_->Read(offset + pos, allowed, opts, &tmp_result,
-                          scratch + pos, nullptr);
+          io_s = file_->Read(offset + pos, allowed, opts, &tmp_result,
+                             scratch + pos, nullptr);
        }
 #ifndef ROCKSDB_LITE
        if (ShouldNotifyListeners()) {
          auto finish_ts = FileOperationInfo::FinishNow();
          NotifyOnFileReadFinish(offset + pos, tmp_result.size(), start_ts,
-                                 finish_ts, s);
+                                 finish_ts, io_s);
        }
 #endif

@@ -166,11 +166,11 @@ Status RandomAccessFileReader::Read(const IOOptions& opts, uint64_t offset,
          assert(tmp_result.data() == res_scratch + pos);
        }
        pos += tmp_result.size();
-        if (!s.ok() || tmp_result.size() < allowed) {
+        if (!io_s.ok() || tmp_result.size() < allowed) {
          break;
        }
      }
-      *result = Slice(res_scratch, s.ok() ? pos : 0);
+      *result = Slice(res_scratch, io_s.ok() ? pos : 0);
    }
    IOSTATS_ADD_IF_POSITIVE(bytes_read, result->size());
    SetPerfLevel(prev_perf_level);
@@ -179,7 +179,7 @@ Status RandomAccessFileReader::Read(const IOOptions& opts, uint64_t offset,
    file_read_hist_->Add(elapsed);
  }

-  return s;
+  return io_s;
 }

 size_t End(const FSReadRequest& r) {
@@ -208,13 +208,13 @@ bool TryMerge(FSReadRequest* dest, const FSReadRequest& src) {
  return true;
 }

-Status RandomAccessFileReader::MultiRead(const IOOptions& opts,
-                                         FSReadRequest* read_reqs,
-                                         size_t num_reqs,
-                                         AlignedBuf* aligned_buf) const {
+IOStatus RandomAccessFileReader::MultiRead(const IOOptions& opts,
+                                           FSReadRequest* read_reqs,
+                                           size_t num_reqs,
+                                           AlignedBuf* aligned_buf) const {
  (void)aligned_buf;  // suppress warning of unused variable in LITE mode
  assert(num_reqs > 0);
-  Status s;
+  IOStatus io_s;
  uint64_t elapsed = 0;
  {
    StopWatch sw(clock_, stats_, hist_type_,
@@ -280,7 +280,7 @@ Status RandomAccessFileReader::MultiRead(const IOOptions& opts,

    {
      IOSTATS_CPU_TIMER_GUARD(cpu_read_nanos, clock_);
-      s = file_->MultiRead(fs_reqs, num_fs_reqs, opts, nullptr);
+      io_s = file_->MultiRead(fs_reqs, num_fs_reqs, opts, nullptr);
    }

 #ifndef ROCKSDB_LITE
@@ -321,7 +321,7 @@ Status RandomAccessFileReader::MultiRead(const IOOptions& opts,
    file_read_hist_->Add(elapsed);
  }

-  return s;
+  return io_s;
 }

 IOStatus RandomAccessFileReader::PrepareIOOptions(const ReadOptions& ro,
@@ -38,7 +38,7 @@ FSReadRequest Align(const FSReadRequest& r, size_t alignment);
 // Otherwise, do nothing and return false.
 bool TryMerge(FSReadRequest* dest, const FSReadRequest& src);

-// RandomAccessFileReader is a wrapper on top of Env::RnadomAccessFile. It is
+// RandomAccessFileReader is a wrapper on top of Env::RandomAccessFile. It is
 // responsible for:
 // - Handling Buffered and Direct reads appropriately.
 // - Rate limiting compaction reads.
@@ -103,10 +103,10 @@ class RandomAccessFileReader {
 #endif
  }

-  static Status Create(const std::shared_ptr<FileSystem>& fs,
-                       const std::string& fname, const FileOptions& file_opts,
-                       std::unique_ptr<RandomAccessFileReader>* reader,
-                       IODebugContext* dbg);
+  static IOStatus Create(const std::shared_ptr<FileSystem>& fs,
+                         const std::string& fname, const FileOptions& file_opts,
+                         std::unique_ptr<RandomAccessFileReader>* reader,
+                         IODebugContext* dbg);
  RandomAccessFileReader(const RandomAccessFileReader&) = delete;
  RandomAccessFileReader& operator=(const RandomAccessFileReader&) = delete;

@@ -120,19 +120,19 @@ class RandomAccessFileReader {
  // 2. Otherwise, scratch is not used and can be null, the aligned_buf owns
  // the internally allocated buffer on return, and the result refers to a
  // region in aligned_buf.
-  Status Read(const IOOptions& opts, uint64_t offset, size_t n, Slice* result,
-              char* scratch, AlignedBuf* aligned_buf,
-              bool for_compaction = false) const;
+  IOStatus Read(const IOOptions& opts, uint64_t offset, size_t n, Slice* result,
+                char* scratch, AlignedBuf* aligned_buf,
+                bool for_compaction = false) const;

  // REQUIRES:
  // num_reqs > 0, reqs do not overlap, and offsets in reqs are increasing.
  // In non-direct IO mode, aligned_buf should be null;
  // In direct IO mode, aligned_buf stores the aligned buffer allocated inside
  // MultiRead, the result Slices in reqs refer to aligned_buf.
-  Status MultiRead(const IOOptions& opts, FSReadRequest* reqs, size_t num_reqs,
-                   AlignedBuf* aligned_buf) const;
+  IOStatus MultiRead(const IOOptions& opts, FSReadRequest* reqs,
+                     size_t num_reqs, AlignedBuf* aligned_buf) const;

-  Status Prefetch(uint64_t offset, size_t n) const {
+  IOStatus Prefetch(uint64_t offset, size_t n) const {
    return file_->Prefetch(offset, n, IOOptions(), nullptr);
  }

@@ -38,12 +38,12 @@ class RandomAccessFileReaderTest : public testing::Test {
  }

  void Read(const std::string& fname, const FileOptions& opts,
-                std::unique_ptr<RandomAccessFileReader>* reader) {
+            std::unique_ptr<RandomAccessFileReader>* reader) {
    std::string fpath = Path(fname);
    std::unique_ptr<FSRandomAccessFile> f;
    ASSERT_OK(fs_->NewRandomAccessFile(fpath, opts, &f, nullptr));
-    (*reader).reset(new RandomAccessFileReader(std::move(f), fpath,
-                                               env_->GetSystemClock().get()));
+    reader->reset(new RandomAccessFileReader(std::move(f), fpath,
+                                             env_->GetSystemClock().get()));
  }

  void AssertResult(const std::string& content,
@@ -705,7 +705,7 @@ struct AdvancedColumnFamilyOptions {
  // updated from the file system.
  // Pre-req: This needs max_open_files to be set to -1.
  // In Level: Non-bottom-level files older than TTL will go through the
-  //           compation process.
+  //           compaction process.
  // In FIFO: Files older than TTL will be deleted.
  // unit: seconds. Ex: 1 day = 1 * 24 * 60 * 60
  // In FIFO, this option will have the same meaning as
@@ -151,7 +151,7 @@ class Cache {
  //   - Name-value option pairs -- "capacity=1M; num_shard_bits=4;
  //     For the LRUCache, the values are defined in LRUCacheOptions.
  // @param result The new Cache object
-  // @return OK if the cache was sucessfully created
+  // @return OK if the cache was successfully created
  // @return NotFound if an invalid name was specified in the value
  // @return InvalidArgument if either the options were not valid
  static Status CreateFromString(const ConfigOptions& config_options,
@@ -33,7 +33,7 @@ class ConcurrentTaskLimiter {
  virtual int32_t GetOutstandingTask() const = 0;
 };

-// Create a ConcurrentTaskLimiter that can be shared with mulitple CFs
+// Create a ConcurrentTaskLimiter that can be shared with multiple CFs
 // across RocksDB instances to control concurrent tasks.
 //
 // @param name: Name of the limiter.
@@ -28,7 +28,7 @@ struct DBOptions;
 // standard way of configuring objects.  A Configurable object can:
 //   -> Populate itself given:
 //        - One or more "name/value" pair strings
-//        - A string repesenting the set of name=value properties
+//        - A string representing the set of name=value properties
 //        - A map of name/value properties.
 //   -> Convert itself into its string representation
 //   -> Dump itself to a Logger
@@ -166,7 +166,7 @@ class Configurable {
  // This is the inverse of ConfigureFromString.
  // @param config_options Controls how serialization happens.
  // @param result The string representation of this object.
-  // @return OK If the options for this object wer successfully serialized.
+  // @return OK If the options for this object were successfully serialized.
  // @return InvalidArgument If one or more of the options could not be
  // serialized.
  Status GetOptionString(const ConfigOptions& config_options,
@@ -276,7 +276,7 @@ class Configurable {
  // Classes may override this method to provide further specialization (such as
  // returning a sub-option)
  //
-  // The default implemntation looks at the registered options.  If the
+  // The default implementation looks at the registered options.  If the
  // input name matches that of a registered option, the pointer registered
  // with that name is returned.
  // e.g,, RegisterOptions("X", &my_ptr, ...); GetOptionsPtr("X") returns
@@ -93,7 +93,7 @@ struct ConfigOptions {
 #ifndef ROCKSDB_LITE

 // The following set of functions provide a way to construct RocksDB Options
-// from a string or a string-to-string map.  Here're the general rule of
+// from a string or a string-to-string map.  Here is the general rule of
 // setting option values from strings by type.  Some RocksDB types are also
 // supported in these APIs.  Please refer to the comment of the function itself
 // to find more information about how to config those RocksDB types.
@@ -149,7 +149,7 @@ struct ConfigOptions {
 // ColumnFamilyOptions "new_options".
 //
 // Below are the instructions of how to config some non-primitive-typed
-// options in ColumnFOptions:
+// options in ColumnFamilyOptions:
 //
 // * table_factory:
 //   table_factory can be configured using our custom nested-option syntax.
@@ -191,7 +191,7 @@ struct ConfigOptions {
 //     * {"memtable", "skip_list:5"} is equivalent to setting
 //       memtable to SkipListFactory(5).
 //   - PrefixHash:
-//     Pass "prfix_hash:<hash_bucket_count>" to config memtable
+//     Pass "prefix_hash:<hash_bucket_count>" to config memtable
 //     to use PrefixHash, or simply "prefix_hash" to use the default
 //     PrefixHash.
 //     [Example]:
@@ -112,7 +112,7 @@ struct RangePtr {
 };

 // It is valid that files_checksums and files_checksum_func_names are both
-// empty (no checksum informaiton is provided for ingestion). Otherwise,
+// empty (no checksum information is provided for ingestion). Otherwise,
 // their sizes should be the same as external_files. The file order should
 // be the same in three vectors and guaranteed by the caller.
 struct IngestExternalFileArg {
@@ -205,11 +205,11 @@ class DB {
  // to open the primary instance.
  // The secondary_path argument points to a directory where the secondary
  // instance stores its info log.
-  // The column_families argument specifieds a list of column families to open.
+  // The column_families argument specifies a list of column families to open.
  // If any of the column families does not exist, the function returns non-OK
  // status.
  // The handles is an out-arg corresponding to the opened database column
-  // familiy handles.
+  // family handles.
  // The dbptr is an out-arg corresponding to the opened secondary instance.
  // The pointer points to a heap-allocated database, and the caller should
  // delete it after use. Before deleting the dbptr, the user should also
@@ -745,7 +745,7 @@ class DB {
    static const std::string kCFStats;

    //  "rocksdb.cfstats-no-file-histogram" - returns a multi-line string with
-    //      general columm family stats per-level over db's lifetime ("L<n>"),
+    //      general column family stats per-level over db's lifetime ("L<n>"),
    //      aggregated over db's lifetime ("Sum"), and aggregated over the
    //      interval since the last retrieval ("Int").
    static const std::string kCFStatsNoFileHistogram;
@@ -1025,7 +1025,7 @@ class DB {
                                     uint64_t* sizes) = 0;

  // Simpler versions of the GetApproximateSizes() method above.
-  // The include_flags argumenbt must of type DB::SizeApproximationFlags
+  // The include_flags argument must of type DB::SizeApproximationFlags
  // and can not be NONE.
  virtual Status GetApproximateSizes(ColumnFamilyHandle* column_family,
                                     const Range* ranges, int n,
@@ -1612,7 +1612,7 @@ class DB {
  }

  // IO Tracing operations. Use EndIOTrace() to stop tracing.
-  virtual Status StartIOTrace(Env* /*env*/, const TraceOptions& /*options*/,
+  virtual Status StartIOTrace(const TraceOptions& /*options*/,
                              std::unique_ptr<TraceWriter>&& /*trace_writer*/) {
    return Status::NotSupported("StartIOTrace() is not implemented.");
  }
@@ -437,7 +437,7 @@ class Env {
  virtual Status GetTestDirectory(std::string* path) = 0;

  // Create and returns a default logger (an instance of EnvLogger) for storing
-  // informational messages. Derived classes can overide to provide custom
+  // informational messages. Derived classes can override to provide custom
  // logger.
  virtual Status NewLogger(const std::string& fname,
                           std::shared_ptr<Logger>* result);
@@ -546,6 +546,13 @@ class Env {
      const EnvOptions& env_options,
      const ImmutableDBOptions& db_options) const;

+  // OptimizeForBlobFileRead will create a new EnvOptions object that
+  // is a copy of the EnvOptions in the parameters, but is optimized for reading
+  // blob files.
+  virtual EnvOptions OptimizeForBlobFileRead(
+      const EnvOptions& env_options,
+      const ImmutableDBOptions& db_options) const;
+
  // Returns the status of all threads that belong to the current Env.
  virtual Status GetThreadList(std::vector<ThreadStatus>* /*thread_list*/) {
    return Status::NotSupported("Env::GetThreadList() not supported.");
@@ -798,7 +805,7 @@ class WritableFile {
  virtual ~WritableFile();

  // Append data to the end of the file
-  // Note: A WriteabelFile object must support either Append or
+  // Note: A WriteableFile object must support either Append or
  // PositionedAppend, so the users cannot mix the two.
  virtual Status Append(const Slice& data) = 0;

@@ -1495,6 +1502,11 @@ class EnvWrapper : public Env {
      const ImmutableDBOptions& db_options) const override {
    return target_->OptimizeForCompactionTableRead(env_options, db_options);
  }
+  EnvOptions OptimizeForBlobFileRead(
+      const EnvOptions& env_options,
+      const ImmutableDBOptions& db_options) const override {
+    return target_->OptimizeForBlobFileRead(env_options, db_options);
+  }
  Status GetFreeSpace(const std::string& path, uint64_t* diskfree) override {
    return target_->GetFreeSpace(path, diskfree);
  }
@@ -73,7 +73,7 @@ class BlockCipher {
  //   - ROT13         Create a ROT13 Cipher
  //   - ROT13:nn      Create a ROT13 Cipher with block size of nn
  // @param result The new cipher object
-  // @return OK if the cipher was sucessfully created
+  // @return OK if the cipher was successfully created
  // @return NotFound if an invalid name was specified in the value
  // @return InvalidArgument if either the options were not valid
  static Status CreateFromString(const ConfigOptions& config_options,
@@ -118,7 +118,7 @@ class EncryptionProvider {
  //   - CTR         Create a CTR provider
  //   - test://CTR Create a CTR provider and initialize it for tests.
  // @param result The new provider object
-  // @return OK if the provider was sucessfully created
+  // @return OK if the provider was successfully created
  // @return NotFound if an invalid name was specified in the value
  // @return InvalidArgument if either the options were not valid
  static Status CreateFromString(const ConfigOptions& config_options,
@@ -116,7 +116,7 @@ class FileChecksumList {
 // Create a new file checksum list.
 extern FileChecksumList* NewFileChecksumList();

-// Return a shared_ptr of the builtin Crc32c based file checksum generatory
+// Return a shared_ptr of the builtin Crc32c based file checksum generator
 // factory object, which can be shared to create the Crc32c based checksum
 // generator object.
 // Note: this implementation is compatible with many other crc32c checksum
--- a/Show More
+++ b/Show More