Wide-column blob separation: lazy resolution through read, compaction, and write paths (#14386)

Summary:
Wide-column blob separation: lazy resolution through read, compaction, and write paths

Extend blob direct write to support wide-column entities (PutEntity), and add
lazy blob resolution for wide-column values across all read and compaction paths.

**Write path -- PutEntity blob separation:**
- BlobWriteBatchTransformer::PutEntityCF now extracts large column values
  (>= min_blob_size) to blob files and serializes V2 entities with BlobIndex
  references, matching the existing Put behavior.
- Add MaybePreprocessWideColumns() static helper to share blob extraction
  logic between the WriteBatch transformer and the new PutEntity fast path.
- Add PutEntityFastPath() in DBImpl that preprocesses columns (sort, blob
  extract, serialize) before calling WriteImpl, skipping the redundant
  WriteBatch transformation pass. Trace batch preserves the original columns.

**Read path -- blob resolution for Get/MultiGet/Iterator:**
- GetContext::SaveValue resolves V2 entity blob columns eagerly: for
  value (Get), resolves the default column's blob reference; for columns
  (GetEntity), resolves all blob columns and re-serializes as V1.
- DBIter::SetValueAndColumnsFromEntity detects V2 entities, deserializes
  with DeserializeV2, and eagerly resolves all blob columns via a new
  ReadPathBlobResolver. Resolved values are cached in the resolver and
  wide_columns_ Slices point into the cache, avoiding copies.
- Add ReadPathBlobResolver (new file) -- on-demand blob fetcher for the
  read path with per-column caching, used by both DBIter and GetContext.
- BlobFetcher gains allow_write_path_fallback to read from in-flight
  direct-write blob files not yet visible through Version (pre-flush reads).
- Memtable lookups for Get(key) on V2 entities with a blob default column
  now return the blob index with is_blob_index=true, triggering the
  existing BDW resolution in MaybeResolveWritePathValue.
- MaybeResolveWritePathValue (renamed from MaybeResolveDirectWriteBlobIndex)
  now also resolves V2 entity blob columns for GetEntity/MultiGetEntity,
  re-serializing as V1 after resolution.

**Compaction path -- filter, GC, and extraction:**
- CompactionIterator::InvokeFilterIfNeeded handles V2 entities: FilterV3
  gets eagerly-resolved column values for backward compatibility; FilterV4
  gets a CompactionBlobResolver for lazy on-demand resolution.
- Add CompactionFilter::FilterV4 with WideColumnBlobResolver* parameter
  and SupportsFilterV4() opt-in. Default delegates to FilterV3.
- CompactionBlobResolver (new class) implements WideColumnBlobResolver
  for the compaction path with stats tracking.
- ExtractLargeColumnValuesIfNeeded extracts inline columns to blob files
  during compaction (entities without existing blob columns only).
- GarbageCollectEntityBlobsIfNeeded relocates blob values from old blob
  files to new ones during compaction GC, with helpers FetchBlobsNeedingGC,
  RelocateBlobValues, and SerializeEntityAfterGC.
- PrepareOutput unified entity deserialization: single DeserializeV2 call
  reused by both filter and GC/extraction paths via entity_deserialized_
  flag, avoiding redundant parsing.

**Merge path -- V2 entity base value resolution:**
- MergeHelper::MergeUntil, GetContext::MergeWithWideColumnBaseValue, and
  DBIter::MergeWithWideColumnBaseValue resolve V2 blob columns before
  calling TimedFullMerge, using ResolveEntityForMerge.

**Blob garbage accounting:**
- BlobGarbageMeter tracks blob file in/out flow for V2 entity blob
  columns via ForEachBlobFileNumber, used for accurate GC decisions.
- FileMetaData::UpdateBoundaries tracks oldest_blob_file_number for
  V2 entities, ensuring blob files referenced by entities are not
  prematurely deleted.

**Serialization improvements:**
- WideColumnSerialization::SerializeV2Impl allocates serialized_blob_indices
  only for actual blob columns (not all columns) and uses autovector for
  name/value sizes.
- Add ForEachBlobFileNumber for lightweight blob file number extraction
  without full deserialization.
- Add ResolveEntityForMerge helper for merge-path resolution.
- Add section-size validation in DeserializeV2Impl.
- Add empty blob index and column type validation.
- blob_column_resolver_util.h -- shared helpers (FindBlobColumn, FindInCache,
  CacheInlinedBlob) used by both ReadPathBlobResolver and CompactionBlobResolver.

**Testing:**
- db_blob_direct_write_test: end-to-end PutEntity with BDW before/after flush,
  verifying Get, GetEntity, MultiGetEntity, and Iterator.
- db_blob_index_test: ~1550 lines covering V2 entity blob resolution through
  Get, GetEntity, MultiGet, Iterator, compaction filter (V3 compat and V4 lazy),
  merge with blob base, and compaction GC/extraction.
- compaction_iterator_test: ~950 lines testing entity blob GC, extraction,
  filter interaction, and combined GC+filter scenarios.
- db_wide_basic_test: ~1200 lines for wide-column lazy blob resolution through
  all read paths plus compaction round-trips.
- db_open_with_config_test: ~450 lines for BDW entity config validation.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14386

Reviewed By: anand1976

Differential Revision: D99739701

Pulled By: xingbowang

fbshipit-source-id: 6badd89b577f3054802eaaa654738468efb9dbdb
This commit is contained in:
Xingbo Wang
2026-04-08 18:58:40 -07:00
committed by meta-codesync[bot]
parent 9d609dd6fe
commit 3070f73e97
66 changed files with 10195 additions and 543 deletions
+13
View File
@@ -111,6 +111,7 @@ cpp_library_wrapper(name="rocksdb_lib", srcs=[
"db/version_util.cc",
"db/wal_edit.cc",
"db/wal_manager.cc",
"db/wide/read_path_blob_resolver.cc",
"db/wide/wide_column_serialization.cc",
"db/wide/wide_columns.cc",
"db/wide/wide_columns_helper.cc",
@@ -4951,6 +4952,12 @@ cpp_unittest_wrapper(name="db_merge_operator_test",
extra_compiler_flags=[])
cpp_unittest_wrapper(name="db_open_with_config_test",
srcs=["db/db_open_with_config_test.cc"],
deps=[":rocksdb_test_lib"],
extra_compiler_flags=[])
cpp_unittest_wrapper(name="db_options_test",
srcs=["db/db_options_test.cc"],
deps=[":rocksdb_test_lib"],
@@ -5041,6 +5048,12 @@ cpp_unittest_wrapper(name="db_wide_basic_test",
extra_compiler_flags=[])
cpp_unittest_wrapper(name="db_wide_blob_direct_write_test",
srcs=["db/wide/db_wide_blob_direct_write_test.cc"],
deps=[":rocksdb_test_lib"],
extra_compiler_flags=[])
cpp_unittest_wrapper(name="db_with_timestamp_basic_test",
srcs=["db/db_with_timestamp_basic_test.cc"],
deps=[":rocksdb_test_lib"],