mirror of
https://github.com/facebook/rocksdb.git
synced 2026-07-07 14:47:40 +08:00
main
41 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Peter Dillinger
|
a004c2d850 |
Add experimental embedded blob SST support (#14851)
Summary:
Add EXPERIMENTAL embedded blob SST support for SstFileWriter through
OpenWithEmbeddedBlobs(). Eligible large values are written as same-file blob
records inline in a block-based SST as values are added (interleaved with data
blocks), while table entries store same-file BlobIndex references that readers
resolve for Get, MultiGet, and iteration, including mixed embedded and
non-embedded wide-column values.
Embedded-blob handling is folded directly into BlockBasedTableBuilder rather than
living in SstFileWriter or a separate table-builder wrapper: SstFileWriter only
selects the mode (via TableBuilderOptions::embedded_blob_options), and the
builder writes blob records inline using its own file writer and running offset.
This is enabled by disabling index-value delta encoding for these SSTs — delta
encoding reconstructs a data block's offset from the previous block and so
requires byte-contiguous data blocks, which interleaved blob records would break.
With full (non-delta) index values, blob records can sit between data blocks, so
no entry buffering/replay is needed. To keep inline blob appends correctly
ordered with data-block writes, these SSTs use single-threaded (non-parallel)
compression. The mode is the only entry point today but the placement keeps it
open to generalization beyond SstFileWriter; regardless, this experimental
feature is expected only to have niche applications.
(Previous revisions of this change allowed delta-encoded index blocks by
putting all blobs at the beginning of the file, but that was a more awkward
and memory-hungry implementation due to buffering all the data blocks before
writing.)
The on-disk record format (SimpleGen2Blob: payload bytes followed by a 5-byte
trailer of a compression marker plus a builtin checksum that is context-modified
by the record's absolute file offset) lives in db/blob/blob_gen2_format.{h,cc},
which now owns both the read (ReadAndVerifySimpleGen2BlobRecord) and write
(WriteSimpleGen2BlobRecord) sides so the format is defined in one place. This is
expected to be reused for upcoming "blog file" support.
Readers need no record-layout metadata: same-file blob resolution is purely
absolute-offset keyed, and the per-record offset-modified checksum (plus a cheap
"record fits within the file" bound) is the corruption guard. The reader's only
embedded-blob metadata is presence: a best-effort auxiliary table property
(blob count and payload-byte totals, for diagnostics) whose mere presence signals
that the SST contains embedded blobs.
Reads route through the column family's BlobSource when a DB is attached, so
embedded payloads are served from / inserted into the blob value cache and
recorded in BLOB_DB_* statistics; the cache key is derived from the
SimpleGen2Blob offset scheme (the same scheme block-based SST blocks use), so
embedded blob records stay collision-free with data blocks even when the blob
cache and block cache are the same cache. Non-DB openers (SstFileReader,
sst_dump, repair, ingestion prevalidation) have no BlobSource and fall back to a
direct, uncached read.
Same-file BlobIndex references use blob file number 0 as the marker. That value
also serves as the invalid blob-file-number sentinel in broader metadata code,
but the meanings do not conflict when used carefully: only the embedded-SST
reader/writer path interprets 0 as same-file, while generic file-metadata paths
continue to reject it as invalid. Using 1 would be worse because legacy
"stackable" BlobDB can use low blob file numbers, including 1, so reserving it
would collide with real blob files.
Compression options remain in the public API as placeholders, but embedded blob
compression support is deferred. Integrating compression with
BlockBasedTableBuilder while avoiding copied CompressAndVerifyBlock-style logic
is tricky enough to deserve a separate, focused PR.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14851
Test Plan:
- Added basic feature coverage to the crash test.
- Added BlobIndexTest.SameFileBlobIndex and BlobGarbageMeterTest.SameFileBlobIndex
coverage for same-file BlobIndex encoding, display, recognition, and ignoring
same-file references in blob-garbage accounting.
- Extended FileMetaDataTest.UpdateBoundariesBlobIndex to preserve the generic
zero-file-number corruption check while keeping same-file embedded blob
semantics at the table-reader/writer layers.
- SstFileReader embedded blob coverage: round-trip Get, MultiGet, and iterator
reads; format_version gating; ignored placeholder compression options; the
2048-byte default min_blob_size; wide-column mixed embedded/non-embedded
values; and early append-error surfacing.
- Added an interleaved-layout test (small block_size with alternating
small/large values) asserting the SST property index_value_is_delta_encoded==0,
more than one data block, and that blob records are interspersed with data
blocks (not a strict front prefix), with all values read back correctly via
Get and iteration; replaces the old "ignored bytes before/after the blob record
prefix" test.
- Added an embedded-record corruption test: flipping a byte inside a blob
record's payload yields Corruption on read with verify_checksums (the
offset-keyed record checksum is the guard now that the range pre-check is gone).
- Exercised the cached read path through BlobSource, including blob-cache
hit/miss behavior and the shared blob_cache == block_cache configuration, in
db_blob_index_test.
- Normal-path CPU regression check: release (DEBUG_LEVEL=0) db_bench on a
non-embedded DB comparing this change vs upstream main, 3 interleaved reps of
fillseq, fillrandom, readrandom, and readseq (5M keys, value_size=100,
compression none, DB on /dev/shm). All deltas were within run-to-run noise
(~1%), i.e. no measurable regression from adding the embedded-mode branch to
the builder hot path.
Reviewed By: xingbowang
Differential Revision: D108564468
Pulled By: pdillinger
fbshipit-source-id: 5f01ffb1d40c6fd5b82d2451ec3342abb5040ca6
|
||
|
Maciej Szeszko
|
30f42e00ba |
fix package fault injection log parser in db stress fbpkg (#14874)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14874 ### Problem RocksDB [alert](https://www.internalfb.com/monitoring/alert/?alert_id=twshared38314.04.ldc2%40%23%24host_health.crashlooping_key.tw_agent.tw_agent.task_failure.app.count.60%40%23%24uhc%3A%20tupperware%20task%20crashlooping) fired for `tsp_ldc/RocksDB/rocks_on_ws_stress_test` after the job started crashlooping with `ModuleNotFoundError: No module named fault_injection_log_parser`. ### RCA D101973626 added `import fault_injection_log_parser` to `db_crashtest.py`, but `rocksdb_db_stress_internal_repo` only packaged `crashtest.py` and `rocks_db_stress`, so the `release_test` fbpkg rolled to TW without the new helper module. ### Fix Export `tools/fault_injection_log_parser.py` and include it in the `rocksdb_db_stress_internal_repo` fbpkg `path_actions` so it is installed next to `crashtest.py`; also add the BUCKLINT-required `network_access = network_access_utils.none()` on the existing `db_stress_auto_tuner_test` target. Reviewed By: xingbowang Differential Revision: D109346224 fbshipit-source-id: 3f770a7ad306d246f279641bf0f19142d05560c4 |
||
|
Peter Dillinger
|
3883a8d05e |
Rename db_test2 -> db_etc2_test (#14218)
Summary: When searching/grepping through unit tests, it's convenient to use test.cc suffix to match all unit tests, or to exclude test.cc when excluding unit tests from a code search. (I've even seen my AI assistant grep through `*test.cc`.) So I'm renaming db_test2.cc (as planned in https://github.com/facebook/rocksdb/issues/14076) Pull Request resolved: https://github.com/facebook/rocksdb/pull/14218 Test Plan: existing tests + CI Reviewed By: xingbowang Differential Revision: D90140624 Pulled By: pdillinger fbshipit-source-id: 78aa099290670bbb4093b2c7c02bc47ab3bf5f8e |
||
|
Peter Dillinger
|
fa8f4f1ef0 |
Update buckifier to use folly/coro instead of folly/experimental/coro (#14685)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14685 D100235293 and D102823090 migrated the generated RocksDB BUCK file from `//folly/experimental/coro:*` to `//folly/coro:*`, but did not update the buckifier script that generates it in internal_repo_rocksdb. The next release would revert the change. Update buckify_rocksdb.py to match, so the generated BUCK file stays consistent with the folly coro migration. Reviewed By: nmk70 Differential Revision: D103096688 fbshipit-source-id: 9055769ed5e9893397c7504ada22e21980f59dd2 |
||
|
Xingbo Wang
|
d534974711 |
Fix BUCK file format check failure (#14691)
Summary: BUCK file is out of sync on folly dependency. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14691 Test Plan: Build pass Reviewed By: archang19 Differential Revision: D103233002 Pulled By: xingbowang fbshipit-source-id: 252fa27229126995a0492bc426a90ead1d10d958 |
||
|
Davis Rollman
|
1fd2c202c7 |
Migrate folly coro references from folly/experimental to folly/coro 9
Summary: Migrate coro references across the repo to the non shim version. Differential Revision: D102823090 fbshipit-source-id: 3bec59aed1f2645139a1ad9c18363e6af462e61f |
||
|
xingbowang
|
809ed26be5 |
Revert "Migrate fbcode coro references from folly/experimental to folly/coro (5/6 - fbcode a-m)" (#14652)
Summary: This reverts commit `bad2d5b0af7e160de5ca58869f40941af0f9da95`. The reverted change switched fbcode Buck dependencies from `//folly/experimental/coro:*` to `//folly/coro:*` in: - `BUCK` - `buckifier/buckify_rocksdb.py` That change broke the internal build, so this PR restores the previous dependency paths. ## Testing - Not run locally; this task only fetched, rebased, and verified the existing revert branch. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14652 Reviewed By: archang19 Differential Revision: D101980062 Pulled By: xingbowang fbshipit-source-id: 9a6edf14c226caf87e2c63c7f2806d409dc809ef |
||
|
Davis Rollman
|
bad2d5b0af |
Migrate fbcode coro references from folly/experimental to folly/coro (5/6 - fbcode a-m)
Summary: Migrate coro references in remaining fbcode services (a-m). Reviewed By: iahs Differential Revision: D98841546 fbshipit-source-id: 83bb5a2ef6a29a1ff464efa0e4e0f76b48662ab0 |
||
|
Xingbo Wang
|
3070f73e97 |
Wide-column blob separation: lazy resolution through read, compaction, and write paths (#14386)
Summary: Wide-column blob separation: lazy resolution through read, compaction, and write paths Extend blob direct write to support wide-column entities (PutEntity), and add lazy blob resolution for wide-column values across all read and compaction paths. **Write path -- PutEntity blob separation:** - BlobWriteBatchTransformer::PutEntityCF now extracts large column values (>= min_blob_size) to blob files and serializes V2 entities with BlobIndex references, matching the existing Put behavior. - Add MaybePreprocessWideColumns() static helper to share blob extraction logic between the WriteBatch transformer and the new PutEntity fast path. - Add PutEntityFastPath() in DBImpl that preprocesses columns (sort, blob extract, serialize) before calling WriteImpl, skipping the redundant WriteBatch transformation pass. Trace batch preserves the original columns. **Read path -- blob resolution for Get/MultiGet/Iterator:** - GetContext::SaveValue resolves V2 entity blob columns eagerly: for value (Get), resolves the default column's blob reference; for columns (GetEntity), resolves all blob columns and re-serializes as V1. - DBIter::SetValueAndColumnsFromEntity detects V2 entities, deserializes with DeserializeV2, and eagerly resolves all blob columns via a new ReadPathBlobResolver. Resolved values are cached in the resolver and wide_columns_ Slices point into the cache, avoiding copies. - Add ReadPathBlobResolver (new file) -- on-demand blob fetcher for the read path with per-column caching, used by both DBIter and GetContext. - BlobFetcher gains allow_write_path_fallback to read from in-flight direct-write blob files not yet visible through Version (pre-flush reads). - Memtable lookups for Get(key) on V2 entities with a blob default column now return the blob index with is_blob_index=true, triggering the existing BDW resolution in MaybeResolveWritePathValue. - MaybeResolveWritePathValue (renamed from MaybeResolveDirectWriteBlobIndex) now also resolves V2 entity blob columns for GetEntity/MultiGetEntity, re-serializing as V1 after resolution. **Compaction path -- filter, GC, and extraction:** - CompactionIterator::InvokeFilterIfNeeded handles V2 entities: FilterV3 gets eagerly-resolved column values for backward compatibility; FilterV4 gets a CompactionBlobResolver for lazy on-demand resolution. - Add CompactionFilter::FilterV4 with WideColumnBlobResolver* parameter and SupportsFilterV4() opt-in. Default delegates to FilterV3. - CompactionBlobResolver (new class) implements WideColumnBlobResolver for the compaction path with stats tracking. - ExtractLargeColumnValuesIfNeeded extracts inline columns to blob files during compaction (entities without existing blob columns only). - GarbageCollectEntityBlobsIfNeeded relocates blob values from old blob files to new ones during compaction GC, with helpers FetchBlobsNeedingGC, RelocateBlobValues, and SerializeEntityAfterGC. - PrepareOutput unified entity deserialization: single DeserializeV2 call reused by both filter and GC/extraction paths via entity_deserialized_ flag, avoiding redundant parsing. **Merge path -- V2 entity base value resolution:** - MergeHelper::MergeUntil, GetContext::MergeWithWideColumnBaseValue, and DBIter::MergeWithWideColumnBaseValue resolve V2 blob columns before calling TimedFullMerge, using ResolveEntityForMerge. **Blob garbage accounting:** - BlobGarbageMeter tracks blob file in/out flow for V2 entity blob columns via ForEachBlobFileNumber, used for accurate GC decisions. - FileMetaData::UpdateBoundaries tracks oldest_blob_file_number for V2 entities, ensuring blob files referenced by entities are not prematurely deleted. **Serialization improvements:** - WideColumnSerialization::SerializeV2Impl allocates serialized_blob_indices only for actual blob columns (not all columns) and uses autovector for name/value sizes. - Add ForEachBlobFileNumber for lightweight blob file number extraction without full deserialization. - Add ResolveEntityForMerge helper for merge-path resolution. - Add section-size validation in DeserializeV2Impl. - Add empty blob index and column type validation. - blob_column_resolver_util.h -- shared helpers (FindBlobColumn, FindInCache, CacheInlinedBlob) used by both ReadPathBlobResolver and CompactionBlobResolver. **Testing:** - db_blob_direct_write_test: end-to-end PutEntity with BDW before/after flush, verifying Get, GetEntity, MultiGetEntity, and Iterator. - db_blob_index_test: ~1550 lines covering V2 entity blob resolution through Get, GetEntity, MultiGet, Iterator, compaction filter (V3 compat and V4 lazy), merge with blob base, and compaction GC/extraction. - compaction_iterator_test: ~950 lines testing entity blob GC, extraction, filter interaction, and combined GC+filter scenarios. - db_wide_basic_test: ~1200 lines for wide-column lazy blob resolution through all read paths plus compaction round-trips. - db_open_with_config_test: ~450 lines for BDW entity config validation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14386 Reviewed By: anand1976 Differential Revision: D99739701 Pulled By: xingbowang fbshipit-source-id: 6badd89b577f3054802eaaa654738468efb9dbdb |
||
|
Xingbo Wang
|
af4e32945b |
Blob direct write v1: write-path blob separation with partitioned files (reduced scope) (#14535)
Summary: This PR introduces **blob direct write v1**, a reduced-scope write-path optimization where large values (>= `min_blob_size`) are written directly to blob files during `Put()` and replaced in the memtable with compact `BlobIndex` references. This avoids holding full values in memory until flush time. ### Motivation In the existing BlobDB architecture, values are written to the WAL and memtable in their full form and separated into blob files only at flush time. This means: - Large values are held in memory twice (raw in memtable + blob file at flush) - Blob I/O is serialized through a single flush thread per column family Blob direct write addresses both: values leave the write path as small `BlobIndex` references, and multiple **partitions** (configurable via `blob_direct_write_partitions`) allow concurrent blob writes with independent locks. ### Design (v1 — single-writer, WAL-disabled, reduced scope) The v1 design intentionally keeps scope narrow for correctness and reviewability: - **Single writer thread assumption**: no concurrent writes to the same partition file. One logical writer serializes the batch. - **WAL-disabled**: direct-write blob files are only registered in MANIFEST at flush time. WAL replay cannot recover unregistered blob references, so WAL is disabled for this v1. - **Flush-on-write**: each `AddRecord` call flushes to the OS immediately. - **FIFO generation batching**: each memtable switch creates one generation batch. Direct-write files for that memtable are sealed and registered atomically when the batch is flushed to MANIFEST. - **Round-robin partitions**: blob writes are distributed across `blob_direct_write_partitions` files using an atomic counter. ### New components | Component | Description | |---|---| | `BlobFilePartitionManager` | Owns N partition files per CF. Manages open/seal/register lifecycle tied to memtable generations. | | `BlobWriteBatchTransformer` | A `WriteBatch::Handler` that rewrites qualifying `Put` values as `BlobIndex` entries before the batch enters the write group. | ### Write path integration 1. `DBImpl::WriteImpl` calls `BlobWriteBatchTransformer::TransformBatch` before entering the writer group (for default write path), or before joining the batch group (for pipelined/unordered write). 2. Values >= `min_blob_size` are written to a partition file; the key is stored with a `BlobIndex` in the transformed batch. A rollback guard marks blob bytes as initial garbage if the write fails. 3. On `SwitchMemtable`, `RotateCurrentGeneration` moves active partitions into the next immutable batch. 4. `FlushMemTableToOutputFile` / `AtomicFlushMemTablesToOutputFiles` call `PrepareFlushAdditions` to seal partition files and collect `BlobFileAddition` + `BlobFileGarbage` entries registered to MANIFEST alongside the flush. 5. Shutdown paths (`CancelAllBackgroundWork`, `WaitForCompact` with `close_db=true`) force-flush all CFs with active direct-write managers to ensure blob files are registered before close. ### Read path - **Get/MultiGet**: `MaybeResolveBlobForWritePath` resolves `BlobIndex` references found in memtable or immutable memtable via `BlobFilePartitionManager::ResolveBlobDirectWriteIndex`, which first checks manifest-visible state and falls back to direct blob-file reads via `BlobFileCache`. - **Iterator**: `DBIter::BlobReader` is extended with a `BlobFilePartitionManager*` to resolve direct-write blob indexes during iteration. The unified `ResolveBlobDirectWriteIndex` path handles both manifest-visible and not-yet-flushed files. ### New options | Option | Default | Description | |---|---|---| | `enable_blob_direct_write` | `false` | Enable write-path blob separation for this CF. Requires `enable_blob_files = true`. Not dynamically changeable. | | `blob_direct_write_partitions` | `1` | Number of parallel partition files per CF. Not dynamically changeable. | ### Feature incompatibilities (reduced v1 scope) The following features are *not supported* when `enable_blob_direct_write = true`, and are enforced both in `db_stress_tool` validation and `db_crashtest.py` sanitization: **Write model constraints:** - `threads` must be 1 (single writer assumption) - `allow_concurrent_memtable_write` = 0 - `enable_pipelined_write` = 0 (transformation done before batch group, but pipelined path supported with pre-transform) - `two_write_queues` = 0 - `unordered_write` = 0 (transformation done before batch group, but unordered path supported with pre-transform) **WAL and recovery:** - `disable_wal` = 1 (required — WAL replay of unregistered blob files is out of v1 scope) - `best_efforts_recovery` = 0 - `reopen` = 0 (no crash-restart with WAL replay) - All WAL-related stress features disabled: `manual_wal_flush_one_in`, `sync_wal_one_in`, `lock_wal_one_in`, `get_sorted_wal_files_one_in`, `get_current_wal_file_one_in`, `track_and_verify_wals`, `rate_limit_auto_wal_flush`, `recycle_log_file_num` **Blob GC and dynamic options:** - `use_blob_db` = 0 (stacked BlobDB not supported) - `allow_setting_blob_options_dynamically` = 0 - `enable_blob_garbage_collection` = 0 - `blob_compaction_readahead_size` = 0 - `blob_file_starting_level` = 0 **Unsupported value types and APIs:** - Merge (`use_merge`, `use_full_merge_v1`) — merge values pass through untransformed - Entity APIs (`use_put_entity_one_in`, `use_get_entity`, `use_multi_get_entity`, `use_attribute_group`) - `use_timed_put_one_in` - User-defined timestamps (`user_timestamp_size`, `persist_user_defined_timestamps`, `create_timestamped_snapshot_one_in`) - Transactions (`use_txn`, `use_optimistic_txn`, `test_multi_ops_txns`, `commit_bypass_memtable_one_in`) — though `WriteCommittedTxn::CommitInternal` falls back from bypass-memtable to normal path when BDW is active - `IngestWriteBatchWithIndex` returns `NotSupported` - `inplace_update_support` = 0 **Fault injection:** - All write/read/metadata fault injection disabled (`sync_fault_injection`, `write_fault_one_in`, `metadata_write_fault_one_in`, `read_fault_one_in`, `metadata_read_fault_one_in`, `open_*_fault_one_in`) **Infrastructure/snapshot APIs:** - `remote_compaction_worker_threads` = 0 - `test_secondary` = 0 - `backup_one_in` = 0 - `checkpoint_one_in` = 0 - `get_live_files_apis_one_in` = 0 - `ingest_external_file_one_in` = 0 - `ingest_wbwi_one_in` = 0 ### Tests - `db/blob/db_blob_basic_test.cc`: ~660 lines of new direct-write unit tests covering basic put/get, multi-partition, flush/compaction, recovery, and error injection. - `db/blob/blob_file_cache_test.cc`: ~96 lines of new tests for direct-write blob file cache behavior. - `db/write_batch_test.cc`: ~96 lines of tests for WriteBatch with blob index entries. - `utilities/transactions/transaction_test.cc`: verifies transaction commit path falls back correctly with direct write enabled. - `db_stress_tool/`: full stress test support with `--enable_blob_direct_write` and `--blob_direct_write_partitions` flags, integrated into `db_crashtest.py` with 10% random selection alongside regular blob params. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14535 Test Plan: ``` make -j128 db_blob_basic_test && ./db_blob_basic_test make -j128 blob_file_cache_test && ./blob_file_cache_test make -j128 write_batch_test && ./write_batch_test make -j128 transaction_test && ./transaction_test make -j128 check ``` Stress test: ``` python3 tools/db_crashtest.py blackbox --enable_blob_direct_write=1 \ --enable_blob_files=1 --blob_direct_write_partitions=4 \ --disable_wal=1 --threads=1 ``` Reviewed By: pdillinger Differential Revision: D98766843 Pulled By: xingbowang fbshipit-source-id: 1577653826913a59d05680a87bce5534ac5a5e69 |
||
|
Danny Chen
|
ae322e0a73 |
Add SortedRunBuilder: use RocksDB as an external… (#14499)
Summary: CONTEXT: MyRocks and other third-party users currently need to understand deep RocksDB internals to sort unsorted data into SST files. The existing pattern requires: opening a full DB instance, configuring VectorRepFactory with universal compaction, disabling auto-compaction, writing unsorted data, manually triggering CompactRange with kForceOptimized, collecting output via GetColumnFamilyMetaData, and finally ingesting via IngestExternalFile with allow_db_generated_files. This is error-prone and requires ~30 lines of RocksDB configuration knowledge. WHAT: This adds a new utility class, SortedRunBuilder, that wraps all of the above into a simple Create/Add/Finish API. Callers feed in unsorted key-value pairs (in any order, from any number of threads) and receive sorted SST files with seqno=0 ready for ingestion -- or can iterate sorted output directly. The name SortedRunBuilder was chosen over alternatives like UnsortedSstFileWriter because this utility targets a broad audience: anyone wanting to use RocksDB as an external sort engine, not just users migrating from SstFileWriter. That said, this explicitly removes the sorted-input requirement that SstFileWriter enforces -- callers no longer need to pre-sort their data before writing SST files. KEY DESIGN DECISIONS: - Zero changes to core RocksDB internals. This is a pure utility layer that exclusively calls existing public APIs: * DB::Open(), DB::Put(), DB::Write(), DB::Flush(), DB::CompactRange(), DB::GetColumnFamilyMetaData(), DB::NewIterator(), DestroyDB() * VectorRepFactory (sort-on-flush memtable) * BottommostLevelCompaction::kForceOptimized (zero seqnos) - No modifications to: compaction logic, memtable implementation, SST file format, ingestion logic, or DB open/close paths - All new code lives in utilities/sorted_run_builder/ and the public header include/rocksdb/utilities/sorted_run_builder.h - Build file changes are limited to registering the new source/test files API SURFACE: SortedRunBuilderOptions opts; opts.temp_dir = "/tmp/sort_work"; std::unique_ptr<SortedRunBuilder> builder; SortedRunBuilder::Create(opts, &builder); builder->Add(key, value); // any order, thread-safe builder->AddBatch(&batch); // WriteBatch for throughput builder->Finish(); // flush + compact + collect builder->GetOutputFiles(); // sorted SSTs for ingestion builder->NewIterator(ro); // iterate sorted output FILES CHANGED: New: include/rocksdb/utilities/sorted_run_builder.h (public header) New: utilities/sorted_run_builder/sorted_run_builder.cc (implementation) New: utilities/sorted_run_builder/sorted_run_builder_test.cc (12 tests) New: docs/plans/sorted_run_builder_plan.md (design plan) New: docs/plans/sorted_run_builder_usage_guide.md (usage guide) Modified: src.mk, CMakeLists.txt, Makefile (register new files only) Pull Request resolved: https://github.com/facebook/rocksdb/pull/14499 Test Plan: $ make clean && make -j$(nproc) sorted_run_builder_test $ ./sorted_run_builder_test [==========] Running 12 tests from 1 test case. [ PASSED ] 12 tests. (688 ms total) Tests cover: basic sort correctness, empty builder, WriteBatch path, concurrent multi-threaded writes, ingestion into a target DB, large random dataset (10K keys), entry/size counters, error cases (Add after Finish, iterator before Finish, empty temp_dir), cleanup verification, and duplicate key handling. Reviewed By: xingbowang Differential Revision: D98935972 Pulled By: dannyhchen fbshipit-source-id: e3bb7f1ea5f6004aab697e4da667fa21292ca250 |
||
|
Xingbo Wang
|
bcaf2794dc |
Fix TSAN data race in InjectedErrorLog by suppressing benign races (#14467)
Summary:
InjectedErrorLog is a lock-free circular ring buffer designed to be safe to call from signal handlers (which cannot use locks). It has an intentional benign data race between Record() (called by worker threads) and PrintAll() (called by the main thread from a signal/termination handler). The code documents this trade-off in comments, but was missing TSAN suppression annotations.
Simply adding TSAN_SUPPRESSION (__attribute__((no_sanitize("thread")))) is insufficient because TSAN still intercepts libc functions like vsnprintf/snprintf -- accesses through these interceptors are still tracked even when the calling function is annotated.
The fix:
1. Add TSAN_SUPPRESSION to both Record() and PrintAll() to suppress direct field reads/writes in the function body.
2. Restructure both functions to use local stack buffers for vsnprintf/snprintf operations instead of operating directly on shared entry data. This avoids passing shared memory through TSAN-intercepted libc functions.
Also adds fault_injection_fs_test with a ConcurrentRecordAndPrintAll test that exercises the concurrent Record() + PrintAll() pattern and verifies no TSAN race is reported.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14467
Test Plan:
- fault_injection_fs_test passes under TSAN (buck2 test fbcode//mode/dbg-tsan)
- Reverted fix, re-ran: Fatal (6 TSAN warnings) -- round-trip confirmed
- fault_injection_fs_test passes under debug mode (no regression)
Reviewed By: mszeszko-meta
Differential Revision: D96948483
Pulled By: xingbowang
fbshipit-source-id: efdd5eafa12a5a82f973e40aa327901cc5f95033
|
||
|
zaidoon
|
ec22903914 |
Support all operation types in User Defined Index (UDI) interface (#14399)
Summary: Remove the restriction that limited UDI to ingest-only, Puts-only use cases. This enables UDI plugins (including the trie index from https://github.com/facebook/rocksdb/issues/14310) to work with all operation types: Put, Delete, Merge, SingleDelete, PutEntity, etc. Part of https://github.com/facebook/rocksdb/issues/12396 Pull Request resolved: https://github.com/facebook/rocksdb/pull/14399 Reviewed By: pdillinger Differential Revision: D96392636 Pulled By: xingbowang fbshipit-source-id: 0f1e6c38531fa72539a0e2c6a3dffff333392b4c |
||
|
Josh Kang
|
0c6f741422 |
Rename write_prepared_transaction_test_seqno to write_prepared_transaction_seqno_test (#14453)
Summary: write_prepared_transaction_test_seqno keeps showing up in git changes test binaries need to end with _test so they are ignored by .gitignore Pull Request resolved: https://github.com/facebook/rocksdb/pull/14453 Reviewed By: xingbowang Differential Revision: D96202671 Pulled By: joshkang97 fbshipit-source-id: 91642226bedde37ca72c6af03f300990dca8ff3a |
||
|
Anand Ananthabhotla
|
3b5cb114e3 |
Refactor MultiScan to use MultiScanIndexIterator (#14401)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14401 Unify the MultiScan and regular iterator codepaths in BlockBasedTableIterator by introducing a MultiScanIndexIterator that implements InternalIteratorBase<IndexValue>. During Prepare(), the original index iterator is swapped out for a MultiScanIndexIterator that wraps the prefetched block handles and scan range metadata. This allows SeekImpl() and FindBlockForward() to use the same code flow for both regular and MultiScan operations, eliminating the need for separate MultiScan-specific methods (SeekMultiScan, FindBlockForwardInMultiScan, MultiScanSeekTargetFromBlock, MultiScanUnexpectedSeekTarget, MultiScanLoadDataBlock, MarkPreparedRangeExhausted). Key changes: - New MultiScanIndexIterator class that manages scan range tracking, block handle iteration, forward-only seek enforcement, and wasted block counting - InitDataBlock() loads blocks from ReadSet when MultiScan is active - FindBlockForward() detects scan range boundaries via IsScanRangeExhausted() after index_iter_->Next() - Disabled reseek optimization for MultiScan so MultiScanIndexIterator::Seek() is always called to update scan range tracking state - Removed MultiScanState struct and all MultiScan-specific methods from BlockBasedTableIterator - No changes to CheckDataBlockWithinUpperBound or CheckOutOfBound — they work as-is through iterate_upper_bound - multi_scan_status_ intentionally not checked in Valid() hot path to avoid performance regression; when status is non-OK, block_iter_points_to_real_block_ is already false - Fixed pre-existing bug in ReadSet::SyncRead() that used the base decompressor without compression dictionary, causing ZSTD data corruption when blocks with dictionary compression needed synchronous fallback reads Reviewed By: xingbowang Differential Revision: D93300655 fbshipit-source-id: 231059208e0cc512bc2ec43ff7055fcb2a2dc72d |
||
|
Josh Kang
|
f25fb41da6 |
Add option to validate sst files in the background on DB open (#14322)
Summary: Add `open_files_async` option for faster DB startup. When enabled, SST file opening and validation is deferred to a background thread after `DB::Open` returns, reducing startup latency for databases with many SST files. WAL recovery remains synchronous. To support this, `FindTable` is extended with a pinning mechanism that stores the cache handle directly on `FileMetaData` via a new `PinnedTableReader` class, and sets the table reader atomically so subsequent reads skip cache lookups. `FileDescriptor::table_reader` is replaced with `PinnedTableReader pinned_reader` which wraps a `std::atomic<TableReader*>` with acquire/release ordering to safely handle concurrent access between the background opener and read threads. Should validations fail, the background opener sets a `kAsyncFileOpen` background error. Future read requests will look up the table reader again via the cache, and if any validations fail there it will get propagated to the user (existing behavior when `max_open_files > 0`). This feature is most useful when `max_open_files=-1`, because otherwise file opening is already capped at 16 files and DB open should be fast. ## Restrictions - This feature also is incompatible with fifo compaction because fifo compaction requires reading table properties under DB mutex. When table reader is unpinned, this may cause a DB hang. - This feature is also incompatible with `skip_stats_update_on_db_open=false` because it will result in even longer DB open ## Key changes - New `open_files_async` DB option with C, Java, and `db_bench` bindings - `BGWorkAsyncFileOpen` background worker that opens all SST files post-`DB::Open`, with shutdown awareness via `shutting_down_` flag - New `PinnedTableReader` class in `version_edit.h` — thread-safe wrapper holding `std::atomic<TableReader*>` and `Cache::Handle*` with proper acquire/release ordering. Replaces the old `FileDescriptor::table_reader` raw pointer and `FileMetaData::table_reader_handle` - Extract `LoadTableHandlersHelper` into `db/version_util.cc` — shared between `VersionBuilder::LoadTableHandlers` (for version edits during recovery) and `BGWorkAsyncFileOpen` (for base storage post-open) - `FindTable` extended with `pin_table_handle` and `out_table_reader` params — when pinning is enabled, the table reader is stored on `FileMetaData` so Get/MultiGet/Iterator skip redundant cache lookups. `FindTable` now performs the pinned-reader fast-path check internally instead of requiring callers to check `fd.table_reader` beforehand - Note: pinning is explicit (not default) because some callers create temporary `FileMetaData`s that would need to properly clean up table handles - `CompactedDBImpl` updated to use `FindTable` + pinning instead of raw `fd.table_reader` access for Get/MultiGet - New `kAsyncFileOpen` background error reason in `listener.h` and `error_handler.cc` - Add a check in ~DBImpl to ensure async file open task has not been forgotten to be scheduled in (future) subclasses of DBImpl. Certain subclasses that never use it will need to explicitly mark it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14322 Test Plan: - `OpenFilesAsyncTest` parameterized over `num_flushes` (1, 20), `ReadType` (Get, MultiGet, Iterator), `max_open_files` (-1, 10), and `read_only` (true, false) - **ConcurrentFileAccess**: concurrent reads and compactions race with async opener - **AfterRead**: reads happen before async opener, verifying lazy open and that the opener sees already-pinned readers - **BeforeRead**: async opener completes first, verifying reads use pre-loaded table readers - **Shutdown**: DB closes before async opener starts, verifying clean cancellation with 0 file opens - **Error**: corrupted SST files, verifying `kAsyncFileOpen` background error is set and reads return corruption - **DropColumnFamily**: CF dropped before async opener runs, verifying the opener gracefully skips dropped CFs - Added to crash test ### Benchmark To simulate a high-latency remote filesystem, I set up a virtual filesystem with dm-delay using 10ms reads, 0 ms writes. ``` # Generate a DB with many L0 files TEST_TMPDIR=/data/users/jkangs/dm-delay-test/mnt ./db_bench -benchmarks=fillseq -disable_auto_compactions=true -write_buffer_size=1000 -num=1000000 ``` ``` ./db_bench -use_existing_db=true -db=/data/users/jkangs/dm-delay-test/mnt/dbbench -benchmarks=readrandom -reads=1 -report_open_timing=true -open_files_async=true -use_direct_reads -file_opening_threads=1 -skip_stats_update_on_db_open OpenDb: 25.1419 milliseconds ``` ``` ./db_bench -use_existing_db=true -db=/data/users/jkangs/dm-delay-test/mnt/dbbench -benchmarks=readrandom -reads=1 -report_open_timing=true -open_files_async=false -use_direct_reads -file_opening_threads=1 -skip_stats_update_on_db_open OpenDb: 23109.4 milliseconds ``` ### No read regressions On main branch ``` ./db_bench -use_existing_db=true -db=/dev/shm/dbbench -benchmarks=readrandom -seed=1 -threads=8 -duration=30 readrandom : 4.827 micros/op 1657100 ops/sec 30.005 seconds 49720992 operations; 183.3 MB/s (6198999 of 6198999 found) ``` On this branch ``` ./db_bench -use_existing_db=true -db=/dev/shm/dbbench -benchmarks=readrandom -seed=1 -threads=8 -duration=30 readrandom : 4.863 micros/op 1644808 ops/sec 30.007 seconds 49354992 operations; 182.0 MB/s (6099999 of 6099999 found) ./db_bench -use_existing_db=true -db=/dev/shm/dbbench -benchmarks=readrandom -seed=1 -threads=8 -duration=30 -open_files_async=true readrandom : 4.803 micros/op 1665392 ops/sec 30.004 seconds 49968992 operations; 184.2 MB/s (6222999 of 6222999 found) ``` Reviewed By: pdillinger, xingbowang Differential Revision: D93538033 Pulled By: joshkang97 fbshipit-source-id: 32ac70c112cd733b7c1e1c1e2e7ce6422318a5ae |
||
|
zaidoon
|
ce0fff9f41 |
Add trie-based User Defined Index (UDI) plugin (#14310)
Summary: Implement a Fast Succinct Trie (FST) index based on LOUDS encoding as a User Defined Index plugin for RocksDB's block-based tables. First step toward trie-based indexing per https://github.com/facebook/rocksdb/issues/12396. The trie uses hybrid LOUDS-Dense (upper levels, 256-bit bitmaps) + LOUDS-Sparse (lower levels, label arrays) encoding inspired by the [SuRF paper](https://www.pdl.cmu.edu/PDL-FTP/Storage/surf_sigmod18.pdf) (Zhang et al., SIGMOD 2018). The boundary between dense and sparse levels is automatically chosen to minimize total space. Key Components - **Bitvector** with O(1) rank and O(log n) select using a rank LUT sampled every 256 bits with popcount intrinsics. Uses uint32_t rank LUT entries (halving memory vs uint64_t). Includes word-level `AppendWord()` for efficient dense bitmap construction and `AppendMultiple()` optimized at word granularity for bulk bit fills. - **Streaming trie builder** using flat per-level arrays with deferred internal marking, handle migration for prefix keys, and lazy node creation. Infers trie structure directly from sorted keys via LCP analysis in a single pass (no intermediate tree). - **LoudsTrie** for immutable querying with BFS-ordered handle reordering built into single-pass level-by-level serialization. Move-only semantics with correct pointer re-seating after `std::string` move. - **LoudsTrieIterator** with rank-based traversal, key reconstruction from trie path, and stack-based backtracking for `Next()`. Uses packed 8-byte `LevelPos` (is_dense flag in bit 63) and `autovector<LevelPos, 24>` to avoid heap allocation. Key reconstruction uses a raw char buffer allocated once to `MaxDepth()+1` bytes. - **TrieIndexFactory/Builder/Reader/Iterator** implementing the `UserDefinedIndexFactory` interface. - **Zero-copy block handle loading** using two fixed-width uint64_t arrays (offsets + sizes) with 8-byte alignment, enabling O(1) initialization via direct pointer assignment. Seek Hot Path Optimizations - **Fanout-1 sparse fast path**: Most sparse nodes in tries built from zero-padded numeric keys have exactly one child. Detected via `start_pos + 1 == end_pos` and inlined as a single byte comparison, avoiding the full `SparseSeekLabel` call. - **Linear scan for small sparse nodes**: `SparseSeekLabel` uses sequential scan for nodes with ≤16 labels instead of binary search. Faster for common 10-child digit nodes where branch misprediction cost outweighs linear scan cost. - **Rank reuse**: `DenseLeafIndexFromRankAndHasChildRank` and `SparseLeafIndexFromHasChildRank` overloads accept pre-computed `has_child_rank` from the Seek descent, avoiding redundant `Rank1` calls. General Performance Optimizations - **Select-free sparse traversal**: Precomputed child position lookup tables (`s_child_start_pos_`/`s_child_end_pos_`) eliminate `Select1` calls during Seek. Sparse traversal tracks `(start_pos, end_pos)` directly, using only `Rank1` (O(1)) + array lookup (O(1)) for child descent. - **Cached label_rank pattern**: Eliminates redundant `Rank1` calls in hot paths (Seek, Next, Advance all cache and reuse the label_rank computed for has_child checking). - **Leaf index fast path**: When no prefix keys exist (common case), `SparseLeafIndex` and `DenseLeafIndexFromRank` skip the prefix-key `Rank1` calls entirely, reducing from 3 to 1 `Rank1` call. - **Popcount-based Select64** via 6-step binary search within 64-bit words. - MSVC portability using RocksDB's `BitsSetToOne`/`CountTrailingZeroBits`. Benchmark Results Trie Seek at 32K keys (16-byte keys, 5M lookups, median of 5 runs): | Configuration | ns/op | |---|---:| | Trie (optimized) | **118** | | Binary search (native) | 134 | Trie Seek is **~12% faster** than native binary search index at 32K keys per block. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14310 Reviewed By: anand1976 Differential Revision: D93921511 Pulled By: xingbowang fbshipit-source-id: ba18604bc6bd4f575311a1ae3047a541274869b6 |
||
|
anand76
|
653fd9c65b |
Bug fix for bg error recovery in TransactionDB (#14313)
Summary: This PR fixes a bug in the interaction between WritePrepared/WriteUnprepared TransactionDB (with two_write_queues=true) and background error recovery. This bug caused crash tests to fail with a "sequence number going backwards" error during DB open. Root Cause ------------ When two_write_queues=true, sequence numbers are allocated via FetchAddLastAllocatedSequence() before a write completes, but are only published via SetLastSequence() after the write succeeds. If a background error occurs (e.g., a MANIFEST write failure during flush), the error recovery path in DBImpl::ResumeImpl creates new memtables and WAL files. The new WAL's starting sequence number is based on LastSequence() (the published value), which can be lower than already-allocated sequence numbers that were written to the old WAL. On subsequent recovery, RocksDB detects that sequence numbers in the new WAL are lower than those in the old WAL and reports a "sequence number going backwards" corruption error, causing the DB to fail to open. Fix --- The fix adds a call to a new VersionSet::SyncLastSequenceWithAllocated() method at the beginning of DBImpl::ResumeImpl, before any new memtables or WALs are created. This method advances last_sequence_ to match last_allocated_sequence_ if the latter is higher, ensuring the new WAL starts with a sequence number that is at least as high as any previously allocated one. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14313 Test Plan: --------- Add new unit tests in write_prepared_transaction_test_seqno Reviewed By: pdillinger Differential Revision: D92746944 Pulled By: anand1976 fbshipit-source-id: 34385fc13fd74435dd1c3283637eb118f45d887e |
||
|
Xingbo Wang
|
656b734a5f |
Support abort background compaction jobs. (#14227)
Summary: This adds a new public API to allow applications to abort all running compactions and prevent new ones from starting. Unlike DisableManualCompaction() which only pauses manual compactions and waits for them to finish naturally, AbortAllCompactions() actively signals running compactions (both automatic and manual) to terminate early and waits for them to complete before returning. The abort signal is checked periodically during compaction (every 100 keys), so ongoing compactions abort quickly. Any output files from aborted compactions are automatically cleaned up to prevent partial results from being installed. This is useful for scenarios where applications need to quickly stop all compaction activity, such as during graceful shutdown or when performing maintenance operations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14227 Test Plan: - Unit tests in db_compaction_abort_test.cc cover various abort scenarios including: abort before/during compaction, abort with multiple subcompactions, nested abort/resume calls, abort with CompactFiles API, abort across multiple column families, and timing guarantees - Updated compaction_job_test.cc to include the new parameter Reviewed By: anand1976 Differential Revision: D91480994 Pulled By: xingbowang fbshipit-source-id: 36837971d8a540cd34d3ec28a78bc94b582625b0 |
||
|
Ryan Hancock
|
2b28885c80 |
Introducing IO Dispatcher (#14135)
Summary: This diff introduces the IO Dispatcher, which will be used to simplify the code path for MultiScan, while further providing a centralized place to enact policy on how MultiScan is done (i.e., limit memory usage and pinned buffers for example). Right now this diff only encapsulates the functionality done during the Prepare of MultiScan. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14135 Reviewed By: anand1976 Differential Revision: D87837261 Pulled By: krhancoc fbshipit-source-id: 2698910ade02bc3d182413ae07ce69fe7abb7ec5 |
||
|
Jay Huh
|
8c8586aa23 |
Add oncall to BUCK file (#14134)
Summary: As title Pull Request resolved: https://github.com/facebook/rocksdb/pull/14134 Test Plan: The following command generated the BUCK file correctly ``` python3 buckifier/buckify_rocksdb.py ``` Reviewed By: anand1976 Differential Revision: D87469877 Pulled By: jaykorean fbshipit-source-id: 9ec330084cfe96ad9b71aa13c8eb16593256a5ac |
||
|
Peter Dillinger
|
37176a4a44 |
Auto-tune manifest file size (#14076)
Summary:
Adds auto-tuning of manifest file size to avoid the need to scale `max_manifest_file_size` in proportion to things like number of SST files to properly balance (a) manifest file write amp and new file creation, vs. (b) manifest file space amp and replay time, including non-incremental space usage in backups. (Manifest file write amp comes from re-writing a "live" record when the manifest file is re-created, or "compacted"; space amp is usage beyond what would be used by a compacted manifest file.) In more detail,
* Add new option `max_manifest_space_amp_pct` with default value of 500, which defaults to 0.2 write amp and up to roughly 5.0 space amp, except `max_manifest_file_size` is treated as the "minimum" size before re-creating ("compacting") the manifest file.
* `max_manifest_file_size` in a way means the same thing, with the same default of 1GB, but in a way has taken on a new role. What is the same is that we do not re-create the manifest file before reaching this size (except for DB re-open), and so users are very unlikely to see a change in default behavior (auto-tuning only kicking in if auto-tuning would exceed 1GB for effective max size for the current manifest file). The new role is as a file size lower bound before auto-tuning kicks in, to minimize churn in files considered "negligibly small." We recommend a new setting of around 1MB or even smaller like 64KB, and expect something like this to become the default soon.
* These two options along with `manifest_preallocation_size` are now mutable with SetDBOptions. The effect is nearly immediate, affecting the next write to the current manifest file.
Also in this PR:
* Refactoring of VersionSet to allow it to get (more) settings from MutableDBOptions. This touches a number of files in not very interesting ways, but notably we have to be careful about thread-safe access to MutableDBOptions fields, and even fields within VersionSet. I have decided to save copies of relevant fields from MutableDBOptions to simplify testing, etc. by not saving a reference to MutableDBOptions but getting notified of updates.
* Updated some logging in VersionSet to provide some basic data about final and compacted manifest sizes (effects of auto-tuning), making sure to avoid I/O while holding DB mutex.
* Added db_etc3_test.cc which is intended as a successor to db_test and db_test2, but having "test.cc" in its name for easier exclusion of test files when using `git grep`. Intended follow-up: rename db_test2 to db_etc2_test
* Moved+updated `ManifestRollOver` test to the new file to be closer to other manifest file rollover testing.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14076
Test Plan:
As for correctness, new unit test AutoTuneManifestSize is pretty thorough. Some other unit tests updated appropriately. Manual tests in the performance section were also audited for expected behavior based on the new logging in the DB LOG. Example LOG data with -max_manifest_file_size=2048 -max_manifest_space_amp_pct=500:
```
2025/10/24-11:12:48.979472 2150678 [/version_set.cc:5927] Created manifest 5, compacted+appended from 52 to 116
2025/10/24-11:12:49.626441 2150682 [/version_set.cc:5927] Created manifest 24, compacted+appended from 2169 to 1801
2025/10/24-11:12:52.194592 2150682 [/version_set.cc:5927] Created manifest 91, compacted+appended from 10913 to 8707
2025/10/24-11:13:02.969944 2150682 [/version_set.cc:5927] Created manifest 362, compacted+appended from 52259 to 13321
2025/10/24-11:13:18.815120 2150681 [/version_set.cc:5927] Created manifest 765, compacted+appended from 80064 to 13304
2025/10/24-11:13:35.590905 2150681 [/version_set.cc:5927] Created manifest 1167, compacted+appended from 79863 to 13304
```
As you can see, it only took a few iterations of ramp-up to settle on the auto-tuned max manifest size for tracking ~122 live SST files, around 80KB and compacting down to about 13KB. (13KB * (500 + 100) / 100 = 78KB). With the default large setting for max_manifest_file_size, we end up with a 232KB manifest, which is more than 90% wasted space. (A long-running DB would be much worse.)
As for performance, we don't expect a difference, even with TransactionDB because actual writing of the manifest is done without holding the DB mutex. I was not able to see a performance regression using db_bench with FIFO compaction and >1000 ~10MB SST files, including settings of -max_manifest_file_size=2048 -max_manifest_space_amp_pct={500,10,0}. No "hiccups" visible with -histogram either.
I also tried seeding a 1 second delay in writing new manifest files (other than the first). This had no significant effect at -max_manifest_space_amp_pct=500 but at 100 started causing write stalls in my test. In many ways this is kind of a worst case scenario and out-of-proportion test, but gives me more confidence that a higher number like 500 is probably the best balance in general.
Reviewed By: xingbowang
Differential Revision: D85445178
Pulled By: pdillinger
fbshipit-source-id: 1e6e07e89c586762dd65c65bb7cb2b8b719513f9
|
||
|
ngina
|
7603712a88 |
Introduce tail estimation to prevent oversized compaction files (#14051)
Summary: **Summary:** This change introduces tail size estimation during SST construction to improve compaction file cutting accuracy to prevent oversized files. The BlockBasedTableBuilder now estimates the SST tail size (index and filter blocks) and uses this estimate, in addition to the data size, to determine when to cut files during compaction. **Problem:** Currently, file cutting logic only considers data size when determining where to cut a file, failing to reserve space for index and filter blocks that are added when the file is finalized. This often leads to SST files that exceed target file size limits. **Behavior Change:** Implement size estimation methods for index and filter builders, and integrate these estimates into BlockBasedTableBuilder via a new EstimatedTailSize() method. This method aggregates estimates from all tail components and is used for file cutting decisions during compaction. **Performance Considerations:** To minimize CPU overhead, size estimates are updated when data blocks are finalized rather than on every key add. For index builders, estimates are updated when index entries are added (one per data block). For filter builders, the OnDataBlockFinalized() hook triggers estimate updates when data blocks are cut/finalized. This approach provides: * Minimal impact to compaction hot path (key additions) * Near real-time estimates for file cutting decisions * Meaningful estimate changes only when data blocks are finalized **Usage:** * Set true mutable cf option `compaction_use_tail_size_estimation` to use tail size estimation for compaction file cutting decisions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14051 Test Plan: * Assert tail size estimate is an overestimate in BlockBasedTableBuilder::Finish * Add new test to verify compaction output file is below target file size **Next steps:** * Enable tail size estimation for compaction file cutting by default (and other improvements) Reviewed By: pdillinger, cbi42 Differential Revision: D84852285 Pulled By: nmk70 fbshipit-source-id: c43cf5dbd2cb2f623a0622591ef24eee30ce0c87 |
||
|
Hui Xiao
|
e32c14eb56 |
Stress/crash test improvement to remote compaction with resumable compaction (#14041)
Summary: **Context/Summary:** - Add resumable compaction to stress test with adaptive progress cancellation - Add fault injection to remote compaction - Fix a real minor bug in a couple testing framework bugs with remote compaction Pull Request resolved: https://github.com/facebook/rocksdb/pull/14041 Test Plan: - Rehearsal stress test, finding bugs for https://github.com/facebook/rocksdb/pull/13984 effectively and did not create new failures. Reviewed By: jaykorean Differential Revision: D84524194 Pulled By: hx235 fbshipit-source-id: 42b4264e428c6739631ed9aa5eb02723367510bc |
||
|
ngina
|
13172e2be3 |
Add method to estimate index size (#14010)
Summary: This method will be used to improve the compaction logic by accounting for the tail size, in addition to the data size, when determining when to cut a file. Problem: Currently the file cutting logic only considers data size when determining where to cut a file, failing to reserve space for index and filter blocks that are added when the file is finalized. Key changes: - Add EstimateCurrentIndexSize() to IndexBuilder interface - Implement in ShortenedIndexBuilder with buffer that accounts for the next index entry. The buffer addresses under-estimation where the current index size doesn't account for the next index entry associated with the data block currently being built. The 2x multiplier bounds the estimate in the right direction and handles outlier cases with large keys. - Add num_index_entries_ member to track added index entries (== data blocks emitted). This is thread-safe since it's updated/read in the serialized emit step. Next steps: - Partitioned index size estimation implementation - Update compaction file cutting logic to consider index size estimation Pull Request resolved: https://github.com/facebook/rocksdb/pull/14010 Test Plan: Added a new test class with unit tests for new builder size estimation across all IndexBuilder implementations. Reviewed By: pdillinger Differential Revision: D83501741 Pulled By: nmk70 fbshipit-source-id: d58fc2a9e92e12a162f6244d4abd707a9c9e1885 |
||
|
Xingbo Wang
|
1aca60c089 |
Improve efficiency in PointLockManager by using separate Condvar (#13731)
Summary: PointLockManager manages point lock per key. The old implementation partition the per key lock into 16 stripes. Each stripe handles the point lock for a subset of keys. Each stripe have only one conditional variable. This conditional variable is used by all the transactions that are waiting for its turn to acquire a lock of a key that belongs to this stripe. In production, we notified that when there are multiple transactions trying to write to the same key, all of them will wait on the same conditional variables. When the previous lock holder released the key, all of the transactions are woken up, but only one of them could proceed, and the rest goes back to sleep. This wasted a lot of CPU cycles. In addition, when there are other keys being locked/unlocked on the same lock stripe, the problem becomes even worse. In order to solve this issue, we implemented a new PerKeyPointLockManager that keeps a transaction waiter queue at per key level. When a transaction could not acquire a lock immediately, it joins the waiter queue of the key and waits on a dedicated conditional variable. When previous lock holder released the lock, it wakes up the next set of transactions that are eligible to acquire the lock from the waiting queue. The queue respect FIFO order, except it prioritizes lock upgrade/downgrade operation. However, this waiter queue change increases the deadlock detection cost, because the transaction waiting in the queue also needs to be considered during deadlock detection. To resolve this issue, a new deadlock_timeout_us (microseconds) configuration is introduced in transaction option. Essentially, when a transaction is waiting on a lock, it will join the wait queue and wait for the duration configured by deadlock_timeout_us without perform deadlock detection. If the transaction didn't get the lock after the deadlock_timeout_us timeout is reached, it will then perform deadlock detection and wait until lock_timeout is reached. This optimization takes the heuristic where majority of the transaction would be able to get the lock without perform deadlock detection. The deadlock_timeout_us configuration needs to be tuned for different workload, if the likelihood of deadlock is very low, the deadlock_timeout_us could be configured close to a big higher than the average transaction execution time, so that majority of the transaction would be able to acquire the lock without performing deadlock detection. If the likelihood of deadlock is high, deadlock_timeout_us could be configured with lower value, so that deadlock would get detected faster. The new PerKeyPointLockManager is disabled by default. It can be enabled by TransactionDBOptions.use_per_key_point_lock_mgr. The deadlock_timeout_us is only effective when PerKeyPointLockManager is used. When deadlock_timeout_us is set to 0, transaction will perform deadlock detection immediately before wait. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13731 Test Plan: Unit test. Stress unit test that validates deadlock detection and exclusive, shared lock guarantee. A new point_lock_bench binary is created to help perform performance test. Reviewed By: pdillinger Differential Revision: D77353607 Pulled By: xingbowang fbshipit-source-id: 21cf93354f9a367a78c8666596ed14013ac7240b |
||
|
Hui Xiao
|
8d2f420db2 |
Shorten the lifetime of statistics object in db stress (#13899)
Summary: **Context/Summary:** Clear statistics reference from options_ to intentionally shorten the statistics object lifetime to be same as the db object (which is the common case in practice) and detect if RocksDB access the statistics beyond its lifetime. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13899 Test Plan: - [Ongoing] Stress test rehearsal Reviewed By: pdillinger Differential Revision: D80985435 Pulled By: hx235 fbshipit-source-id: ab238231cd81f47fa451aea12a0c85fa11d9ac81 |
||
|
Changyu Bi
|
13f054febb |
Support DbStressCustomCompressionManager in ldb and sst_dump (#13827)
Summary: while debugging stress test failure, I noticed that sst_dump and ldb do not work if custom db_stress compression manager is used. This PR adds support for it. ``` ./sst_dump --command=raw --show_properties --file=/tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst options.env is 0x7f2b1f4b9000 Process /tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst Sst file format: block-based /tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst: Not implemented: Could not load CompressionManager: DbStressCustom1 /tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst is not a valid SST file ./ldb idump --db=/tmp/rocksdb_crashtest_whiteboxy_emah11 --ignore_unknown_options --hex >> /tmp/i_dump Failed: Not implemented: Could not load CompressionManager: DbStressCustom1 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/13827 Test Plan: manually tested that ldb and sst_dump work with DbStressCustomCompressionManager after this PR Reviewed By: pdillinger Differential Revision: D79461175 Pulled By: cbi42 fbshipit-source-id: c8c092b10b4fde3a295b00751057749e8f0cf095 |
||
|
Ryan Hancock
|
7c5c37a1a4 |
IntervalSet Data Structure (#13787)
Summary: This diff introduces the IntervalSet data structure, which will be used to help create sets of non overlapping sets of intervals for MultiScan scan options. Specifically, we add specializations for Slices to assist in this. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13787 Test Plan: Added test to catch various cases within adding intervals. Reviewed By: anand1976 Differential Revision: D78624970 Pulled By: krhancoc fbshipit-source-id: 9a3e4a28738ab8428788467540fc05ab5c1a1b67 |
||
|
anand76
|
8b84390517 |
Add upper bound support for forward scans in MultiScan (#13723)
Summary: Respect the scan upper bound/limit, if specified, in `MultiScan`. This applies to block based table and other native RocksDB SSTs. In order to properly support it, the `MultiScan` object caches the `ReadOptions` passed by the user and sets the `iterate_upper_bound` as appropriate. We optimize for the case of either all scans specifying the upper bound, or none of them. In case of mixed scans, we reallocate the DB iterator anytime `ReadOptions` has to be updated. Tests: New unit tests in `db_iterator_test` Pull Request resolved: https://github.com/facebook/rocksdb/pull/13723 Reviewed By: cbi42 Differential Revision: D77385049 Pulled By: anand1976 fbshipit-source-id: 9c02d125770cbedbe6e8c10767ba537e7f7540e1 |
||
|
Sujit Maharjan
|
fd95bc8f5a |
Custom Compressor for predicting the CPU and IO cost of the block level compression (#13711)
Summary: This pull request implements the prediction aspect of auto-tuning compression in RocksDB, as part of Milestone 2. The goal is to optimize compression decisions to meet a given CPU and IO budget, based on the predicted CPU time and result compression ratio for compression decisions on a data block. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13711 Test Plan: Ran benchmark tests to evaluate performance impact of new algorithm Verified that optimization does not compromise overall system performance ```bash SUFFIX=`tty | sed 's|/|_|g'`; for ARGS in "-compression_parallel_threads=1 -compression_type=zstd -compression_manager=none" "-compression_parallel_threads=4 -compression_type=zstd -compression_manager=none" "-compression_parallel_threads=1 -compression_type=zstd -compression_manager=costpredictor" "-compression_parallel_threads=4 -compression_type=zstd -compression_manager=costpredictor" ; do echo $ARGS; (for I in `seq 1 20`; do ./db_bench -db=/dev/shm/dbbench$SUFFIX --benchmarks=fillseq -num=10000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=1000 -fifo_compaction_allow_compaction=0 -disable_wal -write_buffer_size=12000000 $ARGS 2>&1 | grep micros/op; done) | awk '{n++; sum += $5;} END { print int(sum / n); }'; done ``` parallel threads | 1 | 4 -- | -- | -- master branch | 1076660.5 ops | 1668411.3 ops new code compression manager="none" | 1057155.35 ops (-1.81%) | 1648664.2 ops (-1.18%) new code compression manager="costpredictor" | 1080794.8 ops (0.38%)| 1652720.35 ops (-0.94%) Used the mean absolute percentage error (MAPE) to show accuracy of the predictor. ```bash ./db_bench --db=/dev/shm/dbbench$SUFFIX --benchmarks=fillseq --compaction_style=2 --num=10000000 --fifo_compaction_max_table_files_size_mb=1000 --fifo_compaction_allow_compaction=0 --disable_wal --write_buffer_size=12000000 --statistics --stats_level=5 --value_size=2000 --compression_manager=costpredictor --compression_type=zstd --progress_reports=false 2>&1 | tee /tmp/predict.log ``` compression_name | compression_level | MAPE (cpu cost) | MAPE (io cost) | average measured_time (micro sec) | average predicted_time (micro sec) | average measured_io (bytes) | average predicted_io (bytes) -- | -- | -- | -- | -- | -- | -- | -- Snappy | 0 | 16.979548 | 3.138885 | 3.639488 | 2.98755 | 2257.655152 | 2178.070375 LZ4 | 1 | 15.508632 | 3.103681 | 4.733639 | 4.010361 | 2257.803299 | 2179.82233 LZ4 | 4 | 15.471204 | 3.102158 | 4.731955 | 4.006011 | 2258.529203 | 2179.778441 LZ4 | 9 | 15.429305 | 3.09599 | 4.729104 | 4.007059 | 2257.822368 | 2179.927506 LZ4HC | 1 | 7.254545 | 3.112858 | 79.64412 | 76.603272 | 2258.636774 | 2177.464922 LZ4HC | 4 | 7.249132 | 3.085802 | 79.591264 | 76.576416 | 2255.098757 | 2176.126082 LZ4HC | 9 | 7.248921 | 3.09695 | 79.719061 | 76.614155 | 2253.772057 | 2175.882686 ZSTD | 1 | 8.728305 | 3.223971 | 18.93434 | 17.882706 | 1957.773706 | 1890.895071 ZSTD | 15 | 4.853552 | 3.238199 | 329.396574 | 318.277613 | 1918.021616 | 1853.833546 ZSTD | 22 | 4.275209 | 3.243137 | 625.471394 | 596.254939 | 1919.035477 | 1853.44902 ```bash ./db_bench --db=/dev/shm/dbbench$SUFFIX --benchmarks=fillseq --compaction_style=2 --num=10000000 --fifo_compaction_max_table_files_size_mb=1000 --fifo_compaction_allow_compaction=0 --disable_wal --write_buffer_size=12000000 --statistics --stats_level=5 --value_size=2000 --compression_manager=costpredictor --compression_type=zstd --progress_reports=false --write_buffer_size=140737488355328 --block_size=16382 ``` Increasing the block size i.e. doubling the measured time reduces the MAPE by half. compression_name | compression_level | MAPE (cpu cost) | MAPE (io cost) | average measured_time (micro sec) | average predicted_time (micro sec) | average measured_io (bytes) | average predicted_io (bytes) -- | -- | -- | -- | -- | -- | -- | -- Snappy | 0 | 7.933944 | 0.061173 | 7.187587 | 6.815071 | 4466.536629 | 4465.925648 LZ4 | 1 | 5.614279 | 0.050215 | 8.526641 | 8.14445 | 4473.768752 | 4473.159792 LZ4 | 4 | 5.617925 | 0.050317 | 8.525155 | 8.144209 | 4473.772343 | 4473.159782 LZ4 | 9 | 5.65519 | 0.050249 | 8.530569 | 8.14836 | 4473.762187 | 4473.150695 LZ4HC | 1 | 4.259648 | 0.028564 | 98.273778 | 97.820515 | 4471.691596 | 4471.05918 LZ4HC | 4 | 4.269529 | 0.027665 | 98.240579 | 97.788721 | 4465.537078 | 4464.901328 LZ4HC | 9 | 4.274553 | 0.027555 | 98.319357 | 97.8637 | 4465.539437 | 4464.903889 ZSTD | 1 | 4.909716 | 0.155441 | 29.503133 | 29.047057 | 3713.562704 | 3712.978633 ZSTD | 15 | 1.310407 | 0.162864 | 643.803097 | 635.960631 | 3797.544307 | 3705.772419 ZSTD | 22 | 1.011497 | 0.155876 | 1221.189822 | 1220.693678 | 3705.556448 | 3704.972332 Reviewed By: hx235 Differential Revision: D77065528 Pulled By: shubhajeet fbshipit-source-id: f7f4ae018f786bfeae3eacf0135055c63e142610 |
||
|
Sujit Maharjan
|
504ff4ed55 |
Auto skip Compression (#13674)
Summary: **Context:** RocksDB's current compression approach rejects blocks if the compressed size exceeds a predefined threshold. To optimize performance, we aim to develop an algorithm that dynamically stops and resumes block compression attempts based on past rejection data. **Summary:** The goal of this milestone is to design, implement, and evaluate an algorithm that intelligently skips and resumes block compression attempts in RocksDB. The algorithm tracks whether randomly selected blocks was rejected, compressed or bypassed and using data of window size to determine the current rejection rate. The calculate rejection rate is used to decide whether to pause and resume compression attempts. We measure the effectiveness of skipping and resuming compression using DB bench and identify any concerning regressions in correctness and performance. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13674 Test Plan: 1. Test case to see if it can automatically start compression on compression friendly workload and see if it can automatically stop compression on non-compression friendly workload (auto_skip_compresor_test.cc) 3. Regression analysis to prove that no significant performance attempt ```bash SUFFIX=`tty | sed 's|/|_|g'`; for ARGS in "-compression_parallel_threads=1 -compression_type=zstd -compression_manager=none" "-compression_parallel_threads=4 -compression_type=zstd -compression_manager=none" "-compression_parallel_threads=1 -compression_type=zstd -compression_manager=autoskip" "-compression_parallel_threads=4 -compression_type=zstd -compression_manager=autoskip" ; do echo $ARGS; (for I in `seq 1 20`; do ./db_bench -db=/dev/shm/dbbench$SUFFIX --benchmarks=fillseq -num=10000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=1000 -fifo_compaction_allow_compaction=0 -disable_wal -write_buffer_size=12000000 $ARGS 2>&1 | grep micros/op; done) | awk '{n++; sum += $5;} END { print int(sum / n); }'; done ``` Measurement experiment | throughput (% change from main branch) | |---------------|--------------------------------| compression manager = none (main branch) | 1106890.35 ops/s compression manager = none (auto skip) | 1097574.55 ops/s (-0.84%) compression manager = auto skip (auto skip branch) | 1133432.9 ops/s (+2.4%) Reviewed By: hx235 Differential Revision: D76220795 Pulled By: shubhajeet fbshipit-source-id: 0f46ab34da1b451f8907306afba221503e6e22a5 |
||
|
Sujit Maharjan
|
adb750fdf4 |
Separating into cc and header file for simple_mixed_compressor.h (#13665)
Summary: **Summary** This pull request fixes the issue of having a single file simple_mixed_compressor.h containing both implementation and declaration. To improve code organization and follow best practices, I have separated the implementation into a new file simple_mixed_compressor.cc and updated the original file to only contain the necessary declarations. **Testing** Testing was performed by verifying the stdout output from both RoundRobinCompressor and BuiltInCompressorV2. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13665 Reviewed By: pdillinger Differential Revision: D76060831 Pulled By: shubhajeet fbshipit-source-id: c034868be51ea7b89c1a8dd12082b0159f49f588 |
||
|
anand76
|
14c949df8b |
Initial implementation of ExternalTableBuilder (#13434)
Summary: This PR adds the ability to use an ExternalTableBuilder through the SstFileWriter to create external tables. This is a counterpart to https://github.com/facebook/rocksdb/issues/13401 , which adds the ExternalTableReader. The support for external tables is confined to ingestion only DBs, with external table files ingested into the bottommost level only. https://github.com/facebook/rocksdb/issues/13431 enforces ingestion only DBs by adding a disallow_memtable_writes column family option. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13434 Test Plan: New unit tests in table_test.cc Reviewed By: pdillinger Differential Revision: D70532054 Pulled By: anand1976 fbshipit-source-id: a837487eadfabed9627a0eceb403bfc5fc2c427c |
||
|
anand76
|
920d25e34e |
Initial version of an external table reader interface (#13401)
Summary: This PR introduces an interface to plug in an external table file reader into RocksDB. The external table reader may support custom file formats that might work better for a specific use case compared to RocksDB native formats. This initial version allows the external table file to be loaded and queried using an `SstFileReader`. In the near future, we will allow it to be used with a limited RocksDB instance that allows bulkload but not live writes. The model of a DB using an external table reader is a read only database allowing bulkload and atomic replace in the bottommost level only. Live writes, if supported in the future, are expected to use block based table files in higher levels. Tombstones, merge operands, and non-zero sequence numbers are expected to be present only in non-bottommost levels. External table files are assumed to have only Puts, and all keys implicitly have sequence number 0. TODO (in future PRs) - 1. Add support for external file ingestion, with safety mechanisms to prevent accidental writes 2. Add support for atomic column family replace 3. Allow custom table file extensions 4. Add a TableBuilder interface for use with `SstFileWriter` Pull Request resolved: https://github.com/facebook/rocksdb/pull/13401 Reviewed By: pdillinger Differential Revision: D69689351 Pulled By: anand1976 fbshipit-source-id: c5d5b92d56fd4d0fc43a77c4ceb0463d4f479bda |
||
|
Levi Tamasi
|
833a2266a3 |
Expose a simple secondary index implementation (#13370)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/13370 We have a class called `DefaultSecondaryIndex` in `TransactionTest.SecondaryIndexPutDelete` that contains generally useful functionality. The patch generalizes it a bit to make the column name configurable, renames it to `SimpleSecondaryIndex`, and moves it to the public API so applications can use it. Reviewed By: jowlyzhang Differential Revision: D69147890 fbshipit-source-id: 0d2d1cc5adcde01f3978a450ec841c9e990d2170 |
||
|
Ryan Hancock
|
a6322b9ec6 |
Allow for Customizable DB Open Hooks for DB Bench (#13326)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/13326 This diff introduces ToolHooks, a class which allows for users to interpose their own set of logic for various functionality with db_bench_tool (i.e., various OpenDB implementations). Reviewed By: anand1976 Differential Revision: D67868126 fbshipit-source-id: df433b0c8a064a86735b92a8ef5f38527dbc9112 |
||
|
Levi Tamasi
|
4ac85f0a79 |
Add a public factory method for SecondaryIndexIterator (#13327)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/13327 The patch adds a public API method `NewSecondaryIndexIterator` that can be leveraged by users providing their own `SecondaryIndex` implementations. Reviewed By: jaykorean Differential Revision: D68569198 fbshipit-source-id: 07f77837c3ce7ab8ea2d9bac172df3d64ce4f745 |
||
|
Levi Tamasi
|
b339d089ed |
Write-side support for FAISS IVF indices (#13197)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/13197 The patch adds initial support for backing FAISS's inverted file based indices with data stored in RocksDB. It introduces a `SecondaryIndex` implementation called `FaissIVFIndex` which takes ownership of a `faiss::IndexIVF` object. During indexing, `FaissIVFIndex` treats the original value of the specified primary column as an embedding vector, and passes it to the provided FAISS index object to perform quantization. It replaces the original embedding vector with the result of the coarse quantizer (i.e. the inverted list id), and puts the result of the fine quantizer (if any) into the secondary index value. Note that this patch is only one half of the equation; it provides a way of storing FAISS inverted lists in RocksDB but there is currently no retrieval/search support (this will be a follow-up change). Also, the integration currently works only with our internal Buck build. I plan to add support for `cmake` / `make` based builds similarly to how we handle Folly. Reviewed By: jowlyzhang Differential Revision: D66907065 fbshipit-source-id: 63fdf29895d5feeffc230254a7ddfb0aac050967 |
||
|
Maciej Szeszko
|
be7703b27d |
Offline file checksum manifest retriever (#13178)
Summary: This change introduces a new, lightweight _experimental_ API that reconstructs the [file # -> file checksum -> file checksum function] 1-1-1 mapping directly from the `MANIFEST` file considered `CURRENT` in scope of specific DB instance at the time. The goal is to provide a cheap alternative to `DB::GetLiveFilesMetaData` that doesn't require opening the database, reconstructing version sets and/or accessing files that are _potentially_ in disaggregated storage. ### Housekeeping: 1. Moved the `GetCurrentManifestPath` out of `version_set` to a new `manifest_ops` file(s) dedicated to manifest related operations. 2. Introduced new `Env::IOActivity::kReadManifest` to better reflect the IO intent in offline file checksum retrieving function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13178 Test Plan: Added a unit test comparing the outcome of newly introduced API against the established `GetLiveFilesMetaData`: ```hcl ./db_test2 --gtest_filter="*GetFileChecksumsFromCurrentManifest_CRC32*" ``` Reviewed By: pdillinger Differential Revision: D66711910 Pulled By: mszeszko-meta fbshipit-source-id: 57091c550a14ac2e832bf7eea136dab5450e71bc |
||
|
Facebook GitHub Bot
|
10958102aa |
Re-sync with internal repository
The internal and external repositories are out of sync. This Pull Request attempts to brings them back in sync by patching the GitHub repository. Please carefully review this patch. You must disable ShipIt for your project in order to merge this pull request. DO NOT IMPORT this pull request. Instead, merge it directly on GitHub using the MERGE BUTTON. Re-enable ShipIt after merging. |