Hui Xiao
1e5fa69c99
Summary: ### Context/Summary: Flow of resuming: DB::OpenAndCompact() -> Compaction progress file -> SubcompactionProgress -> CompactionJob Flow of persistence: CompactionJob -> SubcompactionProgress -> Compaction progress file -> DB that is called with OpenAndCompact() This PR focuses on SubcompactionProgress -> CompactionJob and CompactionJob -> SubcompactionProgress -> Compaction progress file. For now only single subcompaction is supported as OpenAndCompact() does not partition compaction anyway. The actual triggering of progress persistence and resuming (i.e, integration) is through DB::OpenAndCompact() in the upcoming PR. **Resume Flow** 1. input_iter->Seek(next_internal_key_to_compact) // Position iterator 2. ReadTableProperties() // Validate existing outputs 3. RestoreCompactionOutputs() in CompactionOutputs // Rebuild output file metadata 4. Restore critical statistics about processed input and output records count for verification later 5. AdvanceFileNumbers() // Prevent file number conflicts 6. Continue normal compaction from positioned iterator or fallback to not resuming compaction in limited case or fail the compaction entirely **Persistence Strategy** 1. When: At each SST file completion (FinishCompactionOutputFile()). This is the simplest but most expensive frequency. See below for benchmarking and potential follow-up items 2. What: Serialize, write and sync the in-memory SubcompactionProgress to a dedicated manifest-like file 3. For simplicity: Only persist at "clean" boundaries (no overlapping user keys, no range deletions, no timestamp for now) Pull Request resolved: https://github.com/facebook/rocksdb/pull/13983 Test Plan: - New unit test in CompactionJob level to cover basic compaction progress resumption - Existing UTs and stress/crash test to test no correctness regression to existing compaction code - Run benchmark to ensure no performance regression to existing compaction code ``` ./db_bench --benchmarks=fillseq[-X10] --db=$db --disable_auto_compactions=true --num=100000 --value_size=25000 --compression_type=none --target_file_size_base=268435456 --write_buffer_size=268435456 ``` Pre-PR: fillseq [AVG 10 runs] : 45127 (± 799) ops/sec; 1076.6 (± 19.1) MB/sec fillseq [MEDIAN 10 runs] : 45375 ops/sec; 1082.5 MB/sec Post-PR (regressed 0.057%, ignorable) fillseq [AVG 10 runs] : 45101 (± 920) ops/sec; 1076.0 (± 22.0) MB/sec fillseq [MEDIAN 10 runs] : 45385 ops/sec; 1082.8 MB/sec Reviewed By: jaykorean Differential Revision: D82889188 Pulled By: hx235 fbshipit-source-id: 8553fd478f134969d331af2c5a125b94bd747268
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.
Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples
See the github wiki for more explanation.
The public interface is in include/. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.
License
RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.