mirror of
https://github.com/facebook/rocksdb.git
synced 2026-07-07 14:47:40 +08:00
Resumable Remote Compaction Blog Post (#14759)
Summary: **Summary:** as titled Pull Request resolved: https://github.com/facebook/rocksdb/pull/14759 Test Plan: local server rendering test <img width="914" height="764" alt="Screenshot 2026-05-19 at 2 18 41 PM" src="https://github.com/user-attachments/assets/0e5353d4-df41-476c-8902-28d7975330d6" /> <img width="914" height="764" alt="Screenshot 2026-05-19 at 2 18 59 PM" src="https://github.com/user-attachments/assets/a0f8692d-6ca8-4c67-b7a0-b47146080c57" /> <img width="895" height="477" alt="Screenshot 2026-05-19 at 2 19 20 PM" src="https://github.com/user-attachments/assets/7479dd54-5bc4-41da-99f3-f081a714ba7c" /> Reviewed By: jaykorean Differential Revision: D105665739 Pulled By: hx235 fbshipit-source-id: 0701a627eb2b18b9bfd3dd22397aeae9553f1903
This commit is contained in:
Hui Xiao
committed by
meta-codesync[bot]
meta-codesync[bot]
parent
07a5a0a804
commit
367f2b0fdf
@@ -0,0 +1,55 @@
|
||||
---
|
||||
title: Resumable Remote Compaction
|
||||
layout: post
|
||||
author: hx235
|
||||
category: blog
|
||||
---
|
||||
|
||||
## Background
|
||||
|
||||
RocksDB can offload compaction work to remote workers through the `CompactionService` API. In this model, the **primary RocksDB instance** selects the input files and sends a serialized `CompactionServiceInput` to a worker; the **remote worker** runs `DB::OpenAndCompact()`, writes output SSTs to `output_directory`, and returns a serialized `CompactionServiceResult` that the primary RocksDB instance installs into its LSM tree. See the [Remote Compaction wiki](https://github.com/facebook/rocksdb/wiki/Remote-Compaction) for the full architecture. This lets operators scale compaction throughput with stateless workers while keeping the primary RocksDB instance's CPU and I/O available for serving reads and writes. However, remote compaction jobs can be long-running—sometimes processing hundreds of gigabytes of input. When a worker crashes, gets preempted, or times out, the entire compaction must restart from scratch, wasting all output produced before the interruption and increasing compaction debt on the primary RocksDB instance.
|
||||
|
||||
## How Resumable Remote Compaction Works
|
||||
|
||||
Resumable remote compaction introduces a **checkpoint-and-resume** mechanism. During a compaction, the worker periodically saves its progress to the `output_directory`. If the compaction is interrupted, a subsequent call to `OpenAndCompact()` with the same output directory can pick up from the last checkpoint rather than starting over.
|
||||
|
||||
### Checkpointing
|
||||
|
||||
After each output SST file is completed, the worker persists a progress checkpoint to a **compaction progress file** in the output directory `output_directory`. The checkpoint records which internal key to resume from and the metadata of all completed output files. Progress records use **delta encoding**—each record only contains files completed since the last checkpoint—to keep serialization cost linear.
|
||||
|
||||
|
||||
|
||||
The worker skips checkpointing at boundaries where resuming could be unsafe or requires complicated handling: when range deletions span the file boundary or when adjacent output files share the same user key. These constraints ensure that resuming produces the same results as if the compaction was not interrupted.
|
||||
|
||||
### Resuming
|
||||
|
||||
When `OpenAndCompact()` is called with `allow_resumption = true`, it scans the output directory for a valid progress file. If one is found, it loads the checkpointed state, seeks the input iterator to the recorded resume key, restores the output file state, and continues compaction from that point. If the progress file is corrupted or missing, the system falls back to a fresh compaction by cleaning the directory.
|
||||
|
||||
|
||||
|
||||
## How to Enable It
|
||||
|
||||
On the primary RocksDB instance, set a `CompactionService` implementation on the DB options. On the remote worker, pass `allow_resumption = true` in `OpenAndCompactOptions` when calling `DB::OpenAndCompact()`. The `output_directory` must be the same across retries for resumption to work—each retry call with the same directory will automatically detect and resume from the previous checkpoint. The `REMOTE_COMPACT_RESUMED_BYTES` statistics ticker tracks the total bytes of output files reused from a previous interrupted run, giving visibility into how much work resumption saved.
|
||||
|
||||
```cpp
|
||||
// Primary RocksDB instance
|
||||
DBOptions db_options;
|
||||
db_options.compaction_service = std::make_shared<MyCompactionService>();
|
||||
|
||||
// Remote worker
|
||||
OpenAndCompactOptions options;
|
||||
options.allow_resumption = true;
|
||||
|
||||
std::string result;
|
||||
Status s = DB::OpenAndCompact(
|
||||
options,
|
||||
db_path, // source database path
|
||||
output_directory, // where output SSTs and progress are stored
|
||||
compaction_input, // serialized CompactionServiceInput
|
||||
&result, // serialized CompactionServiceResult
|
||||
override_options);
|
||||
```
|
||||
|
||||
## Future Work
|
||||
|
||||
Today this feature targets remote compaction. The same checkpoint-and-resume mechanism could also support **local compaction** after a crash. The core persistence and resume logic is already in `CompactionJob`; the remaining work is to integrate it with local compaction scheduling and recovery.
|
||||
@@ -0,0 +1,65 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 390" style="background-color: #fafafa;">
|
||||
<defs>
|
||||
<marker id="cp-arrowhead" markerWidth="10" markerHeight="7" refX="10" refY="3.5" orient="auto">
|
||||
<polygon points="0 0, 10 3.5, 0 7" fill="#355070" />
|
||||
</marker>
|
||||
<filter id="cp-shadow" x="-20%" y="-20%" width="140%" height="140%">
|
||||
<feDropShadow dx="2" dy="2" stdDeviation="2" flood-color="#00000020"/>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<text x="500" y="36" font-family="Arial, sans-serif" font-size="22" font-weight="bold" text-anchor="middle" fill="#25364d">
|
||||
Checkpointing
|
||||
</text>
|
||||
|
||||
<g transform="translate(50,80)">
|
||||
<rect x="0" y="0" width="360" height="220" rx="18" fill="#eef5ff" stroke="#9cb9dd" stroke-width="2" filter="url(#cp-shadow)"/>
|
||||
<text x="180" y="38" font-family="Arial, sans-serif" font-size="18" font-weight="bold" text-anchor="middle" fill="#25364d">
|
||||
During DB::OpenAndCompact()
|
||||
</text>
|
||||
|
||||
<g transform="translate(28,86)">
|
||||
<rect x="0" y="0" width="90" height="72" rx="12" fill="#daf3e5" stroke="#4f9d69" stroke-width="2"/>
|
||||
<text x="45" y="30" font-family="Arial, sans-serif" font-size="15" font-weight="bold" text-anchor="middle" fill="#214e32">1.sst</text>
|
||||
<text x="45" y="51" font-family="Arial, sans-serif" font-size="12" text-anchor="middle" fill="#214e32">complete</text>
|
||||
</g>
|
||||
|
||||
<g transform="translate(136,86)">
|
||||
<rect x="0" y="0" width="90" height="72" rx="12" fill="#daf3e5" stroke="#4f9d69" stroke-width="2"/>
|
||||
<text x="45" y="30" font-family="Arial, sans-serif" font-size="15" font-weight="bold" text-anchor="middle" fill="#214e32">2.sst</text>
|
||||
<text x="45" y="51" font-family="Arial, sans-serif" font-size="12" text-anchor="middle" fill="#214e32">complete</text>
|
||||
</g>
|
||||
|
||||
<g transform="translate(244,86)">
|
||||
<rect x="0" y="0" width="90" height="72" rx="12" fill="#fff1d6" stroke="#d08a2e" stroke-width="2" stroke-dasharray="6 4"/>
|
||||
<text x="45" y="28" font-family="Arial, sans-serif" font-size="15" font-weight="bold" text-anchor="middle" fill="#7a5015">3.sst</text>
|
||||
<text x="45" y="49" font-family="Arial, sans-serif" font-size="12" text-anchor="middle" fill="#7a5015">in progress</text>
|
||||
</g>
|
||||
</g>
|
||||
|
||||
<line x1="432" y1="190" x2="558" y2="190" stroke="#355070" stroke-width="3" marker-end="url(#cp-arrowhead)"/>
|
||||
<text x="495" y="168" font-family="Arial, sans-serif" font-size="13" font-weight="bold" text-anchor="middle" fill="#25364d">
|
||||
After each completed SST
|
||||
</text>
|
||||
|
||||
<g transform="translate(590,80)">
|
||||
<rect x="0" y="0" width="360" height="240" rx="18" fill="#fff8e8" stroke="#d4a24a" stroke-width="2" filter="url(#cp-shadow)"/>
|
||||
<text x="180" y="38" font-family="Arial, sans-serif" font-size="18" font-weight="bold" text-anchor="middle" fill="#6e4a16">
|
||||
Compaction Progress File
|
||||
</text>
|
||||
<g transform="translate(26,76)">
|
||||
<rect x="0" y="0" width="308" height="72" rx="12" fill="#fffdf8" stroke="#d4a24a" stroke-width="1.5"/>
|
||||
<text x="154" y="20" font-family="Courier New, monospace" font-size="12" text-anchor="middle" fill="#4e3a19">record 1</text>
|
||||
<text x="154" y="38" font-family="Courier New, monospace" font-size="12" text-anchor="middle" fill="#4e3a19">resume_from = key_after_1.sst</text>
|
||||
<text x="154" y="56" font-family="Courier New, monospace" font-size="12" text-anchor="middle" fill="#4e3a19">new_files += [1.sst metadata]</text>
|
||||
</g>
|
||||
|
||||
<g transform="translate(26,158)">
|
||||
<rect x="0" y="0" width="308" height="72" rx="12" fill="#fffdf8" stroke="#d4a24a" stroke-width="1.5"/>
|
||||
<text x="154" y="20" font-family="Courier New, monospace" font-size="12" text-anchor="middle" fill="#4e3a19">record 2</text>
|
||||
<text x="154" y="38" font-family="Courier New, monospace" font-size="12" text-anchor="middle" fill="#4e3a19">resume_from = key_after_2.sst</text>
|
||||
<text x="154" y="56" font-family="Courier New, monospace" font-size="12" text-anchor="middle" fill="#4e3a19">new_files += [2.sst metadata]</text>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.0 KiB |
@@ -0,0 +1,97 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 720" style="background-color: #fafafa;">
|
||||
<defs>
|
||||
<marker id="rf-arrowhead" markerWidth="10" markerHeight="7" refX="10" refY="3.5" orient="auto">
|
||||
<polygon points="0 0, 10 3.5, 0 7" fill="#355070" />
|
||||
</marker>
|
||||
<filter id="rf-shadow" x="-20%" y="-20%" width="140%" height="140%">
|
||||
<feDropShadow dx="2" dy="2" stdDeviation="2" flood-color="#00000020"/>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<text x="500" y="36" font-family="Arial, sans-serif" font-size="22" font-weight="bold" text-anchor="middle" fill="#25364d">
|
||||
Resuming a Remote Compaction
|
||||
</text>
|
||||
|
||||
<g transform="translate(275,70)">
|
||||
<rect x="0" y="0" width="450" height="64" rx="18" fill="#edf4ff" stroke="#9cb9dd" stroke-width="2" filter="url(#rf-shadow)"/>
|
||||
<text x="225" y="28" font-family="Arial, sans-serif" font-size="18" font-weight="bold" text-anchor="middle" fill="#25364d">
|
||||
OpenAndCompact(allow_resumption = true)
|
||||
</text>
|
||||
<text x="225" y="48" font-family="Arial, sans-serif" font-size="12" text-anchor="middle" fill="#5b6573">
|
||||
Retry with the same output_directory
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<line x1="500" y1="134" x2="500" y2="180" stroke="#355070" stroke-width="3" marker-end="url(#rf-arrowhead)"/>
|
||||
|
||||
<g transform="translate(275,190)">
|
||||
<rect x="0" y="0" width="450" height="78" rx="18" fill="#edf4ff" stroke="#9cb9dd" stroke-width="2" filter="url(#rf-shadow)"/>
|
||||
<text x="225" y="30" font-family="Arial, sans-serif" font-size="18" font-weight="bold" text-anchor="middle" fill="#25364d">
|
||||
Scan output_directory
|
||||
</text>
|
||||
<text x="225" y="52" font-family="Arial, sans-serif" font-size="12" text-anchor="middle" fill="#5b6573">
|
||||
Inspect the progress file and any existing SST files
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<line x1="500" y1="268" x2="500" y2="316" stroke="#355070" stroke-width="3" marker-end="url(#rf-arrowhead)"/>
|
||||
|
||||
<polygon points="500,316 650,386 500,456 350,386" fill="#fff8e8" stroke="#d4a24a" stroke-width="2" filter="url(#rf-shadow)"/>
|
||||
<text x="500" y="392" font-family="Arial, sans-serif" font-size="18" font-weight="bold" text-anchor="middle" fill="#6e4a16">
|
||||
Valid progress file?
|
||||
</text>
|
||||
|
||||
<line x1="350" y1="386" x2="220" y2="386" stroke="#355070" stroke-width="3" marker-end="url(#rf-arrowhead)"/>
|
||||
<text x="308" y="372" font-family="Arial, sans-serif" font-size="13" font-weight="bold" text-anchor="middle" fill="#25364d">No</text>
|
||||
|
||||
<g transform="translate(40,344)">
|
||||
<rect x="0" y="0" width="240" height="72" rx="18" fill="#fff0f0" stroke="#d97070" stroke-width="2" filter="url(#rf-shadow)"/>
|
||||
<text x="120" y="30" font-family="Arial, sans-serif" font-size="17" font-weight="bold" text-anchor="middle" fill="#7f2323">
|
||||
Start fresh
|
||||
</text>
|
||||
<text x="120" y="50" font-family="Arial, sans-serif" font-size="12" text-anchor="middle" fill="#7f2323">
|
||||
Clean output_directory
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<line x1="650" y1="386" x2="780" y2="386" stroke="#355070" stroke-width="3" marker-end="url(#rf-arrowhead)"/>
|
||||
<text x="694" y="372" font-family="Arial, sans-serif" font-size="13" font-weight="bold" text-anchor="middle" fill="#25364d">Yes</text>
|
||||
|
||||
<g transform="translate(720,310)">
|
||||
<rect x="0" y="0" width="230" height="106" rx="18" fill="#eef8ef" stroke="#7db28e" stroke-width="2" filter="url(#rf-shadow)"/>
|
||||
<text x="115" y="30" font-family="Arial, sans-serif" font-size="17" font-weight="bold" text-anchor="middle" fill="#24503a">
|
||||
Resume from checkpoint
|
||||
</text>
|
||||
<text x="115" y="52" font-family="Arial, sans-serif" font-size="12" text-anchor="middle" fill="#24503a">
|
||||
Load progress state
|
||||
</text>
|
||||
<text x="115" y="70" font-family="Arial, sans-serif" font-size="12" text-anchor="middle" fill="#24503a">
|
||||
Seek the iterator to the resume key
|
||||
</text>
|
||||
<text x="115" y="88" font-family="Arial, sans-serif" font-size="12" text-anchor="middle" fill="#24503a">
|
||||
Restore output file state
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<line x1="160" y1="416" x2="160" y2="560" stroke="#355070" stroke-width="3" marker-end="url(#rf-arrowhead)"/>
|
||||
<line x1="835" y1="416" x2="835" y2="560" stroke="#355070" stroke-width="3" marker-end="url(#rf-arrowhead)"/>
|
||||
<line x1="160" y1="560" x2="355" y2="560" stroke="#355070" stroke-width="3"/>
|
||||
<line x1="835" y1="560" x2="645" y2="560" stroke="#355070" stroke-width="3"/>
|
||||
|
||||
<g transform="translate(355,526)">
|
||||
<rect x="0" y="0" width="290" height="50" rx="18" fill="#edf4ff" stroke="#9cb9dd" stroke-width="2" filter="url(#rf-shadow)"/>
|
||||
<text x="145" y="31" font-family="Arial, sans-serif" font-size="18" font-weight="bold" text-anchor="middle" fill="#25364d">
|
||||
Continue compaction
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<line x1="500" y1="576" x2="500" y2="646" stroke="#355070" stroke-width="3" marker-end="url(#rf-arrowhead)"/>
|
||||
|
||||
<g transform="translate(310,656)">
|
||||
<rect x="0" y="0" width="380" height="44" rx="16" fill="#eef8ef" stroke="#7db28e" stroke-width="2" filter="url(#rf-shadow)"/>
|
||||
<text x="190" y="28" font-family="Arial, sans-serif" font-size="17" font-weight="bold" text-anchor="middle" fill="#24503a">
|
||||
Return CompactionServiceResult
|
||||
</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.1 KiB |
Reference in New Issue
Block a user