Files
rocksdb/buckifier/targets_builder.py
T
zaidoon 1cec28d82d Finish C API code generation (continues #14572) (#14868)
Summary:
This continues and finishes **https://github.com/facebook/rocksdb/issues/14572** ("Add semi-automated code generation for RocksDB C API bindings") by xingbowang. The original author is unavailable to finish it, so I've taken it over. **All 13 of the original commits are preserved** (this branch was created from the PR head and builds on top of it — `git log` shows the original `Xingbo Wang` authorship intact); my follow-up work is in the commits prefixed `C API codegen:`.

The underlying design is unchanged and is the original author's: hand-written source templates (`tools/c_api_gen/c_base.h` / `c_base.cc`) plus two generators (auto-discovery from the C++ headers + a spec-driven generator) are inlined into a single, self-contained, `generated` `include/rocksdb/c.h` and `db/c.cc`. This grows the public C API by **668 functions** while keeping `c.h` a single includable header with no `-I` requirement (so `bindgen` and other FFI tools keep working unchanged).

This branch reconciles the PR with ~4 months of `main` and addresses the outstanding review feedback (clang-tidy bot, the automated code review, the `c.h` self-containedness discussion, and pdillinger's points about `include/rocksdb` hygiene and `generated` marking).

## What changed on top of the original PR

### Reconciled with current `main`
- Merged current `main` (conflicts were confined to the generated/test files) and regenerated. Reconciled the 14 C API functions `main` added since the merge-base (e.g. `rocksdb_set_db_options`, the backup-engine rate limiters, `memtable_batch_lookup_optimization`, `optimize_multiget_for_io`, …) and restored 5 enum constants that upstream had added by hand (`rocksdb_txndb_write_policy_*`, `..._index_block_search_type_auto`, `rocksdb_blob_cache_read_byte`).

### Maintainer feedback (pdillinger)
- **No non-user-includable files in `include/rocksdb`.** Moved the hand-written templates out of `include/rocksdb/` and `db/` to `tools/c_api_gen/c_base.{h,cc}`. They were `#include`-ing generated fragments, which broke `make check-headers` and was shipped by `make install`. `include/rocksdb/` now contains only the user-facing, self-contained, `generated` `c.h`.
- `c.h` / `c.cc` carry the `// generated` marker.

### Backward compatibility (zero ABI break)
- The generator derived each wrapper's C type purely from the C++ field, which had silently changed **5 already-shipped signatures** (e.g. `rocksdb_writeoptions_disable_WAL` `int` → `unsigned char`). Added an ABI type-pinning layer (`tools/c_api_gen/abi_type_overrides.json`) so already-shipped functions keep their historical C signature (the body still casts to the real field type). A repo-wide diff against the merge-base now reports **0 ABI drift**.
- New `check_api_compatibility.py` gate (wired into CI + `make`) fails on any removed/changed public function **or** removed enum/typedef symbol, vs a reference revision. Intentional changes go in an allowlist with a reason.

### Correctness (from the automated review)
- Restored 5 option setters that were declared in `c.h` but **defined nowhere** (link failure for downstream bindings such as `rust-rocksdb`). Added `check_api_completeness.py` (dependency-free; runs in CI + `make`) asserting every declared function has exactly one definition — this is the gate that would have caught it.
- `CopyStringVector` now null-checks `malloc`; the WAL filter `std::move`s the `WriteBatch`; the backup exclude-files callback captures by value instead of the wrapper pointer.

### Build / CI robustness
- Removed the dead `C_API_CODEGEN_STAMP` Makefile prerequisite (it was a silent no-op).
- The `make check` staleness check is now opt-out-able (behind `SKIP_FORMAT_BUCK_CHECKS`) and skips gracefully when `clang++` is unavailable, so `make check` works without the codegen toolchain. CI remains the authoritative gate.
- Pinned `clang-format` consistently through `regen_all.py` / `verify_generated_up_to_date.py` (CI uses clang-format-21) so regeneration is byte-reproducible across environments.
- Cleared all 20 `clang-tidy` warnings the bot reported on `db/c.cc` changed lines (fixed in the `c_base.cc` template, not the generated output).
- Updated the internal Buck `c_test_bin` wrapper to expose generated `c_api_gen/*.inc` fragments as headers, so sandboxed Buck builds can compile `db/c_test.c` after the generated round-trip tests are included.

### Test coverage
- Added `gen_roundtrip_tests.py`, which derives **462 set→get→assert round-trip checks across 25 option objects** from the same generated fragments and wires them into `db/c_test.c`. Coverage now tracks the generated surface automatically.

### Docs
- Added the `unreleased_history/public_api_changes` note and fixed `claude_md/add_public_api.md`, which still told contributors to hand-edit the now-`generated` `c.h`/`c.cc`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14868

Test Plan:
- `make c_test && ./c_test` — **passes**, including the 462 generated round-trip assertions (a successful link also confirms the API is complete).
- `python3 tools/c_api_gen/check_api_completeness.py` — all 1737 declared functions defined exactly once.
- `python3 tools/c_api_gen/check_api_compatibility.py --ref <release>` — 1070 reference functions + 229 enum/typedef symbols preserved, 0 removed/changed.
- `python3 tools/c_api_gen/verify_generated_up_to_date.py` — generated output is stable.
- `include/rocksdb/c.h` confirmed self-contained (only `<stdbool.h>`, `<stddef.h>`, `<stdint.h>`).

cc xingbowang

- `buck2 build --flagfile fbcode//mode/dev fbcode//internal_repo_rocksdb/repo:c_test_bin` — passes.
- `buck2 build --flagfile fbcode//mode/dev --config fbcode.arch=aarch64 fbcode//internal_repo_rocksdb/repo:c_test_bin` — passes.

Reviewed By: pdillinger

Differential Revision: D109149150

Pulled By: xingbowang

fbshipit-source-id: 3417375345f360a4c78bdfe27e9850b89d0a226a
2026-06-24 10:45:42 -07:00

182 lines
5.9 KiB
Python

# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
try:
from builtins import object, str
except ImportError:
from __builtin__ import object, str
import pprint
import targets_cfg
class LiteralValue:
def __init__(self, value):
self.value = value
def __str__(self):
return str(self.value)
def smart_quote_value(val):
if isinstance(val, LiteralValue):
return str(val)
return '"%s"' % val
def pretty_list(lst, indent=8):
if lst is None or len(lst) == 0:
return ""
if len(lst) == 1:
return smart_quote_value(lst[0])
separator = ',\n%s' % (" " * indent)
res = separator.join(sorted(map(smart_quote_value, lst)))
res = "\n" + (" " * indent) + res + ',\n' + (" " * (indent - 4))
return res
class TARGETSBuilder:
def __init__(self, path, extra_argv):
self.path = path
header = targets_cfg.rocksdb_target_header_template.format(
extra_argv=extra_argv
)
with open(path, "wb") as targets_file:
targets_file.write(header.encode("utf-8"))
self.total_lib = 0
self.total_bin = 0
self.total_test = 0
self.tests_cfg = ""
def add_oncall(self, oncall):
with open(self.path, "ab") as targets_file:
targets_file.write(targets_cfg.oncall_template.format(name=oncall).encode("utf-8"))
def add_library(
self,
name,
srcs,
deps=None,
headers=None,
extra_external_deps="",
link_whole=False,
external_dependencies=None,
extra_test_libs=False,
):
if headers is not None:
if isinstance(headers, LiteralValue):
headers = str(headers)
else:
headers = "[" + pretty_list(headers) + "]"
else:
headers = "[]"
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.library_template.format(
name=name,
srcs=pretty_list(srcs),
headers=headers,
deps=pretty_list(deps),
extra_external_deps=extra_external_deps,
link_whole=link_whole,
external_dependencies=pretty_list(external_dependencies),
extra_test_libs=extra_test_libs,
).encode("utf-8")
)
self.total_lib = self.total_lib + 1
def add_rocksdb_library(self, name, srcs, headers=None, external_dependencies=None):
headers = "[" + pretty_list(headers) + "]"
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.rocksdb_library_template.format(
name=name,
srcs=pretty_list(srcs),
headers=headers,
external_dependencies=pretty_list(external_dependencies),
).encode("utf-8")
)
self.total_lib = self.total_lib + 1
def add_binary(
self,
name,
srcs,
deps=None,
extra_preprocessor_flags=None,
extra_bench_libs=False,
):
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.binary_template.format(
name=name,
srcs=pretty_list(srcs),
deps=pretty_list(deps),
extra_preprocessor_flags=pretty_list(extra_preprocessor_flags),
extra_bench_libs=extra_bench_libs,
).encode("utf-8")
)
self.total_bin = self.total_bin + 1
def add_c_test(self):
# The actual c_test_bin target is defined by add_c_test_wrapper in the
# internal //rocks/buckifier:defs.bzl (not in this OSS repo). db/c_test.c
# #includes the generated c_api_gen/*.inc fragments, so under Buck's
# hermetic sandbox that wrapper must expose them as headers, e.g.
# headers = native.glob(["c_api_gen/**/*.inc"])
# (Make/CMake resolve the include via -I. / PROJECT_SOURCE_DIR.)
with open(self.path, "ab") as targets_file:
targets_file.write(
b"""
add_c_test_wrapper()
"""
)
def add_test_header(self):
with open(self.path, "ab") as targets_file:
targets_file.write(
b"""
# Generate a test rule for each entry in ROCKS_TESTS
# Do not build the tests in opt mode, since SyncPoint and other test code
# will not be included.
"""
)
def add_fancy_bench_config(
self,
name,
bench_config,
slow,
expected_runtime,
sl_iterations,
regression_threshold,
):
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.fancy_bench_template.format(
name=name,
bench_config=pprint.pformat(bench_config),
slow=slow,
expected_runtime=expected_runtime,
sl_iterations=sl_iterations,
regression_threshold=regression_threshold,
).encode("utf-8")
)
def register_test(self, test_name, src, deps, extra_compiler_flags):
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.unittests_template.format(
test_name=test_name,
test_cc=str(src),
deps=deps,
extra_compiler_flags=extra_compiler_flags,
).encode("utf-8")
)
self.total_test = self.total_test + 1
def export_file(self, name):
with open(self.path, "a") as targets_file:
targets_file.write(
targets_cfg.export_file_template.format(name=name)
)