mirror of
https://github.com/facebook/rocksdb.git
synced 2026-07-07 14:47:40 +08:00
f6fabdb64f
Summary: Local Make builds silently reuse object files even when build parameters (DEBUG_LEVEL, sanitizers, ASSERT_STATUS_CHECKED, RTTI, etc.) change, since all object files share the same paths. This leads to confusing linker errors, sanitizer false negatives, ODR violations, and "phantom" bugs that are easy to miss -- a pitfall for humans and especially for AI agents driving builds non-interactively. This change improves the Make experience on three fronts: 1. Build-parameter change detection in the Makefile. After flags are fully resolved, we hash the effective compile/link inputs (CC|CXX|CFLAGS|CXXFLAGS|LDFLAGS|EXEC_LDFLAGS) into a per-OBJ_DIR .build_signature stamp and compare against the previous build: - Default: stop with a clear error when parameters changed. This is optimized for the expert who might not intend a full rebuild, and for avoiding CI inefficiency. - AUTO_CLEAN=1: automatically run clean-rocks and proceed. - ALLOW_BUILD_PARAMETER_CHANGE=1: skip the check (e.g. intentionally mixing DEBUG_LEVEL=1 and DEBUG_LEVEL=2 objects). The check is skipped for dry runs (-n) and for non-building / self-cleaning targets (clean*, format, check-*, tags, gen-pc, check-progress, watch-log, release, coverage, the asan_*/ubsan_* variants, etc.). make_config.mk is removed only by the top-level `clean` target, not by clean-rocks, so the auto-clean does not delete the make_config.mk already included by the running make. The signature is removed last in clean-rocks so an interrupted clean keeps change detection meaningful. 3. New build_tools/rockstest.sh and build_tools/rocksptest.sh helpers that build and run unit tests in one step. They set AUTO_CLEAN=1 and build with -j<NCORES> (computed like the Makefile). rocksptest.sh runs one or more binaries under the checked-in gtest-parallel (sharing one parallel worker pool); rockstest.sh runs a single binary directly for a small number of cases. Both reject a leading-option first argument; `rockstest.sh install` writes ~/bin/rockstest and ~/bin/rocksptest shims that defer to the in-tree scripts (with a friendly message when run outside a source root). This is a big win for AI-agent workflows: one command instead of a separate `make` then test-run invocation, removing the overhead of monitoring two long commands, while avoiding serial builds and serial test execution. 4. CLAUDE.md guidance updated to recommend AUTO_CLEAN=1 for manual make invocations and the rocks(p)test.sh helpers for running tests Although there might be drive to consolidate around the BUCK build for internal use, I'll note that last I checked it was around ~20 minutes to run all the unit tests under buck and ~2 minutes under `make check`. Bonus: the first revision of this had copyright/license mistakes, and this change corrects similar errors elsewhere, with CLAUDE.md advice updated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14883 Test Plan: Manual verification of the Makefile change detection: - Confirmed the resolved-flags hash is deterministic across runs for the same goal, and differs between configs (e.g. default/shared vs static_lib, and with/without ASSERT_STATUS_CHECKED). - Unchanged parameters: rebuild does not error. - Changed parameters: errors by default; AUTO_CLEAN=1 runs clean-rocks and proceeds; ALLOW_BUILD_PARAMETER_CHANGE=1 bypasses the check. - Dry runs (make -n) and exempt targets (e.g. `make check-progress`, list_all_tests, gen-pc) do not trip the check and leave the stamp untouched. - Reproduced the ASSERT_STATUS_CHECKED=1 auto-clean path end to end and confirmed it no longer fails with "No rule to make target 'make_config.mk'"; verified `make clean` removes make_config.mk while `clean-rocks` does not, and that the signature is deleted as the final clean-rocks step. Manual verification of the helper scripts: - shellcheck-clean and `bash -n` pass for both scripts. - Usage/guard behavior: missing argument and leading-option first argument are rejected with a clear message. - `rockstest.sh install` (validated against a temp HOME) generates both shims; running a shim outside a source root prints "(Not in a rocksdb source root directory?)" and exits non-zero. - rocksptest.sh argument splitting verified for single binary, multiple binaries, and binaries followed by gtest-parallel/test args. - .gitignore patterns verified with `git check-ignore` for .build_signature (root and jl/jls) and build_tools/__pycache__/. Reviewed By: xingbowang Differential Revision: D109713512 Pulled By: pdillinger fbshipit-source-id: a801d29afa91f53a2dce717db9fe5e5d485cc154
77 lines
2.6 KiB
Bash
Executable File
77 lines
2.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Copyright (c) Meta Platforms, Inc. and affiliates.
|
|
# This source code is licensed under both the GPLv2 (found in the
|
|
# COPYING file in the root directory) and Apache 2.0 License
|
|
# (found in the LICENSE.Apache file in the root directory).
|
|
#
|
|
# Build a single RocksDB unit test binary and run it directly (serially).
|
|
# Only recommended with --gtest_filter=... because of speed. See also
|
|
# build_tools/rocksptest.sh
|
|
#
|
|
# Hardened version of a simple `make <bin> && ./<bin>` helper:
|
|
# * AUTO_CLEAN=1 so the Makefile automatically runs a clean when the build
|
|
# parameters (DEBUG_LEVEL, sanitizers, ASSERT_STATUS_CHECKED, RTTI, etc.)
|
|
# have changed since the last build, preventing stale/mixed object files.
|
|
# * Parallel build with -j<NCORES>, computed the same way the Makefile does.
|
|
# * `set -euo pipefail` so any failure stops the script.
|
|
#
|
|
# Run from the repository root.
|
|
#
|
|
# Example usage:
|
|
# build_tools/rockstest.sh db_test --gtest_filter=*MixedSlowdown*
|
|
# build_tools/rockstest.sh env_test # Slow to run many tests serially
|
|
#
|
|
# Install mode:
|
|
# build_tools/rockstest.sh install
|
|
# Creates ~/bin/rockstest and ~/bin/rocksptest shims that defer to
|
|
# build_tools/rockstest.sh / rocksptest.sh in whatever directory you run
|
|
# them from, so you can just type `rockstest db_test` from any rocksdb
|
|
# source root.
|
|
|
|
set -euo pipefail
|
|
|
|
# Install mode: write thin ~/bin shims that defer to the build_tools scripts in
|
|
# the current directory.
|
|
if [ "${1:-}" = "install" ]; then
|
|
mkdir -p "$HOME/bin"
|
|
for name in rockstest rocksptest; do
|
|
dest="$HOME/bin/$name"
|
|
cat > "$dest" <<EOF
|
|
#!/usr/bin/env bash
|
|
# Auto-generated by 'build_tools/rockstest.sh install'. Defers to
|
|
# build_tools/$name.sh in the current rocksdb source root.
|
|
if [ -x build_tools/$name.sh ]; then
|
|
exec build_tools/$name.sh "\$@"
|
|
else
|
|
echo "build_tools/$name.sh not found (Not in a rocksdb source root directory?)" >&2
|
|
exit 1
|
|
fi
|
|
EOF
|
|
chmod +x "$dest"
|
|
echo "Installed $dest"
|
|
done
|
|
exit 0
|
|
fi
|
|
|
|
if [ "$#" -lt 1 ]; then
|
|
echo "usage: $0 <test_binary> [test args...]" >&2
|
|
echo " $0 install # create ~/bin/rockstest and ~/bin/rocksptest shims" >&2
|
|
echo "example: $0 db_test --gtest_filter=*MixedSlowdown*" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# First argument must be a test binary, not an option (catches a forgotten name).
|
|
if [ "${1#-}" != "$1" ]; then
|
|
echo "$0: first argument must be a test binary name, not an option ('$1')" >&2
|
|
exit 1
|
|
fi
|
|
|
|
BIN=$1
|
|
shift
|
|
|
|
# Compute parallelism the same way the Makefile computes NCORES.
|
|
NCORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
|
|
|
|
make AUTO_CLEAN=1 -j"$NCORES" "$BIN"
|
|
./"$BIN" "$@"
|