Files
Peter Dillinger 9f474a1034 Enforce < 4GB sizes for keys, values, and some BlockBuilder inputs (#14461)
Summary:
BlockBuilder was somewhat inconsistent in its treatment of Slices whose size exceeds 4GB, which in a random corruption case could lead to (for example) serializing only the bottom 32 bits of a value size (uncorrupted) but appending the full multi-GB (corrupted) size. Because this is inner loop code, we don't want to pay CPU for extra conversions, data movements, or Status plumbing (already ruled out by Xingbo).

In this change we make BlockBuilder more internally consistent and lift the 32-bit size requirement to callers (of a few functions specifically, for now). To ensure that's satisfied, I've added additional checks near the perimeter of RocksDB to ensure keys and values do not exceed 4GB, plus an extra random corruption or backstop check in BlockBasedTableBuilder. In detail,

BlockBuilder (block_builder.cc/h):
* Add API comments documenting the < 4GB assumption on Add/AddWithLastKey
* Add debug assertions verifying input slice sizes < 4GB
* Simplify AddWithLastKeyImpl to use uint32_t locals, reducing static_cast
* Use only bottom 32 bits when appending derivative slices for consistency
* Update MaybeStripTimestampFromKey to also truncate to 32-bit size
* Add FIXME comments where buffer_/values_buffer_ sizes are truncated

BlockBasedTableBuilder (block_based_table_builder.cc):
* Add value size check (> uint32_t max) as a safety net against random corruption, since we have seen such corruptions in production

WriteBatch (write_batch.cc):
* Tighten existing key size checks to account for kNumInternalBytes (8-byte internal key suffix), using new kMaxWriteBatchKeySize constant
* Add missing size checks to Delete, SingleDelete, and DeleteRange (both Slice and SliceParts variants)

SstFileWriter (sst_file_writer.cc):
* Add key and value size checks in AddImpl and DeleteRangeImpl, which bypass WriteBatch and go directly to the table builder

MergeHelper (merge_helper.cc):
* Add merge result size check (> uint32_t max) in both overloads of TimedFullMergeImpl, returning Corruption if exceeded
* Add PartialMergeMulti result size check in MergeUntil

CompactionIterator (compaction_iterator.cc):
* Add size check on compaction filter output values (kChangeValue and kChangeWideColumnEntity), returning Corruption if > 4GB

meta_blocks.cc:
* Use Slice with uint32_t-truncated sizes in PropertyBlockBuilder::Finish to handle potentially oversized user property collector output

Needed follow-up:
* Check for blocks that are mis-encoded due to overflowing 32 bits for restart point offsets (and similar). See FIXME comments in block_builder.cc for why this is tricky.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14461

Test Plan:
New extreme-size unit tests, along with some testing infrastructure improvements:

Added test::HasBigMem() (in test_util/testharness.h) which returns true when the system has ≥128GB RAM (via sysconf(_SC_PHYS_PAGES) on Linux/macOS) or when ROCKSDB_BIGMEM_TESTS is set. All extreme-size tests use HasBigMem() to skip gracefully on smaller machines rather than being permanently disabled.

Where possible, tests use MemMapping::AllocateLazyZeroed() to supply large keys/values as Slices backed by anonymous mmap (cleaner with new MemMapping::AsSlice()). On Linux, read-only access to these pages maps to the shared kernel zero page, so the source data consumes no physical RAM — only the destination copy (e.g., WriteBatch::rep_) materializes, cutting peak memory roughly in half vs. std::string.

Tests (all enabled, skip via HasBigMem()):

./write_batch_test --gtest_filter='*LargeKeyValueSizeLimit*'
./external_sst_file_basic_test --gtest_filter='*LargeSizeSstFileWriter*'
./merge_test --gtest_filter='*LargeMergeResultRejected*'
./merge_helper_test --gtest_filter='*LargePartialMergeResultRejected*'
./db_compaction_test --gtest_filter='*CompactionFilterLargeValueRejected*'

All 5 pass (verified on a 128GB+ machine). On smaller machines, all 5 bypass cleanly with "insufficient memory for reliable continuous testing".

Reviewed By: xingbowang

Differential Revision: D96521899

Pulled By: pdillinger

fbshipit-source-id: 70f4b5e6a23ab074d60e653fbb7ddc5edbe162ab
2026-04-10 17:04:47 -07:00

168 lines
5.5 KiB
C++

// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "test_util/testharness.h"
#include <cstdlib>
#include <regex>
#include <string>
#include <thread>
#ifndef OS_WIN
#include <unistd.h>
#endif
#ifndef NDEBUG
#include "test_util/sync_point.h"
#endif
namespace {
#ifndef NDEBUG
// Global gtest event listener that cleans up SyncPoint state after every
// test. Many tests set SyncPoint callbacks with captured local variables
// but forget to disable/clear them. Under sharded execution (multiple
// tests per process), stale callbacks cause segfaults or corruption.
class SyncPointCleanupListener : public ::testing::EmptyTestEventListener {
void OnTestEnd(const ::testing::TestInfo& /*test_info*/) override {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearTrace();
// LoadDependency({}) clears successors_, predecessors_, and
// cleared_points_ maps. Without this, stale dependencies from a
// previous test can block SyncPoint::Process() in the next test
// (e.g. a background compaction thread hitting CompactFilesImpl:1
// whose predecessor was never fired).
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({});
}
};
// Auto-register the listener via static initialization.
// This runs before main() and before any test fixtures are constructed.
static int RegisterSyncPointCleanup() noexcept {
::testing::TestEventListeners& listeners =
::testing::UnitTest::GetInstance()->listeners();
listeners.Append(new SyncPointCleanupListener());
return 0;
}
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
[[maybe_unused]] static int sync_point_cleanup_registered_ =
RegisterSyncPointCleanup();
#endif // !NDEBUG
} // namespace
namespace ROCKSDB_NAMESPACE::test {
#ifdef OS_WIN
#include <windows.h>
std::string GetPidStr() { return std::to_string(GetCurrentProcessId()); }
#else
std::string GetPidStr() { return std::to_string(getpid()); }
#endif
::testing::AssertionResult AssertStatus(const char* s_expr, const Status& s) {
if (s.ok()) {
return ::testing::AssertionSuccess();
} else {
return ::testing::AssertionFailure() << s_expr << std::endl << s.ToString();
}
}
std::string TmpDir(Env* env) {
std::string dir;
Status s = env->GetTestDirectory(&dir);
EXPECT_OK(s);
return dir;
}
std::string PerThreadDBPath(std::string dir, std::string name) {
size_t tid = std::hash<std::thread::id>()(std::this_thread::get_id());
return dir + "/" + name + "_" + GetPidStr() + "_" + std::to_string(tid);
}
std::string PerThreadDBPath(std::string name) {
return PerThreadDBPath(test::TmpDir(), name);
}
std::string PerThreadDBPath(Env* env, std::string name) {
return PerThreadDBPath(test::TmpDir(env), name);
}
int RandomSeed() {
const char* env = getenv("TEST_RANDOM_SEED");
int result = (env != nullptr ? atoi(env) : 301);
if (result <= 0) {
result = 301;
}
return result;
}
bool HasBigMem() {
const char* env = getenv("ROCKSDB_BIGMEM_TESTS");
if (env != nullptr && env[0] != '\0') {
return true;
}
#ifdef _SC_PHYS_PAGES
// Check whether the system has at least 128GB of physical RAM.
// _SC_PHYS_PAGES is available on Linux and macOS but is not standard POSIX.
long pages = sysconf(_SC_PHYS_PAGES);
long page_size = sysconf(_SC_PAGE_SIZE);
if (pages > 0 && page_size > 0) {
size_t total_bytes =
static_cast<size_t>(pages) * static_cast<size_t>(page_size);
return total_bytes >= size_t{128} << 30;
}
#endif
return false;
}
TestRegex::TestRegex(const std::string& pattern)
: impl_(std::make_shared<Impl>(pattern)), pattern_(pattern) {}
TestRegex::TestRegex(const char* pattern)
: impl_(std::make_shared<Impl>(pattern)), pattern_(pattern) {}
const std::string& TestRegex::GetPattern() const { return pattern_; }
class TestRegex::Impl : public std::regex {
public:
using std::regex::basic_regex;
};
bool TestRegex::Matches(const std::string& str) const {
if (impl_) {
return std::regex_match(str, *impl_);
} else {
// Should not call Matches on unset Regex
assert(false);
return false;
}
}
::testing::AssertionResult AssertMatchesRegex(const char* str_expr,
const char* pattern_expr,
const std::string& str,
const TestRegex& pattern) {
if (pattern.Matches(str)) {
return ::testing::AssertionSuccess();
} else if (TestRegex("\".*\"").Matches(pattern_expr)) {
// constant regex string
return ::testing::AssertionFailure()
<< str << " (" << str_expr << ")" << std::endl
<< "does not match regex " << pattern.GetPattern();
} else {
// runtime regex string
return ::testing::AssertionFailure()
<< str << " (" << str_expr << ")" << std::endl
<< "does not match regex" << std::endl
<< pattern.GetPattern() << " (" << pattern_expr << ")";
}
}
} // namespace ROCKSDB_NAMESPACE::test