mirror of
https://github.com/facebook/rocksdb.git
synced 2026-07-07 14:47:40 +08:00
9f474a1034
Summary: BlockBuilder was somewhat inconsistent in its treatment of Slices whose size exceeds 4GB, which in a random corruption case could lead to (for example) serializing only the bottom 32 bits of a value size (uncorrupted) but appending the full multi-GB (corrupted) size. Because this is inner loop code, we don't want to pay CPU for extra conversions, data movements, or Status plumbing (already ruled out by Xingbo). In this change we make BlockBuilder more internally consistent and lift the 32-bit size requirement to callers (of a few functions specifically, for now). To ensure that's satisfied, I've added additional checks near the perimeter of RocksDB to ensure keys and values do not exceed 4GB, plus an extra random corruption or backstop check in BlockBasedTableBuilder. In detail, BlockBuilder (block_builder.cc/h): * Add API comments documenting the < 4GB assumption on Add/AddWithLastKey * Add debug assertions verifying input slice sizes < 4GB * Simplify AddWithLastKeyImpl to use uint32_t locals, reducing static_cast * Use only bottom 32 bits when appending derivative slices for consistency * Update MaybeStripTimestampFromKey to also truncate to 32-bit size * Add FIXME comments where buffer_/values_buffer_ sizes are truncated BlockBasedTableBuilder (block_based_table_builder.cc): * Add value size check (> uint32_t max) as a safety net against random corruption, since we have seen such corruptions in production WriteBatch (write_batch.cc): * Tighten existing key size checks to account for kNumInternalBytes (8-byte internal key suffix), using new kMaxWriteBatchKeySize constant * Add missing size checks to Delete, SingleDelete, and DeleteRange (both Slice and SliceParts variants) SstFileWriter (sst_file_writer.cc): * Add key and value size checks in AddImpl and DeleteRangeImpl, which bypass WriteBatch and go directly to the table builder MergeHelper (merge_helper.cc): * Add merge result size check (> uint32_t max) in both overloads of TimedFullMergeImpl, returning Corruption if exceeded * Add PartialMergeMulti result size check in MergeUntil CompactionIterator (compaction_iterator.cc): * Add size check on compaction filter output values (kChangeValue and kChangeWideColumnEntity), returning Corruption if > 4GB meta_blocks.cc: * Use Slice with uint32_t-truncated sizes in PropertyBlockBuilder::Finish to handle potentially oversized user property collector output Needed follow-up: * Check for blocks that are mis-encoded due to overflowing 32 bits for restart point offsets (and similar). See FIXME comments in block_builder.cc for why this is tricky. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14461 Test Plan: New extreme-size unit tests, along with some testing infrastructure improvements: Added test::HasBigMem() (in test_util/testharness.h) which returns true when the system has ≥128GB RAM (via sysconf(_SC_PHYS_PAGES) on Linux/macOS) or when ROCKSDB_BIGMEM_TESTS is set. All extreme-size tests use HasBigMem() to skip gracefully on smaller machines rather than being permanently disabled. Where possible, tests use MemMapping::AllocateLazyZeroed() to supply large keys/values as Slices backed by anonymous mmap (cleaner with new MemMapping::AsSlice()). On Linux, read-only access to these pages maps to the shared kernel zero page, so the source data consumes no physical RAM — only the destination copy (e.g., WriteBatch::rep_) materializes, cutting peak memory roughly in half vs. std::string. Tests (all enabled, skip via HasBigMem()): ./write_batch_test --gtest_filter='*LargeKeyValueSizeLimit*' ./external_sst_file_basic_test --gtest_filter='*LargeSizeSstFileWriter*' ./merge_test --gtest_filter='*LargeMergeResultRejected*' ./merge_helper_test --gtest_filter='*LargePartialMergeResultRejected*' ./db_compaction_test --gtest_filter='*CompactionFilterLargeValueRejected*' All 5 pass (verified on a 128GB+ machine). On smaller machines, all 5 bypass cleanly with "insufficient memory for reliable continuous testing". Reviewed By: xingbowang Differential Revision: D96521899 Pulled By: pdillinger fbshipit-source-id: 70f4b5e6a23ab074d60e653fbb7ddc5edbe162ab
168 lines
5.5 KiB
C++
168 lines
5.5 KiB
C++
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
//
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
#include "test_util/testharness.h"
|
|
|
|
#include <cstdlib>
|
|
#include <regex>
|
|
#include <string>
|
|
#include <thread>
|
|
|
|
#ifndef OS_WIN
|
|
#include <unistd.h>
|
|
#endif
|
|
|
|
#ifndef NDEBUG
|
|
#include "test_util/sync_point.h"
|
|
#endif
|
|
|
|
namespace {
|
|
#ifndef NDEBUG
|
|
// Global gtest event listener that cleans up SyncPoint state after every
|
|
// test. Many tests set SyncPoint callbacks with captured local variables
|
|
// but forget to disable/clear them. Under sharded execution (multiple
|
|
// tests per process), stale callbacks cause segfaults or corruption.
|
|
class SyncPointCleanupListener : public ::testing::EmptyTestEventListener {
|
|
void OnTestEnd(const ::testing::TestInfo& /*test_info*/) override {
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearTrace();
|
|
// LoadDependency({}) clears successors_, predecessors_, and
|
|
// cleared_points_ maps. Without this, stale dependencies from a
|
|
// previous test can block SyncPoint::Process() in the next test
|
|
// (e.g. a background compaction thread hitting CompactFilesImpl:1
|
|
// whose predecessor was never fired).
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({});
|
|
}
|
|
};
|
|
|
|
// Auto-register the listener via static initialization.
|
|
// This runs before main() and before any test fixtures are constructed.
|
|
static int RegisterSyncPointCleanup() noexcept {
|
|
::testing::TestEventListeners& listeners =
|
|
::testing::UnitTest::GetInstance()->listeners();
|
|
listeners.Append(new SyncPointCleanupListener());
|
|
return 0;
|
|
}
|
|
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
|
|
[[maybe_unused]] static int sync_point_cleanup_registered_ =
|
|
RegisterSyncPointCleanup();
|
|
#endif // !NDEBUG
|
|
} // namespace
|
|
|
|
namespace ROCKSDB_NAMESPACE::test {
|
|
|
|
#ifdef OS_WIN
|
|
#include <windows.h>
|
|
|
|
std::string GetPidStr() { return std::to_string(GetCurrentProcessId()); }
|
|
#else
|
|
std::string GetPidStr() { return std::to_string(getpid()); }
|
|
#endif
|
|
|
|
::testing::AssertionResult AssertStatus(const char* s_expr, const Status& s) {
|
|
if (s.ok()) {
|
|
return ::testing::AssertionSuccess();
|
|
} else {
|
|
return ::testing::AssertionFailure() << s_expr << std::endl << s.ToString();
|
|
}
|
|
}
|
|
|
|
std::string TmpDir(Env* env) {
|
|
std::string dir;
|
|
Status s = env->GetTestDirectory(&dir);
|
|
EXPECT_OK(s);
|
|
return dir;
|
|
}
|
|
|
|
std::string PerThreadDBPath(std::string dir, std::string name) {
|
|
size_t tid = std::hash<std::thread::id>()(std::this_thread::get_id());
|
|
return dir + "/" + name + "_" + GetPidStr() + "_" + std::to_string(tid);
|
|
}
|
|
|
|
std::string PerThreadDBPath(std::string name) {
|
|
return PerThreadDBPath(test::TmpDir(), name);
|
|
}
|
|
|
|
std::string PerThreadDBPath(Env* env, std::string name) {
|
|
return PerThreadDBPath(test::TmpDir(env), name);
|
|
}
|
|
|
|
int RandomSeed() {
|
|
const char* env = getenv("TEST_RANDOM_SEED");
|
|
int result = (env != nullptr ? atoi(env) : 301);
|
|
if (result <= 0) {
|
|
result = 301;
|
|
}
|
|
return result;
|
|
}
|
|
|
|
bool HasBigMem() {
|
|
const char* env = getenv("ROCKSDB_BIGMEM_TESTS");
|
|
if (env != nullptr && env[0] != '\0') {
|
|
return true;
|
|
}
|
|
#ifdef _SC_PHYS_PAGES
|
|
// Check whether the system has at least 128GB of physical RAM.
|
|
// _SC_PHYS_PAGES is available on Linux and macOS but is not standard POSIX.
|
|
long pages = sysconf(_SC_PHYS_PAGES);
|
|
long page_size = sysconf(_SC_PAGE_SIZE);
|
|
if (pages > 0 && page_size > 0) {
|
|
size_t total_bytes =
|
|
static_cast<size_t>(pages) * static_cast<size_t>(page_size);
|
|
return total_bytes >= size_t{128} << 30;
|
|
}
|
|
#endif
|
|
return false;
|
|
}
|
|
|
|
TestRegex::TestRegex(const std::string& pattern)
|
|
: impl_(std::make_shared<Impl>(pattern)), pattern_(pattern) {}
|
|
TestRegex::TestRegex(const char* pattern)
|
|
: impl_(std::make_shared<Impl>(pattern)), pattern_(pattern) {}
|
|
|
|
const std::string& TestRegex::GetPattern() const { return pattern_; }
|
|
|
|
class TestRegex::Impl : public std::regex {
|
|
public:
|
|
using std::regex::basic_regex;
|
|
};
|
|
|
|
bool TestRegex::Matches(const std::string& str) const {
|
|
if (impl_) {
|
|
return std::regex_match(str, *impl_);
|
|
} else {
|
|
// Should not call Matches on unset Regex
|
|
assert(false);
|
|
return false;
|
|
}
|
|
}
|
|
|
|
::testing::AssertionResult AssertMatchesRegex(const char* str_expr,
|
|
const char* pattern_expr,
|
|
const std::string& str,
|
|
const TestRegex& pattern) {
|
|
if (pattern.Matches(str)) {
|
|
return ::testing::AssertionSuccess();
|
|
} else if (TestRegex("\".*\"").Matches(pattern_expr)) {
|
|
// constant regex string
|
|
return ::testing::AssertionFailure()
|
|
<< str << " (" << str_expr << ")" << std::endl
|
|
<< "does not match regex " << pattern.GetPattern();
|
|
} else {
|
|
// runtime regex string
|
|
return ::testing::AssertionFailure()
|
|
<< str << " (" << str_expr << ")" << std::endl
|
|
<< "does not match regex" << std::endl
|
|
<< pattern.GetPattern() << " (" << pattern_expr << ")";
|
|
}
|
|
}
|
|
|
|
} // namespace ROCKSDB_NAMESPACE::test
|