Xingbo Wang
3b66de3fac
Summary:
It contains 3 commits.
Commit 1: Fix /dev/shm exhaustion during make check
Fix /dev/shm exhaustion during make check
1. Add space-heavy tests to NON_PARALLEL_TEST: perf_context_test (1GB
write_buffer_size), obsolete_files_test (~1GB), backup_engine_test
(1GB), prefetch_test (1GB), and db_io_failure_test (256MB).
2. Add per-test cleanup on success: each parallel test script now
removes its test directory after passing. Failed test directories
are preserved for debugging.
3. Add age-based stale directory cleanup: at the start of make check,
remove /dev/shm/rocksdb.* directories older than 3 hours from
previous failed runs, using age-based filtering to avoid disturbing
concurrent runs.
Commit 2: Fix SIGSEGV in prefetch_test due to stale SyncPoint callbacks
Both FilePrefetchBufferTest and FSBufferPrefetchTest fixtures set up
SyncPoint callbacks capturing local variables by reference and enable
processing, but neither fixture's TearDown() clears them. When a
subsequent test runs, the stale callbacks fire with dangling
references, causing memory corruption and SIGSEGV.
Fixed by adding DisableProcessing() and ClearAllCallBacks() to both
fixtures' TearDown() methods.
Commit 3: Fix OpenFilesAsyncTest using excessive disk space
Fix OpenFilesAsyncTest using excessive disk space in /dev/shm
OpenFilesAsyncTest::SetupData() set write_buffer_size to SIZE_MAX to
prevent automatic flushes. This caused each of its ~20 parallel
instances to consume 6-43 GB in /dev/shm (validated: a single
Shutdown/3 instance used 43 GB), totaling ~233 GB and filling the
disk. This was the primary cause of "No space left on device" errors
in make check. The sst flushed is small, but it causes issue in WAL
space reservation, which bloated the disk usage.
The SIZE_MAX is unnecessary — the test writes only 4 tiny key-value
pairs and does explicit Flush() after each. The default 64 MB
write_buffer_size will never auto-flush for such small writes.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14425
Test Plan: Unit Test
Reviewed By: joshkang97
Differential Revision: D95253518
Pulled By: xingbowang
fbshipit-source-id: 03907df59b3d89d90413a0e33996ec205944d48b
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.
Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples
See the github wiki for more explanation.
The public interface is in include/. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.
License
RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.