Summary:
Some tests were failing due to apparent missing include of iomanip. I suspect this was from a gtest upgrade, because in open source, the include iomanip comes from gtest.h. To ensure we maintain compatibility with older gtest as well as the newer one, I pulled the include iomanip out of the in-repo gtest.h. Note that other places in gtest code only instantiate floating-point related templates with `float` and `double` types.
Also, to avoid `make format` being insanely slow on gtest.h, I've excluded third-party from the formatting check.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13661
Test Plan: make check, internal CI, manually ensure formatting check works outside of third-party/
Reviewed By: jaykorean
Differential Revision: D75963897
Pulled By: pdillinger
fbshipit-source-id: ed5737dd456e74068185f1ac5d57046d7509df7a
Summary:
**Summary**
This pull request introduces a mixed compressor, RoundRobinManager and RoundRobinCompressor, which selects algorithms in a loop. This implementation replaces the current hacky approach to round-robin compression in BuiltInCompressorV2. Additionally, it configures RocksDB to optionally utilize this customized compressor in the db stress test.
**Testing**
Testing was performed by verifying the stdout output from both RoundRobinCompressor and BuiltInCompressorV2.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13647
Reviewed By: pdillinger
Differential Revision: D75921997
Pulled By: shubhajeet
fbshipit-source-id: 8f42ac46f08ba982b2cd70241bd7dc13ff5a1225
Summary:
... to support SmallEnumSet over CompressionType with allowed custom compression types using most of the available byte. This is accomplished using an std::array<uint64_t> in place of just uint64_t. Also adds an std::bitset-like count() operation.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13657
Test Plan: unit tests included
Reviewed By: hx235
Differential Revision: D75827601
Pulled By: pdillinger
fbshipit-source-id: 519ae97ac671fd9885d6485976abbd969d1392d3
Summary:
**Summary**
This pull request aims to populate num_input_files and total_input_bytes in the CompactionJobStats object, which is accessible through EventListener::OnCompactionBegin(DB*, const CompactionJobInfo&). This change will enable RocksDB users to access accurate compaction input information.
**Context/Goals**
Provide accurate compaction input statistics to RocksDB users
Populate num_input_files and total_input_bytes in CompactionJobStats
Ensure correct population of these fields before EventListener::OnCompactionBegin() is called
**Test Plan**
Added test code to capture num_input_file and total_num_bytes when EventHandler is triggered
Asserted that these values are populated correctly
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13637
Reviewed By: cbi42
Differential Revision: D75690774
Pulled By: shubhajeet
fbshipit-source-id: 8236546f8ce7743f46048b302b376b7ef6429887
Summary:
Somehow this was previously not being tested in our Windows CI jobs so was accidentally broken in https://github.com/facebook/rocksdb/pull/13540 This fix will need to be backported to 10.3.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13649
Test Plan: CI
Reviewed By: hx235
Differential Revision: D75655418
Pulled By: pdillinger
fbshipit-source-id: a56bb213270904a1b7a13b905c2cc1919116df1c
Summary:
Also revamping test
GeneralTableTest::ApproximateOffsetOfCompressed so that it's not sensitive to adding new metadata to SST files
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13646
Test Plan: manually inspect new table property, which is not parsed anywhere, just for information to human reader
Reviewed By: hx235
Differential Revision: D75561241
Pulled By: pdillinger
fbshipit-source-id: c076c01a8b540bc4cb771964d48fa919c4c48ae4
Summary:
to make it easier to use 0 for disabled. And deprecate the use of txn db option `txn_commit_bypass_memtable_threshold`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13636
Test Plan: updated unit tests.
Reviewed By: jowlyzhang
Differential Revision: D75262136
Pulled By: cbi42
fbshipit-source-id: 9040e5a9c918c1d0906a2db4600cc012d2436b22
Summary:
Larger key/values can cause memtable write to take longer time. Add new option `TransactionOptions::large_txn_commit_optimize_byte_threshold` that enables the optimization by transaction write batch size.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13634
Test Plan:
- new unit test
- added option to stress test and ran stress test for some time: `python3 ./tools/db_crashtest.py --txn blackbox --txn_write_policy=0 --commit_bypass_memtable_one_in=50 --test_batches_snapshots=0`
Reviewed By: jowlyzhang
Differential Revision: D75248126
Pulled By: cbi42
fbshipit-source-id: 9522db93457729ba60e4176f7d47f7c2c7778567
Summary:
This exposes CompressionManager and related classes to the public API and adds `ColumnFamilyOptions::compression_manager` for tying a custom compression strategy to a column family. At the moment, this does not support custom/pluggable compression algorithms, just custom strategies around the built-in algorithms, e.g. which compression to use when and where.
A large part of the change is moving code from internal compression.h to a new public header advanced_compression.h, with some minor changes:
* `Decompressor::ExtractUncompressedSize()` is out-of-lined
* CompressionManager inherits Customizable and some related changes to members of CompressionManager are made. (Core functionality of CompressionManager is unchanged.)
This depends on a smart pointer I'm calling `ManagedPtr` which I'm adding to data_structure.h.
Additionally, advanced_compression.h gets CompressorWrapper and CompressionManagerWrapper as building blocks for overriding aspects of compression strategy while leveraging existing compression algorithms / schemas.
Some pieces needed to support the `compression_manager` option and rudimentary Customizable implementation are included. More work will be needed to make this general and well-behaved (see e.g. https://github.com/facebook/rocksdb/issues/8641; I still hit inscrutible problems every time I touch Customizable).
I'll add a release note for the experimental feature once pluggable compression algorithms and more of the Customizable things are working.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13626
Test Plan:
Added a unit test demonstrating how a custom compressor can "bypass" or "reject" compressions.
Expected next follow-up (probably someone else): use a custom CompressionManager/Compressor to replace the internal hack for testing mixed compressions.
Reviewed By: hx235
Differential Revision: D75028850
Pulled By: pdillinger
fbshipit-source-id: 8565bb8ba4b5fa923b1e29e76b4f7bb4faa42381
Summary:
Some specific old versions around RocksDB 2.5 would compress the metaindex and properties blocks. This hasn't been done since, probably because it interferes with the properties block indicating how to set up for decompression (so the reader can read those blocks before doing any decompression).
To fix backward compatibility, we establish a decompressor early if format_version indicates the file could come from a sufficiently old version.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13628
Test Plan: local and CI runs of tools/check_format_compatible.sh. (I don't believe we need special code to set up a unit test for this case.)
Reviewed By: jowlyzhang
Differential Revision: D75107623
Pulled By: pdillinger
fbshipit-source-id: 97132b8c5e0602e8e27254a11386d866b23cb4f5
Summary:
This PR introduces a new CF option, `memtable_avg_op_scan_flush_trigger`, to support triggering a memtable flush when an iterator skips too many invisible keys from the active memtable. This is a follow up to https://github.com/facebook/rocksdb/pull/13523#discussion_r2038261975, which introduced the option `memtable_op_scan_flush_trigger` for a single expensive iterator step. This PR focus on an expensive stretch of iterator steps, between Seeks and until iterator destruction. To avoid triggering a memtable flush for a stretch that is too small, this option only takes effect when the total number of entries skipped from the active memtable in a stretch of iterator steps exceeds `memtable_op_scan_flush_trigger`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13593
Test Plan:
* New unit tests covering the new option
* Add the option to the crash test.
Reviewed By: hx235
Differential Revision: D74434263
Pulled By: cbi42
fbshipit-source-id: 64f1101efb79c7498e2038eff630713ead8f6f41
Summary:
Instead of using FileSystem::GetFileSize() for each CompactionOutputFile, use the file size that is being tracked internally as part of the output file's metadata. FileSize is now part of `CompactionServiceOutputFile` and serialized in the `CompactionServiceResult`
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13620
Test Plan:
Tested with logging Meta's internal offload Infra
```
./compaction_job_test
```
Reviewed By: jowlyzhang
Differential Revision: D75006961
Pulled By: jaykorean
fbshipit-source-id: 008f9dc22bd672746ac180380ada4188713a6b85
Summary:
Usual release steps
* Release notes from 10.3 branch
* Update version.h
* Add 10.3.fb to check_format_compatible.sh
* Update folly commit hash. Added a few hacks to fix build errors.
Bonus:
* Add a check_format_compatible.sh sanity check to the per-PR GitHub actions jobs. It should be quick enough and catch typos in release diffs as we've seen in the past.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13622
Test Plan: CI
Reviewed By: jowlyzhang
Differential Revision: D74943843
Pulled By: pdillinger
fbshipit-source-id: 4ff1db9a635e111f8830cadff2d3ee51cf2de512
Summary:
showing up in the crash test after https://github.com/facebook/rocksdb/issues/13540
* For an assertion `dict_samples.sample_data.size() <= opts_.max_dict_bytes` we needed to ensure that `zstd_max_train_bytes` only takes effect with kZSTD compression.
* For an assertion with `r->table_options.verify_compression == (verify_decomp != nullptr)` we needed to ensure that `data_block_verify_decompressor` is set even when dictionary compression is attempted but not used.
* Noticed along the way: finish an optimization in `CompressAndVerifyBlock` that was incomplete.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13621
Test Plan:
Both failures were reproducible with hard-coding of some crash test params, and now not getting a failure.
```
--compression_type=zstd --compression_max_dict_bytes=16384 --compression_zstd_max_train_bytes=65536 --compression_max_dict_buffer_bytes=131071 --compression_use_zstd_dict_trainer=1
```
Write performance test like in https://github.com/facebook/rocksdb/issues/13540 shows essentially no change, maybe slightly faster (+0.4%) with verify_compression.
Reviewed By: virajthakur
Differential Revision: D74939103
Pulled By: pdillinger
fbshipit-source-id: 8bac8891bc08e1356eff52cc524e5bb409b0f86f
Summary:
[internal use] Allow the application to pass a request_id per read request to RocksDB and pass it down to the FileSystem (via IODebugContext)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13616
Test Plan:
./db_test --gtest_filter=DBTest.RequestIdPlumbingTest
Validates that RocksDB Api calls with request_id set result in request_id being passed to the filesystem through IODebugContext
Reviewed By: pdillinger
Differential Revision: D74912824
Pulled By: virajthakur
fbshipit-source-id: 4f15fef3ff7b5d700563f993f9b211c991020fb6
Summary:
Remove the dependency on `allow_db_generated_files` option in `IngestExternalFile` to be set for ingesting external tables. The files are created by SstFileWriter, and we should be able to ingest them. We could make it work by having the external table implementation provide the version and global sequence number related properties, but its safer to have RocksDB generate the table properties block and store it as is in the file.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13608
Test Plan: Add unit test to test basic ingestion and ingestion with atomic_replace_range
Reviewed By: pdillinger
Differential Revision: D74830707
Pulled By: anand1976
fbshipit-source-id: 4a9bea4a4f38f7c24c584262095c5c98cd771ddc
Summary:
Add stats to monitor the large transaction optimization. A stat is added for how many times wbwi ingestion is used. A histogram is added to track transaction size. We could also just track write batch size for all writes but I don't want to add the overhead to all writes yet.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13611
Test Plan:
ran `python3 ./tools/db_crashtest.py --txn blackbox --txn_write_policy=0 --commit_bypass_memtable_one_in=50 --test_batches_snapshots=0 --stats_dump_period_sec=2 --dump_malloc_stats=0 --statistics=1` and manually check LOG files
```
rocksdb.number.wbwi.ingest COUNT : 57
...
rocksdb.num.op.per.transaction P50 : 1.000000 P95 : 1.000000 P99 : 1.000000 P100 : 1.000000 COUNT : 2265 SUM : 2265
```
Reviewed By: jowlyzhang
Differential Revision: D74829087
Pulled By: cbi42
fbshipit-source-id: 5a9c3ab2d4cb6071cedfc47201ce2cf65a77d3c6
Summary:
Similar to https://github.com/facebook/rocksdb/pull/13555, add more info, ColumnFamily Id and name, to `CompactionServiceJobInfo`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13615
Test Plan:
Updated Unit Test
```
./compaction_service_test
```
Reviewed By: archang19
Differential Revision: D74845661
Pulled By: jaykorean
fbshipit-source-id: e2fc61006092b9febec1c6637b92cb00fb6cb73e
Summary:
Adds new classes etc. in internal compression.h that are intended to become public APIs for supporting custom/pluggable compression. Some steps remain to allow for pluggable compression and to remove a lot of legacy code (e.g. now called `OLD_CompressData` and `OLD_UncompressData`), but this change refactors the key integration points of SST building and reading and compressed secondary cache over to the new APIs.
Compared with the proposed https://github.com/facebook/rocksdb/issues/7650, this fixes a number of issues including
* Making a clean divide between public and internal APIs (currently just indicated with comments)
* Enough generality that built-in compressions generally fit into the framework rather than needing special treatment
* Avoid exposing obnoxious idioms like `compress_format_version` to the user.
* Enough generality that a compressor mixing algorithms/strategies from other compressors is pretty well supported without an extra schema layer
* Explicit thread-safety contracts (carefully considered)
* Contract details around schema compatibility and extension with code changes (more detail in next PR)
* Customizable "working areas" (e.g. for ZSTD "context")
* Decompression into an arbitrary memory location (rather than involving the decompressor in memory allocation; should facilitate reducing number of objects in block cache)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13540
Test Plan:
This is currently an internal refactor. More testing will come when the new API is migrated to the public API. A test in db_block_cache_test is updated to meaningfully cover a case (cache warming compression dictionary block) that was previously only covered in the crash test.
SST write performance test, like https://github.com/facebook/rocksdb/issues/13583. Compile with CLANG, run before & after simultaneously:
```
SUFFIX=`tty | sed 's|/|_|g'`; for ARGS in "-compression_parallel_threads=1 -compression_type=none" "-compression_parallel_threads=1 -compression_type=snappy" "-compression_parallel_threads=1 -compression_type=zstd" "-compression_parallel_threads=1 -compression_type=zstd -verify_compression=1" "-compression_parallel_threads=1 -compression_type=zstd -compression_max_dict_bytes=8180" "-compression_parallel_threads=4 -compression_type=snappy"; do echo $ARGS; (for I in `seq 1 20`; do ./db_bench -db=/dev/shm/dbbench$SUFFIX --benchmarks=fillseq -num=10000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=1000 -fifo_compaction_allow_compaction=0 -disable_wal -write_buffer_size=12000000 $ARGS 2>&1 | grep micros/op; done) | awk '{n++; sum += $5;} END { print int(sum / n); }'; done
```
Before (this PR and with https://github.com/facebook/rocksdb/issues/13583 reverted):
-compression_parallel_threads=1 -compression_type=none
1908372
-compression_parallel_threads=1 -compression_type=snappy
1926093
-compression_parallel_threads=1 -compression_type=zstd
1208259
-compression_parallel_threads=1 -compression_type=zstd -verify_compression=1
997583
-compression_parallel_threads=1 -compression_type=zstd -compression_max_dict_bytes=8180
934246
-compression_parallel_threads=4 -compression_type=snappy
1644849
After:
-compression_parallel_threads=1 -compression_type=none
1956054 (+2.5%)
-compression_parallel_threads=1 -compression_type=snappy
1911433 (-0.8%)
-compression_parallel_threads=1 -compression_type=zstd
1205668 (-0.3%)
-compression_parallel_threads=1 -compression_type=zstd -verify_compression=1
999263 (+0.2%)
-compression_parallel_threads=1 -compression_type=zstd -compression_max_dict_bytes=8180
934322 (+0.0%)
-compression_parallel_threads=4 -compression_type=snappy
1642519 (-0.2%)
Pretty neutral change(s) overall.
SST read performance test (related to https://github.com/facebook/rocksdb/issues/13583). Set up:
```
for COMP in none snappy zstd; do echo $ARGS; ./db_bench -db=/dev/shm/dbbench-$COMP --benchmarks=fillseq,flush -num=10000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=1000 -fifo_compaction_allow_compaction=0 -disable_wal -write_buffer_size=12000000 -compression_type=$COMP; done
```
Test (compile with CLANG, run before & after simultaneously):
```
for COMP in none snappy zstd; do echo $COMP; (for I in `seq 1 5`; do ./db_bench -readonly -db=/dev/shm/dbbench-$COMP --benchmarks=readrandom -num=10000000 -duration=20 -threads=8 2>&1 | grep micros/op; done) | awk '{n++; sum += $5;} END { print int(sum / n); }'; done
```
Before (this PR and with https://github.com/facebook/rocksdb/issues/13583 reverted):
none
1495646
snappy
1172443
zstd
706036
zstd (after constructing with -compression_max_dict_bytes=8180)
656182
After:
none
1494981 (-0.0%)
snappy
1171846 (-0.1%)
zstd
696363 (-1.4%)
zstd (after constructing with -compression_max_dict_bytes=8180)
667585 (+1.7%)
Pretty neutral.
Reviewed By: hx235
Differential Revision: D74626863
Pulled By: pdillinger
fbshipit-source-id: dc8ff3178da9b4eaa7c16aa1bb910c872afaf14a
Summary:
Addresses https://github.com/facebook/rocksdb/issues/13587.
This PR exposes the optimized implementation of batched reads through a `Transaction` object to Java clients.
The latency improvement of transactional multiget on production workload achieved by switching the implementation is roughly:
```
quantile=0.2: 21%
quantile=0.5: 28%
quantile=0.8: 46%
quantile=1.0: 239%
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13589
Reviewed By: jaykorean
Differential Revision: D74660169
Pulled By: cbi42
fbshipit-source-id: d01780173e0500c96e5e431ff6645008cbf6e8b5
Summary:
Expose pinned WriteBatchWithIndex::GetFromBatchAndDB through C bindings so that one can read data from the `WriteBatchWithIndex` and db w/o copying the data.
This fixes https://github.com/facebook/rocksdb/issues/12969.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12970
Reviewed By: cbi42
Differential Revision: D74586418
Pulled By: jaykorean
fbshipit-source-id: a5a4d2e8ce3ddf4c2371fdfdb4e9c3309966a05d
Summary:
As titled.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13605
Test Plan: This is removing a test
Reviewed By: mszeszko-meta
Differential Revision: D74660230
Pulled By: jowlyzhang
fbshipit-source-id: 9c1d46b56d2f9ee43eba645563d4f954645d1ace
Summary:
We saw some crash test failure for secondary db. It happens during crash recovery verification. This PR logs the manifest number when such failure happens. This PR also includes a small fix in `TryCatchUpWithPrimary()` that could incorrectly check WAL not found case.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13594
Test Plan: monitor further secondary DB crash test failure.
Reviewed By: archang19
Differential Revision: D74488769
Pulled By: cbi42
fbshipit-source-id: 226e55b2f99a739e93abda3ee91c05b80f59bf6a
Summary:
This PR fixes a bug where the file checksum for an external table file was not being calculated by SstFileWriter. The checksum is calculated in WritableFileWriter, so we need to pass that the the external table builder rather than the FSWritableFile pointer directly. However, WritableFileWriter is private to RocksDB, so wrap it in an FSWritableFile and pass it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13591
Test Plan: Add a new test in table_test.cc
Reviewed By: jaykorean
Differential Revision: D74410563
Pulled By: anand1976
fbshipit-source-id: c7fa8142e20da8836589dee5fa50919951cf4046
Summary:
Prior to this PR, for FIFO kChangeTemperature compaction was done by iterating and reading thru the input sst and generate the output sst. This was wasteful since for FIFO we could apply the "trivial" move by copying the input sst to the out sst without need decompress/compress and reading thru the input sst content at all. This PR added "allow_trivial_copy_when_change_temperature" to the CompactionOptionsFIFO.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13562
Reviewed By: cbi42
Differential Revision: D73295404
Pulled By: mikechuangmeta
fbshipit-source-id: 02241c7389797730ecd4a3b636837cb5f912b424
Summary:
Allow specifying ReadOptions for WBWI iterator when creating it through the C bindings. This allows to specify upper and lower bounds for the created iterator.
This fixes https://github.com/facebook/rocksdb/issues/12963.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12968
Reviewed By: pdillinger
Differential Revision: D74188049
Pulled By: jaykorean
fbshipit-source-id: 970d9910472dfedaa29a800c6d52bec14c656f3c
Summary:
clarify in comments and fix one implementation under NewIterator where option `memtable_op_scan_flush_trigger` does not work correctly with tailing iterator yet. This is because tailing iterator can rebuild iterator internally which reads from a newer memtable, and DBIter's reference to active memtable needs to be refreshed. This PR clarifies that `memtable_op_scan_flush_trigger` will have no effect on tailing iterator. We can add the support in the future if needed.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13586
Test Plan: existing tests.
Reviewed By: jaykorean
Differential Revision: D74108099
Pulled By: cbi42
fbshipit-source-id: 7c6608485d57755abc44f3be0b3c5d82a7bc5ca9
Summary:
While working on some compression refactoring, I noticed that `NotifyCollectTableCollectorsOnBlockAdd()` was being called from multiple threads (with `parallel_threads` > 1), meaning we were violating the promise that TablePropertiesCollectors need not be thread safe (and typically will not be, for efficiency).
Fixing this is a bit awkward or intrusive. Even though it seems weird to expose `block_compressed_bytes_fast` and `block_compressed_bytes_fast` in the public `BlockAdd()` function, and NOT the actual compressed block size used, there are some Meta-internal uses that would at least require negotiation / coordination to deprecate and remove. So it's probably easiest to just keep the awkward functionality and do the necessary modifications to call from a single thread.
The simplest solution that preserves the functionality with `parallel_threads` > 1 (provide the sampling data, expected ordering between `BlockAdd()` and `AddUserKey()`, no races) is to do the compression sampling in the thread building uncompressed blocks. Specifically, moving `NotifyCollectTableCollectorsOnBlockAdd()` and the compression sampling from `CompressAndVerifyBlock()`, which is called in parallel, to table builder `Flush()`, which is only called serially (per file). Even though this adds some compression to that single thread when sampling is enabled, that should be tolerable without complicating the code or regressing performance. Some related or nearby optimizations are included to ensure this.
* Got rid of a lot of unnecessary indirection and unnecessary fields in BlockRep, which should be a step in improving parallel compression performance (still bad IMHO).
* Restructured some `if`s etc. to streamline some logic
This satisfies my original refactoring need to moving the sampling code higher up the stack from `CompressBlock()`, to set up some other upcoming refactorings. The other caller of `CompressBlock()` (legacy BlobDB) doesn't need it, and in fact is better off calling `CompressData()` directly because it does not appear to be dealing with the various "no compression" outcomes introduced by `CompressBlock()`.
Eventual follow-up:
* Performance data below shows how the overhead of parallel compression can make it slower, with available CPUs, compared to serial compression. This infrastructure should be re-designed/re-engineered to reduce thread creation, context switches, etc. Also, more of the processing such as checksumming could be parallelized. (Things dependent on the block location in the file, such as ChecksumModifierForContext and cache warming, cannot be parallelized.)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13583
ThreadSanitizer: data race /data/users/peterd/rocksdb/./db_stress_tool/db_stress_table_properties_collector.h:36:5 in rocksdb::DbStressTablePropertiesCollector::BlockAdd(unsigned long, unsigned long, unsigned long)
```
Performance:
```
SUFFIX=`tty | sed 's|/|_|g'`; for ARGS in "-compression_parallel_threads=1 -compression_type=none" "-compression_parallel_threads=1 -compression_type=snappy" "-compression_parallel_threads=4 -compression_type=snappy"; do echo $ARGS; (for I in `seq 1 100`; do ./db_bench -db=/dev/shm/dbbench$SUFFIX --benchmarks=fillseq -num=10000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=1000 -fifo_compaction_allow_compaction=0 -disable_wal -write_buffer_size=12000000 $ARGS 2>&1 | grep micros/op; done) | awk '{n++; sum += $5;} END { print int(sum / n); }'; done
```
Average ops/s of 100 runs, running before & after at the same time, using clang DEBUG_LEVEL=0:
-compression_parallel_threads=1 -compression_type=none
Before: 1976319
After: 1983840 (+0.3%)
-compression_parallel_threads=1 -compression_type=snappy
Before: 1945576
After: 1953473 (+0.4%)
-compression_parallel_threads=4 -compression_type=snappy
Before: 1573190
After: 1611881 (+2.4%)
-compression_parallel_threads=4 -sample_for_compression=100 (pretty high sample rate)
Before: 1577167
After: 1589704 (+0.8%)
-compression_parallel_threads=4 -sample_for_compression=10 (crazy high sample rate)
Before: 1581276
After: 1393453 (-11.9%)
As seen, you need a very very high compression sample rate to see a regression. I would expect a setting like 1000 to be more typical.
Test Plan:
Along with existing unit tests + CI, expanded crash test to make its TablePropertiesCollector non-trivial, to exercise the bug (and other potential bugs), which was confirmed with local run of whitebox_crash_test with TSAN:
```
Reviewed By: hx235
Differential Revision: D73944593
Pulled By: pdillinger
fbshipit-source-id: f1dcba4ebdc01e735251037395003945c9b34e62
Summary:
I added `TransactionDBOptions::txn_commit_bypass_memtable_threshold` previously but per DB option is not dynamically changeable. Adding it as a per transaction option to make it easier to use. The option naming is updated to make it easier for customer to understand `large_txn_commit_optimize_threshold`. The transaction DB option `TransactionDBOptions::txn_commit_bypass_memtable_threshold` is marked as deprecated.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13582
Test Plan:
- new unit test
- updated stress test to use this new transaction option
Reviewed By: jowlyzhang
Differential Revision: D73960981
Pulled By: cbi42
fbshipit-source-id: 406f6e0f5f4eb6b336976f9a93b0bc08e61a9662
Summary:
AcquireLocked() returns transaction ids that currently hold the lock for deadlock detection purpose. We should not include the id of the transaction that is trying to acquire the lock, since this would lead to a false-positive deadlock detection where the deadlock is a self-loop. Note that since `wait_ids` is never cleared, there is another bug where if AcquireLocked() fails with kLockLimit, we could do deadlock detection based on `wait_ids` from a previous lock acquire attempt. This PR fixes both bugs.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13575
Test Plan: added a unit test repro that shows deadlock status can be incorrectly returned.
Reviewed By: jaykorean
Differential Revision: D73617887
Pulled By: cbi42
fbshipit-source-id: a6388b3ec53db13e2c502d60199378ea95885841
Summary:
**Context/Summary:**
Similar to https://github.com/facebook/rocksdb/commit/0a43d8a261b9c633c0a4e369b1ef33aa5ee32810, this is to verify flush output file contains the exact number of keys (represented by its `TableProperties::num_entries`) as added to table builder for block-based and plain table format. The implementation reuses a temporary compaction stats to record output record and existing input record (with some refactoring)
**Bonus:**
following https://github.com/facebook/rocksdb/commit/0a43d8a261b9c633c0a4e369b1ef33aa5ee32810#r154313564, limit compaction output record count check within block based table and plain table format as well as removing extra test setting; fix some typo
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13556
Test Plan: New test
Reviewed By: jaykorean
Differential Revision: D73229644
Pulled By: hx235
fbshipit-source-id: 2a7796450048b3bcb2d5c38f2b5fc6b53e4aae37
Summary:
we want to limit the maximum disk space used by RocksDB in one of our Go services, as it runs on a highly disk-constrained network switch.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13404
Reviewed By: cbi42
Differential Revision: D73517940
Pulled By: jaykorean
fbshipit-source-id: ae91fc7a4992399e20f06cc67dad8130cf19049e
Peter Dillinger