Summary:
- propagate lower-level read and merge failures through `GetContext` via `read_status`, so `Get` and `GetEntity` preserve the original error instead of synthesizing `Corruption` when blob-backed reads or merge resolution fail
- teach `GetMergeOperands` to resolve blob-backed default columns from wide-column entities, covering both the direct base-value path and the merge-plus-base path
- add regression coverage for blob-read IO errors during `Get`/`GetEntity` merge resolution and for `GetMergeOperands` on blob-backed wide-column entities
- fix the `DBFlushTest.MemPurgeCorrectLogNumberAndSSTFileCreation` test race by waiting for flush callbacks and cleaning up sync points
## Testing
- `make db_blob_basic_test -j14`
- `/usr/bin/perl -e 'alarm shift; exec ARGV' 60 ./db_blob_basic_test --gtest_filter='DBBlobBasicTest/DBBlobBasicIOErrorTest.GetBlob_IOError/*:DBBlobBasicTest/DBBlobBasicIOErrorTest.GetEntityMergeWithBlobBaseIOError/*'`
## Task
T265824017, T265415808
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14640
Reviewed By: anand1976
Differential Revision: D101690700
Pulled By: xingbowang
fbshipit-source-id: 2b6fc357b37a01efa72a2d54dcff55be8992f42a
Summary:
Based on passing address of uninit variable in ReadOnlyMemTable::Get() in memtable.h. The contract and other implementations suggest it is a pure out parameter that is always overwritten, so we initialize it in the function before checking its value in a loop
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13486
Test Plan: watch build-linux-valgrind in CI
Reviewed By: cbi42
Differential Revision: D71819843
Pulled By: pdillinger
fbshipit-source-id: 1e06f3ee6998099791af27de5b2872eb476ceb7c
Summary:
added merge support for WBWIMemTable. Most of the preparation work is done in https://github.com/facebook/rocksdb/issues/13387 and https://github.com/facebook/rocksdb/issues/13400. The main code change to support merge is in wbwi_memtable.cc to support reading the Merge value type. The rest of the changes are mostly comment change and tests.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13410
Test Plan:
- new unit test
- ran `python3 ./tools/db_crashtest.py --txn blackbox --txn_write_policy=0 --commit_bypass_memtable_one_in=100 --test_batches_snapshots=0 --use_merge=1` for several runs.
Reviewed By: jowlyzhang
Differential Revision: D69885868
Pulled By: cbi42
fbshipit-source-id: b127d95a3027dc35910f6e5d65f3409ba27e2b6b
Summary:
This is a preparation for supporting merge in `WBWIMemTable`. This PR updates the sequence number assignment method so that it allows efficient and simple assignment when there are multiple entries with the same user key. This can happen when the WBWI contains Merge operations. This assignment relies on tracking the number of updates issued for each key in each WBWI entry (`WriteBatchIndexEntry::update_count`). Some refactoring is done in WBWI to remove `last_entry_offset` as part of the WBWI state which I find it harder to use correctly.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13400
Test Plan: updated unit tests to check that update count is tracked correctly and WBWIMemTable is assigning sequence number as expected.
Reviewed By: pdillinger
Differential Revision: D69666462
Pulled By: cbi42
fbshipit-source-id: 9b18291825017a67c4da3318e8a556aa2971326b
Summary:
add a new transaction option `TransactionOptions::commit_bypass_memtable` that will ingest the transaction into a DB as an immutable memtables, skipping memtable writes during transaction commit. This helps to reduce the blocking time of committing a large transaction, which is mostly spent on memtable writes. The ingestion is done by creating WBWIMemTable using transaction's underlying WBWI, and ingest it as the latest immutable memtable. The feature will be experimental.
Major changes are:
1. write path change to ingest the transaction, mostly in WriteImpl() and IngestWBWI() in db_impl_write.cc.
2. WBWI changes to track some per CF stats like entry count and overwritten single deletion count, and track which keys have overwritten single deletions (see 3.). Per CF stat is used to precompute the number of entries in each WBWIMemTable.
3. WBWIMemTable Iterator changes to emit overwritten single deletions. The motivation is explained in the comment above class WBWIMemTable definition. The rest of the changes in WBWIMemTable are moving the iterator definition around.
Some intended follow ups:
1. support for merge operations
2. stats/logging around this option
3. tests improvement, including stress test support for the more comprehensive no_batched_op_stress.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13144
Test Plan:
* added new unit tests
* enabled in multi_ops_txns_stress test
* Benchmark: applying the change in 8222c0cafc4c6eb3a0d05807f7014b44998acb7a, I tested txn size of 10k and check perf context for write_memtable_time, write_wal_time and key_lock_wait_time(repurposed for transaction unlock time). Though the benchmark result number can be flaky, this shows memtable write time improved a lot (more than 100 times). The benchmark also shows that the remaining commit latency is from transaction unlock.
```
./db_bench --benchmarks=fillrandom --seed=1727376962 --threads=1 --disable_auto_compactions=1 --max_write_buffer_number=100 --min_write_buffer_number_to_merge=100 --writes=100000 --batch_size=10000 --transaction_db=1 --perf_level=4 --enable_pipelined_write=false --commit_bypass_memtable=1
commit_bypass_memtable = false
fillrandom : 3.982 micros/op 251119 ops/sec 0.398 seconds 100000 operations; 27.8 MB/s PERF_CONTEXT:
write_memtable_time = 116950422
write_wal_time = 8535565
txn unlock time = 32979883
commit_bypass_memtable = true
fillrandom : 2.627 micros/op 380559 ops/sec 0.263 seconds 100000 operations; 42.1 MB/s PERF_CONTEXT:
write_memtable_time = 740784
write_wal_time = 11993119
txn unlock time = 21735685
```
Reviewed By: jowlyzhang
Differential Revision: D66307632
Pulled By: cbi42
fbshipit-source-id: 6619af58c4c537aed1f76c4a7e869fb3f5098999
Summary:
introduce the class WBWIMemTable that implements ReadOnlyMemTable interface with data stored in a WriteBatchWithIndex object.
This PR implements the main read path: Get, MultiGet and Iterator. It only supports Put, Delete and SingleDelete operations for now. All the keys in the WBWIMemTable will be assigned a global sequence number through WBWIMemTable::SetGlobalSequenceNumber().
Planned follow up PRs:
- Create WBWIMemTable with a transaction's WBWI and ingest it into a DB during Transaction::Commit()
- Support for Merge. This will be more complicated since we can have multiple updates with the same user key for Merge.
- Support for other operations like WideColumn and other ReadOnlyMemTable methods.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13123
Test Plan: * A mini-stress test for the read path is added as a new unit test
Reviewed By: jowlyzhang
Differential Revision: D65633419
Pulled By: cbi42
fbshipit-source-id: 0684fe47260b41f51ca39c300eb72ca5bc9c5a3b
Xingbo Wang