Compare commits

..

28 Commits

Author SHA1 Message Date
Yanqin Jin 2df3e90028 Fix the logic of setting read_amp_bytes_per_bit from OPTIONS file (#7680)
Summary:
Instead of using `EncodeFixed32` which always serialize a integer to
little endian, we should use the local machine's endianness when
populating a native data structure during options parsing.
Without this fix, `read_amp_bytes_per_bit` may be populated incorrectly
on big-endian machines.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7680

Test Plan: make check

Reviewed By: pdillinger

Differential Revision: D24999166

Pulled By: riversand963

fbshipit-source-id: dc603cff6e17f8fa32479ce6df93b93082e6b0c4
2020-11-17 10:34:41 -08:00
Yanqin Jin 6a23aaf86e Bump version and update HISTORY.md 2020-11-15 15:30:48 -08:00
Yanqin Jin bd39be1859 Hack to load OPTIONS file for read_amp_bytes_per_bit (#7659)
Summary:
A temporary hack to work around a bug in 6.10, 6.11, 6.12, 6.13 and
6.14. The bug will write out 8 bytes to OPTIONS file from the starting
address of BlockBasedTableOptions.read_amp_bytes_per_bit which is
actually a uint32. Consequently, the value of read_amp_bytes_per_bit
written in the OPTIONS file is wrong. From 6.15, RocksDB will
try to parse the read_amp_bytes_per_bit from OPTIONS file as a uint32.
To be able to load OPTIONS file generated by affected releases before
the fix, we need to manually parse read_amp_bytes_per_bit with this hack.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7659

Test Plan:
Generate a db with current 6.14.fb (head at https://github.com/facebook/rocksdb/commit/b6db05dbb5364c658c5401a8078d73697bb5f31d). Maybe use db_stress.

Checkout this PR, run
```
 ~/rocksdb/ldb --db=. --try_load_options --ignore_unknown_options idump --count_only
```
Expect success, and should not see
```
Failed: Invalid argument: Error parsing read_amp_bytes_per_bit:17179869184
```

Also
make check

Reviewed By: anand1976

Differential Revision: D24954752

Pulled By: riversand963

fbshipit-source-id: c7b802fc3e52acd050a4fc1cd475016122234394
2020-11-15 15:29:14 -08:00
Huisheng Liu 62cf055775 fix read_amp_bytes_per_bit field size (#7651)
Summary:
The field in BlockBasedTableOptions is 4 bytes:
  // Default: 0 (disabled)
  uint32_t read_amp_bytes_per_bit = 0;

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7651

Reviewed By: ltamasi

Differential Revision: D24844994

Pulled By: riversand963

fbshipit-source-id: e2695e55532256ef8996dd6939cad06987a80293
2020-11-15 15:27:17 -08:00
Jay Zhuang 97bf78721b Update HISTORY.md and version.h for 6.13.3 2020-10-14 09:29:38 -07:00
Jay Zhuang 0bda0e3dfa Fix StallWrite crash with mixed of slowdown/no_slowdown writes (#7508)
Summary:
`BeginWriteStall()` removes no_slowdown write from the write
list and updates `link_newer`, which makes `CreateMissingNewerLinks()`
thought all write list has valid `link_newer` and failed to create link
for all writers.
It caused flaky test and SegFault for release build.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7508

Test Plan: Add unittest to reproduce the issue.

Reviewed By: anand1976

Differential Revision: D24126601

Pulled By: jay-zhuang

fbshipit-source-id: f8ac5dba653f7ee1b0950296427d4f5f8ee34a06
2020-10-13 17:44:02 -07:00
Andrew Kryczka 950b72c743 Update HISTORY.md and version.h for 6.13.2 2020-10-13 09:41:35 -07:00
Andrew Kryczka 6eccfbbf46 Fix bug when DeleteRange() used with paranoid_file_checks == true
Backports part of 7508175558.

The unit test comes from https://github.com/facebook/rocksdb/pull/7521.
2020-10-13 09:40:12 -07:00
Andrew Kryczka ed5c64e0d9 update HISTORY.md and version.h for 6.13.1 2020-10-12 14:06:52 -07:00
Andrew Kryczka 0dd184c446 fix TARGETS
ran `python3 buckifier/buckify_rocksdb.py`
2020-10-12 14:06:28 -07:00
Andrew Kryczka 6676d31d3f Fix bug in pinned partitioned indexes with some reads bypassing block cache
Backports part of 75d3b6fdf0.
2020-10-12 13:55:59 -07:00
Andrew Kryczka 82c5a09241 Fix bug in pinned partitioned user key indexes
Backports part of 75d3b6fdf0.
2020-10-12 13:54:35 -07:00
Andrew Kryczka ac39f413d8 Add missing release note 2020-10-12 13:54:11 -07:00
Yanqin Jin d6cd5d1e3c Add a rocksdb lib target with link_whole=True (#7466)
Summary:
We would like to build a shared library with all fbcode dependencies statically linked within.
This resulting .so should not drop any symbols definitions in the building process.
To ensure that, we use `link_whole=True` according to
https://buck.build/rule/cxx_library.html#link_whole.
Since `link_whole` is `False` by default, adding a `link_whole=False` to existing libraries won't
change any behavior.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7466

Test Plan: build a .so and test internally.

Reviewed By: pdillinger

Differential Revision: D24009780

Pulled By: riversand963

fbshipit-source-id: d18804d495da7195ed72a2040e1a5de4fd336519
2020-10-05 11:26:16 -07:00
Yanqin Jin 5a86c2e725 Re-add extra compiler flags when building unittests (#7437)
Summary:
Re-add extra_compiler_flags when building unit tests for fbcode.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7437

Test Plan: Integrate with buck and run internal tests.

Reviewed By: pdillinger

Differential Revision: D23943924

Pulled By: riversand963

fbshipit-source-id: b92b7ad003e06e0860c45efc5f7f9684233d0c55
2020-09-25 16:48:46 -07:00
Peter Dillinger e29a510f45 Add Unreleased to HISTORY.md for patches 2020-09-24 08:21:36 -07:00
Zhichao Cao 41a2789e18 Add AppendWithVerify and PositionedAppendWithVerify to Env and FileSystem (#7419)
Summary:
Add new AppendWithVerify and PositionedAppendWithVerify APIs to Env and FileSystem to bring the data verification information (data checksum information) from upper layer (e.g., WritableFileWriter) to the storage layer. This PR only include the API definition, no functional codes are added to unblock other developers which depend on these APIs.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7419

Test Plan: make -j32

Reviewed By: pdillinger

Differential Revision: D23883196

Pulled By: zhichao-cao

fbshipit-source-id: 94676c26bc56144cc32e3661f84f21eccd790411
2020-09-23 20:44:00 -07:00
Andrew Kryczka 2e5f8bd434 Fix scope of ReadOptions in SstFileReader (#7432)
Summary:
a4a4a2dabd changed the contract of `TableReader::NewIterator()` to require
`ReadOptions` outlive the returned iterator. But I didn't notice that
`SstFileReader` violates the new contract and needs to be adapted. The unit test
provided here exposes the problem when run under ASAN.

```
$ ./sst_file_reader_test --gtest_filter=SstFileReaderTest.ReadOptionsOutOfScope
Note: Google Test filter = SstFileReaderTest.ReadOptionsOutOfScope
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from SstFileReaderTest
[ RUN      ] SstFileReaderTest.ReadOptionsOutOfScope
=================================================================
==3238048==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffd6189e158 at pc 0x000001298350 bp 0x7ffd6189c280 sp 0x7ffd6189c278
READ of size 8 at 0x7ffd6189e158 thread T0
    #0 0x129834f in rocksdb::BlockBasedTableIterator::InitDataBlock() table/block_based/block_based_table_iterator.cc:236
    https://github.com/facebook/rocksdb/issues/1 0x12b01f7 in rocksdb::BlockBasedTableIterator::SeekImpl(rocksdb::Slice const*) table/block_based/block_based_table_iterator.cc:77
    https://github.com/facebook/rocksdb/issues/2 0x844d28 in rocksdb::IteratorWrapperBase<rocksdb::Slice>::SeekToFirst() table/iterator_wrapper.h:116
    https://github.com/facebook/rocksdb/issues/3 0x844d28 in rocksdb::DBIter::SeekToFirst() db/db_iter.cc:1352
    https://github.com/facebook/rocksdb/issues/4 0x52482b in rocksdb::SstFileReaderTest_ReadOptionsOutOfScope_Test::TestBody() table/sst_file_reader_test.cc:150
    https://github.com/facebook/rocksdb/issues/5 0x5f433c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:3899
    https://github.com/facebook/rocksdb/issues/6 0x5f433c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:3935
    https://github.com/facebook/rocksdb/issues/7 0x5cc2de in testing::Test::Run() third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:3973
    https://github.com/facebook/rocksdb/issues/8 0x5cc988 in testing::Test::Run() third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:3965
    https://github.com/facebook/rocksdb/issues/9 0x5cc988 in testing::TestInfo::Run() third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:4149
    https://github.com/facebook/rocksdb/issues/10 0x5cce9a in testing::TestInfo::Run() third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:4124
    https://github.com/facebook/rocksdb/issues/11 0x5cce9a in testing::TestCase::Run() third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:4267
    https://github.com/facebook/rocksdb/issues/12 0x5ce696 in testing::TestCase::Run() third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:4253
    https://github.com/facebook/rocksdb/issues/13 0x5ce696 in testing::internal::UnitTestImpl::RunAllTests() third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:6633
    https://github.com/facebook/rocksdb/issues/14 0x5f541c in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:3899
    https://github.com/facebook/rocksdb/issues/15 0x5f541c in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:3935
    https://github.com/facebook/rocksdb/issues/16 0x5cee74 in testing::UnitTest::Run() third-party/gtest-1.8.1/fused-src/gtest/gtest-all.cc:6242
    https://github.com/facebook/rocksdb/issues/17 0x4c0332 in RUN_ALL_TESTS() third-party/gtest-1.8.1/fused-src/gtest/gtest.h:22104
    https://github.com/facebook/rocksdb/issues/18 0x4c0332 in main table/sst_file_reader_test.cc:213
    https://github.com/facebook/rocksdb/issues/19 0x7fb0263281a5 in __libc_start_main (/usr/local/fbcode/platform007/lib/libc.so.6+0x211a5)
    https://github.com/facebook/rocksdb/issues/20 0x523e56  (/data/users/andrewkr/rocksdb/sst_file_reader_test+0x523e56)

Address 0x7ffd6189e158 is located in stack of thread T0 at offset 568 in frame
    #0 0x52428f in rocksdb::SstFileReaderTest_ReadOptionsOutOfScope_Test::TestBody() table/sst_file_reader_test.cc:131

  This frame has 9 object(s):
    [32, 40) 'reader'
    [96, 104) '<unknown>'
    [160, 168) '<unknown>'
    [224, 232) 'iter'
    [288, 304) 'gtest_ar'
    [352, 368) '<unknown>'
    [416, 440) 'keys'
    [480, 512) '<unknown>'
    [544, 680) 'ropts' <== Memory access at offset 568 is inside this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
 AddressSanitizer: stack-use-after-scope table/block_based/block_based_table_iterator.cc:236 in rocksdb::BlockBasedTableIterator::InitDataBlock()
...
```

The fix is to use `ArenaWrappedDBIter` which has support for holding a
`ReadOptions` in an `Arena` whose lifetime is tied to the iterator.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7432

Test Plan: verified the provided unit test no longer fails

Reviewed By: pdillinger

Differential Revision: D23880043

Pulled By: ajkr

fbshipit-source-id: 9464c37408f7bd7c9c4a90ceffb04d9f0ca7a494
2020-09-23 15:28:50 -07:00
Peter Dillinger e041a40a23 Add and fix clang -Wshift-sign-overflow (#7431)
Summary:
This option is apparently used by some teams within Facebook
(internal ref T75998621)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7431

Test Plan: USE_CLANG=1 make check before (fails) and after

Reviewed By: jay-zhuang

Differential Revision: D23876584

Pulled By: pdillinger

fbshipit-source-id: abb8b67a1f1aac75327944d266e284b2b6727191
2020-09-23 15:28:47 -07:00
Peter Dillinger 74be8f3ea2 Fix/minimize mock_time_env.h dependencies (#7426)
Summary:
(a) own copy of kMicrosInSecond
(b) out-of-line sync point code

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7426

Test Plan: FB internal

Reviewed By: ajkr

Differential Revision: D23861363

Pulled By: pdillinger

fbshipit-source-id: de6b1621dca2f7391c5ff72bad04a7613dc27527
2020-09-23 11:42:15 -07:00
Peter Dillinger cda8a74eb7 Less I/O for incremental backups, slightly better corruption detection (#7413)
Summary:
Two relatively simple functional changes to incremental backup
behavior, integrated with a minor refactoring to reduce code redundancy and
improve error/log message. There are nuances to the impact of these changes,
but I believe they are fundamentally good and generally safe. Those functional
changes:

* Incremental backups no longer read DB table files that are already saved to a
shared part of the backup directory, unless `share_files_with_checksum` is used
with `kLegacyCrc32cAndFileSize` naming (discouraged) where crc32c full file
checksums are needed to determine file naming.
  * Justification: incremental backups should not need to read the whole DB,
especially without rate limiting. (Although other BackupEngine reads are not
rate limited either, other non-trivial reads are generally limited by a
corresponding write, as in copying files.) Also, the fact that this is not
already fixed was arguably a bug/oversight in the implementation of https://github.com/facebook/rocksdb/issues/7110.

* When considering whether a table file is already backed up in a shared part
of backup directory, BackupEngine would already query the sizes of source (DB)
and pre-existing destination (backup) files. BackupEngine now uses these file
sizes to detect corruption, as at least one of (a) old backup, (b) backup in
progress, or (c) current DB is corrupt if there's a size mismatch.
  * Justification: a random related fix that also helps to cover a small hole
in corruption checking uncovered by the other functional change:
  * For `share_table_files` without "checksum" (not recommended), the other
change regresses in detecting fundamentally unsafe use of this option
combination: when you might generate different versions of same SST file
number. As demonstrated by `BackupableDBTest.FailOverwritingBackups,` this
regression is greatly mitigated by the new file size checking. Nevertheless,
almost no reason to use `share_files_with_checksum=false` should remain, and
comments are updated appropriately.

Also, this change renames internal function `CalculateChecksum` to
`ReadFileAndComputeChecksum` to make the performance impact of this function
clear in code reviews.

It is not clear what 'same_path' is for in backupable_db.cc, and I suspect it
cannot be true for a DB with unique file names (like DBImpl). Nevertheless,
I've tried to keep its functionality intact when `true` to minimize risk for
now, despite having no unit tests for which it is true.

Select impact details (much more in unit tests): For
`share_files_with_checksum`, I am confident there is no regression (vs.
pre-6.12) in detecting DB or backup corruption at backup creation time, mostly
because the old design did not leverage this extra checksum computation for
detecting inconsistencies at backup creation time. (With computed checksums in
names, a recently corrupted file just looked like a different file vs. what was
already backed up.)

Even in the hypothetical case of DB session id collision (~100 bits entropy
collision), file size in name and/or our file size check add an extra layer of
protection against false success in creating an accurate new backup. (Unit test
included.)

`DB::VerifyChecksum` and `BackupEngine::VerifyBackup` with checksum checking
are still able to catch corruptions that `CreateNewBackup` does not. Note that
when custom file checksum support is added to BackupEngine, that will
essentially give the same power as `DB::VerifyChecksum` into `CreateNewBackup`.
We could add options for `CreateNewBackup` to cover some of what would be
caught by `VerifyBackup` with checksum checking.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7413

Test Plan:
Two new unit tests included, both of which fail without these
changes. Although we don't test the I/O improvement directly, we test it
indirectly in DB corruption detection power that was inadvertently unlocked
with new backup file naming PLUS computing current content checksums (now
removed). (I don't think that case of DB corruption detection justifies reading
the whole DB on incremental backup.)

Reviewed By: zhichao-cao

Differential Revision: D23818480

Pulled By: pdillinger

fbshipit-source-id: 148aff16f001af5b9fd4b22f155311c2461f1bac
2020-09-22 10:20:22 -07:00
Peter Dillinger fb98398ca9 Postponing custom checksum support in BackupEngine (#7411)
Summary:
This change reverts BackupEngine to 6.12 state to accommodate a
higher-priority fix that does not easily merge with this custom checksum
support. We intend to reinstate this support soon, by merging a revert
of this change.

For backupable_db_test, I've removed the tests depending on this
feature.

I've also removed relevant HISTORY.md entry.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7411

Test Plan: unit tests

Reviewed By: ajkr

Differential Revision: D23793835

Pulled By: pdillinger

fbshipit-source-id: 7e861436539584799b13d1a8ae559b81b6d08052
2020-09-22 10:18:30 -07:00
Peter Dillinger 2dbb90a064 Update HISTORY.md for #7346 (#7417)
Summary:
Copied from Andrew's entry for 6.12.3. Inserted here
retroactive to 6.12

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7417

Test Plan: no code change

Reviewed By: jay-zhuang

Differential Revision: D23815980

Pulled By: pdillinger

fbshipit-source-id: 3c8a052cdb61be1215d311556c9487f9ea5c8cb0
2020-09-21 09:49:45 -07:00
Peter Dillinger b4f29f7515 Restore file size in backup table file names (and other cleanup) (#7400)
Summary:
Prior to 6.12, backup files using share_files_with_checksum had
the file size encoded in the file name, after the last '\_' and before
the last '.'. We considered this an implementation detail subject to
change, and indeed removed this information from the file name (with an
option to use old behavior) because it was considered
ineffective/inefficient for file name uniqueness. However, some
downstream RocksDB users were relying on this information since the file
size is not explicitly in the backup manifest file.

This primary purpose of this change is "retrofitting" the 6.12 release
(not yet a public release) to simultaneously support the benefits of the
new naming scheme (I/O performance and data correctness at scale) and
preserve the file size information, both as default behaviors. With this
change, we are essentially making the file size information encoded in
the file name an official, though obscure, extension of the backup meta
file format.

We preserve an option (kLegacyCrc32cAndFileSize) to use the original
"legacy" naming scheme, with its caveats, and make it easy to omit the
file size information (no kFlagIncludeFileSize), for more compact file
names. But note that changing the naming scheme used on an existing db
and backup directory can lead to transient space amplification, as some
files will be stored under two names in the shared_checksum directory.
Because some backups were saved using the original 6.12 naming scheme,
we offer two ways of dealing with those files: SST files generated by
older 6.12 versions can either use the default naming scheme in effect
when the SST files were generated (kFlagMatchInterimNaming, default, no
transient space amplification) or can use a new naming scheme (no
kFlagMatchInterimNaming, potential space amplification because some
already stored files getting a new name).

We don't have a natural way to detect which files were generated by
previous 6.12 versions, but this change hacks one in by changing DB
session ids to now use a more concise encoding, reducing file name
length, saving ~dozen bytes from SST files, and making them visually
distinct from DB ids so that they are less likely to be mixed up.

Two final auxiliary notes:
Recognizing that the backup file names have become a de facto part of
the backup meta schema, this change makes them easier to parse and
extend by putting a distinct marker, 's', before DB session ids embedded
in the name. When we extend this to allow custom checksums in the name,
they can get their own marker to ensure safe parsing. For backward
compatibility, file size does not get a marker but is assumed for
`_[0-9]+[.]`

Another change from initial 6.12 default behavior is never including
file custom checksum in the file name. Looking ahead to 6.13, we do not
want the default behavior to cause backup space amplification for
someone turning on file custom checksum checking in BackupEngine; we
want that to be an easy decision. When implemented, including file
custom checksums in backup file names will be a non-default option.

Actual file name patterns and priorities, as regexes:

    kLegacyCrc32cAndFileSize OR pre-6.12 SST file ->
      [0-9]+_[0-9]+_[0-9]+[.]sst
    kFlagMatchInterimNaming set (default) AND early 6.12 SST file ->
      [0-9]+_[0-9a-fA-F-]+[.]sst
    kUseDbSessionId AND NOT kFlagIncludeFileSize ->
      [0-9]+_s[0-9A-Z]{20}[.]sst
    kUseDbSessionId AND kFlagIncludeFileSize (default) ->
      [0-9]+_s[0-9A-Z]{20}_[0-9]+[.]sst

We might add opt-in options for more '\_' separated data in the name,
but embedded file size, if present, will always be after last '\_' and
before '.sst'.

This change was originally applied to version 6.12. (See https://github.com/facebook/rocksdb/issues/7390)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7400

Test Plan:
unit tests included. Sync point callbacks are used to mimic
previous version SST files.

Reviewed By: ajkr

Differential Revision: D23759587

Pulled By: pdillinger

fbshipit-source-id: f62d8af4e0978de0a34f26288cfbe66049b70025
2020-09-21 08:05:23 -07:00
Zhichao Cao 731f022dc8 fix the flaky test failure (#7415)
Summary:
Fix the flaky test failure in error_handler_fs_test. Add the sync point, solve the dependency.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7415

Test Plan: make asan_check, ~/gtest-parallel/gtest-parallel -r 100 ./error_handler_fs_test

Reviewed By: siying

Differential Revision: D23804330

Pulled By: zhichao-cao

fbshipit-source-id: 5175108651f7652e47e15978f2a9c1669ef59d80
2020-09-21 07:59:48 -07:00
Zhichao Cao a02495cf80 Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310)
Summary:
In the current implementation, any retryable IO error happens during Flush is mapped to a hard error. In this case, DB is stopped and write is stalled unless the background error is cleaned. In this PR, if WAL is DISABLED, the retryable IO error during FLush is mapped to a soft error. Such that, the memtable can continue receive the writes. At the same time, if auto resume is triggered, SwtichMemtable will not be called during Flush when resuming the DB to avoid to many small memtables. Testing cases are added.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7310

Test Plan: adding new unit test, pass make check.

Reviewed By: anand1976

Differential Revision: D23710892

Pulled By: zhichao-cao

fbshipit-source-id: bc4ca50d11c6b23b60d2c0cb171d86d542b038e9
2020-09-21 07:59:19 -07:00
anand76 5a0cef9259 Update HISTORY.md with IO fencing error code (#7402)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7402

Reviewed By: pdillinger

Differential Revision: D23761689

Pulled By: anand1976

fbshipit-source-id: 59e10f0aaa80f6c0f5a46dc99467138c4cee0511
2020-09-17 11:52:23 -07:00
anand76 bb95ed284d Add a new IOStatus subcode to indicate that writes are fenced off (#7374)
Summary:
In a distributed file system, directory ownership is enforced by fencing
off the previous owner once they've been preempted by a new owner. This
PR adds a IOStatus subcode for ```StatusCode::IOError``` to indicate this.
Once this error is returned for a file write, the DB is put in read-only
mode and not allowed to resume in read-write mode.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7374

Test Plan: Add new unit tests in ```error_handler_fs_test```

Reviewed By: riversand963

Differential Revision: D23687777

Pulled By: anand1976

fbshipit-source-id: bef948642089dc0af399057864d9a8ca339e8b2f
2020-09-17 11:51:31 -07:00
292 changed files with 4676 additions and 13992 deletions
+117 -201
View File
@@ -2,235 +2,159 @@ version: 2.1
orbs:
win: circleci/windows@2.4.0
slack: circleci/slack@3.4.2
aliases:
- &notify-on-master-failure
fail_only: true
only_for_branches: master
commands:
install-pyenv-on-macos:
steps:
- run:
name: Install pyenv on macos
command: |
HOMEBREW_NO_AUTO_UPDATE=1 brew install pyenv
increase-max-open-files-on-macos:
steps:
- run:
name: Increase max open files
command: |
sudo sysctl -w kern.maxfiles=1048576
sudo sysctl -w kern.maxfilesperproc=1048576
sudo launchctl limit maxfiles 1048576
pre-steps:
steps:
- checkout
- run: pyenv install --skip-existing 3.5.9
- run: pyenv global 3.5.9
- run:
name: Setup Environment Variables
command: |
echo "export GTEST_THROW_ON_FAILURE=0" >> $BASH_ENV
echo "export GTEST_OUTPUT=\"xml:/tmp/test-results/\"" >> $BASH_ENV
echo "export SKIP_FORMAT_BUCK_CHECKS=1" >> $BASH_ENV
echo "export PRINT_PARALLEL_OUTPUTS=1" >> $BASH_ENV
post-steps:
steps:
- slack/status: *notify-on-master-failure
- store_test_results: # store test result if there's any
path: /tmp/test-results
- store_artifacts: # store LOG for debugging if there's any
path: LOG
install-clang-10:
steps:
- run:
name: Install Clang 10
command: |
echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
echo "APT::Acquire::Retries \"10\";" | sudo tee -a /etc/apt/apt.conf.d/80-retries # llvm.org unreliable
sudo apt-get update -y && sudo apt-get install -y clang-10
install-gflags:
steps:
- run:
name: Install gflags
command: |
sudo apt-get update -y && sudo apt-get install -y libgflags-dev
install-gflags-on-macos:
steps:
- run:
name: Install gflags on macos
command: |
HOMEBREW_NO_AUTO_UPDATE=1 brew install gflags
install-gtest-parallel:
steps:
- run:
name: Install gtest-parallel
command: |
git clone --single-branch --branch master --depth 1 https://github.com/google/gtest-parallel.git ~/gtest-parallel
echo "export PATH=$HOME/gtest-parallel:$PATH" >> $BASH_ENV
executors:
windows-2xlarge:
machine:
image: 'windows-server-2019-vs2019:stable'
image: 'windows-server-2019-vs2019:201908-06'
resource_class: windows.2xlarge
shell: bash.exe
jobs:
build-macos:
macos:
xcode: 11.3.0
steps:
- increase-max-open-files-on-macos
- install-pyenv-on-macos
- pre-steps
- install-gflags-on-macos
- run: ulimit -S -n 1048576 && OPT=-DCIRCLECI make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
- post-steps
build-linux:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- pre-steps
- install-gflags
- run: make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
- post-steps
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
- store_test_results:
path: /tmp/test-results
build-linux-shared_lib-alt_namespace-status_checked:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- pre-steps
- install-gflags
- run: ASSERT_STATUS_CHECKED=1 TEST_UINT128_COMPAT=1 LIB_MODE=shared OPT="-DROCKSDB_NAMESPACE=alternative_rocksdb_ns" make V=1 -j32 all check_some | .circleci/cat_ignore_eagain
- post-steps
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 ASSERT_STATUS_CHECKED=1 TEST_UINT128_COMPAT=1 LIB_MODE=shared OPT="-DROCKSDB_NAMESPACE=alternative_rocksdb_ns" GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 -j32 all check_some | .circleci/cat_ignore_eagain
- store_test_results:
path: /tmp/test-results
build-linux-release:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- install-gflags
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: ./db_stress --version # ensure with gflags
- post-steps
build-linux-release-rtti:
build-linux-lite:
machine:
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 LITE=1 GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
- store_test_results:
path: /tmp/test-results
build-linux-lite-release:
machine:
image: ubuntu-1604:201903-01
resource_class: large
steps:
- checkout # check out the code in the project directory
- run: make clean
- run: USE_RTTI=1 DEBUG_LEVEL=0 make V=1 -j16 static_lib tools db_bench | .circleci/cat_ignore_eagain
- run: LITE=1 make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: make clean
- run: USE_RTTI=1 DEBUG_LEVEL=0 make V=1 -j16 static_lib tools db_bench | .circleci/cat_ignore_eagain
- run: ./db_stress --version # ensure with gflags
build-linux-lite:
machine:
image: ubuntu-1604:202007-01
resource_class: 2xlarge
steps:
- pre-steps
- install-gflags
- run: LITE=1 make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
- post-steps
build-linux-lite-release:
machine:
image: ubuntu-1604:202007-01
resource_class: large
steps:
- checkout # check out the code in the project directory
- run: LITE=1 make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- install-gflags
- run: LITE=1 make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: ./db_stress --version # ensure with gflags
- post-steps
build-linux-clang-no-test:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: sudo apt-get update -y && sudo apt-get install -y clang libgflags-dev
- run: CC=clang CXX=clang++ USE_CLANG=1 PORTABLE=1 make V=1 -j32 all | .circleci/cat_ignore_eagain
- post-steps
build-linux-clang10-no-test:
machine:
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "APT::Acquire::Retries \"10\";" | sudo tee -a /etc/apt/apt.conf.d/80-retries # llvm.org unreliable
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
- run: CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j32 all | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
build-linux-clang10-asan:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- pre-steps
- install-gflags
- install-clang-10
- run: COMPILE_WITH_ASAN=1 CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j32 check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
- post-steps
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "APT::Acquire::Retries \"10\";" | sudo tee -a /etc/apt/apt.conf.d/80-retries # llvm.org unreliable
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 COMPILE_WITH_ASAN=1 CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 PRINT_PARALLEL_OUTPUTS=1 GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 -j32 check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
- store_test_results:
path: /tmp/test-results
build-linux-clang10-mini-tsan:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- pre-steps
- install-gflags
- install-clang-10
- run: COMPILE_WITH_TSAN=1 CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j32 check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out.
- post-steps
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "APT::Acquire::Retries \"10\";" | sudo tee -a /etc/apt/apt.conf.d/80-retries # llvm.org unreliable
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 COMPILE_WITH_TSAN=1 CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 PRINT_PARALLEL_OUTPUTS=1 EXCLUDE_TESTS_REGEX="TransactionStressTest|SnapshotConcurrentAccess|SeqAdvanceConcurrent|DeadlockStress|MultiThreadedDBTest.MultiThreaded|WriteUnpreparedStressTest.ReadYourOwnWriteStress|DBAsBaseDB/TransactionStressTest|FlushCloseWALFiles|BackgroundPurgeCFDropTest" GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 -j32 check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out. Exclude FlushCloseWALFiles and BackgroundPurgeCFDropTest for occasional failures.
- store_test_results:
path: /tmp/test-results
build-linux-clang10-ubsan:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- pre-steps
- install-gflags
- install-clang-10
- run: COMPILE_WITH_UBSAN=1 OPT="-fsanitize-blacklist=.circleci/ubsan_suppression_list.txt" CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j32 ubsan_check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
- post-steps
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "APT::Acquire::Retries \"10\";" | sudo tee -a /etc/apt/apt.conf.d/80-retries # llvm.org unreliable
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 COMPILE_WITH_UBSAN=1 OPT="-fsanitize-blacklist=.circleci/ubsan_suppression_list.txt" CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 PRINT_PARALLEL_OUTPUTS=1 make V=1 -j32 ubsan_check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
build-linux-clang10-clang-analyze:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- pre-steps
- install-gflags
- install-clang-10
- run: sudo apt-get update -y && sudo apt-get install -y clang-tools-10
- run: CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 CLANG_ANALYZER="/usr/bin/clang++-10" CLANG_SCAN_BUILD=scan-build-10 USE_CLANG=1 make V=1 -j32 analyze | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out. For unknown, reason passing "clang++-10" as CLANG_ANALYZER doesn't work, and we need a full path.
- post-steps
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "APT::Acquire::Retries \"10\";" | sudo tee -a /etc/apt/apt.conf.d/80-retries # llvm.org unreliable
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev clang-tools-10
- run: SKIP_FORMAT_BUCK_CHECKS=1 CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 CLANG_ANALYZER="/usr/bin/clang++-10" CLANG_SCAN_BUILD=scan-build-10 USE_CLANG=1 PRINT_PARALLEL_OUTPUTS=1 make V=1 -j32 analyze | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out. For unknown, reason passing "clang++-10" as CLANG_ANALYZER doesn't work, and we need a full path.
build-linux-cmake:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: (mkdir build && cd build && cmake -DWITH_GFLAGS=0 .. && make V=1 -j32) | .circleci/cat_ignore_eagain
- post-steps
build-linux-unity:
docker: # executor type
@@ -240,17 +164,16 @@ jobs:
- checkout # check out the code in the project directory
- run: apt-get update -y && apt-get install -y libgflags-dev
- run: TEST_TMPDIR=/dev/shm && make V=1 -j16 unity_test | .circleci/cat_ignore_eagain
- post-steps
build-linux-gcc-4-8:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: large
steps:
- pre-steps
- checkout
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install gcc-4.8 g++-4.8 libgflags-dev
- run: CC=gcc-4.8 CXX=g++-4.8 V=1 SKIP_LINK=1 make -j4 all | .circleci/cat_ignore_eagain # Linking broken because libgflags compiled with newer ABI
- post-steps
build-windows:
executor: windows-2xlarge
@@ -311,42 +234,42 @@ jobs:
name: "Test RocksDB"
shell: powershell.exe
command: |
build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,db_test,db_test2,db_merge_operand_test,bloom_test,c_test,coding_test,crc32c_test,dynamic_bloom_test,env_basic_test,env_test,hash_test,random_test -Concurrency 16
build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,db_test,db_test2,env_basic_test,env_test,db_merge_operand_test -Concurrency 16
build-linux-java:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- pre-steps
- install-gflags
- checkout
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run:
name: "Build RocksDBJava"
command: |
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0
export PATH=$JAVA_HOME/bin:$PATH
echo "JAVA_HOME=${JAVA_HOME}"
which java && java -version
which javac && javac -version
make V=1 J=32 -j32 rocksdbjava jtest | .circleci/cat_ignore_eagain
- post-steps
SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 make V=1 J=32 -j32 rocksdbjava jtest | .circleci/cat_ignore_eagain
build-examples:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: medium
steps:
- pre-steps
- install-gflags
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run:
name: "Build examples"
command: |
OPT=-DTRAVIS V=1 make -j4 static_lib && cd examples && make -j4 | ../.circleci/cat_ignore_eagain
- post-steps
build-linux-non-shm:
machine:
image: ubuntu-1604:202007-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
parameters:
start_test:
@@ -356,23 +279,13 @@ jobs:
default: ""
type: string
steps:
- pre-steps
- install-gflags
- install-gtest-parallel
- checkout # checkout the code in the project directory
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run:
name: "Build unit tests"
name: "Build and unit tests on non-shm"
command: |
echo "env: $(env)"
echo "** done env"
ROCKSDBTESTS_START=<<parameters.start_test>> ROCKSDBTESTS_END=<<parameters.end_test>> ROCKSDBTESTS_SUBSET_TESTS_TO_FILE=/tmp/test_list make V=1 -j32 --output-sync=target build_subset_tests
- run:
name: "Run unit tests in parallel"
command: |
sed -i 's/[[:space:]]*$//; s/ / \.\//g; s/.*/.\/&/' /tmp/test_list
cat /tmp/test_list
export TEST_TMPDIR=/tmp/rocksdb_test_tmp
/usr/bin/python ../gtest-parallel/gtest-parallel $(</tmp/test_list) --output_dir=/tmp | cat # pipe to cat to continuously output status on circleci UI. Otherwise, no status will be printed while the job is running.
- post-steps
TMPD=/tmp/rocksdb_test_tmp ROCKSDBTESTS_START=<<parameters.start_test>> ROCKSDBTESTS_END=<<parameters.end_test>> make V=1 J=32 -j32 check_some | .circleci/cat_ignore_eagain
workflows:
build-linux:
@@ -387,15 +300,15 @@ workflows:
build-linux-release:
jobs:
- build-linux-release
build-linux-release-rtti:
jobs:
- build-linux-release-rtti
build-linux-lite-release:
jobs:
- build-linux-lite-release
build-linux-clang-no-test:
jobs:
- build-linux-clang-no-test
build-linux-clang10-no-test:
jobs:
- build-linux-clang10-no-test
build-linux-clang10-asan:
jobs:
- build-linux-clang10-asan
@@ -437,23 +350,26 @@ workflows:
build-examples:
jobs:
- build-examples
build-linux-non-shm:
build-linux-non-shm-1:
jobs:
- build-linux-non-shm:
start_test: ""
end_test: "db_options_test" # make sure unique in src.mk
end_test: "db_tailing_iter_test" # make sure unique in src.mk
build-linux-non-shm-2:
jobs:
- build-linux-non-shm:
start_test: "db_options_test" # make sure unique in src.mk
end_test: "filename_test" # make sure unique in src.mk
start_test: "db_tailing_iter_test" # make sure unique in src.mk
end_test: "db_test2" # make sure unique in src.mk
build-linux-non-shm-3:
jobs:
- build-linux-non-shm:
start_test: "filename_test" # make sure unique in src.mk
end_test: "statistics_test" # make sure unique in src.mk
start_test: "db_test2" # make sure unique in src.mk
end_test: "arena_test" # make sure unique in src.mk
build-linux-non-shm-4:
jobs:
- build-linux-non-shm:
start_test: "statistics_test" # make sure unique in src.mk
start_test: "compact_on_deletion_collector_test" # make sure unique in src.mk
end_test: ""
build-linux-gcc-4-8:
jobs:
- build-linux-gcc-4-8
build-macos:
jobs:
- build-macos
+2 -2
View File
@@ -100,11 +100,11 @@ matrix:
arch: amd64
env: JOB_NAME=cmake-gcc9
# Exclude most osx, arm64 and ppc64le tests for pull requests, but build in branches
# Temporarily disable ppc64le cmake test while snapd is broken
# Temporarily disable ppc64le unit tests in PRs until Travis gets its act together (#6653)
- if: type = pull_request AND commit_message !~ /FULL_CI/
os: linux
arch: ppc64le
env: JOB_NAME=cmake
env: TEST_GROUP=platform_dependent
# NB: the cmake build is a partial java test
- if: type = pull_request AND commit_message !~ /FULL_CI/
os: osx
+3 -17
View File
@@ -391,12 +391,6 @@ if(NOT WITH_DYNAMIC_EXTENSION)
add_definitions(-DROCKSDB_NO_DYNAMIC_EXTENSION)
endif()
option(ASSERT_STATUS_CHECKED "build with assert status checked" OFF)
if (ASSERT_STATUS_CHECKED)
message(STATUS "Build with assert status checked")
add_definitions(-DROCKSDB_ASSERT_STATUS_CHECKED)
endif()
if(DEFINED USE_RTTI)
if(USE_RTTI)
message(STATUS "Enabling RTTI")
@@ -573,9 +567,8 @@ set(SOURCES
db/blob/blob_file_builder.cc
db/blob/blob_file_garbage.cc
db/blob/blob_file_meta.cc
db/blob/blob_file_reader.cc
db/blob/blob_log_format.cc
db/blob/blob_log_sequential_reader.cc
db/blob/blob_log_reader.cc
db/blob/blob_log_writer.cc
db/builder.cc
db/c.cc
@@ -621,8 +614,6 @@ set(SOURCES
db/memtable_list.cc
db/merge_helper.cc
db/merge_operator.cc
db/output_validator.cc
db/periodic_work_scheduler.cc
db/range_del_aggregator.cc
db/range_tombstone_fragmenter.cc
db/repair.cc
@@ -680,12 +671,12 @@ set(SOURCES
monitoring/perf_level.cc
monitoring/persistent_stats_history.cc
monitoring/statistics.cc
monitoring/stats_dump_scheduler.cc
monitoring/thread_status_impl.cc
monitoring/thread_status_updater.cc
monitoring/thread_status_util.cc
monitoring/thread_status_util_debug.cc
options/cf_options.cc
options/configurable.cc
options/db_options.cc
options/options.cc
options/options_helper.cc
@@ -736,7 +727,6 @@ set(SOURCES
table/sst_file_dumper.cc
table/sst_file_reader.cc
table/sst_file_writer.cc
table/table_factory.cc
table/table_properties.cc
table/two_level_iterator.cc
test_util/sync_point.cc
@@ -745,7 +735,6 @@ set(SOURCES
test_util/transaction_test_util.cc
tools/block_cache_analyzer/block_cache_trace_analyzer.cc
tools/dump/db_dump_tool.cc
tools/io_tracer_parser_tool.cc
tools/ldb_cmd.cc
tools/ldb_tool.cc
tools/sst_dump_tool.cc
@@ -1045,7 +1034,6 @@ if(WITH_TESTS)
db/blob/blob_file_addition_test.cc
db/blob/blob_file_builder_test.cc
db/blob/blob_file_garbage_test.cc
db/blob/blob_file_reader_test.cc
db/blob/db_blob_index_test.cc
db/column_family_test.cc
db/compact_files_test.cc
@@ -1105,7 +1093,6 @@ if(WITH_TESTS)
db/merge_test.cc
db/options_file_test.cc
db/perf_context_test.cc
db/periodic_work_scheduler_test.cc
db/plain_table_db_test.cc
db/prefix_test.cc
db/range_del_aggregator_test.cc
@@ -1137,8 +1124,8 @@ if(WITH_TESTS)
monitoring/histogram_test.cc
monitoring/iostats_context_test.cc
monitoring/statistics_test.cc
monitoring/stats_dump_scheduler_test.cc
monitoring/stats_history_test.cc
options/configurable_test.cc
options/options_settable_test.cc
options/options_test.cc
table/block_based/block_based_filter_block_test.cc
@@ -1157,7 +1144,6 @@ if(WITH_TESTS)
test_util/testutil_test.cc
trace_replay/io_tracer_test.cc
tools/block_cache_analyzer/block_cache_trace_analyzer_test.cc
tools/io_tracer_parser_test.cc
tools/ldb_cmd_test.cc
tools/reduce_levels_test.cc
tools/sst_dump_test.cc
+18 -29
View File
@@ -1,36 +1,24 @@
# Rocksdb Change Log
## Unreleased
## 6.13.4 (11/15/2020)
### Bug Fixes
* Fixed a bug of encoding and parsing BlockBasedTableOptions::read_amp_bytes_per_bit as a 64-bit integer.
* Fixed the logic of populating native data structure for `read_amp_bytes_per_bit` during OPTIONS file parsing on big-endian architecture. Without this fix, original code introduced in PR7659, when running on big-endian machine, can mistakenly store read_amp_bytes_per_bit (an uint32) in little endian format. Future access to `read_amp_bytes_per_bit` will give wrong values. Little endian architecture is not affected.
## 6.13.3 (10/14/2020)
### Bug Fixes
* Fix a bug that could cause a stalled write to crash with mixed of slowdown and no_slowdown writes (`WriteOptions.no_slowdown=true`).
## 6.13.2 (10/13/2020)
### Bug Fixes
* Fix false positive flush/compaction `Status::Corruption` failure when `paranoid_file_checks == true` and range tombstones were written to the compaction output files.
## 6.13.1 (10/12/2020)
### Bug Fixes
* Since 6.12, memtable lookup should report unrecognized value_type as corruption (#7121).
* Fixed a bug in the following combination of features: indexes with user keys (`format_version >= 3`), indexes are partitioned (`index_type == kTwoLevelIndexSearch`), and some index partitions are pinned in memory (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache`). The bug could cause keys to be truncated when read from the index leading to wrong read results or other unexpected behavior.
* Fixed a bug when indexes are partitioned (`index_type == kTwoLevelIndexSearch`), some index partitions are pinned in memory (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache`), and partitions reads could be mixed between block cache and directly from the file (e.g., with `enable_index_compression == 1` and `mmap_read == 1`, partitions that were stored uncompressed due to poor compression ratio would be read directly from the file via mmap, while partitions that were stored compressed would be read from block cache). The bug could cause index partitions to be mistakenly considered empty during reads leading to wrong read results.
* Since 6.12, memtable lookup should report unrecognized value_type as corruption (#7121).
* Since 6.14, fix false positive flush/compaction `Status::Corruption` failure when `paranoid_file_checks == true` and range tombstones were written to the compaction output files.
### Public API Change
* Deprecate `BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache` and `BlockBasedTableOptions::pin_top_level_index_and_filter`. These options still take effect until users migrate to the replacement APIs in `BlockBasedTableOptions::metadata_cache_options`. Migration guidance can be found in the API comments on the deprecated options.
## 6.14 (10/09/2020)
### Bug fixes
* Fixed a bug after a `CompactRange()` with `CompactRangeOptions::change_level` set fails due to a conflict in the level change step, which caused all subsequent calls to `CompactRange()` with `CompactRangeOptions::change_level` set to incorrectly fail with a `Status::NotSupported("another thread is refitting")` error.
* Fixed a bug that the bottom most level compaction could still be a trivial move even if `BottommostLevelCompaction.kForce` or `kForceOptimized` is set.
### Public API Change
* The methods to create and manage EncrypedEnv have been changed. The EncryptionProvider is now passed to NewEncryptedEnv as a shared pointer, rather than a raw pointer. Comparably, the CTREncryptedProvider now takes a shared pointer, rather than a reference, to a BlockCipher. CreateFromString methods have been added to BlockCipher and EncryptionProvider to provide a single API by which different ciphers and providers can be created, respectively.
* The internal classes (CTREncryptionProvider, ROT13BlockCipher, CTRCipherStream) associated with the EncryptedEnv have been moved out of the public API. To create a CTREncryptionProvider, one can either use EncryptionProvider::NewCTRProvider, or EncryptionProvider::CreateFromString("CTR"). To create a new ROT13BlockCipher, one can either use BlockCipher::NewROT13Cipher or BlockCipher::CreateFromString("ROT13").
* The EncryptionProvider::AddCipher method has been added to allow keys to be added to an EncryptionProvider. This API will allow future providers to support multiple cipher keys.
* Add a new option "allow_data_in_errors". When this new option is set by users, it allows users to opt-in to get error messages containing corrupted keys/values. Corrupt keys, values will be logged in the messages, logs, status etc. that will help users with the useful information regarding affected data. By default value of this option is set false to prevent users data to be exposed in the messages so currently, data will be redacted from logs, messages, status by default.
* AdvancedColumnFamilyOptions::force_consistency_checks is now true by default, for more proactive DB corruption detection at virtually no cost (estimated two extra CPU cycles per million on a major production workload). Corruptions reported by these checks now mention "force_consistency_checks" in case a false positive corruption report is suspected and the option needs to be disabled (unlikely). Since existing column families have a saved setting for force_consistency_checks, only new column families will pick up the new default.
### General Improvements
* The settings of the DBOptions and ColumnFamilyOptions are now managed by Configurable objects (see New Features). The same convenience methods to configure these options still exist but the backend implementation has been unified under a common implementation.
### New Features
* Methods to configure serialize, and compare -- such as TableFactory -- are exposed directly through the Configurable base class (from which these objects inherit). This change will allow for better and more thorough configuration management and retrieval in the future. The options for a Configurable object can be set via the ConfigureFromMap, ConfigureFromString, or ConfigureOption method. The serialized version of the options of an object can be retrieved via the GetOptionString, ToString, or GetOption methods. The list of options supported by an object can be obtained via the GetOptionNames method. The "raw" object (such as the BlockBasedTableOption) for an option may be retrieved via the GetOptions method. Configurable options can be compared via the AreEquivalent method. The settings within a Configurable object may be validated via the ValidateOptions method. The object may be intialized (at which point only mutable options may be updated) via the PrepareOptions method.
* Introduce options.check_flush_compaction_key_order with default value to be true. With this option, during flush and compaction, key order will be checked when writing to each SST file. If the order is violated, the flush or compaction will fail.
* Added is_full_compaction to CompactionJobStats, so that the information is available through the EventListener interface.
* Add more stats for MultiGet in Histogram to get number of data blocks, index blocks, filter blocks and sst files read from file system per level.
## 6.13 (09/12/2020)
## 6.13 (09/24/2020)
### Bug fixes
* Fix a performance regression introduced in 6.4 that makes a upper bound check for every Next() even if keys are within a data block that is within the upper bound.
* Fix a possible corruption to the LSM state (overlapping files within a level) when a `CompactRange()` for refitting levels (`CompactRangeOptions::change_level == true`) and another manual compaction are executed in parallel.
@@ -65,7 +53,7 @@
### Behavior Changes
* File abstraction `FSRandomAccessFile.Prefetch()` default return status is changed from `OK` to `NotSupported`. If the user inherited file doesn't implement prefetch, RocksDB will create internal prefetch buffer to improve read performance.
* When retryabel IO error happens during Flush (manifest write error is excluded) and WAL is disabled, originally it is mapped to kHardError. Now,it is mapped to soft error. So DB will not stall the writes unless the memtable is full. At the same time, when auto resume is triggered to recover the retryable IO error during Flush, SwitchMemtable is not called to avoid generating to many small immutable memtables. If WAL is enabled, no behavior changes.
* When retryable IO error happens during Flush (manifest write error is excluded) and WAL is disabled, originally it is mapped to kHardError. Now,it is mapped to soft error. So DB will not stall the writes unless the memtable is full. At the same time, when auto resume is triggered to recover the retryable IO error during Flush, SwitchMemtable is not called to avoid generating to many small immutable memtables. If WAL is enabled, no behavior changes.
* When considering whether a table file is already backed up in a shared part of backup directory, BackupEngine would already query the sizes of source (DB) and pre-existing destination (backup) files. BackupEngine now uses these file sizes to detect corruption, as at least one of (a) old backup, (b) backup in progress, or (c) current DB is corrupt if there's a size mismatch.
### Others
@@ -80,6 +68,7 @@
* `DB::OpenForReadOnly()` now returns `Status::NotFound` when the specified DB directory does not exist. Previously the error returned depended on the underlying `Env`. This change is available in all 6.11 releases as well.
* A parameter `verify_with_checksum` is added to `BackupEngine::VerifyBackup`, which is false by default. If it is ture, `BackupEngine::VerifyBackup` verifies checksums and file sizes of backup files. Pass `false` for `verify_with_checksum` to maintain the previous behavior and performance of `BackupEngine::VerifyBackup`, by only verifying sizes of backup files.
### Behavior Changes
* Best-efforts recovery ignores CURRENT file completely. If CURRENT file is missing during recovery, best-efforts recovery still proceeds with MANIFEST file(s).
* In best-efforts recovery, an error that is not Corruption or IOError::kNotFound or IOError::kPathNotFound will be overwritten silently. Fix this by checking all non-ok cases and return early.
+13 -80
View File
@@ -560,7 +560,6 @@ PARALLEL_TEST = \
ifeq ($(USE_FOLLY_DISTRIBUTED_MUTEX),1)
TESTS += folly_synchronization_distributed_mutex_test
PARALLEL_TEST += folly_synchronization_distributed_mutex_test
TESTS_PASSING_ASC = folly_synchronization_distributed_mutex_test
endif
# options_settable_test doesn't pass with UBSAN as we use hack in the test
@@ -578,7 +577,6 @@ ifdef ASSERT_STATUS_CHECKED
blob_file_addition_test \
blob_file_builder_test \
blob_file_garbage_test \
blob_file_reader_test \
bloom_test \
cassandra_format_test \
cassandra_row_merge_test \
@@ -587,14 +585,6 @@ ifdef ASSERT_STATUS_CHECKED
coding_test \
crc32c_test \
dbformat_test \
db_basic_test \
db_flush_test \
db_with_timestamp_basic_test \
db_with_timestamp_compaction_test \
db_options_test \
db_properties_test \
db_secondary_test \
options_file_test \
defer_test \
filename_test \
dynamic_bloom_test \
@@ -602,10 +592,7 @@ ifdef ASSERT_STATUS_CHECKED
env_test \
env_logger_test \
event_logger_test \
error_handler_fs_test \
auto_roll_logger_test \
file_indexer_test \
flush_job_test \
hash_table_test \
hash_test \
heap_test \
@@ -613,61 +600,33 @@ ifdef ASSERT_STATUS_CHECKED
inlineskiplist_test \
io_posix_test \
iostats_context_test \
ldb_cmd_test \
memkind_kmem_allocator_test \
merger_test \
mock_env_test \
object_registry_test \
prefix_test \
repair_test \
configurable_test \
options_settable_test \
options_test \
random_test \
range_del_aggregator_test \
sst_file_reader_test \
range_tombstone_fragmenter_test \
repeatable_thread_test \
skiplist_test \
slice_test \
sst_dump_test \
statistics_test \
stats_history_test \
thread_local_test \
trace_analyzer_test \
env_timed_test \
filelock_test \
timer_queue_test \
timer_test \
options_util_test \
persistent_cache_test \
util_merge_operators_test \
block_cache_trace_analyzer_test \
block_cache_tracer_test \
cache_simulator_test \
sim_cache_test \
version_builder_test \
version_edit_test \
work_queue_test \
write_controller_test \
compaction_iterator_test \
compaction_job_test \
compaction_job_stats_test \
io_tracer_test \
merge_helper_test \
memtable_list_test \
flush_job_test \
block_based_filter_block_test \
block_fetcher_test \
full_filter_block_test \
partitioned_filter_block_test \
column_family_test \
file_reader_writer_test \
corruption_test \
db_universal_compaction_test \
import_column_family_test \
memory_test \
table_test \
ifeq ($(USE_FOLLY_DISTRIBUTED_MUTEX),1)
TESTS_PASSING_ASC += folly_synchronization_distributed_mutex_test
@@ -675,13 +634,13 @@ endif
# Enable building all unit tests, but use check_some to run only tests
# known to pass ASC (ASSERT_STATUS_CHECKED)
ROCKSDBTESTS_SUBSET ?= $(TESTS_PASSING_ASC)
SUBSET := $(TESTS_PASSING_ASC)
# Alternate: only build unit tests known to pass ASC, and run them
# with make check
#TESTS := $(filter $(TESTS_PASSING_ASC),$(TESTS))
#PARALLEL_TEST := $(filter $(TESTS_PASSING_ASC),$(PARALLEL_TEST))
else
ROCKSDBTESTS_SUBSET ?= $(TESTS)
SUBSET := $(TESTS)
endif
# Not necessarily well thought out or up-to-date, but matches old list
TESTS_PLATFORM_DEPENDENT := \
@@ -711,22 +670,22 @@ TESTS_PLATFORM_DEPENDENT := \
iostats_context_test \
db_wal_test \
# Sort ROCKSDBTESTS_SUBSET for filtering, except db_test is special (expensive)
# so is placed first (out-of-order)
ROCKSDBTESTS_SUBSET := $(filter db_test, $(ROCKSDBTESTS_SUBSET)) $(sort $(filter-out db_test, $(ROCKSDBTESTS_SUBSET)))
# Sort SUBSET for filtering, except db_test is special (expensive) so
# is placed first (out-of-order)
SUBSET := $(filter db_test, $(SUBSET)) $(sort $(filter-out db_test, $(SUBSET)))
ifdef ROCKSDBTESTS_START
ROCKSDBTESTS_SUBSET := $(shell echo $(ROCKSDBTESTS_SUBSET) | sed 's/^.*$(ROCKSDBTESTS_START)/$(ROCKSDBTESTS_START)/')
SUBSET := $(shell echo $(SUBSET) | sed 's/^.*$(ROCKSDBTESTS_START)/$(ROCKSDBTESTS_START)/')
endif
ifdef ROCKSDBTESTS_END
ROCKSDBTESTS_SUBSET := $(shell echo $(ROCKSDBTESTS_SUBSET) | sed 's/$(ROCKSDBTESTS_END).*//')
SUBSET := $(shell echo $(SUBSET) | sed 's/$(ROCKSDBTESTS_END).*//')
endif
ifeq ($(ROCKSDBTESTS_PLATFORM_DEPENDENT), only)
ROCKSDBTESTS_SUBSET := $(filter $(TESTS_PLATFORM_DEPENDENT), $(ROCKSDBTESTS_SUBSET))
SUBSET := $(filter $(TESTS_PLATFORM_DEPENDENT), $(SUBSET))
else ifeq ($(ROCKSDBTESTS_PLATFORM_DEPENDENT), exclude)
ROCKSDBTESTS_SUBSET := $(filter-out $(TESTS_PLATFORM_DEPENDENT), $(ROCKSDBTESTS_SUBSET))
SUBSET := $(filter-out $(TESTS_PLATFORM_DEPENDENT), $(SUBSET))
endif
# bench_tool_analyer main is in bench_tool_analyzer_tool, or this would be simpler...
@@ -828,7 +787,7 @@ endif # PLATFORM_SHARED_EXT
all: $(LIBRARY) $(BENCHMARKS) tools tools_lib test_libs $(TESTS)
all_but_some_tests: $(LIBRARY) $(BENCHMARKS) tools tools_lib test_libs $(ROCKSDBTESTS_SUBSET)
all_but_some_tests: $(LIBRARY) $(BENCHMARKS) tools tools_lib test_libs $(SUBSET)
static_lib: $(STATIC_LIBRARY)
@@ -1046,8 +1005,8 @@ ifndef SKIP_FORMAT_BUCK_CHECKS
endif
# TODO add ldb_tests
check_some: $(ROCKSDBTESTS_SUBSET)
for t in $(ROCKSDBTESTS_SUBSET); do echo "===== Running $$t (`date`)"; ./$$t || exit 1; done
check_some: $(SUBSET)
for t in $(SUBSET); do echo "===== Running $$t (`date`)"; ./$$t || exit 1; done
.PHONY: ldb_tests
ldb_tests: ldb
@@ -1151,9 +1110,6 @@ ubsan_crash_test_with_best_efforts_recovery: clean
valgrind_test:
ROCKSDB_VALGRIND_RUN=1 DISABLE_JEMALLOC=1 $(MAKE) valgrind_check
valgrind_test_some:
ROCKSDB_VALGRIND_RUN=1 DISABLE_JEMALLOC=1 $(MAKE) valgrind_check_some
valgrind_check: $(TESTS)
$(MAKE) DRIVER="$(VALGRIND_VER) $(VALGRIND_OPTS)" gen_parallel_tests
$(AM_V_GEN)if test "$(J)" != 1 \
@@ -1172,14 +1128,6 @@ valgrind_check: $(TESTS)
done; \
fi
valgrind_check_some: $(ROCKSDBTESTS_SUBSET)
for t in $(ROCKSDBTESTS_SUBSET); do \
$(VALGRIND_VER) $(VALGRIND_OPTS) ./$$t; \
ret_code=$$?; \
if [ $$ret_code -ne 0 ]; then \
exit $$ret_code; \
fi; \
done
ifneq ($(PAR_TEST),)
parloop:
@@ -1781,9 +1729,6 @@ thread_list_test: $(OBJ_DIR)/util/thread_list_test.o $(TEST_LIBRARY) $(LIBRARY)
compact_files_test: $(OBJ_DIR)/db/compact_files_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
configurable_test: options/configurable_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
options_test: $(OBJ_DIR)/options/options_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
@@ -1916,13 +1861,10 @@ blob_file_builder_test: $(OBJ_DIR)/db/blob/blob_file_builder_test.o $(TEST_LIBRA
blob_file_garbage_test: $(OBJ_DIR)/db/blob/blob_file_garbage_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
blob_file_reader_test: $(OBJ_DIR)/db/blob/blob_file_reader_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
timer_test: $(OBJ_DIR)/util/timer_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
periodic_work_scheduler_test: $(OBJ_DIR)/db/periodic_work_scheduler_test.o $(TEST_LIBRARY) $(LIBRARY)
stats_dump_scheduler_test: $(OBJ_DIR)/monitoring/stats_dump_scheduler_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
testutil_test: $(OBJ_DIR)/test_util/testutil_test.o $(TEST_LIBRARY) $(LIBRARY)
@@ -1934,12 +1876,6 @@ io_tracer_test: $(OBJ_DIR)/trace_replay/io_tracer_test.o $(OBJ_DIR)/trace_replay
prefetch_test: $(OBJ_DIR)/file/prefetch_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
io_tracer_parser_test: $(OBJ_DIR)/tools/io_tracer_parser_test.o $(OBJ_DIR)/tools/io_tracer_parser_tool.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
io_tracer_parser: $(OBJ_DIR)/tools/io_tracer_parser.o $(TOOLS_LIBRARY) $(LIBRARY)
$(AM_LINK)
#-------------------------------------------------
# make install related stuff
PREFIX ?= /usr/local
@@ -2377,9 +2313,6 @@ else
depend: $(DEPFILES)
endif
build_subset_tests: $(ROCKSDBTESTS_SUBSET)
$(AM_V_GEN)if [ -n "$${ROCKSDBTESTS_SUBSET_TESTS_TO_FILE}" ]; then echo "$(ROCKSDBTESTS_SUBSET)" > "$${ROCKSDBTESTS_SUBSET_TESTS_TO_FILE}"; else echo "$(ROCKSDBTESTS_SUBSET)"; fi
# if the make goal is either "clean" or "format", we shouldn't
# try to import the *.d files.
# TODO(kailiu) The unfamiliarity of Make's conditions leads to the ugly
+17 -72
View File
@@ -25,16 +25,13 @@ ROCKSDB_EXTERNAL_DEPS = [
("gflags", None, "gflags"),
("lz4", None, "lz4"),
("zstd", None),
("tbb", None),
]
ROCKSDB_OS_DEPS = [
(
"linux",
["third-party//numa:numa", "third-party//liburing:uring", "third-party//tbb:tbb"],
),
(
"macos",
["third-party//tbb:tbb"],
["third-party//numa:numa", "third-party//liburing:uring"],
),
]
@@ -52,27 +49,17 @@ ROCKSDB_OS_PREPROCESSOR_FLAGS = [
"-DHAVE_SSE42",
"-DLIBURING",
"-DNUMA",
"-DROCKSDB_PLATFORM_POSIX",
"-DROCKSDB_LIB_IO_POSIX",
"-DTBB",
],
),
(
"macos",
[
"-DOS_MACOSX",
"-DROCKSDB_PLATFORM_POSIX",
"-DROCKSDB_LIB_IO_POSIX",
"-DTBB",
],
),
(
"windows",
[ "-DOS_WIN", "-DWIN32", "-D_MBCS", "-DWIN64", "-DNOMINMAX" ]
["-DOS_MACOSX"],
),
]
ROCKSDB_PREPROCESSOR_FLAGS = [
"-DROCKSDB_PLATFORM_POSIX",
"-DROCKSDB_LIB_IO_POSIX",
"-DROCKSDB_SUPPORT_THREAD_LOCAL",
# Flags to enable libs we include
@@ -83,6 +70,7 @@ ROCKSDB_PREPROCESSOR_FLAGS = [
"-DZSTD",
"-DZSTD_STATIC_LINKING_ONLY",
"-DGFLAGS=gflags",
"-DTBB",
# Added missing flags from output of build_detect_platform
"-DROCKSDB_BACKTRACE",
@@ -137,9 +125,8 @@ cpp_library(
"db/blob/blob_file_builder.cc",
"db/blob/blob_file_garbage.cc",
"db/blob/blob_file_meta.cc",
"db/blob/blob_file_reader.cc",
"db/blob/blob_log_format.cc",
"db/blob/blob_log_sequential_reader.cc",
"db/blob/blob_log_reader.cc",
"db/blob/blob_log_writer.cc",
"db/builder.cc",
"db/c.cc",
@@ -185,8 +172,6 @@ cpp_library(
"db/memtable_list.cc",
"db/merge_helper.cc",
"db/merge_operator.cc",
"db/output_validator.cc",
"db/periodic_work_scheduler.cc",
"db/range_del_aggregator.cc",
"db/range_tombstone_fragmenter.cc",
"db/repair.cc",
@@ -247,25 +232,19 @@ cpp_library(
"monitoring/perf_level.cc",
"monitoring/persistent_stats_history.cc",
"monitoring/statistics.cc",
"monitoring/stats_dump_scheduler.cc",
"monitoring/thread_status_impl.cc",
"monitoring/thread_status_updater.cc",
"monitoring/thread_status_updater_debug.cc",
"monitoring/thread_status_util.cc",
"monitoring/thread_status_util_debug.cc",
"options/cf_options.cc",
"options/configurable.cc",
"options/db_options.cc",
"options/options.cc",
"options/options_helper.cc",
"options/options_parser.cc",
"port/port_posix.cc",
"port/stack_trace.cc",
"port/win/env_default.cc",
"port/win/env_win.cc",
"port/win/io_win.cc",
"port/win/port_win.cc",
"port/win/win_logger.cc",
"port/win/win_thread.cc",
"table/adaptive/adaptive_table_factory.cc",
"table/block_based/binary_search_index_reader.cc",
"table/block_based/block.cc",
@@ -311,14 +290,12 @@ cpp_library(
"table/sst_file_dumper.cc",
"table/sst_file_reader.cc",
"table/sst_file_writer.cc",
"table/table_factory.cc",
"table/table_properties.cc",
"table/two_level_iterator.cc",
"test_util/sync_point.cc",
"test_util/sync_point_impl.cc",
"test_util/transaction_test_util.cc",
"tools/dump/db_dump_tool.cc",
"tools/io_tracer_parser_tool.cc",
"tools/ldb_cmd.cc",
"tools/ldb_tool.cc",
"tools/sst_dump_tool.cc",
@@ -425,9 +402,8 @@ cpp_library(
"db/blob/blob_file_builder.cc",
"db/blob/blob_file_garbage.cc",
"db/blob/blob_file_meta.cc",
"db/blob/blob_file_reader.cc",
"db/blob/blob_log_format.cc",
"db/blob/blob_log_sequential_reader.cc",
"db/blob/blob_log_reader.cc",
"db/blob/blob_log_writer.cc",
"db/builder.cc",
"db/c.cc",
@@ -473,8 +449,6 @@ cpp_library(
"db/memtable_list.cc",
"db/merge_helper.cc",
"db/merge_operator.cc",
"db/output_validator.cc",
"db/periodic_work_scheduler.cc",
"db/range_del_aggregator.cc",
"db/range_tombstone_fragmenter.cc",
"db/repair.cc",
@@ -535,25 +509,19 @@ cpp_library(
"monitoring/perf_level.cc",
"monitoring/persistent_stats_history.cc",
"monitoring/statistics.cc",
"monitoring/stats_dump_scheduler.cc",
"monitoring/thread_status_impl.cc",
"monitoring/thread_status_updater.cc",
"monitoring/thread_status_updater_debug.cc",
"monitoring/thread_status_util.cc",
"monitoring/thread_status_util_debug.cc",
"options/cf_options.cc",
"options/configurable.cc",
"options/db_options.cc",
"options/options.cc",
"options/options_helper.cc",
"options/options_parser.cc",
"port/port_posix.cc",
"port/stack_trace.cc",
"port/win/env_default.cc",
"port/win/env_win.cc",
"port/win/io_win.cc",
"port/win/port_win.cc",
"port/win/win_logger.cc",
"port/win/win_thread.cc",
"table/adaptive/adaptive_table_factory.cc",
"table/block_based/binary_search_index_reader.cc",
"table/block_based/block.cc",
@@ -599,14 +567,12 @@ cpp_library(
"table/sst_file_dumper.cc",
"table/sst_file_reader.cc",
"table/sst_file_writer.cc",
"table/table_factory.cc",
"table/table_properties.cc",
"table/two_level_iterator.cc",
"test_util/sync_point.cc",
"test_util/sync_point_impl.cc",
"test_util/transaction_test_util.cc",
"tools/dump/db_dump_tool.cc",
"tools/io_tracer_parser_tool.cc",
"tools/ldb_cmd.cc",
"tools/ldb_tool.cc",
"tools/sst_dump_tool.cc",
@@ -864,13 +830,6 @@ ROCKS_TESTS = [
[],
[],
],
[
"blob_file_reader_test",
"db/blob/blob_file_reader_test.cc",
"serial",
[],
[],
],
[
"block_based_filter_block_test",
"table/block_based/block_based_filter_block_test.cc",
@@ -1039,13 +998,6 @@ ROCKS_TESTS = [
[],
[],
],
[
"configurable_test",
"options/configurable_test.cc",
"serial",
[],
[],
],
[
"corruption_test",
"db/corruption_test.cc",
@@ -1515,13 +1467,6 @@ ROCKS_TESTS = [
[],
[],
],
[
"io_tracer_parser_test",
"tools/io_tracer_parser_test.cc",
"serial",
[],
[],
],
[
"io_tracer_test",
"trace_replay/io_tracer_test.cc",
@@ -1690,13 +1635,6 @@ ROCKS_TESTS = [
[],
[],
],
[
"periodic_work_scheduler_test",
"db/periodic_work_scheduler_test.cc",
"serial",
[],
[],
],
[
"persistent_cache_test",
"utilities/persistent_cache/persistent_cache_test.cc",
@@ -1830,6 +1768,13 @@ ROCKS_TESTS = [
[],
[],
],
[
"stats_dump_scheduler_test",
"monitoring/stats_dump_scheduler_test.cc",
"serial",
[],
[],
],
[
"stats_history_test",
"monitoring/stats_history_test.cc",
+10 -10
View File
@@ -27,9 +27,9 @@ def pretty_list(lst, indent=8):
class TARGETSBuilder(object):
def __init__(self, path):
self.path = path
self.targets_file = open(path, 'wb')
self.targets_file = open(path, 'w')
header = targets_cfg.rocksdb_target_header_template
self.targets_file.write(header.encode("utf-8"))
self.targets_file.write(header)
self.total_lib = 0
self.total_bin = 0
self.total_test = 0
@@ -53,7 +53,7 @@ class TARGETSBuilder(object):
headers=headers,
deps=pretty_list(deps),
extra_external_deps=extra_external_deps,
link_whole=link_whole).encode("utf-8"))
link_whole=link_whole))
self.total_lib = self.total_lib + 1
def add_rocksdb_library(self, name, srcs, headers=None):
@@ -67,18 +67,18 @@ class TARGETSBuilder(object):
name=name,
srcs=pretty_list(srcs),
headers_attr_prefix=headers_attr_prefix,
headers=headers).encode("utf-8"))
headers=headers))
self.total_lib = self.total_lib + 1
def add_binary(self, name, srcs, deps=None):
self.targets_file.write(targets_cfg.binary_template.format(
name=name,
srcs=pretty_list(srcs),
deps=pretty_list(deps)).encode("utf-8"))
self.targets_file.write(targets_cfg.binary_template % (
name,
pretty_list(srcs),
pretty_list(deps)))
self.total_bin = self.total_bin + 1
def add_c_test(self):
self.targets_file.write(b"""
self.targets_file.write("""
if not is_opt_mode:
cpp_binary(
name = "c_test_bin",
@@ -120,5 +120,5 @@ if not is_opt_mode:
self.total_test = self.total_test + 1
def flush_tests(self):
self.targets_file.write(targets_cfg.unittests_template.format(tests=self.tests_cfg).encode("utf-8"))
self.targets_file.write(targets_cfg.unittests_template % self.tests_cfg)
self.tests_cfg = ""
+10 -22
View File
@@ -32,16 +32,13 @@ ROCKSDB_EXTERNAL_DEPS = [
("gflags", None, "gflags"),
("lz4", None, "lz4"),
("zstd", None),
("tbb", None),
]
ROCKSDB_OS_DEPS = [
(
"linux",
["third-party//numa:numa", "third-party//liburing:uring", "third-party//tbb:tbb"],
),
(
"macos",
["third-party//tbb:tbb"],
["third-party//numa:numa", "third-party//liburing:uring"],
),
]
@@ -59,27 +56,17 @@ ROCKSDB_OS_PREPROCESSOR_FLAGS = [
"-DHAVE_SSE42",
"-DLIBURING",
"-DNUMA",
"-DROCKSDB_PLATFORM_POSIX",
"-DROCKSDB_LIB_IO_POSIX",
"-DTBB",
],
),
(
"macos",
[
"-DOS_MACOSX",
"-DROCKSDB_PLATFORM_POSIX",
"-DROCKSDB_LIB_IO_POSIX",
"-DTBB",
],
),
(
"windows",
[ "-DOS_WIN", "-DWIN32", "-D_MBCS", "-DWIN64", "-DNOMINMAX" ]
["-DOS_MACOSX"],
),
]
ROCKSDB_PREPROCESSOR_FLAGS = [
"-DROCKSDB_PLATFORM_POSIX",
"-DROCKSDB_LIB_IO_POSIX",
"-DROCKSDB_SUPPORT_THREAD_LOCAL",
# Flags to enable libs we include
@@ -90,6 +77,7 @@ ROCKSDB_PREPROCESSOR_FLAGS = [
"-DZSTD",
"-DZSTD_STATIC_LINKING_ONLY",
"-DGFLAGS=gflags",
"-DTBB",
# Added missing flags from output of build_detect_platform
"-DROCKSDB_BACKTRACE",
@@ -167,12 +155,12 @@ cpp_library(
binary_template = """
cpp_binary(
name = "{name}",
srcs = [{srcs}],
name = "%s",
srcs = [%s],
arch_preprocessor_flags = ROCKSDB_ARCH_PREPROCESSOR_FLAGS,
compiler_flags = ROCKSDB_COMPILER_FLAGS,
preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
deps = [{deps}],
deps = [%s],
external_deps = ROCKSDB_EXTERNAL_DEPS,
)
"""
@@ -189,7 +177,7 @@ test_cfg_template = """ [
unittests_template = """
# [test_name, test_src, test_type, extra_deps, extra_compiler_flags]
ROCKS_TESTS = [
{tests}]
%s]
# Generate a test rule for each entry in ROCKS_TESTS
# Do not build the tests in opt mode, since SyncPoint and other test code
+9 -6
View File
@@ -10,7 +10,7 @@
#include "rocksdb/cache.h"
#include "cache/lru_cache.h"
#include "rocksdb/utilities/options_type.h"
#include "options/options_helper.h"
#include "util/string_util.h"
namespace ROCKSDB_NAMESPACE {
@@ -19,19 +19,22 @@ static std::unordered_map<std::string, OptionTypeInfo>
lru_cache_options_type_info = {
{"capacity",
{offsetof(struct LRUCacheOptions, capacity), OptionType::kSizeT,
OptionVerificationType::kNormal, OptionTypeFlags::kMutable}},
OptionVerificationType::kNormal, OptionTypeFlags::kMutable,
offsetof(struct LRUCacheOptions, capacity)}},
{"num_shard_bits",
{offsetof(struct LRUCacheOptions, num_shard_bits), OptionType::kInt,
OptionVerificationType::kNormal, OptionTypeFlags::kMutable}},
OptionVerificationType::kNormal, OptionTypeFlags::kMutable,
offsetof(struct LRUCacheOptions, num_shard_bits)}},
{"strict_capacity_limit",
{offsetof(struct LRUCacheOptions, strict_capacity_limit),
OptionType::kBoolean, OptionVerificationType::kNormal,
OptionTypeFlags::kMutable}},
OptionTypeFlags::kMutable,
offsetof(struct LRUCacheOptions, strict_capacity_limit)}},
{"high_pri_pool_ratio",
{offsetof(struct LRUCacheOptions, high_pri_pool_ratio),
OptionType::kDouble, OptionVerificationType::kNormal,
OptionTypeFlags::kMutable}},
};
OptionTypeFlags::kMutable,
offsetof(struct LRUCacheOptions, high_pri_pool_ratio)}}};
#endif // ROCKSDB_LITE
Status Cache::CreateFromString(const ConfigOptions& config_options,
+1 -1
View File
@@ -136,7 +136,7 @@ struct KeyGen {
for (uint32_t i = 0; i < FLAGS_skew; ++i) {
raw = std::min(raw, rnd.Next());
}
uint64_t key = FastRange64(raw, max_key);
uint64_t key = fastrange64(raw, max_key);
// Variable size and alignment
size_t off = key % 8;
key_data[0] = char{42};
+3 -15
View File
@@ -31,13 +31,12 @@ BlobFileBuilder::BlobFileBuilder(
int job_id, uint32_t column_family_id,
const std::string& column_family_name, Env::IOPriority io_priority,
Env::WriteLifeTimeHint write_hint,
std::vector<std::string>* blob_file_paths,
std::vector<BlobFileAddition>* blob_file_additions)
: BlobFileBuilder([versions]() { return versions->NewFileNumber(); }, env,
fs, immutable_cf_options, mutable_cf_options,
file_options, job_id, column_family_id,
column_family_name, io_priority, write_hint,
blob_file_paths, blob_file_additions) {}
blob_file_additions) {}
BlobFileBuilder::BlobFileBuilder(
std::function<uint64_t()> file_number_generator, Env* env, FileSystem* fs,
@@ -46,7 +45,6 @@ BlobFileBuilder::BlobFileBuilder(
int job_id, uint32_t column_family_id,
const std::string& column_family_name, Env::IOPriority io_priority,
Env::WriteLifeTimeHint write_hint,
std::vector<std::string>* blob_file_paths,
std::vector<BlobFileAddition>* blob_file_additions)
: file_number_generator_(std::move(file_number_generator)),
env_(env),
@@ -61,7 +59,6 @@ BlobFileBuilder::BlobFileBuilder(
column_family_name_(column_family_name),
io_priority_(io_priority),
write_hint_(write_hint),
blob_file_paths_(blob_file_paths),
blob_file_additions_(blob_file_additions),
blob_count_(0),
blob_bytes_(0) {
@@ -70,10 +67,7 @@ BlobFileBuilder::BlobFileBuilder(
assert(fs_);
assert(immutable_cf_options_);
assert(file_options_);
assert(blob_file_paths_);
assert(blob_file_paths_->empty());
assert(blob_file_additions_);
assert(blob_file_additions_->empty());
}
BlobFileBuilder::~BlobFileBuilder() = default;
@@ -151,7 +145,7 @@ Status BlobFileBuilder::OpenBlobFileIfNeeded() {
assert(immutable_cf_options_);
assert(!immutable_cf_options_->cf_paths.empty());
std::string blob_file_path = BlobFileName(
const std::string blob_file_path = BlobFileName(
immutable_cf_options_->cf_paths.front().path, blob_file_number);
std::unique_ptr<FSWritableFile> file;
@@ -167,12 +161,6 @@ Status BlobFileBuilder::OpenBlobFileIfNeeded() {
}
}
// Note: files get added to blob_file_paths_ right after the open, so they
// can be cleaned up upon failure. Contrast this with blob_file_additions_,
// which only contains successfully written files.
assert(blob_file_paths_);
blob_file_paths_->emplace_back(std::move(blob_file_path));
assert(file);
file->SetIOPriority(io_priority_);
file->SetWriteLifeTimeHint(write_hint_);
@@ -180,7 +168,7 @@ Status BlobFileBuilder::OpenBlobFileIfNeeded() {
Statistics* const statistics = immutable_cf_options_->statistics;
std::unique_ptr<WritableFileWriter> file_writer(new WritableFileWriter(
std::move(file), blob_file_paths_->back(), *file_options_, env_,
std::move(file), blob_file_path, *file_options_, env_,
nullptr /*IOTracer*/, statistics, immutable_cf_options_->listeners,
immutable_cf_options_->file_checksum_gen_factory));
-3
View File
@@ -36,7 +36,6 @@ class BlobFileBuilder {
const std::string& column_family_name,
Env::IOPriority io_priority,
Env::WriteLifeTimeHint write_hint,
std::vector<std::string>* blob_file_paths,
std::vector<BlobFileAddition>* blob_file_additions);
BlobFileBuilder(std::function<uint64_t()> file_number_generator, Env* env,
@@ -48,7 +47,6 @@ class BlobFileBuilder {
const std::string& column_family_name,
Env::IOPriority io_priority,
Env::WriteLifeTimeHint write_hint,
std::vector<std::string>* blob_file_paths,
std::vector<BlobFileAddition>* blob_file_additions);
BlobFileBuilder(const BlobFileBuilder&) = delete;
@@ -81,7 +79,6 @@ class BlobFileBuilder {
std::string column_family_name_;
Env::IOPriority io_priority_;
Env::WriteLifeTimeHint write_hint_;
std::vector<std::string>* blob_file_paths_;
std::vector<BlobFileAddition>* blob_file_additions_;
std::unique_ptr<BlobLogWriter> writer_;
uint64_t blob_count_;
+46 -111
View File
@@ -14,7 +14,7 @@
#include "db/blob/blob_file_addition.h"
#include "db/blob/blob_index.h"
#include "db/blob/blob_log_format.h"
#include "db/blob/blob_log_sequential_reader.h"
#include "db/blob/blob_log_reader.h"
#include "env/composite_env_wrapper.h"
#include "env/mock_env.h"
#include "file/filename.h"
@@ -42,15 +42,17 @@ class BlobFileBuilderTest : public testing::Test {
protected:
BlobFileBuilderTest() : mock_env_(Env::Default()), fs_(&mock_env_) {}
void VerifyBlobFile(uint64_t blob_file_number,
const std::string& blob_file_path,
uint32_t column_family_id,
void VerifyBlobFile(const ImmutableCFOptions& immutable_cf_options,
uint64_t blob_file_number, uint32_t column_family_id,
CompressionType blob_compression_type,
const std::vector<std::pair<std::string, std::string>>&
expected_key_value_pairs,
const std::vector<std::string>& blob_indexes) {
assert(expected_key_value_pairs.size() == blob_indexes.size());
const std::string blob_file_path = BlobFileName(
immutable_cf_options.cf_paths.front().path, blob_file_number);
std::unique_ptr<FSRandomAccessFile> file;
constexpr IODebugContext* dbg = nullptr;
ASSERT_OK(
@@ -61,8 +63,8 @@ class BlobFileBuilderTest : public testing::Test {
&mock_env_));
constexpr Statistics* statistics = nullptr;
BlobLogSequentialReader blob_log_reader(std::move(file_reader), &mock_env_,
statistics);
BlobLogReader blob_log_reader(std::move(file_reader), &mock_env_,
statistics);
BlobLogHeader header;
ASSERT_OK(blob_log_reader.ReadHeader(&header));
@@ -77,7 +79,7 @@ class BlobFileBuilderTest : public testing::Test {
uint64_t blob_offset = 0;
ASSERT_OK(blob_log_reader.ReadRecord(
&record, BlobLogSequentialReader::kReadHeaderKeyBlob, &blob_offset));
&record, BlobLogReader::kReadHeaderKeyBlob, &blob_offset));
// Check the contents of the blob file
const auto& expected_key_value = expected_key_value_pairs[i];
@@ -135,14 +137,12 @@ TEST_F(BlobFileBuilderTest, BuildAndCheckOneFile) {
constexpr Env::IOPriority io_priority = Env::IO_HIGH;
constexpr Env::WriteLifeTimeHint write_hint = Env::WLTH_MEDIUM;
std::vector<std::string> blob_file_paths;
std::vector<BlobFileAddition> blob_file_additions;
BlobFileBuilder builder(TestFileNumberGenerator(), &mock_env_, &fs_,
&immutable_cf_options, &mutable_cf_options,
&file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint,
&blob_file_paths, &blob_file_additions);
BlobFileBuilder builder(
TestFileNumberGenerator(), &mock_env_, &fs_, &immutable_cf_options,
&mutable_cf_options, &file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint, &blob_file_additions);
std::vector<std::pair<std::string, std::string>> expected_key_value_pairs(
number_of_blobs);
@@ -168,20 +168,12 @@ TEST_F(BlobFileBuilderTest, BuildAndCheckOneFile) {
ASSERT_OK(builder.Finish());
// Check the metadata generated
constexpr uint64_t blob_file_number = 2;
ASSERT_EQ(blob_file_paths.size(), 1);
const std::string& blob_file_path = blob_file_paths[0];
ASSERT_EQ(blob_file_path,
BlobFileName(immutable_cf_options.cf_paths.front().path,
blob_file_number));
ASSERT_EQ(blob_file_additions.size(), 1);
const auto& blob_file_addition = blob_file_additions[0];
constexpr uint64_t blob_file_number = 2;
ASSERT_EQ(blob_file_addition.GetBlobFileNumber(), blob_file_number);
ASSERT_EQ(blob_file_addition.GetTotalBlobCount(), number_of_blobs);
ASSERT_EQ(
@@ -189,7 +181,7 @@ TEST_F(BlobFileBuilderTest, BuildAndCheckOneFile) {
number_of_blobs * (BlobLogRecord::kHeaderSize + key_size + value_size));
// Verify the contents of the new blob file as well as the blob references
VerifyBlobFile(blob_file_number, blob_file_path, column_family_id,
VerifyBlobFile(immutable_cf_options, blob_file_number, column_family_id,
kNoCompression, expected_key_value_pairs, blob_indexes);
}
@@ -218,14 +210,12 @@ TEST_F(BlobFileBuilderTest, BuildAndCheckMultipleFiles) {
constexpr Env::IOPriority io_priority = Env::IO_HIGH;
constexpr Env::WriteLifeTimeHint write_hint = Env::WLTH_MEDIUM;
std::vector<std::string> blob_file_paths;
std::vector<BlobFileAddition> blob_file_additions;
BlobFileBuilder builder(TestFileNumberGenerator(), &mock_env_, &fs_,
&immutable_cf_options, &mutable_cf_options,
&file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint,
&blob_file_paths, &blob_file_additions);
BlobFileBuilder builder(
TestFileNumberGenerator(), &mock_env_, &fs_, &immutable_cf_options,
&mutable_cf_options, &file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint, &blob_file_additions);
std::vector<std::pair<std::string, std::string>> expected_key_value_pairs(
number_of_blobs);
@@ -251,19 +241,12 @@ TEST_F(BlobFileBuilderTest, BuildAndCheckMultipleFiles) {
ASSERT_OK(builder.Finish());
// Check the metadata generated
ASSERT_EQ(blob_file_paths.size(), number_of_blobs);
ASSERT_EQ(blob_file_additions.size(), number_of_blobs);
for (size_t i = 0; i < number_of_blobs; ++i) {
const uint64_t blob_file_number = i + 2;
ASSERT_EQ(blob_file_paths[i],
BlobFileName(immutable_cf_options.cf_paths.front().path,
blob_file_number));
const auto& blob_file_addition = blob_file_additions[i];
ASSERT_EQ(blob_file_addition.GetBlobFileNumber(), blob_file_number);
ASSERT_EQ(blob_file_addition.GetBlobFileNumber(), i + 2);
ASSERT_EQ(blob_file_addition.GetTotalBlobCount(), 1);
ASSERT_EQ(blob_file_addition.GetTotalBlobBytes(),
BlobLogRecord::kHeaderSize + key_size + value_size);
@@ -275,8 +258,8 @@ TEST_F(BlobFileBuilderTest, BuildAndCheckMultipleFiles) {
expected_key_value_pairs[i]};
std::vector<std::string> blob_index{blob_indexes[i]};
VerifyBlobFile(i + 2, blob_file_paths[i], column_family_id, kNoCompression,
expected_key_value_pair, blob_index);
VerifyBlobFile(immutable_cf_options, i + 2, column_family_id,
kNoCompression, expected_key_value_pair, blob_index);
}
}
@@ -303,14 +286,12 @@ TEST_F(BlobFileBuilderTest, InlinedValues) {
constexpr Env::IOPriority io_priority = Env::IO_HIGH;
constexpr Env::WriteLifeTimeHint write_hint = Env::WLTH_MEDIUM;
std::vector<std::string> blob_file_paths;
std::vector<BlobFileAddition> blob_file_additions;
BlobFileBuilder builder(TestFileNumberGenerator(), &mock_env_, &fs_,
&immutable_cf_options, &mutable_cf_options,
&file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint,
&blob_file_paths, &blob_file_additions);
BlobFileBuilder builder(
TestFileNumberGenerator(), &mock_env_, &fs_, &immutable_cf_options,
&mutable_cf_options, &file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint, &blob_file_additions);
for (size_t i = 0; i < number_of_blobs; ++i) {
const std::string key = std::to_string(i);
@@ -327,7 +308,6 @@ TEST_F(BlobFileBuilderTest, InlinedValues) {
ASSERT_OK(builder.Finish());
// Check the metadata generated
ASSERT_TRUE(blob_file_paths.empty());
ASSERT_TRUE(blob_file_additions.empty());
}
@@ -355,14 +335,12 @@ TEST_F(BlobFileBuilderTest, Compression) {
constexpr Env::IOPriority io_priority = Env::IO_HIGH;
constexpr Env::WriteLifeTimeHint write_hint = Env::WLTH_MEDIUM;
std::vector<std::string> blob_file_paths;
std::vector<BlobFileAddition> blob_file_additions;
BlobFileBuilder builder(TestFileNumberGenerator(), &mock_env_, &fs_,
&immutable_cf_options, &mutable_cf_options,
&file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint,
&blob_file_paths, &blob_file_additions);
BlobFileBuilder builder(
TestFileNumberGenerator(), &mock_env_, &fs_, &immutable_cf_options,
&mutable_cf_options, &file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint, &blob_file_additions);
const std::string key("1");
const std::string uncompressed_value(value_size, 'x');
@@ -375,20 +353,12 @@ TEST_F(BlobFileBuilderTest, Compression) {
ASSERT_OK(builder.Finish());
// Check the metadata generated
constexpr uint64_t blob_file_number = 2;
ASSERT_EQ(blob_file_paths.size(), 1);
const std::string& blob_file_path = blob_file_paths[0];
ASSERT_EQ(blob_file_path,
BlobFileName(immutable_cf_options.cf_paths.front().path,
blob_file_number));
ASSERT_EQ(blob_file_additions.size(), 1);
const auto& blob_file_addition = blob_file_additions[0];
constexpr uint64_t blob_file_number = 2;
ASSERT_EQ(blob_file_addition.GetBlobFileNumber(), blob_file_number);
ASSERT_EQ(blob_file_addition.GetTotalBlobCount(), 1);
@@ -411,7 +381,7 @@ TEST_F(BlobFileBuilderTest, Compression) {
{key, compressed_value}};
std::vector<std::string> blob_indexes{blob_index};
VerifyBlobFile(blob_file_number, blob_file_path, column_family_id,
VerifyBlobFile(immutable_cf_options, blob_file_number, column_family_id,
kSnappyCompression, expected_key_value_pairs, blob_indexes);
}
@@ -437,14 +407,12 @@ TEST_F(BlobFileBuilderTest, CompressionError) {
constexpr Env::IOPriority io_priority = Env::IO_HIGH;
constexpr Env::WriteLifeTimeHint write_hint = Env::WLTH_MEDIUM;
std::vector<std::string> blob_file_paths;
std::vector<BlobFileAddition> blob_file_additions;
BlobFileBuilder builder(TestFileNumberGenerator(), &mock_env_, &fs_,
&immutable_cf_options, &mutable_cf_options,
&file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint,
&blob_file_paths, &blob_file_additions);
BlobFileBuilder builder(
TestFileNumberGenerator(), &mock_env_, &fs_, &immutable_cf_options,
&mutable_cf_options, &file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint, &blob_file_additions);
SyncPoint::GetInstance()->SetCallBack("CompressData:TamperWithReturnValue",
[](void* arg) {
@@ -462,15 +430,6 @@ TEST_F(BlobFileBuilderTest, CompressionError) {
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
constexpr uint64_t blob_file_number = 2;
ASSERT_EQ(blob_file_paths.size(), 1);
ASSERT_EQ(blob_file_paths[0],
BlobFileName(immutable_cf_options.cf_paths.front().path,
blob_file_number));
ASSERT_TRUE(blob_file_additions.empty());
}
TEST_F(BlobFileBuilderTest, Checksum) {
@@ -514,14 +473,12 @@ TEST_F(BlobFileBuilderTest, Checksum) {
constexpr Env::IOPriority io_priority = Env::IO_HIGH;
constexpr Env::WriteLifeTimeHint write_hint = Env::WLTH_MEDIUM;
std::vector<std::string> blob_file_paths;
std::vector<BlobFileAddition> blob_file_additions;
BlobFileBuilder builder(TestFileNumberGenerator(), &mock_env_, &fs_,
&immutable_cf_options, &mutable_cf_options,
&file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint,
&blob_file_paths, &blob_file_additions);
BlobFileBuilder builder(
TestFileNumberGenerator(), &mock_env_, &fs_, &immutable_cf_options,
&mutable_cf_options, &file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint, &blob_file_additions);
const std::string key("1");
const std::string value("deadbeef");
@@ -534,20 +491,12 @@ TEST_F(BlobFileBuilderTest, Checksum) {
ASSERT_OK(builder.Finish());
// Check the metadata generated
constexpr uint64_t blob_file_number = 2;
ASSERT_EQ(blob_file_paths.size(), 1);
const std::string& blob_file_path = blob_file_paths[0];
ASSERT_EQ(blob_file_path,
BlobFileName(immutable_cf_options.cf_paths.front().path,
blob_file_number));
ASSERT_EQ(blob_file_additions.size(), 1);
const auto& blob_file_addition = blob_file_additions[0];
constexpr uint64_t blob_file_number = 2;
ASSERT_EQ(blob_file_addition.GetBlobFileNumber(), blob_file_number);
ASSERT_EQ(blob_file_addition.GetTotalBlobCount(), 1);
ASSERT_EQ(blob_file_addition.GetTotalBlobBytes(),
@@ -560,7 +509,7 @@ TEST_F(BlobFileBuilderTest, Checksum) {
{key, value}};
std::vector<std::string> blob_indexes{blob_index};
VerifyBlobFile(blob_file_number, blob_file_path, column_family_id,
VerifyBlobFile(immutable_cf_options, blob_file_number, column_family_id,
kNoCompression, expected_key_value_pairs, blob_indexes);
}
@@ -612,14 +561,13 @@ TEST_P(BlobFileBuilderIOErrorTest, IOError) {
constexpr Env::IOPriority io_priority = Env::IO_HIGH;
constexpr Env::WriteLifeTimeHint write_hint = Env::WLTH_MEDIUM;
std::vector<std::string> blob_file_paths;
std::vector<BlobFileAddition> blob_file_additions;
BlobFileBuilder builder(TestFileNumberGenerator(), &fault_injection_env_,
&fs_, &immutable_cf_options, &mutable_cf_options,
&file_options_, job_id, column_family_id,
column_family_name, io_priority, write_hint,
&blob_file_paths, &blob_file_additions);
&blob_file_additions);
SyncPoint::GetInstance()->SetCallBack(sync_point_, [this](void* /* arg */) {
fault_injection_env_.SetFilesystemActive(false,
@@ -636,19 +584,6 @@ TEST_P(BlobFileBuilderIOErrorTest, IOError) {
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
if (sync_point_ == "BlobFileBuilder::OpenBlobFileIfNeeded:NewWritableFile") {
ASSERT_TRUE(blob_file_paths.empty());
} else {
constexpr uint64_t blob_file_number = 2;
ASSERT_EQ(blob_file_paths.size(), 1);
ASSERT_EQ(blob_file_paths[0],
BlobFileName(immutable_cf_options.cf_paths.front().path,
blob_file_number));
}
ASSERT_TRUE(blob_file_additions.empty());
}
} // namespace ROCKSDB_NAMESPACE
-423
View File
@@ -1,423 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#include "db/blob/blob_file_reader.h"
#include <cassert>
#include <string>
#include "db/blob/blob_log_format.h"
#include "file/filename.h"
#include "options/cf_options.h"
#include "rocksdb/file_system.h"
#include "rocksdb/slice.h"
#include "rocksdb/status.h"
#include "test_util/sync_point.h"
#include "util/compression.h"
#include "util/crc32c.h"
namespace ROCKSDB_NAMESPACE {
Status BlobFileReader::Create(
const ImmutableCFOptions& immutable_cf_options,
const FileOptions& file_options, uint32_t column_family_id,
HistogramImpl* blob_file_read_hist, uint64_t blob_file_number,
std::unique_ptr<BlobFileReader>* blob_file_reader) {
assert(blob_file_reader);
assert(!*blob_file_reader);
uint64_t file_size = 0;
std::unique_ptr<RandomAccessFileReader> file_reader;
{
const Status s =
OpenFile(immutable_cf_options, file_options, blob_file_read_hist,
blob_file_number, &file_size, &file_reader);
if (!s.ok()) {
return s;
}
}
assert(file_reader);
CompressionType compression_type = kNoCompression;
{
const Status s =
ReadHeader(file_reader.get(), column_family_id, &compression_type);
if (!s.ok()) {
return s;
}
}
{
const Status s = ReadFooter(file_size, file_reader.get());
if (!s.ok()) {
return s;
}
}
blob_file_reader->reset(
new BlobFileReader(std::move(file_reader), file_size, compression_type));
return Status::OK();
}
Status BlobFileReader::OpenFile(
const ImmutableCFOptions& immutable_cf_options,
const FileOptions& file_opts, HistogramImpl* blob_file_read_hist,
uint64_t blob_file_number, uint64_t* file_size,
std::unique_ptr<RandomAccessFileReader>* file_reader) {
assert(file_size);
assert(file_reader);
const auto& cf_paths = immutable_cf_options.cf_paths;
assert(!cf_paths.empty());
const std::string blob_file_path =
BlobFileName(cf_paths.front().path, blob_file_number);
FileSystem* const fs = immutable_cf_options.fs;
assert(fs);
constexpr IODebugContext* dbg = nullptr;
{
TEST_SYNC_POINT("BlobFileReader::OpenFile:GetFileSize");
const Status s =
fs->GetFileSize(blob_file_path, IOOptions(), file_size, dbg);
if (!s.ok()) {
return s;
}
}
if (*file_size < BlobLogHeader::kSize + BlobLogFooter::kSize) {
return Status::Corruption("Malformed blob file");
}
std::unique_ptr<FSRandomAccessFile> file;
{
TEST_SYNC_POINT("BlobFileReader::OpenFile:NewRandomAccessFile");
const Status s =
fs->NewRandomAccessFile(blob_file_path, file_opts, &file, dbg);
if (!s.ok()) {
return s;
}
}
assert(file);
if (immutable_cf_options.advise_random_on_open) {
file->Hint(FSRandomAccessFile::kRandom);
}
file_reader->reset(new RandomAccessFileReader(
std::move(file), blob_file_path, immutable_cf_options.env,
std::shared_ptr<IOTracer>(), immutable_cf_options.statistics,
BLOB_DB_BLOB_FILE_READ_MICROS, blob_file_read_hist,
immutable_cf_options.rate_limiter, immutable_cf_options.listeners));
return Status::OK();
}
Status BlobFileReader::ReadHeader(const RandomAccessFileReader* file_reader,
uint32_t column_family_id,
CompressionType* compression_type) {
assert(file_reader);
assert(compression_type);
Slice header_slice;
Buffer buf;
AlignedBuf aligned_buf;
{
TEST_SYNC_POINT("BlobFileReader::ReadHeader:ReadFromFile");
constexpr uint64_t read_offset = 0;
constexpr size_t read_size = BlobLogHeader::kSize;
const Status s = ReadFromFile(file_reader, read_offset, read_size,
&header_slice, &buf, &aligned_buf);
if (!s.ok()) {
return s;
}
TEST_SYNC_POINT_CALLBACK("BlobFileReader::ReadHeader:TamperWithResult",
&header_slice);
}
BlobLogHeader header;
{
const Status s = header.DecodeFrom(header_slice);
if (!s.ok()) {
return s;
}
}
constexpr ExpirationRange no_expiration_range;
if (header.has_ttl || header.expiration_range != no_expiration_range) {
return Status::Corruption("Unexpected TTL blob file");
}
if (header.column_family_id != column_family_id) {
return Status::Corruption("Column family ID mismatch");
}
*compression_type = header.compression;
return Status::OK();
}
Status BlobFileReader::ReadFooter(uint64_t file_size,
const RandomAccessFileReader* file_reader) {
assert(file_size >= BlobLogHeader::kSize + BlobLogFooter::kSize);
assert(file_reader);
Slice footer_slice;
Buffer buf;
AlignedBuf aligned_buf;
{
TEST_SYNC_POINT("BlobFileReader::ReadFooter:ReadFromFile");
const uint64_t read_offset = file_size - BlobLogFooter::kSize;
constexpr size_t read_size = BlobLogFooter::kSize;
const Status s = ReadFromFile(file_reader, read_offset, read_size,
&footer_slice, &buf, &aligned_buf);
if (!s.ok()) {
return s;
}
TEST_SYNC_POINT_CALLBACK("BlobFileReader::ReadFooter:TamperWithResult",
&footer_slice);
}
BlobLogFooter footer;
{
const Status s = footer.DecodeFrom(footer_slice);
if (!s.ok()) {
return s;
}
}
constexpr ExpirationRange no_expiration_range;
if (footer.expiration_range != no_expiration_range) {
return Status::Corruption("Unexpected TTL blob file");
}
return Status::OK();
}
Status BlobFileReader::ReadFromFile(const RandomAccessFileReader* file_reader,
uint64_t read_offset, size_t read_size,
Slice* slice, Buffer* buf,
AlignedBuf* aligned_buf) {
assert(slice);
assert(buf);
assert(aligned_buf);
assert(file_reader);
Status s;
if (file_reader->use_direct_io()) {
constexpr char* scratch = nullptr;
s = file_reader->Read(IOOptions(), read_offset, read_size, slice, scratch,
aligned_buf);
} else {
buf->reset(new char[read_size]);
constexpr AlignedBuf* aligned_scratch = nullptr;
s = file_reader->Read(IOOptions(), read_offset, read_size, slice,
buf->get(), aligned_scratch);
}
if (!s.ok()) {
return s;
}
if (slice->size() != read_size) {
return Status::Corruption("Failed to read data from blob file");
}
return Status::OK();
}
BlobFileReader::BlobFileReader(
std::unique_ptr<RandomAccessFileReader>&& file_reader, uint64_t file_size,
CompressionType compression_type)
: file_reader_(std::move(file_reader)),
file_size_(file_size),
compression_type_(compression_type) {
assert(file_reader_);
}
BlobFileReader::~BlobFileReader() = default;
Status BlobFileReader::GetBlob(const ReadOptions& read_options,
const Slice& user_key, uint64_t offset,
uint64_t value_size,
CompressionType compression_type,
PinnableSlice* value) const {
assert(value);
const uint64_t key_size = user_key.size();
if (!IsValidBlobOffset(offset, key_size, value_size, file_size_)) {
return Status::Corruption("Invalid blob offset");
}
if (compression_type != compression_type_) {
return Status::Corruption("Compression type mismatch when reading blob");
}
// Note: if verify_checksum is set, we read the entire blob record to be able
// to perform the verification; otherwise, we just read the blob itself. Since
// the offset in BlobIndex actually points to the blob value, we need to make
// an adjustment in the former case.
const uint64_t adjustment =
read_options.verify_checksums
? BlobLogRecord::CalculateAdjustmentForRecordHeader(key_size)
: 0;
assert(offset >= adjustment);
Slice record_slice;
Buffer buf;
AlignedBuf aligned_buf;
{
TEST_SYNC_POINT("BlobFileReader::GetBlob:ReadFromFile");
const uint64_t record_offset = offset - adjustment;
const uint64_t record_size = value_size + adjustment;
const Status s = ReadFromFile(file_reader_.get(), record_offset,
static_cast<size_t>(record_size),
&record_slice, &buf, &aligned_buf);
if (!s.ok()) {
return s;
}
TEST_SYNC_POINT_CALLBACK("BlobFileReader::GetBlob:TamperWithResult",
&record_slice);
}
if (read_options.verify_checksums) {
const Status s = VerifyBlob(record_slice, user_key, value_size);
if (!s.ok()) {
return s;
}
}
const Slice value_slice(record_slice.data() + adjustment, value_size);
{
const Status s =
UncompressBlobIfNeeded(value_slice, compression_type, value);
if (!s.ok()) {
return s;
}
}
return Status::OK();
}
Status BlobFileReader::VerifyBlob(const Slice& record_slice,
const Slice& user_key, uint64_t value_size) {
BlobLogRecord record;
const Slice header_slice(record_slice.data(), BlobLogRecord::kHeaderSize);
{
const Status s = record.DecodeHeaderFrom(header_slice);
if (!s.ok()) {
return s;
}
}
if (record.key_size != user_key.size()) {
return Status::Corruption("Key size mismatch when reading blob");
}
if (record.value_size != value_size) {
return Status::Corruption("Value size mismatch when reading blob");
}
record.key =
Slice(record_slice.data() + BlobLogRecord::kHeaderSize, record.key_size);
if (record.key != user_key) {
return Status::Corruption("Key mismatch when reading blob");
}
record.value = Slice(record.key.data() + record.key_size, value_size);
{
TEST_SYNC_POINT_CALLBACK("BlobFileReader::VerifyBlob:CheckBlobCRC",
&record);
const Status s = record.CheckBlobCRC();
if (!s.ok()) {
return s;
}
}
return Status::OK();
}
Status BlobFileReader::UncompressBlobIfNeeded(const Slice& value_slice,
CompressionType compression_type,
PinnableSlice* value) {
assert(value);
if (compression_type == kNoCompression) {
SaveValue(value_slice, value);
return Status::OK();
}
UncompressionContext context(compression_type);
UncompressionInfo info(context, UncompressionDict::GetEmptyDict(),
compression_type);
size_t uncompressed_size = 0;
constexpr uint32_t compression_format_version = 2;
constexpr MemoryAllocator* allocator = nullptr;
CacheAllocationPtr output =
UncompressData(info, value_slice.data(), value_slice.size(),
&uncompressed_size, compression_format_version, allocator);
TEST_SYNC_POINT_CALLBACK(
"BlobFileReader::UncompressBlobIfNeeded:TamperWithResult", &output);
if (!output) {
return Status::Corruption("Unable to uncompress blob");
}
SaveValue(Slice(output.get(), uncompressed_size), value);
return Status::OK();
}
void BlobFileReader::SaveValue(const Slice& src, PinnableSlice* dst) {
assert(dst);
if (dst->IsPinned()) {
dst->Reset();
}
dst->PinSelf(src);
}
} // namespace ROCKSDB_NAMESPACE
-81
View File
@@ -1,81 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#include <cinttypes>
#include <memory>
#include "file/random_access_file_reader.h"
#include "rocksdb/compression_type.h"
#include "rocksdb/rocksdb_namespace.h"
namespace ROCKSDB_NAMESPACE {
class Status;
struct ImmutableCFOptions;
struct FileOptions;
class HistogramImpl;
struct ReadOptions;
class Slice;
class PinnableSlice;
class BlobFileReader {
public:
static Status Create(const ImmutableCFOptions& immutable_cf_options,
const FileOptions& file_options,
uint32_t column_family_id,
HistogramImpl* blob_file_read_hist,
uint64_t blob_file_number,
std::unique_ptr<BlobFileReader>* reader);
BlobFileReader(const BlobFileReader&) = delete;
BlobFileReader& operator=(const BlobFileReader&) = delete;
~BlobFileReader();
Status GetBlob(const ReadOptions& read_options, const Slice& user_key,
uint64_t offset, uint64_t value_size,
CompressionType compression_type, PinnableSlice* value) const;
private:
BlobFileReader(std::unique_ptr<RandomAccessFileReader>&& file_reader,
uint64_t file_size, CompressionType compression_type);
static Status OpenFile(const ImmutableCFOptions& immutable_cf_options,
const FileOptions& file_opts,
HistogramImpl* blob_file_read_hist,
uint64_t blob_file_number, uint64_t* file_size,
std::unique_ptr<RandomAccessFileReader>* file_reader);
static Status ReadHeader(const RandomAccessFileReader* file_reader,
uint32_t column_family_id,
CompressionType* compression_type);
static Status ReadFooter(uint64_t file_size,
const RandomAccessFileReader* file_reader);
using Buffer = std::unique_ptr<char[]>;
static Status ReadFromFile(const RandomAccessFileReader* file_reader,
uint64_t read_offset, size_t read_size,
Slice* slice, Buffer* buf,
AlignedBuf* aligned_buf);
static Status VerifyBlob(const Slice& record_slice, const Slice& user_key,
uint64_t value_size);
static Status UncompressBlobIfNeeded(const Slice& value_slice,
CompressionType compression_type,
PinnableSlice* value);
static void SaveValue(const Slice& src, PinnableSlice* dst);
std::unique_ptr<RandomAccessFileReader> file_reader_;
uint64_t file_size_;
CompressionType compression_type_;
};
} // namespace ROCKSDB_NAMESPACE
-770
View File
@@ -1,770 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#include "db/blob/blob_file_reader.h"
#include <cassert>
#include <string>
#include "db/blob/blob_log_format.h"
#include "db/blob/blob_log_writer.h"
#include "env/mock_env.h"
#include "file/filename.h"
#include "file/read_write_util.h"
#include "file/writable_file_writer.h"
#include "options/cf_options.h"
#include "rocksdb/env.h"
#include "rocksdb/file_system.h"
#include "rocksdb/options.h"
#include "test_util/testharness.h"
#include "util/compression.h"
#include "utilities/fault_injection_env.h"
namespace ROCKSDB_NAMESPACE {
namespace {
// Creates a test blob file with a single blob in it. Note: this method
// makes it possible to test various corner cases by allowing the caller
// to specify the contents of various blob file header/footer fields.
void WriteBlobFile(const ImmutableCFOptions& immutable_cf_options,
uint32_t column_family_id, bool has_ttl,
const ExpirationRange& expiration_range_header,
const ExpirationRange& expiration_range_footer,
uint64_t blob_file_number, const Slice& key,
const Slice& blob, CompressionType compression_type,
uint64_t* blob_offset, uint64_t* blob_size) {
assert(!immutable_cf_options.cf_paths.empty());
assert(blob_offset);
assert(blob_size);
const std::string blob_file_path = BlobFileName(
immutable_cf_options.cf_paths.front().path, blob_file_number);
std::unique_ptr<FSWritableFile> file;
ASSERT_OK(NewWritableFile(immutable_cf_options.fs, blob_file_path, &file,
FileOptions()));
std::unique_ptr<WritableFileWriter> file_writer(
new WritableFileWriter(std::move(file), blob_file_path, FileOptions(),
immutable_cf_options.env));
constexpr Statistics* statistics = nullptr;
constexpr bool use_fsync = false;
BlobLogWriter blob_log_writer(std::move(file_writer),
immutable_cf_options.env, statistics,
blob_file_number, use_fsync);
BlobLogHeader header(column_family_id, compression_type, has_ttl,
expiration_range_header);
ASSERT_OK(blob_log_writer.WriteHeader(header));
std::string compressed_blob;
Slice blob_to_write;
if (compression_type == kNoCompression) {
blob_to_write = blob;
*blob_size = blob.size();
} else {
CompressionOptions opts;
CompressionContext context(compression_type);
constexpr uint64_t sample_for_compression = 0;
CompressionInfo info(opts, context, CompressionDict::GetEmptyDict(),
compression_type, sample_for_compression);
constexpr uint32_t compression_format_version = 2;
ASSERT_TRUE(
CompressData(blob, info, compression_format_version, &compressed_blob));
blob_to_write = compressed_blob;
*blob_size = compressed_blob.size();
}
uint64_t key_offset = 0;
ASSERT_OK(
blob_log_writer.AddRecord(key, blob_to_write, &key_offset, blob_offset));
BlobLogFooter footer;
footer.blob_count = 1;
footer.expiration_range = expiration_range_footer;
std::string checksum_method;
std::string checksum_value;
ASSERT_OK(
blob_log_writer.AppendFooter(footer, &checksum_method, &checksum_value));
}
} // anonymous namespace
class BlobFileReaderTest : public testing::Test {
protected:
BlobFileReaderTest() : mock_env_(Env::Default()) {}
MockEnv mock_env_;
};
TEST_F(BlobFileReaderTest, CreateReaderAndGetBlob) {
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_,
"BlobFileReaderTest_CreateReaderAndGetBlob"),
0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = false;
constexpr ExpirationRange expiration_range;
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range, expiration_range, blob_file_number, key, blob,
kNoCompression, &blob_offset, &blob_size);
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
ASSERT_OK(BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader));
// Make sure the blob can be retrieved with and without checksum verification
ReadOptions read_options;
read_options.verify_checksums = false;
{
PinnableSlice value;
ASSERT_OK(reader->GetBlob(read_options, key, blob_offset, blob_size,
kNoCompression, &value));
ASSERT_EQ(value, blob);
}
read_options.verify_checksums = true;
{
PinnableSlice value;
ASSERT_OK(reader->GetBlob(read_options, key, blob_offset, blob_size,
kNoCompression, &value));
ASSERT_EQ(value, blob);
}
// Invalid offset (too close to start of file)
{
PinnableSlice value;
ASSERT_TRUE(reader
->GetBlob(read_options, key, blob_offset - 1, blob_size,
kNoCompression, &value)
.IsCorruption());
}
// Invalid offset (too close to end of file)
{
PinnableSlice value;
ASSERT_TRUE(reader
->GetBlob(read_options, key, blob_offset + 1, blob_size,
kNoCompression, &value)
.IsCorruption());
}
// Incorrect compression type
{
PinnableSlice value;
ASSERT_TRUE(
reader
->GetBlob(read_options, key, blob_offset, blob_size, kZSTD, &value)
.IsCorruption());
}
// Incorrect key size
{
constexpr char shorter_key[] = "k";
PinnableSlice value;
ASSERT_TRUE(reader
->GetBlob(read_options, shorter_key,
blob_offset - (sizeof(key) - sizeof(shorter_key)),
blob_size, kNoCompression, &value)
.IsCorruption());
}
// Incorrect key
{
constexpr char incorrect_key[] = "foo";
PinnableSlice value;
ASSERT_TRUE(reader
->GetBlob(read_options, incorrect_key, blob_offset,
blob_size, kNoCompression, &value)
.IsCorruption());
}
// Incorrect value size
{
PinnableSlice value;
ASSERT_TRUE(reader
->GetBlob(read_options, key, blob_offset, blob_size + 1,
kNoCompression, &value)
.IsCorruption());
}
}
TEST_F(BlobFileReaderTest, Malformed) {
// Write a blob file consisting of nothing but a header, and make sure we
// detect the error when we open it for reading
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_, "BlobFileReaderTest_Malformed"), 0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr uint64_t blob_file_number = 1;
{
constexpr bool has_ttl = false;
constexpr ExpirationRange expiration_range;
const std::string blob_file_path = BlobFileName(
immutable_cf_options.cf_paths.front().path, blob_file_number);
std::unique_ptr<FSWritableFile> file;
ASSERT_OK(NewWritableFile(immutable_cf_options.fs, blob_file_path, &file,
FileOptions()));
std::unique_ptr<WritableFileWriter> file_writer(
new WritableFileWriter(std::move(file), blob_file_path, FileOptions(),
immutable_cf_options.env));
constexpr Statistics* statistics = nullptr;
constexpr bool use_fsync = false;
BlobLogWriter blob_log_writer(std::move(file_writer),
immutable_cf_options.env, statistics,
blob_file_number, use_fsync);
BlobLogHeader header(column_family_id, kNoCompression, has_ttl,
expiration_range);
ASSERT_OK(blob_log_writer.WriteHeader(header));
}
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
ASSERT_TRUE(BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader)
.IsCorruption());
}
TEST_F(BlobFileReaderTest, TTL) {
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_, "BlobFileReaderTest_TTL"), 0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = true;
constexpr ExpirationRange expiration_range;
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range, expiration_range, blob_file_number, key, blob,
kNoCompression, &blob_offset, &blob_size);
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
ASSERT_TRUE(BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader)
.IsCorruption());
}
TEST_F(BlobFileReaderTest, ExpirationRangeInHeader) {
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_,
"BlobFileReaderTest_ExpirationRangeInHeader"),
0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = false;
const ExpirationRange expiration_range_header(
1, 2); // can be made constexpr when we adopt C++14
constexpr ExpirationRange expiration_range_footer;
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range_header, expiration_range_footer,
blob_file_number, key, blob, kNoCompression, &blob_offset,
&blob_size);
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
ASSERT_TRUE(BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader)
.IsCorruption());
}
TEST_F(BlobFileReaderTest, ExpirationRangeInFooter) {
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_,
"BlobFileReaderTest_ExpirationRangeInFooter"),
0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = false;
constexpr ExpirationRange expiration_range_header;
const ExpirationRange expiration_range_footer(
1, 2); // can be made constexpr when we adopt C++14
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range_header, expiration_range_footer,
blob_file_number, key, blob, kNoCompression, &blob_offset,
&blob_size);
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
ASSERT_TRUE(BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader)
.IsCorruption());
}
TEST_F(BlobFileReaderTest, IncorrectColumnFamily) {
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_,
"BlobFileReaderTest_IncorrectColumnFamily"),
0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = false;
constexpr ExpirationRange expiration_range;
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range, expiration_range, blob_file_number, key, blob,
kNoCompression, &blob_offset, &blob_size);
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
constexpr uint32_t incorrect_column_family_id = 2;
ASSERT_TRUE(BlobFileReader::Create(immutable_cf_options, FileOptions(),
incorrect_column_family_id,
blob_file_read_hist, blob_file_number,
&reader)
.IsCorruption());
}
TEST_F(BlobFileReaderTest, BlobCRCError) {
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_, "BlobFileReaderTest_BlobCRCError"), 0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = false;
constexpr ExpirationRange expiration_range;
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range, expiration_range, blob_file_number, key, blob,
kNoCompression, &blob_offset, &blob_size);
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
ASSERT_OK(BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader));
SyncPoint::GetInstance()->SetCallBack(
"BlobFileReader::VerifyBlob:CheckBlobCRC", [](void* arg) {
BlobLogRecord* const record = static_cast<BlobLogRecord*>(arg);
assert(record);
record->blob_crc = 0xfaceb00c;
});
SyncPoint::GetInstance()->EnableProcessing();
PinnableSlice value;
ASSERT_TRUE(reader
->GetBlob(ReadOptions(), key, blob_offset, blob_size,
kNoCompression, &value)
.IsCorruption());
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_F(BlobFileReaderTest, Compression) {
if (!Snappy_Supported()) {
return;
}
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_, "BlobFileReaderTest_Compression"), 0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = false;
constexpr ExpirationRange expiration_range;
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range, expiration_range, blob_file_number, key, blob,
kSnappyCompression, &blob_offset, &blob_size);
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
ASSERT_OK(BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader));
// Make sure the blob can be retrieved with and without checksum verification
ReadOptions read_options;
read_options.verify_checksums = false;
{
PinnableSlice value;
ASSERT_OK(reader->GetBlob(read_options, key, blob_offset, blob_size,
kSnappyCompression, &value));
ASSERT_EQ(value, blob);
}
read_options.verify_checksums = true;
{
PinnableSlice value;
ASSERT_OK(reader->GetBlob(read_options, key, blob_offset, blob_size,
kSnappyCompression, &value));
ASSERT_EQ(value, blob);
}
}
TEST_F(BlobFileReaderTest, UncompressionError) {
if (!Snappy_Supported()) {
return;
}
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_,
"BlobFileReaderTest_UncompressionError"),
0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = false;
constexpr ExpirationRange expiration_range;
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range, expiration_range, blob_file_number, key, blob,
kSnappyCompression, &blob_offset, &blob_size);
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
ASSERT_OK(BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader));
SyncPoint::GetInstance()->SetCallBack(
"BlobFileReader::UncompressBlobIfNeeded:TamperWithResult", [](void* arg) {
CacheAllocationPtr* const output =
static_cast<CacheAllocationPtr*>(arg);
assert(output);
output->reset();
});
SyncPoint::GetInstance()->EnableProcessing();
PinnableSlice value;
ASSERT_TRUE(reader
->GetBlob(ReadOptions(), key, blob_offset, blob_size,
kSnappyCompression, &value)
.IsCorruption());
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
}
class BlobFileReaderIOErrorTest
: public testing::Test,
public testing::WithParamInterface<std::string> {
protected:
BlobFileReaderIOErrorTest()
: mock_env_(Env::Default()),
fault_injection_env_(&mock_env_),
sync_point_(GetParam()) {}
MockEnv mock_env_;
FaultInjectionTestEnv fault_injection_env_;
std::string sync_point_;
};
INSTANTIATE_TEST_CASE_P(BlobFileReaderTest, BlobFileReaderIOErrorTest,
::testing::ValuesIn(std::vector<std::string>{
"BlobFileReader::OpenFile:GetFileSize",
"BlobFileReader::OpenFile:NewRandomAccessFile",
"BlobFileReader::ReadHeader:ReadFromFile",
"BlobFileReader::ReadFooter:ReadFromFile",
"BlobFileReader::GetBlob:ReadFromFile"}));
TEST_P(BlobFileReaderIOErrorTest, IOError) {
// Simulates an I/O error during the specified step
Options options;
options.env = &fault_injection_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&fault_injection_env_,
"BlobFileReaderIOErrorTest_IOError"),
0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = false;
constexpr ExpirationRange expiration_range;
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range, expiration_range, blob_file_number, key, blob,
kNoCompression, &blob_offset, &blob_size);
SyncPoint::GetInstance()->SetCallBack(sync_point_, [this](void* /* arg */) {
fault_injection_env_.SetFilesystemActive(false,
Status::IOError(sync_point_));
});
SyncPoint::GetInstance()->EnableProcessing();
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
const Status s = BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader);
const bool fail_during_create =
(sync_point_ != "BlobFileReader::GetBlob:ReadFromFile");
if (fail_during_create) {
ASSERT_TRUE(s.IsIOError());
} else {
ASSERT_OK(s);
PinnableSlice value;
ASSERT_TRUE(reader
->GetBlob(ReadOptions(), key, blob_offset, blob_size,
kNoCompression, &value)
.IsIOError());
}
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
}
class BlobFileReaderDecodingErrorTest
: public testing::Test,
public testing::WithParamInterface<std::string> {
protected:
BlobFileReaderDecodingErrorTest()
: mock_env_(Env::Default()), sync_point_(GetParam()) {}
MockEnv mock_env_;
std::string sync_point_;
};
INSTANTIATE_TEST_CASE_P(BlobFileReaderTest, BlobFileReaderDecodingErrorTest,
::testing::ValuesIn(std::vector<std::string>{
"BlobFileReader::ReadHeader:TamperWithResult",
"BlobFileReader::ReadFooter:TamperWithResult",
"BlobFileReader::GetBlob:TamperWithResult"}));
TEST_P(BlobFileReaderDecodingErrorTest, DecodingError) {
Options options;
options.env = &mock_env_;
options.cf_paths.emplace_back(
test::PerThreadDBPath(&mock_env_,
"BlobFileReaderDecodingErrorTest_DecodingError"),
0);
options.enable_blob_files = true;
ImmutableCFOptions immutable_cf_options(options);
constexpr uint32_t column_family_id = 1;
constexpr bool has_ttl = false;
constexpr ExpirationRange expiration_range;
constexpr uint64_t blob_file_number = 1;
constexpr char key[] = "key";
constexpr char blob[] = "blob";
uint64_t blob_offset = 0;
uint64_t blob_size = 0;
WriteBlobFile(immutable_cf_options, column_family_id, has_ttl,
expiration_range, expiration_range, blob_file_number, key, blob,
kNoCompression, &blob_offset, &blob_size);
SyncPoint::GetInstance()->SetCallBack(sync_point_, [](void* arg) {
Slice* const slice = static_cast<Slice*>(arg);
assert(slice);
assert(!slice->empty());
slice->remove_prefix(1);
});
SyncPoint::GetInstance()->EnableProcessing();
constexpr HistogramImpl* blob_file_read_hist = nullptr;
std::unique_ptr<BlobFileReader> reader;
const Status s = BlobFileReader::Create(immutable_cf_options, FileOptions(),
column_family_id, blob_file_read_hist,
blob_file_number, &reader);
const bool fail_during_create =
sync_point_ != "BlobFileReader::GetBlob:TamperWithResult";
if (fail_during_create) {
ASSERT_TRUE(s.IsCorruption());
} else {
ASSERT_OK(s);
PinnableSlice value;
ASSERT_TRUE(reader
->GetBlob(ReadOptions(), key, blob_offset, blob_size,
kNoCompression, &value)
.IsCorruption());
}
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+2 -9
View File
@@ -7,9 +7,8 @@
#include <sstream>
#include <string>
#include "rocksdb/compression_type.h"
#include "rocksdb/options.h"
#include "util/coding.h"
#include "util/compression.h"
#include "util/string_util.h"
namespace ROCKSDB_NAMESPACE {
@@ -83,11 +82,6 @@ class BlobIndex {
return size_;
}
CompressionType compression() const {
assert(!IsInlined());
return compression_;
}
Status DecodeFrom(Slice slice) {
static const std::string kErrorMessage = "Error while decoding blob index";
assert(slice.size() > 0);
@@ -123,8 +117,7 @@ class BlobIndex {
oss << "[inlined blob] value:" << value_.ToString(output_hex);
} else {
oss << "[blob ref] file:" << file_number_ << " offset:" << offset_
<< " size:" << size_
<< " compression: " << CompressionTypeToString(compression_);
<< " size:" << size_;
}
if (HasTTL()) {
-4
View File
@@ -95,10 +95,6 @@ Status BlobLogFooter::DecodeFrom(Slice src) {
return Status::OK();
}
uint64_t BlobLogRecord::CalculateAdjustmentForRecordHeader(uint64_t key_size) {
return key_size + kHeaderSize;
}
void BlobLogRecord::EncodeHeaderTo(std::string* dst) {
assert(dst != nullptr);
dst->clear();
-20
View File
@@ -104,11 +104,6 @@ struct BlobLogRecord {
// header include fields up to blob CRC
static constexpr size_t kHeaderSize = 32;
// Note that the offset field of BlobIndex actually points to the blob value
// as opposed to the start of the blob record. The following method can
// be used to calculate the adjustment needed to read the blob record header.
static uint64_t CalculateAdjustmentForRecordHeader(uint64_t key_size);
uint64_t key_size = 0;
uint64_t value_size = 0;
uint64_t expiration = 0;
@@ -128,19 +123,4 @@ struct BlobLogRecord {
Status CheckBlobCRC() const;
};
// Checks whether a blob offset is potentially valid or not.
inline bool IsValidBlobOffset(uint64_t value_offset, uint64_t key_size,
uint64_t value_size, uint64_t file_size) {
if (value_offset <
BlobLogHeader::kSize + BlobLogRecord::kHeaderSize + key_size) {
return false;
}
if (value_offset + value_size + BlobLogFooter::kSize > file_size) {
return false;
}
return true;
}
} // namespace ROCKSDB_NAMESPACE
@@ -4,7 +4,7 @@
// (found in the LICENSE.Apache file in the root directory).
//
#include "db/blob/blob_log_sequential_reader.h"
#include "db/blob/blob_log_reader.h"
#include <algorithm>
@@ -14,19 +14,16 @@
namespace ROCKSDB_NAMESPACE {
BlobLogSequentialReader::BlobLogSequentialReader(
BlobLogReader::BlobLogReader(
std::unique_ptr<RandomAccessFileReader>&& file_reader, Env* env,
Statistics* statistics)
: file_(std::move(file_reader)),
env_(env),
statistics_(statistics),
buffer_(),
next_byte_(0) {}
BlobLogSequentialReader::~BlobLogSequentialReader() = default;
Status BlobLogSequentialReader::ReadSlice(uint64_t size, Slice* slice,
char* buf) {
assert(slice);
Status BlobLogReader::ReadSlice(uint64_t size, Slice* slice, char* buf) {
assert(file_);
StopWatch read_sw(env_, statistics_, BLOB_DB_BLOB_FILE_READ_MICROS);
@@ -43,8 +40,7 @@ Status BlobLogSequentialReader::ReadSlice(uint64_t size, Slice* slice,
return s;
}
Status BlobLogSequentialReader::ReadHeader(BlobLogHeader* header) {
assert(header);
Status BlobLogReader::ReadHeader(BlobLogHeader* header) {
assert(next_byte_ == 0);
static_assert(BlobLogHeader::kSize <= sizeof(header_buf_),
@@ -62,10 +58,8 @@ Status BlobLogSequentialReader::ReadHeader(BlobLogHeader* header) {
return header->DecodeFrom(buffer_);
}
Status BlobLogSequentialReader::ReadRecord(BlobLogRecord* record,
ReadLevel level,
uint64_t* blob_offset) {
assert(record);
Status BlobLogReader::ReadRecord(BlobLogRecord* record, ReadLevel level,
uint64_t* blob_offset) {
static_assert(BlobLogRecord::kHeaderSize <= sizeof(header_buf_),
"Buffer is smaller than BlobLogRecord::kHeaderSize");
@@ -114,8 +108,7 @@ Status BlobLogSequentialReader::ReadRecord(BlobLogRecord* record,
return s;
}
Status BlobLogSequentialReader::ReadFooter(BlobLogFooter* footer) {
assert(footer);
Status BlobLogReader::ReadFooter(BlobLogFooter* footer) {
static_assert(BlobLogFooter::kSize <= sizeof(header_buf_),
"Buffer is smaller than BlobLogFooter::kSize");
@@ -6,26 +6,28 @@
#pragma once
#include <memory>
#include <string>
#include "db/blob/blob_log_format.h"
#include "file/random_access_file_reader.h"
#include "rocksdb/env.h"
#include "rocksdb/slice.h"
#include "rocksdb/statistics.h"
#include "rocksdb/status.h"
namespace ROCKSDB_NAMESPACE {
class RandomAccessFileReader;
class Env;
class Statistics;
class Status;
class SequentialFileReader;
class Logger;
/**
* BlobLogSequentialReader is a general purpose log stream reader
* implementation. The actual job of reading from the device is implemented by
* the RandomAccessFileReader interface.
* BlobLogReader is a general purpose log stream reader implementation. The
* actual job of reading from the device is implemented by the SequentialFile
* interface.
*
* Please see BlobLogWriter for details on the file and record layout.
* Please see Writer for details on the file and record layout.
*/
class BlobLogSequentialReader {
class BlobLogReader {
public:
enum ReadLevel {
kReadHeader,
@@ -33,22 +35,23 @@ class BlobLogSequentialReader {
kReadHeaderKeyBlob,
};
// Create a reader that will return log records from "*file_reader".
BlobLogSequentialReader(std::unique_ptr<RandomAccessFileReader>&& file_reader,
Env* env, Statistics* statistics);
// Create a reader that will return log records from "*file".
// "*file" must remain live while this BlobLogReader is in use.
BlobLogReader(std::unique_ptr<RandomAccessFileReader>&& file_reader, Env* env,
Statistics* statistics);
// No copying allowed
BlobLogSequentialReader(const BlobLogSequentialReader&) = delete;
BlobLogSequentialReader& operator=(const BlobLogSequentialReader&) = delete;
BlobLogReader(const BlobLogReader&) = delete;
BlobLogReader& operator=(const BlobLogReader&) = delete;
~BlobLogSequentialReader();
~BlobLogReader() = default;
Status ReadHeader(BlobLogHeader* header);
// Read the next record into *record. Returns true if read
// successfully, false if we hit end of the input. The contents filled in
// *record will only be valid until the next mutating operation on this
// reader.
// successfully, false if we hit end of the input. May use
// "*scratch" as temporary storage. The contents filled in *record
// will only be valid until the next mutating operation on this
// reader or the next mutation to *scratch.
// If blob_offset is non-null, return offset of the blob through it.
Status ReadRecord(BlobLogRecord* record, ReadLevel level = kReadHeader,
uint64_t* blob_offset = nullptr);
@@ -69,7 +72,7 @@ class BlobLogSequentialReader {
Slice buffer_;
char header_buf_[BlobLogRecord::kHeaderSize];
// which byte to read next
// which byte to read next. For asserting proper usage
uint64_t next_byte_;
};
+29 -69
View File
@@ -13,13 +13,11 @@
#include <deque>
#include <vector>
#include "db/blob/blob_file_builder.h"
#include "db/compaction/compaction_iterator.h"
#include "db/dbformat.h"
#include "db/event_helpers.h"
#include "db/internal_stats.h"
#include "db/merge_helper.h"
#include "db/output_validator.h"
#include "db/range_del_aggregator.h"
#include "db/table_cache.h"
#include "db/version_edit.h"
@@ -69,14 +67,13 @@ TableBuilder* NewTableBuilder(
}
Status BuildTable(
const std::string& dbname, VersionSet* versions, Env* env, FileSystem* fs,
const std::string& dbname, Env* env, FileSystem* fs,
const ImmutableCFOptions& ioptions,
const MutableCFOptions& mutable_cf_options, const FileOptions& file_options,
TableCache* table_cache, InternalIterator* iter,
std::vector<std::unique_ptr<FragmentedRangeTombstoneIterator>>
range_del_iters,
FileMetaData* meta, std::vector<BlobFileAddition>* blob_file_additions,
const InternalKeyComparator& internal_comparator,
FileMetaData* meta, const InternalKeyComparator& internal_comparator,
const std::vector<std::unique_ptr<IntTblPropCollectorFactory>>*
int_tbl_prop_collector_factories,
uint32_t column_family_id, const std::string& column_family_name,
@@ -97,12 +94,9 @@ Status BuildTable(
column_family_name.empty());
// Reports the IOStats for flush for every following bytes.
const size_t kReportFlushIOStatsEvery = 1048576;
OutputValidator output_validator(
internal_comparator,
/*enable_order_check=*/
mutable_cf_options.check_flush_compaction_key_order,
/*enable_hash=*/paranoid_file_checks);
uint64_t paranoid_hash = 0;
Status s;
IOStatus io_s;
meta->fd.file_size = 0;
iter->SeekToFirst();
std::unique_ptr<CompactionRangeDelAggregator> range_del_agg(
@@ -113,7 +107,6 @@ Status BuildTable(
std::string fname = TableFileName(ioptions.cf_paths, meta->fd.GetNumber(),
meta->fd.GetPathId());
std::vector<std::string> blob_file_paths;
std::string file_checksum = kUnknownFileChecksum;
std::string file_checksum_func_name = kUnknownFileChecksumFuncName;
#ifndef ROCKSDB_LITE
@@ -135,8 +128,7 @@ Status BuildTable(
bool use_direct_writes = file_options.use_direct_writes;
TEST_SYNC_POINT_CALLBACK("BuildTable:create_file", &use_direct_writes);
#endif // !NDEBUG
IOStatus io_s = NewWritableFile(fs, fname, &file, file_options);
assert(s.ok());
io_s = NewWritableFile(fs, fname, &file, file_options);
s = io_s;
if (io_status->ok()) {
*io_status = io_s;
@@ -171,31 +163,20 @@ Status BuildTable(
snapshots.empty() ? 0 : snapshots.back(),
snapshot_checker);
std::unique_ptr<BlobFileBuilder> blob_file_builder(
(mutable_cf_options.enable_blob_files && blob_file_additions)
? new BlobFileBuilder(versions, env, fs, &ioptions,
&mutable_cf_options, &file_options, job_id,
column_family_id, column_family_name,
io_priority, write_hint, &blob_file_paths,
blob_file_additions)
: nullptr);
CompactionIterator c_iter(
iter, internal_comparator.user_comparator(), &merge, kMaxSequenceNumber,
&snapshots, earliest_write_conflict_snapshot, snapshot_checker, env,
ShouldReportDetailedTime(env, ioptions.statistics),
true /* internal key corruption is not ok */, range_del_agg.get(),
blob_file_builder.get(), ioptions.allow_data_in_errors);
true /* internal key corruption is not ok */, range_del_agg.get());
c_iter.SeekToFirst();
for (; c_iter.Valid(); c_iter.Next()) {
const Slice& key = c_iter.key();
const Slice& value = c_iter.value();
const ParsedInternalKey& ikey = c_iter.ikey();
// Generate a rolling 64-bit hash of the key and values
s = output_validator.Add(key, value);
if (!s.ok()) {
break;
if (paranoid_file_checks) {
// Generate a rolling 64-bit hash of the key and values
paranoid_hash = Hash64(key.data(), key.size(), paranoid_hash);
paranoid_hash = Hash64(value.data(), value.size(), paranoid_hash);
}
builder->Add(key, value);
meta->UpdateBoundaries(key, value, ikey.sequence, ikey.type);
@@ -207,36 +188,29 @@ Status BuildTable(
ThreadStatus::FLUSH_BYTES_WRITTEN, IOSTATS(bytes_written));
}
}
if (!s.ok()) {
c_iter.status().PermitUncheckedError();
} else if (!c_iter.status().ok()) {
s = c_iter.status();
}
if (s.ok()) {
auto range_del_it = range_del_agg->NewIterator();
for (range_del_it->SeekToFirst(); range_del_it->Valid();
range_del_it->Next()) {
auto tombstone = range_del_it->Tombstone();
auto kv = tombstone.Serialize();
builder->Add(kv.first.Encode(), kv.second);
meta->UpdateBoundariesForRange(kv.first, tombstone.SerializeEndKey(),
tombstone.seq_, internal_comparator);
}
if (blob_file_builder) {
s = blob_file_builder->Finish();
}
auto range_del_it = range_del_agg->NewIterator();
for (range_del_it->SeekToFirst(); range_del_it->Valid();
range_del_it->Next()) {
auto tombstone = range_del_it->Tombstone();
auto kv = tombstone.Serialize();
builder->Add(kv.first.Encode(), kv.second);
meta->UpdateBoundariesForRange(kv.first, tombstone.SerializeEndKey(),
tombstone.seq_, internal_comparator);
}
// Finish and check for builder errors
bool empty = builder->IsEmpty();
s = c_iter.status();
TEST_SYNC_POINT("BuildTable:BeforeFinishBuildTable");
const bool empty = builder->IsEmpty();
if (!s.ok() || empty) {
builder->Abandon();
} else {
s = builder->Finish();
}
io_s = builder->io_status();
if (io_status->ok()) {
*io_status = builder->io_status();
*io_status = io_s;
}
if (s.ok() && !empty) {
@@ -297,15 +271,15 @@ Status BuildTable(
/*allow_unprepared_value*/ false));
s = it->status();
if (s.ok() && paranoid_file_checks) {
OutputValidator file_validator(internal_comparator,
/*enable_order_check=*/true,
/*enable_hash=*/true);
uint64_t check_hash = 0;
for (it->SeekToFirst(); it->Valid(); it->Next()) {
// Generate a rolling 64-bit hash of the key and values
file_validator.Add(it->key(), it->value()).PermitUncheckedError();
check_hash = Hash64(it->key().data(), it->key().size(), check_hash);
check_hash =
Hash64(it->value().data(), it->value().size(), check_hash);
}
s = it->status();
if (s.ok() && !output_validator.CompareValidator(file_validator)) {
if (s.ok() && check_hash != paranoid_hash) {
s = Status::Corruption("Paranoid checksums do not match");
}
}
@@ -318,21 +292,7 @@ Status BuildTable(
}
if (!s.ok() || meta->fd.GetFileSize() == 0) {
constexpr IODebugContext* dbg = nullptr;
Status ignored = fs->DeleteFile(fname, IOOptions(), dbg);
ignored.PermitUncheckedError();
assert(blob_file_additions || blob_file_paths.empty());
if (blob_file_additions) {
for (const std::string& blob_file_path : blob_file_paths) {
ignored = fs->DeleteFile(blob_file_path, IOOptions(), dbg);
ignored.PermitUncheckedError();
}
blob_file_additions->clear();
}
fs->DeleteFile(fname, IOOptions(), nullptr);
}
if (meta->fd.GetFileSize() == 0) {
+2 -5
View File
@@ -27,10 +27,8 @@ namespace ROCKSDB_NAMESPACE {
struct Options;
struct FileMetaData;
class VersionSet;
class Env;
struct EnvOptions;
class BlobFileAddition;
class Iterator;
class SnapshotChecker;
class TableCache;
@@ -65,14 +63,13 @@ TableBuilder* NewTableBuilder(
// @param column_family_name Name of the column family that is also identified
// by column_family_id, or empty string if unknown.
extern Status BuildTable(
const std::string& dbname, VersionSet* versions, Env* env, FileSystem* fs,
const std::string& dbname, Env* env, FileSystem* fs,
const ImmutableCFOptions& options,
const MutableCFOptions& mutable_cf_options, const FileOptions& file_options,
TableCache* table_cache, InternalIterator* iter,
std::vector<std::unique_ptr<FragmentedRangeTombstoneIterator>>
range_del_iters,
FileMetaData* meta, std::vector<BlobFileAddition>* blob_file_additions,
const InternalKeyComparator& internal_comparator,
FileMetaData* meta, const InternalKeyComparator& internal_comparator,
const std::vector<std::unique_ptr<IntTblPropCollectorFactory>>*
int_tbl_prop_collector_factories,
uint32_t column_family_id, const std::string& column_family_name,
+9 -11
View File
@@ -504,13 +504,13 @@ rocksdb_t* rocksdb_open_with_ttl(
return result;
}
rocksdb_t* rocksdb_open_for_read_only(const rocksdb_options_t* options,
const char* name,
unsigned char error_if_wal_file_exists,
char** errptr) {
rocksdb_t* rocksdb_open_for_read_only(
const rocksdb_options_t* options,
const char* name,
unsigned char error_if_log_file_exist,
char** errptr) {
DB* db;
if (SaveError(errptr, DB::OpenForReadOnly(options->rep, std::string(name),
&db, error_if_wal_file_exists))) {
if (SaveError(errptr, DB::OpenForReadOnly(options->rep, std::string(name), &db, error_if_log_file_exist))) {
return nullptr;
}
rocksdb_t* result = new rocksdb_t;
@@ -747,7 +747,7 @@ rocksdb_t* rocksdb_open_for_read_only_column_families(
int num_column_families, const char* const* column_family_names,
const rocksdb_options_t* const* column_family_options,
rocksdb_column_family_handle_t** column_family_handles,
unsigned char error_if_wal_file_exists, char** errptr) {
unsigned char error_if_log_file_exist, char** errptr) {
std::vector<ColumnFamilyDescriptor> column_families;
for (int i = 0; i < num_column_families; i++) {
column_families.push_back(ColumnFamilyDescriptor(
@@ -757,10 +757,8 @@ rocksdb_t* rocksdb_open_for_read_only_column_families(
DB* db;
std::vector<ColumnFamilyHandle*> handles;
if (SaveError(errptr,
DB::OpenForReadOnly(DBOptions(db_options->rep),
std::string(name), column_families,
&handles, &db, error_if_wal_file_exists))) {
if (SaveError(errptr, DB::OpenForReadOnly(DBOptions(db_options->rep),
std::string(name), column_families, &handles, &db, error_if_log_file_exist))) {
return nullptr;
}
+1 -5
View File
@@ -58,11 +58,7 @@ static void StartPhase(const char* name) {
static const char* GetTempDir(void) {
const char* ret = getenv("TEST_TMPDIR");
if (ret == NULL || ret[0] == '\0')
#ifdef OS_WIN
ret = getenv("TEMP");
#else
ret = "/tmp";
#endif
ret = "/tmp";
return ret;
}
#ifdef _MSC_VER
+9 -14
View File
@@ -32,7 +32,7 @@
#include "monitoring/thread_status_util.h"
#include "options/options_helper.h"
#include "port/port.h"
#include "rocksdb/table.h"
#include "table/block_based/block_based_table_factory.h"
#include "table/merging_iterator.h"
#include "util/autovector.h"
#include "util/cast_util.h"
@@ -357,8 +357,8 @@ ColumnFamilyOptions SanitizeOptions(const ImmutableDBOptions& db_options,
result.max_compaction_bytes = result.target_file_size_base * 25;
}
bool is_block_based_table = (result.table_factory->IsInstanceOf(
TableFactory::kBlockBasedTableName()));
bool is_block_based_table =
(result.table_factory->Name() == BlockBasedTableFactory::kName);
const uint64_t kAdjustedTtl = 30 * 24 * 60 * 60;
if (result.ttl == kDefaultTtl) {
@@ -1102,12 +1102,10 @@ Status ColumnFamilyData::RangesOverlapWithMemtables(
super_version->mem->NewRangeTombstoneIterator(read_opts, read_seq);
range_del_agg.AddTombstones(
std::unique_ptr<FragmentedRangeTombstoneIterator>(active_range_del_iter));
Status status;
status = super_version->imm->AddRangeTombstoneIterators(
read_opts, nullptr /* arena */, &range_del_agg);
// AddRangeTombstoneIterators always return Status::OK.
assert(status.ok());
super_version->imm->AddRangeTombstoneIterators(read_opts, nullptr /* arena */,
&range_del_agg);
Status status;
for (size_t i = 0; i < ranges.size() && status.ok() && !*overlap; ++i) {
auto* vstorage = super_version->current->storage_info();
auto* ucmp = vstorage->InternalComparator()->user_comparator();
@@ -1118,8 +1116,7 @@ Status ColumnFamilyData::RangesOverlapWithMemtables(
ParsedInternalKey seek_result;
if (status.ok()) {
if (memtable_iter->Valid() &&
ParseInternalKey(memtable_iter->key(), &seek_result) !=
Status::OK()) {
!ParseInternalKey(memtable_iter->key(), &seek_result)) {
status = Status::Corruption("DB have corrupted keys");
}
}
@@ -1319,8 +1316,7 @@ Status ColumnFamilyData::ValidateOptions(
}
if (cf_options.ttl > 0 && cf_options.ttl != kDefaultTtl) {
if (!cf_options.table_factory->IsInstanceOf(
TableFactory::kBlockBasedTableName())) {
if (cf_options.table_factory->Name() != BlockBasedTableFactory::kName) {
return Status::NotSupported(
"TTL is only supported in Block-Based Table format. ");
}
@@ -1328,8 +1324,7 @@ Status ColumnFamilyData::ValidateOptions(
if (cf_options.periodic_compaction_seconds > 0 &&
cf_options.periodic_compaction_seconds != kDefaultPeriodicCompSecs) {
if (!cf_options.table_factory->IsInstanceOf(
TableFactory::kBlockBasedTableName())) {
if (cf_options.table_factory->Name() != BlockBasedTableFactory::kName) {
return Status::NotSupported(
"Periodic Compaction is only supported in "
"Block-Based Table format. ");
+24 -40
View File
@@ -79,12 +79,7 @@ class ColumnFamilyTestBase : public testing::Test {
std::vector<ColumnFamilyDescriptor> column_families;
for (auto h : handles_) {
ColumnFamilyDescriptor cfdescriptor;
Status s = h->GetDescriptor(&cfdescriptor);
#ifdef ROCKSDB_LITE
EXPECT_TRUE(s.IsNotSupported());
#else
EXPECT_OK(s);
#endif // ROCKSDB_LITE
h->GetDescriptor(&cfdescriptor);
column_families.push_back(cfdescriptor);
}
Close();
@@ -173,7 +168,7 @@ class ColumnFamilyTestBase : public testing::Test {
void Close() {
for (auto h : handles_) {
if (h) {
ASSERT_OK(db_->DestroyColumnFamilyHandle(h));
db_->DestroyColumnFamilyHandle(h);
}
}
handles_.clear();
@@ -284,11 +279,9 @@ class ColumnFamilyTestBase : public testing::Test {
// Verify the CF options of the returned CF handle.
ColumnFamilyDescriptor desc;
ASSERT_OK(handles_[cfi]->GetDescriptor(&desc));
// Need to sanitize the default column family options before comparing
// them.
ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(
ConfigOptions(), desc.options,
SanitizeOptions(dbfull()->immutable_db_options(), current_cf_opt)));
RocksDBOptionsParser::VerifyCFOptions(ConfigOptions(), desc.options,
current_cf_opt);
#endif // !ROCKSDB_LITE
cfi++;
}
@@ -314,7 +307,7 @@ class ColumnFamilyTestBase : public testing::Test {
void DropColumnFamilies(const std::vector<int>& cfs) {
for (auto cf : cfs) {
ASSERT_OK(db_->DropColumnFamily(handles_[cf]));
ASSERT_OK(db_->DestroyColumnFamilyHandle(handles_[cf]));
db_->DestroyColumnFamilyHandle(handles_[cf]);
handles_[cf] = nullptr;
names_[cf] = "";
}
@@ -335,7 +328,7 @@ class ColumnFamilyTestBase : public testing::Test {
ASSERT_OK(Put(cf, key, rnd_.RandomString(key_value_size - 10)));
}
}
ASSERT_OK(db_->FlushWAL(/*sync=*/false));
db_->FlushWAL(false);
}
#ifndef ROCKSDB_LITE // TEST functions in DB are not supported in lite
@@ -657,7 +650,7 @@ TEST_P(FlushEmptyCFTestWithParam, FlushEmptyCFTest) {
Flush(0);
ASSERT_OK(Put(1, "bar", "v3")); // seqID 4
ASSERT_OK(Put(1, "foo", "v4")); // seqID 5
ASSERT_OK(db_->FlushWAL(/*sync=*/false));
db_->FlushWAL(false);
// Preserve file system state up to here to simulate a crash condition.
fault_env->SetFilesystemActive(false);
@@ -720,7 +713,7 @@ TEST_P(FlushEmptyCFTestWithParam, FlushEmptyCFTest2) {
// Write to log file D
ASSERT_OK(Put(1, "bar", "v4")); // seqID 7
ASSERT_OK(Put(1, "bar", "v5")); // seqID 8
ASSERT_OK(db_->FlushWAL(/*sync=*/false));
db_->FlushWAL(false);
// Preserve file system state up to here to simulate a crash condition.
fault_env->SetFilesystemActive(false);
std::vector<std::string> names;
@@ -849,15 +842,13 @@ TEST_P(ColumnFamilyTest, WriteBatchFailure) {
Open();
CreateColumnFamiliesAndReopen({"one", "two"});
WriteBatch batch;
ASSERT_OK(batch.Put(handles_[0], Slice("existing"), Slice("column-family")));
ASSERT_OK(
batch.Put(handles_[1], Slice("non-existing"), Slice("column-family")));
batch.Put(handles_[0], Slice("existing"), Slice("column-family"));
batch.Put(handles_[1], Slice("non-existing"), Slice("column-family"));
ASSERT_OK(db_->Write(WriteOptions(), &batch));
DropColumnFamilies({1});
WriteOptions woptions_ignore_missing_cf;
woptions_ignore_missing_cf.ignore_missing_column_families = true;
ASSERT_OK(
batch.Put(handles_[0], Slice("still here"), Slice("column-family")));
batch.Put(handles_[0], Slice("still here"), Slice("column-family"));
ASSERT_OK(db_->Write(woptions_ignore_missing_cf, &batch));
ASSERT_EQ("column-family", Get(0, "still here"));
Status s = db_->Write(WriteOptions(), &batch);
@@ -896,10 +887,10 @@ TEST_P(ColumnFamilyTest, IgnoreRecoveredLog) {
ASSERT_OK(env_->CreateDirIfMissing(dbname_));
ASSERT_OK(env_->CreateDirIfMissing(backup_logs));
std::vector<std::string> old_files;
ASSERT_OK(env_->GetChildren(backup_logs, &old_files));
env_->GetChildren(backup_logs, &old_files);
for (auto& file : old_files) {
if (file != "." && file != "..") {
ASSERT_OK(env_->DeleteFile(backup_logs + "/" + file));
env_->DeleteFile(backup_logs + "/" + file);
}
}
@@ -927,7 +918,7 @@ TEST_P(ColumnFamilyTest, IgnoreRecoveredLog) {
// copy the logs to backup
std::vector<std::string> logs;
ASSERT_OK(env_->GetChildren(db_options_.wal_dir, &logs));
env_->GetChildren(db_options_.wal_dir, &logs);
for (auto& log : logs) {
if (log != ".." && log != ".") {
CopyFile(db_options_.wal_dir + "/" + log, backup_logs + "/" + log);
@@ -992,7 +983,6 @@ TEST_P(ColumnFamilyTest, FlushTest) {
ASSERT_OK(Put(0, "foofoo", "bar"));
for (auto* it : iterators) {
ASSERT_OK(it->status());
delete it;
}
}
@@ -1090,8 +1080,8 @@ TEST_P(ColumnFamilyTest, CrashAfterFlush) {
CreateColumnFamilies({"one"});
WriteBatch batch;
ASSERT_OK(batch.Put(handles_[0], Slice("foo"), Slice("bar")));
ASSERT_OK(batch.Put(handles_[1], Slice("foo"), Slice("bar")));
batch.Put(handles_[0], Slice("foo"), Slice("bar"));
batch.Put(handles_[1], Slice("foo"), Slice("bar"));
ASSERT_OK(db_->Write(WriteOptions(), &batch));
Flush(0);
fault_env->SetFilesystemActive(false);
@@ -2077,7 +2067,6 @@ std::string IterStatus(Iterator* iter) {
if (iter->Valid()) {
result = iter->key().ToString() + "->" + iter->value().ToString();
} else {
EXPECT_OK(iter->status());
result = "(invalid)";
}
return result;
@@ -2332,7 +2321,7 @@ TEST_P(ColumnFamilyTest, ReadDroppedColumnFamily) {
ASSERT_OK(db_->DropColumnFamily(handles_[2]));
} else {
// delete CF two
ASSERT_OK(db_->DestroyColumnFamilyHandle(handles_[2]));
db_->DestroyColumnFamilyHandle(handles_[2]);
handles_[2] = nullptr;
}
// Make sure iterator created can still be used.
@@ -2388,6 +2377,7 @@ TEST_P(ColumnFamilyTest, LiveIteratorWithDroppedColumnFamily) {
// 1MB should create ~10 files for each CF
int kKeysNum = 10000;
PutRandomData(1, kKeysNum, 100);
{
std::unique_ptr<Iterator> iterator(
db_->NewIterator(ReadOptions(), handles_[1]));
@@ -3038,7 +3028,6 @@ TEST_P(ColumnFamilyTest, IteratorCloseWALFile1) {
ASSERT_OK(Put(1, "fodor", "mirko"));
// Create an iterator holding the current super version.
Iterator* it = db_->NewIterator(ReadOptions(), handles_[1]);
ASSERT_OK(it->status());
// A flush will make `it` hold the last reference of its super version.
Flush(1);
@@ -3091,7 +3080,6 @@ TEST_P(ColumnFamilyTest, IteratorCloseWALFile2) {
ReadOptions ro;
ro.background_purge_on_iterator_cleanup = true;
Iterator* it = db_->NewIterator(ro, handles_[1]);
ASSERT_OK(it->status());
// A flush will make `it` hold the last reference of its super version.
Flush(1);
@@ -3186,8 +3174,6 @@ TEST_P(ColumnFamilyTest, ForwardIteratorCloseWALFile) {
// Deleting the iterator will clear its super version, triggering
// closing all files
it->Seek("");
ASSERT_OK(it->status());
ASSERT_EQ(2, env.num_open_wal_file_.load());
ASSERT_EQ(0, env.delete_count_.load());
@@ -3218,8 +3204,8 @@ TEST_P(ColumnFamilyTest, LogSyncConflictFlush) {
Open();
CreateColumnFamiliesAndReopen({"one", "two"});
ASSERT_OK(Put(0, "", ""));
ASSERT_OK(Put(1, "foo", "bar"));
Put(0, "", "");
Put(1, "foo", "bar");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::SyncWAL:BeforeMarkLogsSynced:1",
@@ -3229,11 +3215,11 @@ TEST_P(ColumnFamilyTest, LogSyncConflictFlush) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::port::Thread thread([&] { ASSERT_OK(db_->SyncWAL()); });
ROCKSDB_NAMESPACE::port::Thread thread([&] { db_->SyncWAL(); });
TEST_SYNC_POINT("ColumnFamilyTest::LogSyncConflictFlush:1");
Flush(1);
ASSERT_OK(Put(1, "foo", "bar"));
Put(1, "foo", "bar");
Flush(1);
TEST_SYNC_POINT("ColumnFamilyTest::LogSyncConflictFlush:2");
@@ -3316,7 +3302,7 @@ TEST_P(ColumnFamilyTest, DISABLED_LogTruncationTest) {
Close();
// cleanup
ASSERT_OK(env_->DeleteDir(backup_logs));
env_->DeleteDir(backup_logs);
}
TEST_P(ColumnFamilyTest, DefaultCfPathsTest) {
@@ -3378,11 +3364,9 @@ TEST_P(ColumnFamilyTest, MultipleCFPathsTest) {
read_options.readahead_size = 0;
auto it = dbi->NewIterator(read_options, handles_[cf]);
for (it->SeekToFirst(); it->Valid(); it->Next()) {
ASSERT_OK(it->status());
Slice key(it->key());
ASSERT_NE(keys_[cf].end(), keys_[cf].find(key.ToString()));
}
ASSERT_OK(it->status());
delete it;
for (const auto& key : keys_[cf]) {
+7 -14
View File
@@ -41,11 +41,8 @@ Status CompactedDBImpl::Get(const ReadOptions& options, ColumnFamilyHandle*,
GetContext::kNotFound, key, value, nullptr, nullptr,
nullptr, true, nullptr, nullptr);
LookupKey lkey(key, kMaxSequenceNumber);
Status s = files_.files[FindFile(key)].fd.table_reader->Get(
options, lkey.internal_key(), &get_context, nullptr);
if (!s.ok() && !s.IsNotFound()) {
return s;
}
files_.files[FindFile(key)].fd.table_reader->Get(options, lkey.internal_key(),
&get_context, nullptr);
if (get_context.State() == GetContext::kFound) {
return Status::OK();
}
@@ -77,14 +74,10 @@ std::vector<Status> CompactedDBImpl::MultiGet(const ReadOptions& options,
GetContext::kNotFound, keys[idx], &pinnable_val,
nullptr, nullptr, nullptr, true, nullptr, nullptr);
LookupKey lkey(keys[idx], kMaxSequenceNumber);
Status s = r->Get(options, lkey.internal_key(), &get_context, nullptr);
if (!s.ok() && !s.IsNotFound()) {
statuses[idx] = s;
} else {
value.assign(pinnable_val.data(), pinnable_val.size());
if (get_context.State() == GetContext::kFound) {
statuses[idx] = Status::OK();
}
r->Get(options, lkey.internal_key(), &get_context, nullptr);
value.assign(pinnable_val.data(), pinnable_val.size());
if (get_context.State() == GetContext::kFound) {
statuses[idx] = Status::OK();
}
}
++idx;
@@ -156,7 +149,7 @@ Status CompactedDBImpl::Open(const Options& options,
std::unique_ptr<CompactedDBImpl> db(new CompactedDBImpl(db_options, dbname));
Status s = db->Init(options);
if (s.ok()) {
db->StartPeriodicWorkScheduler();
db->StartStatsDumpScheduler();
ROCKS_LOG_INFO(db->immutable_db_options_.info_log,
"Opened the db as fully compacted mode");
LogFlush(db->immutable_db_options_.info_log);
+30 -65
View File
@@ -3,11 +3,9 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#include "db/compaction/compaction_iterator.h"
#include <cinttypes>
#include "db/blob/blob_file_builder.h"
#include "db/compaction/compaction_iterator.h"
#include "db/snapshot_checker.h"
#include "port/likely.h"
#include "rocksdb/listener.h"
@@ -38,9 +36,8 @@ CompactionIterator::CompactionIterator(
SequenceNumber earliest_write_conflict_snapshot,
const SnapshotChecker* snapshot_checker, Env* env,
bool report_detailed_time, bool expect_valid_internal_key,
CompactionRangeDelAggregator* range_del_agg,
BlobFileBuilder* blob_file_builder, bool allow_data_in_errors,
const Compaction* compaction, const CompactionFilter* compaction_filter,
CompactionRangeDelAggregator* range_del_agg, const Compaction* compaction,
const CompactionFilter* compaction_filter,
const std::atomic<bool>* shutting_down,
const SequenceNumber preserve_deletes_seqnum,
const std::atomic<int>* manual_compaction_paused,
@@ -49,7 +46,6 @@ CompactionIterator::CompactionIterator(
input, cmp, merge_helper, last_sequence, snapshots,
earliest_write_conflict_snapshot, snapshot_checker, env,
report_detailed_time, expect_valid_internal_key, range_del_agg,
blob_file_builder, allow_data_in_errors,
std::unique_ptr<CompactionProxy>(
compaction ? new CompactionProxy(compaction) : nullptr),
compaction_filter, shutting_down, preserve_deletes_seqnum,
@@ -62,7 +58,6 @@ CompactionIterator::CompactionIterator(
const SnapshotChecker* snapshot_checker, Env* env,
bool report_detailed_time, bool expect_valid_internal_key,
CompactionRangeDelAggregator* range_del_agg,
BlobFileBuilder* blob_file_builder, bool allow_data_in_errors,
std::unique_ptr<CompactionProxy> compaction,
const CompactionFilter* compaction_filter,
const std::atomic<bool>* shutting_down,
@@ -79,7 +74,6 @@ CompactionIterator::CompactionIterator(
report_detailed_time_(report_detailed_time),
expect_valid_internal_key_(expect_valid_internal_key),
range_del_agg_(range_del_agg),
blob_file_builder_(blob_file_builder),
compaction_(std::move(compaction)),
compaction_filter_(compaction_filter),
shutting_down_(shutting_down),
@@ -89,8 +83,7 @@ CompactionIterator::CompactionIterator(
current_user_key_snapshot_(0),
merge_out_iter_(merge_helper_),
current_key_committed_(false),
info_log_(info_log),
allow_data_in_errors_(allow_data_in_errors) {
info_log_(info_log) {
assert(compaction_filter_ == nullptr || compaction_ != nullptr);
assert(snapshots_ != nullptr);
bottommost_level_ = compaction_ == nullptr
@@ -151,11 +144,11 @@ void CompactionIterator::Next() {
if (merge_out_iter_.Valid()) {
key_ = merge_out_iter_.key();
value_ = merge_out_iter_.value();
Status s = ParseInternalKey(key_, &ikey_);
bool valid_key = ParseInternalKey(key_, &ikey_);
// MergeUntil stops when it encounters a corrupt key and does not
// include them in the result, so we expect the keys here to be valid.
assert(s.ok());
if (!s.ok()) {
assert(valid_key);
if (!valid_key) {
ROCKS_LOG_FATAL(info_log_, "Invalid key (%s) in compaction",
key_.ToString(true).c_str());
}
@@ -270,23 +263,15 @@ void CompactionIterator::NextFromInput() {
value_ = input_->value();
iter_stats_.num_input_records++;
Status pikStatus = ParseInternalKey(key_, &ikey_);
if (!pikStatus.ok()) {
if (!ParseInternalKey(key_, &ikey_)) {
iter_stats_.num_input_corrupt_records++;
// If `expect_valid_internal_key_` is false, return the corrupted key
// and let the caller decide what to do with it.
// TODO(noetzli): We should have a more elegant solution for this.
if (expect_valid_internal_key_) {
std::string msg("Corrupted internal key not expected.");
if (allow_data_in_errors_) {
msg.append(" Corrupt key: " + ikey_.user_key.ToString(/*hex=*/true) +
". ");
msg.append("key type: " + std::to_string(ikey_.type) + ".");
msg.append("seq: " + std::to_string(ikey_.sequence) + ".");
}
status_ = Status::Corruption(msg.c_str());
return;
assert(!"Corrupted internal key not expected.");
status_ = Status::Corruption("Corrupted internal key not expected.");
break;
}
key_ = current_key_.SetInternalKey(key_);
has_current_user_key_ = false;
@@ -438,8 +423,7 @@ void CompactionIterator::NextFromInput() {
// Check whether the next key exists, is not corrupt, and is the same key
// as the single delete.
if (input_->Valid() &&
ParseInternalKey(input_->key(), &next_ikey) == Status::OK() &&
if (input_->Valid() && ParseInternalKey(input_->key(), &next_ikey) &&
cmp_->Equal(ikey_.user_key, next_ikey.user_key)) {
// Check whether the next key belongs to the same snapshot as the
// SingleDelete.
@@ -586,8 +570,7 @@ void CompactionIterator::NextFromInput() {
// Skip over all versions of this key that happen to occur in the same snapshot
// range as the delete
while (!IsPausingManualCompaction() && !IsShuttingDown() &&
input_->Valid() &&
(ParseInternalKey(input_->key(), &next_ikey) == Status::OK()) &&
input_->Valid() && ParseInternalKey(input_->key(), &next_ikey) &&
cmp_->Equal(ikey_.user_key, next_ikey.user_key) &&
(prev_snapshot == 0 ||
DEFINITELY_NOT_IN_SNAPSHOT(next_ikey.sequence, prev_snapshot))) {
@@ -595,8 +578,7 @@ void CompactionIterator::NextFromInput() {
}
// If you find you still need to output a row with this key, we need to output the
// delete too
if (input_->Valid() &&
(ParseInternalKey(input_->key(), &next_ikey) == Status::OK()) &&
if (input_->Valid() && ParseInternalKey(input_->key(), &next_ikey) &&
cmp_->Equal(ikey_.user_key, next_ikey.user_key)) {
valid_ = true;
at_next_ = true;
@@ -625,11 +607,11 @@ void CompactionIterator::NextFromInput() {
// These will be correctly set below.
key_ = merge_out_iter_.key();
value_ = merge_out_iter_.value();
pikStatus = ParseInternalKey(key_, &ikey_);
bool valid_key = ParseInternalKey(key_, &ikey_);
// MergeUntil stops when it encounters a corrupt key and does not
// include them in the result, so we expect the keys here to valid.
assert(pikStatus.ok());
if (!pikStatus.ok()) {
assert(valid_key);
if (!valid_key) {
ROCKS_LOG_FATAL(info_log_, "Invalid key (%s) in compaction",
key_.ToString(true).c_str());
}
@@ -679,37 +661,20 @@ void CompactionIterator::NextFromInput() {
void CompactionIterator::PrepareOutput() {
if (valid_) {
if (ikey_.type == kTypeValue) {
if (blob_file_builder_) {
blob_index_.clear();
const Status s =
blob_file_builder_->Add(user_key(), value_, &blob_index_);
if (compaction_filter_ && ikey_.type == kTypeBlobIndex) {
const auto blob_decision = compaction_filter_->PrepareBlobOutput(
user_key(), value_, &compaction_filter_value_);
if (!s.ok()) {
status_ = s;
valid_ = false;
} else if (!blob_index_.empty()) {
value_ = blob_index_;
ikey_.type = kTypeBlobIndex;
current_key_.UpdateInternalKey(ikey_.sequence, ikey_.type);
}
}
} else if (ikey_.type == kTypeBlobIndex) {
if (compaction_filter_) {
const auto blob_decision = compaction_filter_->PrepareBlobOutput(
user_key(), value_, &compaction_filter_value_);
if (blob_decision == CompactionFilter::BlobDecision::kCorruption) {
status_ = Status::Corruption(
"Corrupted blob reference encountered during GC");
valid_ = false;
} else if (blob_decision == CompactionFilter::BlobDecision::kIOError) {
status_ = Status::IOError("Could not relocate blob during GC");
valid_ = false;
} else if (blob_decision ==
CompactionFilter::BlobDecision::kChangeValue) {
value_ = compaction_filter_value_;
}
if (blob_decision == CompactionFilter::BlobDecision::kCorruption) {
status_ = Status::Corruption(
"Corrupted blob reference encountered during GC");
valid_ = false;
} else if (blob_decision == CompactionFilter::BlobDecision::kIOError) {
status_ = Status::IOError("Could not relocate blob during GC");
valid_ = false;
} else if (blob_decision ==
CompactionFilter::BlobDecision::kChangeValue) {
value_ = compaction_filter_value_;
}
}
-10
View File
@@ -21,8 +21,6 @@
namespace ROCKSDB_NAMESPACE {
class BlobFileBuilder;
class CompactionIterator {
public:
// A wrapper around Compaction. Has a much smaller interface, only what
@@ -68,8 +66,6 @@ class CompactionIterator {
const SnapshotChecker* snapshot_checker, Env* env,
bool report_detailed_time, bool expect_valid_internal_key,
CompactionRangeDelAggregator* range_del_agg,
BlobFileBuilder* blob_file_builder,
bool allow_data_in_errors,
const Compaction* compaction = nullptr,
const CompactionFilter* compaction_filter = nullptr,
const std::atomic<bool>* shutting_down = nullptr,
@@ -85,8 +81,6 @@ class CompactionIterator {
const SnapshotChecker* snapshot_checker, Env* env,
bool report_detailed_time, bool expect_valid_internal_key,
CompactionRangeDelAggregator* range_del_agg,
BlobFileBuilder* blob_file_builder,
bool allow_data_in_errors,
std::unique_ptr<CompactionProxy> compaction,
const CompactionFilter* compaction_filter = nullptr,
const std::atomic<bool>* shutting_down = nullptr,
@@ -169,7 +163,6 @@ class CompactionIterator {
bool report_detailed_time_;
bool expect_valid_internal_key_;
CompactionRangeDelAggregator* range_del_agg_;
BlobFileBuilder* blob_file_builder_;
std::unique_ptr<CompactionProxy> compaction_;
const CompactionFilter* compaction_filter_;
const std::atomic<bool>* shutting_down_;
@@ -218,7 +211,6 @@ class CompactionIterator {
// PinnedIteratorsManager used to pin input_ Iterator blocks while reading
// merge operands and then releasing them after consuming them.
PinnedIteratorsManager pinned_iters_mgr_;
std::string blob_index_;
std::string compaction_filter_value_;
InternalKey compaction_filter_skip_until_;
// "level_ptrs" holds indices that remember which file of an associated
@@ -235,8 +227,6 @@ class CompactionIterator {
bool current_key_committed_;
std::shared_ptr<Logger> info_log_;
bool allow_data_in_errors_;
bool IsShuttingDown() {
// This is a best-effort facility, so memory_order_relaxed is sufficient.
return shutting_down_ && shutting_down_->load(std::memory_order_relaxed);
+1 -10
View File
@@ -258,9 +258,7 @@ class CompactionIteratorTest : public testing::TestWithParam<bool> {
iter_.get(), cmp_, merge_helper_.get(), last_sequence, &snapshots_,
earliest_write_conflict_snapshot, snapshot_checker_.get(),
Env::Default(), false /* report_detailed_time */, false,
range_del_agg_.get(), nullptr /* blob_file_builder */,
false /*allow_data_in_errors*/, std::move(compaction), filter,
&shutting_down_));
range_del_agg_.get(), std::move(compaction), filter, &shutting_down_));
}
void AddSnapshot(SequenceNumber snapshot,
@@ -295,7 +293,6 @@ class CompactionIteratorTest : public testing::TestWithParam<bool> {
ASSERT_EQ(expected_values[i], c_iter_->value().ToString()) << info;
c_iter_->Next();
}
ASSERT_OK(c_iter_->status());
ASSERT_FALSE(c_iter_->Valid());
}
@@ -320,7 +317,6 @@ TEST_P(CompactionIteratorTest, EmptyResult) {
test::KeyStr("a", 3, kTypeValue)},
{"", "val"}, {}, {}, 5);
c_iter_->SeekToFirst();
ASSERT_OK(c_iter_->status());
ASSERT_FALSE(c_iter_->Valid());
}
@@ -342,7 +338,6 @@ TEST_P(CompactionIteratorTest, CorruptionAfterSingleDeletion) {
ASSERT_TRUE(c_iter_->Valid());
ASSERT_EQ(test::KeyStr("b", 10, kTypeValue), c_iter_->key().ToString());
c_iter_->Next();
ASSERT_OK(c_iter_->status());
ASSERT_FALSE(c_iter_->Valid());
}
@@ -359,7 +354,6 @@ TEST_P(CompactionIteratorTest, SimpleRangeDeletion) {
ASSERT_TRUE(c_iter_->Valid());
ASSERT_EQ(test::KeyStr("night", 3, kTypeValue), c_iter_->key().ToString());
c_iter_->Next();
ASSERT_OK(c_iter_->status());
ASSERT_FALSE(c_iter_->Valid());
}
@@ -381,7 +375,6 @@ TEST_P(CompactionIteratorTest, RangeDeletionWithSnapshots) {
ASSERT_TRUE(c_iter_->Valid());
ASSERT_EQ(test::KeyStr("night", 40, kTypeValue), c_iter_->key().ToString());
c_iter_->Next();
ASSERT_OK(c_iter_->status());
ASSERT_FALSE(c_iter_->Valid());
}
@@ -475,7 +468,6 @@ TEST_P(CompactionIteratorTest, CompactionFilterSkipUntil) {
ASSERT_EQ(test::KeyStr("h", 91, kTypeValue), c_iter_->key().ToString());
ASSERT_EQ("hv91", c_iter_->value().ToString());
c_iter_->Next();
ASSERT_OK(c_iter_->status());
ASSERT_FALSE(c_iter_->Valid());
// Check that the compaction iterator did the correct sequence of calls on
@@ -669,7 +661,6 @@ TEST_P(CompactionIteratorTest, SingleMergeOperand) {
ASSERT_TRUE(c_iter_->Valid());
ASSERT_EQ("bv1bv2", c_iter_->value().ToString());
c_iter_->Next();
ASSERT_OK(c_iter_->status());
ASSERT_EQ("cv1cv2", c_iter_->value().ToString());
}
+69 -101
View File
@@ -32,7 +32,6 @@
#include "db/memtable_list.h"
#include "db/merge_context.h"
#include "db/merge_helper.h"
#include "db/output_validator.h"
#include "db/range_del_aggregator.h"
#include "db/version_set.h"
#include "file/filename.h"
@@ -125,14 +124,9 @@ struct CompactionJob::SubcompactionState {
// Files produced by this subcompaction
struct Output {
Output(FileMetaData&& _meta, const InternalKeyComparator& _icmp,
bool _enable_order_check, bool _enable_hash)
: meta(std::move(_meta)),
validator(_icmp, _enable_order_check, _enable_hash),
finished(false) {}
FileMetaData meta;
OutputValidator validator;
bool finished;
uint64_t paranoid_hash;
std::shared_ptr<const TableProperties> table_properties;
};
@@ -176,16 +170,17 @@ struct CompactionJob::SubcompactionState {
// Adds the key and value to the builder
// If paranoid is true, adds the key-value to the paranoid hash
Status AddToBuilder(const Slice& key, const Slice& value) {
void AddToBuilder(const Slice& key, const Slice& value, bool paranoid) {
auto curr = current_output();
assert(builder != nullptr);
assert(curr != nullptr);
Status s = curr->validator.Add(key, value);
if (!s.ok()) {
return s;
if (paranoid) {
// Generate a rolling 64-bit hash of the key and values
curr->paranoid_hash = Hash64(key.data(), key.size(), curr->paranoid_hash);
curr->paranoid_hash =
Hash64(value.data(), value.size(), curr->paranoid_hash);
}
builder->Add(key, value);
return Status::OK();
}
// Returns true iff we should stop building the current output
@@ -276,8 +271,10 @@ void CompactionJob::AggregateStatistics() {
compact_->total_bytes += sc.total_bytes;
compact_->num_output_records += sc.num_output_records;
}
for (SubcompactionState& sc : compact_->sub_compact_states) {
compaction_job_stats_->Add(sc.compaction_job_stats);
if (compaction_job_stats_) {
for (SubcompactionState& sc : compact_->sub_compact_states) {
compaction_job_stats_->Add(sc.compaction_job_stats);
}
}
}
@@ -330,7 +327,6 @@ CompactionJob::CompactionJob(
measure_io_stats_(measure_io_stats),
write_hint_(Env::WLTH_NOT_SET),
thread_pri_(thread_pri) {
assert(compaction_job_stats_ != nullptr);
assert(log_buffer_ != nullptr);
const auto* cfd = compact_->compaction->column_family_data();
ThreadStatusUtil::SetColumnFamily(cfd, cfd->ioptions()->env,
@@ -382,9 +378,10 @@ void CompactionJob::ReportStartedCompaction(Compaction* compaction) {
// to ensure GetThreadList() can always show them all together.
ThreadStatusUtil::SetThreadOperation(ThreadStatus::OP_COMPACTION);
compaction_job_stats_->is_manual_compaction =
compaction->is_manual_compaction();
compaction_job_stats_->is_full_compaction = compaction->is_full_compaction();
if (compaction_job_stats_) {
compaction_job_stats_->is_manual_compaction =
compaction->is_manual_compaction();
}
}
void CompactionJob::Prepare() {
@@ -665,20 +662,14 @@ Status CompactionJob::Run() {
auto s = iter->status();
if (s.ok() && paranoid_file_checks_) {
OutputValidator validator(cfd->internal_comparator(),
/*_enable_order_check=*/true,
/*_enable_hash=*/true);
uint64_t hash = 0;
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
s = validator.Add(iter->key(), iter->value());
if (!s.ok()) {
break;
}
// Generate a rolling 64-bit hash of the key and values, using the
hash = Hash64(iter->key().data(), iter->key().size(), hash);
hash = Hash64(iter->value().data(), iter->value().size(), hash);
}
if (s.ok()) {
s = iter->status();
}
if (s.ok() &&
!validator.CompareValidator(files_output[file_idx]->validator)) {
s = iter->status();
if (s.ok() && hash != files_output[file_idx]->paranoid_hash) {
s = Status::Corruption("Paranoid checksums do not match");
}
}
@@ -802,12 +793,14 @@ Status CompactionJob::Install(const MutableCFOptions& mutable_cf_options) {
<< compact_->sub_compact_states.size() << "output_compression"
<< CompressionTypeToString(compact_->compaction->output_compression());
stream << "num_single_delete_mismatches"
<< compaction_job_stats_->num_single_del_mismatch;
stream << "num_single_delete_fallthrough"
<< compaction_job_stats_->num_single_del_fallthru;
if (compaction_job_stats_ != nullptr) {
stream << "num_single_delete_mismatches"
<< compaction_job_stats_->num_single_del_mismatch;
stream << "num_single_delete_fallthrough"
<< compaction_job_stats_->num_single_del_fallthru;
}
if (measure_io_stats_) {
if (measure_io_stats_ && compaction_job_stats_ != nullptr) {
stream << "file_write_nanos" << compaction_job_stats_->file_write_nanos;
stream << "file_range_sync_nanos"
<< compaction_job_stats_->file_range_sync_nanos;
@@ -921,7 +914,6 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
&existing_snapshots_, earliest_write_conflict_snapshot_,
snapshot_checker_, env_, ShouldReportDetailedTime(env_, stats_),
/*expect_valid_internal_key=*/true, &range_del_agg,
/* blob_file_builder */ nullptr, db_options_.allow_data_in_errors,
sub_compact->compaction, compaction_filter, shutting_down_,
preserve_deletes_seqnum_, manual_compaction_paused_,
db_options_.info_log));
@@ -968,10 +960,7 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
break;
}
}
status = sub_compact->AddToBuilder(key, value);
if (!status.ok()) {
break;
}
sub_compact->AddToBuilder(key, value, paranoid_file_checks_);
sub_compact->current_output_file_size =
sub_compact->builder->EstimatedFileSize();
@@ -989,11 +978,13 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
// going to be 1.2MB and max_output_file_size = 1MB, prefer to have 0.6MB
// and 0.6MB instead of 1MB and 0.2MB)
bool output_file_ended = false;
Status input_status;
if (sub_compact->compaction->output_level() != 0 &&
sub_compact->current_output_file_size >=
sub_compact->compaction->max_output_file_size()) {
// (1) this key terminates the file. For historical reasons, the iterator
// status before advancing will be given to FinishCompactionOutputFile().
input_status = input->status();
output_file_ended = true;
}
TEST_SYNC_POINT_CALLBACK(
@@ -1020,6 +1011,7 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
// (2) this key belongs to the next file. For historical reasons, the
// iterator status after advancing will be given to
// FinishCompactionOutputFile().
input_status = input->status();
output_file_ended = true;
}
}
@@ -1029,9 +1021,9 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
next_key = &c_iter->key();
}
CompactionIterationStats range_del_out_stats;
status = FinishCompactionOutputFile(input->status(), sub_compact,
&range_del_agg, &range_del_out_stats,
next_key);
status =
FinishCompactionOutputFile(input_status, sub_compact, &range_del_agg,
&range_del_out_stats, next_key);
RecordDroppedKeys(range_del_out_stats,
&sub_compact->compaction_job_stats);
}
@@ -1087,7 +1079,7 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
CompactionIterationStats range_del_out_stats;
Status s = FinishCompactionOutputFile(status, sub_compact, &range_del_agg,
&range_del_out_stats);
if (!s.ok() && status.ok()) {
if (status.ok()) {
status = s;
}
RecordDroppedKeys(range_del_out_stats, &sub_compact->compaction_job_stats);
@@ -1113,16 +1105,6 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
SetPerfLevel(prev_perf_level);
}
}
#ifdef ROCKSDB_ASSERT_STATUS_CHECKED
if (!status.ok()) {
if (sub_compact->c_iter) {
sub_compact->c_iter->status().PermitUncheckedError();
}
if (input) {
input->status().PermitUncheckedError();
}
}
#endif // ROCKSDB_ASSERT_STATUS_CHECKED
sub_compact->c_iter.reset();
input.reset();
@@ -1392,9 +1374,6 @@ Status CompactionJob::FinishCompactionOutputFile(
}
if (sub_compact->io_status.ok()) {
sub_compact->io_status = io_s;
// Since this error is really a copy of the
// "normal" status, it does not also need to be checked
sub_compact->io_status.PermitUncheckedError();
}
sub_compact->outfile.reset();
@@ -1453,10 +1432,7 @@ Status CompactionJob::FinishCompactionOutputFile(
auto sfm =
static_cast<SstFileManagerImpl*>(db_options_.sst_file_manager.get());
if (sfm && meta != nullptr && meta->fd.GetPathId() == 0) {
Status add_s = sfm->OnAddFile(fname);
if (!add_s.ok() && s.ok()) {
s = add_s;
}
sfm->OnAddFile(fname);
if (sfm->IsMaxAllowedSpaceReached()) {
// TODO(ajkr): should we return OK() if max space was reached by the final
// compaction output file (similarly to how flush works when full)?
@@ -1465,9 +1441,7 @@ Status CompactionJob::FinishCompactionOutputFile(
"CompactionJob::FinishCompactionOutputFile:"
"MaxAllowedSpaceReached");
InstrumentedMutexLock l(db_mutex_);
// Should handle return error?
db_error_handler_->SetBGError(s, BackgroundErrorReason::kCompaction)
.PermitUncheckedError();
db_error_handler_->SetBGError(s, BackgroundErrorReason::kCompaction);
}
}
#endif
@@ -1568,9 +1542,6 @@ Status CompactionJob::OpenCompactionOutputFile(
s = io_s;
if (sub_compact->io_status.ok()) {
sub_compact->io_status = io_s;
// Since this error is really a copy of the io_s that is checked below as s,
// it does not also need to be checked.
sub_compact->io_status.PermitUncheckedError();
}
if (!s.ok()) {
ROCKS_LOG_ERROR(
@@ -1606,17 +1577,14 @@ Status CompactionJob::OpenCompactionOutputFile(
// Initialize a SubcompactionState::Output and add it to sub_compact->outputs
{
FileMetaData meta;
meta.fd = FileDescriptor(file_number,
sub_compact->compaction->output_path_id(), 0);
meta.oldest_ancester_time = oldest_ancester_time;
meta.file_creation_time = current_time;
sub_compact->outputs.emplace_back(
std::move(meta), cfd->internal_comparator(),
/*enable_order_check=*/
sub_compact->compaction->mutable_cf_options()
->check_flush_compaction_key_order,
/*enable_hash=*/paranoid_file_checks_);
SubcompactionState::Output out;
out.meta.fd = FileDescriptor(file_number,
sub_compact->compaction->output_path_id(), 0);
out.meta.oldest_ancester_time = oldest_ancester_time;
out.meta.file_creation_time = current_time;
out.finished = false;
out.paranoid_hash = 0;
sub_compact->outputs.push_back(out);
}
writable_file->SetIOPriority(Env::IOPriority::IO_LOW);
@@ -1669,9 +1637,6 @@ void CompactionJob::CleanupCompaction() {
TableCache::Evict(table_cache_.get(), out.meta.fd.GetNumber());
}
}
// TODO: sub_compact.io_status is not checked like status. Not sure if thats
// intentional. So ignoring the io_status as of now.
sub_compact.io_status.PermitUncheckedError();
}
delete compact_;
compact_ = nullptr;
@@ -1748,29 +1713,32 @@ void CompactionJob::UpdateCompactionInputStatsHelper(int* num_files,
void CompactionJob::UpdateCompactionJobStats(
const InternalStats::CompactionStats& stats) const {
#ifndef ROCKSDB_LITE
compaction_job_stats_->elapsed_micros = stats.micros;
if (compaction_job_stats_) {
compaction_job_stats_->elapsed_micros = stats.micros;
// input information
compaction_job_stats_->total_input_bytes =
stats.bytes_read_non_output_levels + stats.bytes_read_output_level;
compaction_job_stats_->num_input_records = stats.num_input_records;
compaction_job_stats_->num_input_files =
stats.num_input_files_in_non_output_levels +
stats.num_input_files_in_output_level;
compaction_job_stats_->num_input_files_at_output_level =
stats.num_input_files_in_output_level;
// input information
compaction_job_stats_->total_input_bytes =
stats.bytes_read_non_output_levels + stats.bytes_read_output_level;
compaction_job_stats_->num_input_records = stats.num_input_records;
compaction_job_stats_->num_input_files =
stats.num_input_files_in_non_output_levels +
stats.num_input_files_in_output_level;
compaction_job_stats_->num_input_files_at_output_level =
stats.num_input_files_in_output_level;
// output information
compaction_job_stats_->total_output_bytes = stats.bytes_written;
compaction_job_stats_->num_output_records = compact_->num_output_records;
compaction_job_stats_->num_output_files = stats.num_output_files;
// output information
compaction_job_stats_->total_output_bytes = stats.bytes_written;
compaction_job_stats_->num_output_records = compact_->num_output_records;
compaction_job_stats_->num_output_files = stats.num_output_files;
if (compact_->NumOutputFiles() > 0U) {
CopyPrefix(compact_->SmallestUserKey(),
CompactionJobStats::kMaxPrefixLength,
&compaction_job_stats_->smallest_output_key_prefix);
CopyPrefix(compact_->LargestUserKey(), CompactionJobStats::kMaxPrefixLength,
&compaction_job_stats_->largest_output_key_prefix);
if (compact_->NumOutputFiles() > 0U) {
CopyPrefix(compact_->SmallestUserKey(),
CompactionJobStats::kMaxPrefixLength,
&compaction_job_stats_->smallest_output_key_prefix);
CopyPrefix(compact_->LargestUserKey(),
CompactionJobStats::kMaxPrefixLength,
&compaction_job_stats_->largest_output_key_prefix);
}
}
#else
(void)stats;
+22 -19
View File
@@ -459,7 +459,6 @@ class CompactionJobStatsChecker : public EventListener {
ASSERT_EQ(current_stats.num_output_files,
stats.num_output_files);
ASSERT_EQ(current_stats.is_full_compaction, stats.is_full_compaction);
ASSERT_EQ(current_stats.is_manual_compaction,
stats.is_manual_compaction);
@@ -572,7 +571,7 @@ CompactionJobStats NewManualCompactionJobStats(
uint64_t num_input_records, size_t key_size, size_t value_size,
size_t num_output_files, uint64_t num_output_records,
double compression_ratio, uint64_t num_records_replaced,
bool is_full = false, bool is_manual = true) {
bool is_manual = true) {
CompactionJobStats stats;
stats.Reset();
@@ -596,7 +595,6 @@ CompactionJobStats NewManualCompactionJobStats(
stats.total_input_raw_value_bytes =
num_input_records * value_size;
stats.is_full_compaction = is_full;
stats.is_manual_compaction = is_manual;
stats.num_records_replaced = num_records_replaced;
@@ -798,7 +796,7 @@ TEST_P(CompactionJobStatsTest, CompactionJobStatsTest) {
}
ASSERT_OK(Flush(1));
ASSERT_OK(static_cast_with_check<DBImpl>(db_)->TEST_WaitForCompact());
static_cast_with_check<DBImpl>(db_)->TEST_WaitForCompact();
stats_checker->set_verify_next_comp_io_stats(true);
std::atomic<bool> first_prepare_write(true);
@@ -896,7 +894,7 @@ TEST_P(CompactionJobStatsTest, DeletionStatsTest) {
CompactRangeOptions cr_options;
cr_options.change_level = true;
cr_options.target_level = 2;
ASSERT_OK(db_->CompactRange(cr_options, handles_[1], nullptr, nullptr));
db_->CompactRange(cr_options, handles_[1], nullptr, nullptr);
ASSERT_GT(NumTableFilesAtLevel(2, 1), 0);
// Stage 2: Generate files including keys from the entire key range
@@ -983,21 +981,26 @@ TEST_P(CompactionJobStatsTest, UniversalCompactionTest) {
if (num_input_units == 0) {
continue;
}
// A full compaction only happens when the number of flushes equals to
// the number of compaction input runs.
bool is_full = num_flushes == num_input_units;
// The following statement determines the expected smallest key
// based on whether it is a full compaction.
uint64_t smallest_key = is_full ? key_base : key_base * (num_flushes - 1);
// based on whether it is a full compaction. A full compaction only
// happens when the number of flushes equals to the number of compaction
// input runs.
uint64_t smallest_key =
(num_flushes == num_input_units) ?
key_base : key_base * (num_flushes - 1);
stats_checker->AddExpectedStats(NewManualCompactionJobStats(
Key(smallest_key, 10),
Key(smallest_key + key_base * num_input_units - key_interval, 10),
num_input_units, num_input_units > 2 ? num_input_units / 2 : 0,
num_keys_per_table * num_input_units, kKeySize, kValueSize,
num_input_units, num_keys_per_table * num_input_units, 1.0, 0, is_full,
false));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
stats_checker->AddExpectedStats(
NewManualCompactionJobStats(
Key(smallest_key, 10),
Key(smallest_key + key_base * num_input_units - key_interval, 10),
num_input_units,
num_input_units > 2 ? num_input_units / 2 : 0,
num_keys_per_table * num_input_units,
kKeySize, kValueSize,
num_input_units,
num_keys_per_table * num_input_units,
1.0, 0, false));
dbfull()->TEST_WaitForCompact();
}
ASSERT_EQ(stats_checker->NumberOfUnverifiedStats(), 3U);
@@ -1008,7 +1011,7 @@ TEST_P(CompactionJobStatsTest, UniversalCompactionTest) {
&rnd, start_key, start_key + key_base - 1,
kKeySize, kValueSize, key_interval,
compression_ratio, 1);
ASSERT_OK(static_cast_with_check<DBImpl>(db_)->TEST_WaitForCompact());
static_cast_with_check<DBImpl>(db_)->TEST_WaitForCompact();
}
ASSERT_EQ(stats_checker->NumberOfUnverifiedStats(), 0U);
}
+12 -23
View File
@@ -130,7 +130,7 @@ class CompactionJobTest : public testing::Test {
return blob_index;
}
void AddMockFile(const mock::KVVector& contents, int level = 0) {
void AddMockFile(const stl_wrappers::KVMap& contents, int level = 0) {
assert(contents.size() > 0);
bool first_key = true;
@@ -144,7 +144,7 @@ class CompactionJobTest : public testing::Test {
std::string skey;
std::string value;
std::tie(skey, value) = kv;
const Status pikStatus = ParseInternalKey(skey, &key);
bool parsed = ParseInternalKey(skey, &key);
smallest_seqno = std::min(smallest_seqno, key.sequence);
largest_seqno = std::max(largest_seqno, key.sequence);
@@ -162,7 +162,7 @@ class CompactionJobTest : public testing::Test {
first_key = false;
if (pikStatus.ok() && key.type == kTypeBlobIndex) {
if (parsed && key.type == kTypeBlobIndex) {
BlobIndex blob_index;
const Status s = blob_index.DecodeFrom(value);
if (!s.ok()) {
@@ -192,9 +192,8 @@ class CompactionJobTest : public testing::Test {
kUnknownFileChecksum, kUnknownFileChecksumFuncName);
mutex_.Lock();
EXPECT_OK(
versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),
mutable_cf_options_, &edit, &mutex_));
versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),
mutable_cf_options_, &edit, &mutex_);
mutex_.Unlock();
}
@@ -205,8 +204,8 @@ class CompactionJobTest : public testing::Test {
}
// returns expected result after compaction
mock::KVVector CreateTwoFiles(bool gen_corrupted_keys) {
stl_wrappers::KVMap expected_results;
stl_wrappers::KVMap CreateTwoFiles(bool gen_corrupted_keys) {
auto expected_results = mock::MakeMockFile();
const int kKeysPerFile = 10000;
const int kCorruptKeysPerFile = 200;
const int kMatchingKeys = kKeysPerFile / 2;
@@ -232,25 +231,19 @@ class CompactionJobTest : public testing::Test {
test::CorruptKeyType(&internal_key);
test::CorruptKeyType(&bottommost_internal_key);
}
contents.push_back({internal_key.Encode().ToString(), value});
contents.insert({ internal_key.Encode().ToString(), value });
if (i == 1 || k < kMatchingKeys || corrupt_id(k - kMatchingKeys)) {
expected_results.insert(
{bottommost_internal_key.Encode().ToString(), value});
{ bottommost_internal_key.Encode().ToString(), value });
}
}
mock::SortKVVector(&contents);
AddMockFile(contents);
}
SetLastSequence(sequence_number);
mock::KVVector expected_results_kvvector;
for (auto& kv : expected_results) {
expected_results_kvvector.push_back({kv.first, kv.second});
}
return expected_results_kvvector;
return expected_results;
}
void NewDB() {
@@ -291,8 +284,6 @@ class CompactionJobTest : public testing::Test {
// Make "CURRENT" file that points to the new manifest file.
s = SetCurrentFile(fs_.get(), dbname_, 1, nullptr);
ASSERT_OK(s);
std::vector<ColumnFamilyDescriptor> column_families;
cf_options_.table_factory = mock_table_factory_;
cf_options_.merge_operator = merge_op_;
@@ -305,7 +296,7 @@ class CompactionJobTest : public testing::Test {
void RunCompaction(
const std::vector<std::vector<FileMetaData*>>& input_files,
const mock::KVVector& expected_results,
const stl_wrappers::KVMap& expected_results,
const std::vector<SequenceNumber>& snapshots = {},
SequenceNumber earliest_write_conflict_snapshot = kMaxSequenceNumber,
int output_level = 1, bool verify = true,
@@ -351,10 +342,8 @@ class CompactionJobTest : public testing::Test {
Status s;
s = compaction_job.Run();
ASSERT_OK(s);
ASSERT_OK(compaction_job.io_status());
mutex_.Lock();
ASSERT_OK(compaction_job.Install(*cfd->GetLatestMutableCFOptions()));
ASSERT_OK(compaction_job.io_status());
mutex_.Unlock();
if (verify) {
@@ -650,7 +639,7 @@ TEST_F(CompactionJobTest, FilterAllMergeOperands) {
SetLastSequence(11U);
auto files = cfd_->current()->storage_info()->LevelFiles(0);
mock::KVVector empty_map;
stl_wrappers::KVMap empty_map;
RunCompaction({files}, empty_map);
}
+11 -3
View File
@@ -461,6 +461,7 @@ Compaction* UniversalCompactionBuilder::PickCompaction() {
// validate that all the chosen files of L0 are non overlapping in time
#ifndef NDEBUG
SequenceNumber prev_smallest_seqno = 0U;
bool is_first = true;
size_t level_index = 0U;
@@ -470,6 +471,7 @@ Compaction* UniversalCompactionBuilder::PickCompaction() {
if (is_first) {
is_first = false;
}
prev_smallest_seqno = f->fd.smallest_seqno;
}
level_index = 1U;
}
@@ -481,7 +483,16 @@ Compaction* UniversalCompactionBuilder::PickCompaction() {
&largest_seqno);
if (is_first) {
is_first = false;
} else if (prev_smallest_seqno > 0) {
// A level is considered as the bottommost level if there are
// no files in higher levels or if files in higher levels do
// not overlap with the files being compacted. Sequence numbers
// of files in bottommost level can be set to 0 to help
// compression. As a result, the following assert may not hold
// if the prev_smallest_seqno is 0.
assert(prev_smallest_seqno > largest_seqno);
}
prev_smallest_seqno = smallest_seqno;
}
}
#endif
@@ -996,9 +1007,6 @@ Compaction* UniversalCompactionBuilder::PickCompactionToOldest(
comp_reason_print_string = "size amp";
} else {
assert(false);
comp_reason_print_string = "unknown: ";
comp_reason_print_string.append(
std::to_string(static_cast<int>(compaction_reason)));
}
char file_num_buf[256];
+30 -137
View File
@@ -56,8 +56,7 @@ class CorruptionTest : public testing::Test {
options_.wal_recovery_mode = WALRecoveryMode::kTolerateCorruptedTailRecords;
options_.env = &env_;
dbname_ = test::PerThreadDBPath("corruption_test");
Status s = DestroyDB(dbname_, options_);
EXPECT_OK(s);
DestroyDB(dbname_, options_);
db_ = nullptr;
options_.create_if_missing = true;
@@ -112,12 +111,12 @@ class CorruptionTest : public testing::Test {
for (int i = 0; i < n; i++) {
if (flush_every != 0 && i != 0 && i % flush_every == 0) {
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
dbi->TEST_FlushMemTable();
}
//if ((i % 100) == 0) fprintf(stderr, "@ %d of %d\n", i, n);
Slice key = Key(i + start, &key_space);
batch.Clear();
ASSERT_OK(batch.Put(key, Value(i + start, &value_space)));
batch.Put(key, Value(i, &value_space));
ASSERT_OK(db_->Write(WriteOptions(), &batch));
}
}
@@ -138,7 +137,6 @@ class CorruptionTest : public testing::Test {
// occurred.
Iterator* iter = db_->NewIterator(ReadOptions(false, true));
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
ASSERT_OK(iter->status());
uint64_t key;
Slice in(iter->key());
if (!ConsumeDecimalNumber(&in, &key) ||
@@ -155,7 +153,6 @@ class CorruptionTest : public testing::Test {
correct++;
}
}
iter->status().PermitUncheckedError();
delete iter;
fprintf(stderr,
@@ -275,7 +272,7 @@ TEST_F(CorruptionTest, NewFileErrorDuringWrite) {
bool failed = false;
for (int i = 0; i < num; i++) {
WriteBatch batch;
ASSERT_OK(batch.Put("a", Value(100, &value_storage)));
batch.Put("a", Value(100, &value_storage));
s = db_->Write(WriteOptions(), &batch);
if (!s.ok()) {
failed = true;
@@ -291,9 +288,9 @@ TEST_F(CorruptionTest, NewFileErrorDuringWrite) {
TEST_F(CorruptionTest, TableFile) {
Build(100);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr));
ASSERT_OK(dbi->TEST_CompactRange(1, nullptr, nullptr));
dbi->TEST_FlushMemTable();
dbi->TEST_CompactRange(0, nullptr, nullptr);
dbi->TEST_CompactRange(1, nullptr, nullptr);
Corrupt(kTableFile, 100, 1);
Check(99, 99);
@@ -314,9 +311,9 @@ TEST_F(CorruptionTest, VerifyChecksumReadahead) {
Build(10000);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr));
ASSERT_OK(dbi->TEST_CompactRange(1, nullptr, nullptr));
dbi->TEST_FlushMemTable();
dbi->TEST_CompactRange(0, nullptr, nullptr);
dbi->TEST_CompactRange(1, nullptr, nullptr);
senv.count_random_reads_ = true;
senv.random_read_counter_.Reset();
@@ -361,7 +358,7 @@ TEST_F(CorruptionTest, TableFileIndexData) {
// build 2 tables, flush at 5000
Build(10000, 5000);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
dbi->TEST_FlushMemTable();
// corrupt an index block of an entire file
Corrupt(kTableFile, -2000, 500);
@@ -408,8 +405,8 @@ TEST_F(CorruptionTest, SequenceNumberRecovery) {
TEST_F(CorruptionTest, CorruptedDescriptor) {
ASSERT_OK(db_->Put(WriteOptions(), "foo", "hello"));
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr));
dbi->TEST_FlushMemTable();
dbi->TEST_CompactRange(0, nullptr, nullptr);
Corrupt(kDescriptorFile, 0, 1000);
Status s = TryReopen();
@@ -427,9 +424,9 @@ TEST_F(CorruptionTest, CompactionInputError) {
Reopen(&options);
Build(10);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr));
ASSERT_OK(dbi->TEST_CompactRange(1, nullptr, nullptr));
dbi->TEST_FlushMemTable();
dbi->TEST_CompactRange(0, nullptr, nullptr);
dbi->TEST_CompactRange(1, nullptr, nullptr);
ASSERT_EQ(1, Property("rocksdb.num-files-at-level2"));
Corrupt(kTableFile, 100, 1);
@@ -452,12 +449,12 @@ TEST_F(CorruptionTest, CompactionInputErrorParanoid) {
// Fill levels >= 1
for (int level = 1; level < dbi->NumberLevels(); level++) {
ASSERT_OK(dbi->Put(WriteOptions(), "", "begin"));
ASSERT_OK(dbi->Put(WriteOptions(), "~", "end"));
ASSERT_OK(dbi->TEST_FlushMemTable());
dbi->Put(WriteOptions(), "", "begin");
dbi->Put(WriteOptions(), "~", "end");
dbi->TEST_FlushMemTable();
for (int comp_level = 0; comp_level < dbi->NumberLevels() - level;
++comp_level) {
ASSERT_OK(dbi->TEST_CompactRange(comp_level, nullptr, nullptr));
dbi->TEST_CompactRange(comp_level, nullptr, nullptr);
}
}
@@ -465,8 +462,8 @@ TEST_F(CorruptionTest, CompactionInputErrorParanoid) {
dbi = static_cast_with_check<DBImpl>(db_);
Build(10);
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(dbi->TEST_WaitForCompact());
dbi->TEST_FlushMemTable();
dbi->TEST_WaitForCompact();
ASSERT_EQ(1, Property("rocksdb.num-files-at-level0"));
CorruptTableFileAtLevel(0, 100, 1);
@@ -491,7 +488,7 @@ TEST_F(CorruptionTest, CompactionInputErrorParanoid) {
TEST_F(CorruptionTest, UnrelatedKeys) {
Build(10);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
dbi->TEST_FlushMemTable();
Corrupt(kTableFile, 100, 1);
ASSERT_NOK(dbi->VerifyChecksum());
@@ -500,7 +497,7 @@ TEST_F(CorruptionTest, UnrelatedKeys) {
std::string v;
ASSERT_OK(db_->Get(ReadOptions(), Key(1000, &tmp1), &v));
ASSERT_EQ(Value(1000, &tmp2).ToString(), v);
ASSERT_OK(dbi->TEST_FlushMemTable());
dbi->TEST_FlushMemTable();
ASSERT_OK(db_->Get(ReadOptions(), Key(1000, &tmp1), &v));
ASSERT_EQ(Value(1000, &tmp2).ToString(), v);
}
@@ -553,28 +550,26 @@ TEST_F(CorruptionTest, FileSystemStateCorrupted) {
if (iter == 0) { // corrupt file size
std::unique_ptr<WritableFile> file;
env_.NewWritableFile(filename, &file, EnvOptions());
ASSERT_OK(file->Append(Slice("corrupted sst")));
file->Append(Slice("corrupted sst"));
file.reset();
Status x = TryReopen(&options);
ASSERT_TRUE(x.IsCorruption());
} else { // delete the file
ASSERT_OK(env_.DeleteFile(filename));
env_.DeleteFile(filename);
Status x = TryReopen(&options);
ASSERT_TRUE(x.IsPathNotFound());
}
ASSERT_OK(DestroyDB(dbname_, options_));
DestroyDB(dbname_, options_);
}
}
static const auto& corruption_modes = {
mock::MockTableFactory::kCorruptNone, mock::MockTableFactory::kCorruptKey,
mock::MockTableFactory::kCorruptValue,
mock::MockTableFactory::kCorruptReorderKey};
static const auto& corruption_modes = {mock::MockTableFactory::kCorruptNone,
mock::MockTableFactory::kCorruptKey,
mock::MockTableFactory::kCorruptValue};
TEST_F(CorruptionTest, ParanoidFileChecksOnFlush) {
Options options;
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
Status s;
@@ -582,7 +577,6 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnFlush) {
delete db_;
db_ = nullptr;
s = DestroyDB(dbname_, options);
ASSERT_OK(s);
std::shared_ptr<mock::MockTableFactory> mock =
std::make_shared<mock::MockTableFactory>();
options.table_factory = mock;
@@ -603,7 +597,6 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnCompact) {
Options options;
options.paranoid_file_checks = true;
options.create_if_missing = true;
options.check_flush_compaction_key_order = false;
Status s;
for (const auto& mode : corruption_modes) {
delete db_;
@@ -630,7 +623,6 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnCompact) {
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
Options options;
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
for (bool do_flush : {true, false}) {
@@ -662,7 +654,6 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
Options options;
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
for (bool do_flush : {true, false}) {
@@ -697,7 +688,6 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
Options options;
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
for (bool do_flush : {true, false}) {
@@ -727,103 +717,6 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
}
}
TEST_F(CorruptionTest, LogCorruptionErrorsInCompactionIterator) {
Options options;
options.create_if_missing = true;
options.allow_data_in_errors = true;
auto mode = mock::MockTableFactory::kCorruptKey;
delete db_;
db_ = nullptr;
ASSERT_OK(DestroyDB(dbname_, options));
std::shared_ptr<mock::MockTableFactory> mock =
std::make_shared<mock::MockTableFactory>();
mock->SetCorruptionMode(mode);
options.table_factory = mock;
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
Build(100, 2);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
Status s = dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true);
ASSERT_NOK(s);
ASSERT_TRUE(s.IsCorruption());
}
TEST_F(CorruptionTest, CompactionKeyOrderCheck) {
Options options;
options.paranoid_file_checks = false;
options.create_if_missing = true;
options.check_flush_compaction_key_order = false;
delete db_;
db_ = nullptr;
ASSERT_OK(DestroyDB(dbname_, options));
std::shared_ptr<mock::MockTableFactory> mock =
std::make_shared<mock::MockTableFactory>();
options.table_factory = mock;
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
mock->SetCorruptionMode(mock::MockTableFactory::kCorruptReorderKey);
Build(100, 2);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
mock->SetCorruptionMode(mock::MockTableFactory::kCorruptNone);
ASSERT_OK(db_->SetOptions({{"check_flush_compaction_key_order", "true"}}));
ASSERT_NOK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
}
TEST_F(CorruptionTest, FlushKeyOrderCheck) {
Options options;
options.paranoid_file_checks = false;
options.create_if_missing = true;
ASSERT_OK(db_->SetOptions({{"check_flush_compaction_key_order", "true"}}));
ASSERT_OK(db_->Put(WriteOptions(), "foo1", "v1"));
ASSERT_OK(db_->Put(WriteOptions(), "foo2", "v1"));
ASSERT_OK(db_->Put(WriteOptions(), "foo3", "v1"));
ASSERT_OK(db_->Put(WriteOptions(), "foo4", "v1"));
int cnt = 0;
// Generate some out of order keys from the memtable
SyncPoint::GetInstance()->SetCallBack(
"MemTableIterator::Next:0", [&](void* arg) {
MemTableRep::Iterator* mem_iter =
static_cast<MemTableRep::Iterator*>(arg);
if (++cnt == 3) {
mem_iter->Prev();
mem_iter->Prev();
}
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Status s = static_cast_with_check<DBImpl>(db_)->TEST_FlushMemTable();
ASSERT_NOK(s);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_F(CorruptionTest, DisableKeyOrderCheck) {
Options options;
ASSERT_OK(db_->SetOptions({{"check_flush_compaction_key_order", "false"}}));
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
SyncPoint::GetInstance()->SetCallBack(
"OutputValidator::Add:order_check",
[&](void* /*arg*/) { ASSERT_TRUE(false); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(db_->Put(WriteOptions(), "foo1", "v1"));
ASSERT_OK(db_->Put(WriteOptions(), "foo3", "v1"));
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(db_->Put(WriteOptions(), "foo2", "v1"));
ASSERT_OK(db_->Put(WriteOptions(), "foo4", "v1"));
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+89 -178
View File
@@ -29,15 +29,15 @@ namespace ROCKSDB_NAMESPACE {
class DBBasicTest : public DBTestBase {
public:
DBBasicTest() : DBTestBase("/db_basic_test", /*env_do_fsync=*/false) {}
DBBasicTest() : DBTestBase("/db_basic_test", /*env_do_fsync=*/true) {}
};
TEST_F(DBBasicTest, OpenWhenOpen) {
Options options = CurrentOptions();
options.env = env_;
DB* db2 = nullptr;
Status s = DB::Open(options, dbname_, &db2);
ASSERT_NOK(s);
ROCKSDB_NAMESPACE::DB* db2 = nullptr;
ROCKSDB_NAMESPACE::Status s = DB::Open(options, dbname_, &db2);
ASSERT_EQ(Status::Code::kIOError, s.code());
ASSERT_EQ(Status::SubCode::kNone, s.subcode());
ASSERT_TRUE(strstr(s.getState(), "lock ") != nullptr);
@@ -49,13 +49,13 @@ TEST_F(DBBasicTest, UniqueSession) {
Options options = CurrentOptions();
std::string sid1, sid2, sid3, sid4;
ASSERT_OK(db_->GetDbSessionId(sid1));
db_->GetDbSessionId(sid1);
Reopen(options);
ASSERT_OK(db_->GetDbSessionId(sid2));
db_->GetDbSessionId(sid2);
ASSERT_OK(Put("foo", "v1"));
ASSERT_OK(db_->GetDbSessionId(sid4));
db_->GetDbSessionId(sid4);
Reopen(options);
ASSERT_OK(db_->GetDbSessionId(sid3));
db_->GetDbSessionId(sid3);
ASSERT_NE(sid1, sid2);
ASSERT_NE(sid1, sid3);
@@ -73,14 +73,14 @@ TEST_F(DBBasicTest, UniqueSession) {
#ifndef ROCKSDB_LITE
Close();
ASSERT_OK(ReadOnlyReopen(options));
ASSERT_OK(db_->GetDbSessionId(sid1));
db_->GetDbSessionId(sid1);
// Test uniqueness between readonly open (sid1) and regular open (sid3)
ASSERT_NE(sid1, sid3);
Close();
ASSERT_OK(ReadOnlyReopen(options));
ASSERT_OK(db_->GetDbSessionId(sid2));
db_->GetDbSessionId(sid2);
ASSERT_EQ("v1", Get("foo"));
ASSERT_OK(db_->GetDbSessionId(sid3));
db_->GetDbSessionId(sid3);
ASSERT_NE(sid1, sid2);
@@ -88,13 +88,13 @@ TEST_F(DBBasicTest, UniqueSession) {
#endif // ROCKSDB_LITE
CreateAndReopenWithCF({"goku"}, options);
ASSERT_OK(db_->GetDbSessionId(sid1));
db_->GetDbSessionId(sid1);
ASSERT_OK(Put("bar", "e1"));
ASSERT_OK(db_->GetDbSessionId(sid2));
db_->GetDbSessionId(sid2);
ASSERT_EQ("e1", Get("bar"));
ASSERT_OK(db_->GetDbSessionId(sid3));
db_->GetDbSessionId(sid3);
ReopenWithColumnFamilies({"default", "goku"}, options);
ASSERT_OK(db_->GetDbSessionId(sid4));
db_->GetDbSessionId(sid4);
ASSERT_EQ(sid1, sid2);
ASSERT_EQ(sid2, sid3);
@@ -163,7 +163,7 @@ TEST_F(DBBasicTest, ReadOnlyDBWithWriteDBIdToManifestSet) {
assert(options.env == env_);
ASSERT_OK(ReadOnlyReopen(options));
std::string db_id1;
ASSERT_OK(db_->GetDbIdentity(db_id1));
db_->GetDbIdentity(db_id1);
ASSERT_EQ("v3", Get("foo"));
ASSERT_EQ("v2", Get("bar"));
Iterator* iter = db_->NewIterator(ReadOptions());
@@ -186,7 +186,7 @@ TEST_F(DBBasicTest, ReadOnlyDBWithWriteDBIdToManifestSet) {
ASSERT_EQ("v2", Get("bar"));
ASSERT_TRUE(db_->SyncWAL().IsNotSupported());
std::string db_id2;
ASSERT_OK(db_->GetDbIdentity(db_id2));
db_->GetDbIdentity(db_id2);
ASSERT_EQ(db_id1, db_id2);
}
@@ -241,7 +241,7 @@ TEST_F(DBBasicTest, CompactedDB) {
ASSERT_OK(Put("hhh", DummyString(kFileSize / 2, 'h')));
ASSERT_OK(Put("iii", DummyString(kFileSize / 2, 'i')));
ASSERT_OK(Put("jjj", DummyString(kFileSize / 2, 'j')));
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_EQ(3, NumTableFilesAtLevel(1));
Close();
@@ -299,8 +299,8 @@ TEST_F(DBBasicTest, LevelLimitReopen) {
int i = 0;
while (NumTableFilesAtLevel(2, 1) == 0) {
ASSERT_OK(Put(1, Key(i++), value));
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
}
options.num_levels = 1;
@@ -354,8 +354,8 @@ TEST_F(DBBasicTest, EmptyFlush) {
options.disable_auto_compactions = true;
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "a", Slice()));
ASSERT_OK(SingleDelete(1, "a"));
Put(1, "a", Slice());
SingleDelete(1, "a");
ASSERT_OK(Flush(1));
ASSERT_EQ("[ ]", AllEntriesFor("a", 1));
@@ -605,16 +605,16 @@ TEST_F(DBBasicTest, Snapshot) {
options_override.skip_policy = kSkipNoSnapshot;
do {
CreateAndReopenWithCF({"pikachu"}, CurrentOptions(options_override));
ASSERT_OK(Put(0, "foo", "0v1"));
ASSERT_OK(Put(1, "foo", "1v1"));
Put(0, "foo", "0v1");
Put(1, "foo", "1v1");
const Snapshot* s1 = db_->GetSnapshot();
ASSERT_EQ(1U, GetNumSnapshots());
uint64_t time_snap1 = GetTimeOldestSnapshots();
ASSERT_GT(time_snap1, 0U);
ASSERT_EQ(GetSequenceOldestSnapshots(), s1->GetSequenceNumber());
ASSERT_OK(Put(0, "foo", "0v2"));
ASSERT_OK(Put(1, "foo", "1v2"));
Put(0, "foo", "0v2");
Put(1, "foo", "1v2");
env_->MockSleepForSeconds(1);
@@ -622,8 +622,8 @@ TEST_F(DBBasicTest, Snapshot) {
ASSERT_EQ(2U, GetNumSnapshots());
ASSERT_EQ(time_snap1, GetTimeOldestSnapshots());
ASSERT_EQ(GetSequenceOldestSnapshots(), s1->GetSequenceNumber());
ASSERT_OK(Put(0, "foo", "0v3"));
ASSERT_OK(Put(1, "foo", "1v3"));
Put(0, "foo", "0v3");
Put(1, "foo", "1v3");
{
ManagedSnapshot s3(db_);
@@ -631,8 +631,8 @@ TEST_F(DBBasicTest, Snapshot) {
ASSERT_EQ(time_snap1, GetTimeOldestSnapshots());
ASSERT_EQ(GetSequenceOldestSnapshots(), s1->GetSequenceNumber());
ASSERT_OK(Put(0, "foo", "0v4"));
ASSERT_OK(Put(1, "foo", "1v4"));
Put(0, "foo", "0v4");
Put(1, "foo", "1v4");
ASSERT_EQ("0v1", Get(0, "foo", s1));
ASSERT_EQ("1v1", Get(1, "foo", s1));
ASSERT_EQ("0v2", Get(0, "foo", s2));
@@ -698,14 +698,14 @@ TEST_P(DBBasicMultiConfigs, CompactBetweenSnapshots) {
Random rnd(301);
FillLevels("a", "z", 1);
ASSERT_OK(Put(1, "foo", "first"));
Put(1, "foo", "first");
const Snapshot* snapshot1 = db_->GetSnapshot();
ASSERT_OK(Put(1, "foo", "second"));
ASSERT_OK(Put(1, "foo", "third"));
ASSERT_OK(Put(1, "foo", "fourth"));
Put(1, "foo", "second");
Put(1, "foo", "third");
Put(1, "foo", "fourth");
const Snapshot* snapshot2 = db_->GetSnapshot();
ASSERT_OK(Put(1, "foo", "fifth"));
ASSERT_OK(Put(1, "foo", "sixth"));
Put(1, "foo", "fifth");
Put(1, "foo", "sixth");
// All entries (including duplicates) exist
// before any compaction or flush is triggered.
@@ -723,8 +723,7 @@ TEST_P(DBBasicMultiConfigs, CompactBetweenSnapshots) {
// after we release the snapshot1, only two values left
db_->ReleaseSnapshot(snapshot1);
FillLevels("a", "z", 1);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr,
nullptr));
dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr, nullptr);
// We have only one valid snapshot snapshot2. Since snapshot1 is
// not valid anymore, "first" should be removed by a compaction.
@@ -735,8 +734,7 @@ TEST_P(DBBasicMultiConfigs, CompactBetweenSnapshots) {
// after we release the snapshot2, only one value should be left
db_->ReleaseSnapshot(snapshot2);
FillLevels("a", "z", 1);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr,
nullptr));
dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr, nullptr);
ASSERT_EQ("sixth", Get(1, "foo"));
ASSERT_EQ(AllEntriesFor("foo", 1), "[ sixth ]");
}
@@ -792,18 +790,18 @@ TEST_F(DBBasicTest, CompactOnFlush) {
options.disable_auto_compactions = true;
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "foo", "v1"));
Put(1, "foo", "v1");
ASSERT_OK(Flush(1));
ASSERT_EQ(AllEntriesFor("foo", 1), "[ v1 ]");
// Write two new keys
ASSERT_OK(Put(1, "a", "begin"));
ASSERT_OK(Put(1, "z", "end"));
ASSERT_OK(Flush(1));
Put(1, "a", "begin");
Put(1, "z", "end");
Flush(1);
// Case1: Delete followed by a put
ASSERT_OK(Delete(1, "foo"));
ASSERT_OK(Put(1, "foo", "v2"));
Delete(1, "foo");
Put(1, "foo", "v2");
ASSERT_EQ(AllEntriesFor("foo", 1), "[ v2, DEL, v1 ]");
// After the current memtable is flushed, the DEL should
@@ -811,66 +809,66 @@ TEST_F(DBBasicTest, CompactOnFlush) {
ASSERT_OK(Flush(1));
ASSERT_EQ(AllEntriesFor("foo", 1), "[ v2, v1 ]");
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), handles_[1],
nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr,
nullptr);
ASSERT_EQ(AllEntriesFor("foo", 1), "[ v2 ]");
// Case 2: Delete followed by another delete
ASSERT_OK(Delete(1, "foo"));
ASSERT_OK(Delete(1, "foo"));
Delete(1, "foo");
Delete(1, "foo");
ASSERT_EQ(AllEntriesFor("foo", 1), "[ DEL, DEL, v2 ]");
ASSERT_OK(Flush(1));
ASSERT_EQ(AllEntriesFor("foo", 1), "[ DEL, v2 ]");
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), handles_[1],
nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr,
nullptr);
ASSERT_EQ(AllEntriesFor("foo", 1), "[ ]");
// Case 3: Put followed by a delete
ASSERT_OK(Put(1, "foo", "v3"));
ASSERT_OK(Delete(1, "foo"));
Put(1, "foo", "v3");
Delete(1, "foo");
ASSERT_EQ(AllEntriesFor("foo", 1), "[ DEL, v3 ]");
ASSERT_OK(Flush(1));
ASSERT_EQ(AllEntriesFor("foo", 1), "[ DEL ]");
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), handles_[1],
nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr,
nullptr);
ASSERT_EQ(AllEntriesFor("foo", 1), "[ ]");
// Case 4: Put followed by another Put
ASSERT_OK(Put(1, "foo", "v4"));
ASSERT_OK(Put(1, "foo", "v5"));
Put(1, "foo", "v4");
Put(1, "foo", "v5");
ASSERT_EQ(AllEntriesFor("foo", 1), "[ v5, v4 ]");
ASSERT_OK(Flush(1));
ASSERT_EQ(AllEntriesFor("foo", 1), "[ v5 ]");
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), handles_[1],
nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr,
nullptr);
ASSERT_EQ(AllEntriesFor("foo", 1), "[ v5 ]");
// clear database
ASSERT_OK(Delete(1, "foo"));
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), handles_[1],
nullptr, nullptr));
Delete(1, "foo");
dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr,
nullptr);
ASSERT_EQ(AllEntriesFor("foo", 1), "[ ]");
// Case 5: Put followed by snapshot followed by another Put
// Both puts should remain.
ASSERT_OK(Put(1, "foo", "v6"));
Put(1, "foo", "v6");
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(Put(1, "foo", "v7"));
Put(1, "foo", "v7");
ASSERT_OK(Flush(1));
ASSERT_EQ(AllEntriesFor("foo", 1), "[ v7, v6 ]");
db_->ReleaseSnapshot(snapshot);
// clear database
ASSERT_OK(Delete(1, "foo"));
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), handles_[1],
nullptr, nullptr));
Delete(1, "foo");
dbfull()->CompactRange(CompactRangeOptions(), handles_[1], nullptr,
nullptr);
ASSERT_EQ(AllEntriesFor("foo", 1), "[ ]");
// Case 5: snapshot followed by a put followed by another Put
// Only the last put should remain.
const Snapshot* snapshot1 = db_->GetSnapshot();
ASSERT_OK(Put(1, "foo", "v8"));
ASSERT_OK(Put(1, "foo", "v9"));
Put(1, "foo", "v8");
Put(1, "foo", "v9");
ASSERT_OK(Flush(1));
ASSERT_EQ(AllEntriesFor("foo", 1), "[ v9 ]");
db_->ReleaseSnapshot(snapshot1);
@@ -893,7 +891,7 @@ TEST_F(DBBasicTest, FlushOneColumnFamily) {
ASSERT_OK(Put(7, "popovich", "popovich"));
for (int i = 0; i < 8; ++i) {
ASSERT_OK(Flush(i));
Flush(i);
auto tables = ListTableFiles(env_, dbname_);
ASSERT_EQ(tables.size(), i + 1U);
}
@@ -1035,7 +1033,7 @@ class TestEnv : public EnvWrapper {
explicit TestLogger(TestEnv* env_ptr) : Logger() { env = env_ptr; }
~TestLogger() override {
if (!closed_) {
CloseHelper().PermitUncheckedError();
CloseHelper();
}
}
void Logv(const char* /*format*/, va_list /*ap*/) override {}
@@ -1871,95 +1869,6 @@ TEST_F(DBBasicTest, MultiGetBatchedValueSizeMultiLevelMerge) {
}
}
TEST_F(DBBasicTest, MultiGetStats) {
Options options;
options.create_if_missing = true;
options.disable_auto_compactions = true;
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
table_options.block_size = 1;
table_options.index_type =
BlockBasedTableOptions::IndexType::kTwoLevelIndexSearch;
table_options.partition_filters = true;
table_options.no_block_cache = true;
table_options.cache_index_and_filter_blocks = false;
table_options.filter_policy.reset(NewBloomFilterPolicy(10, false));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
CreateAndReopenWithCF({"pikachu"}, options);
int total_keys = 2000;
std::vector<std::string> keys_str(total_keys);
std::vector<Slice> keys(total_keys);
std::vector<PinnableSlice> values(total_keys);
std::vector<Status> s(total_keys);
ReadOptions read_opts;
Random rnd(309);
// Create Multiple SST files at multiple levels.
for (int i = 0; i < 500; ++i) {
keys_str[i] = "k" + std::to_string(i);
keys[i] = Slice(keys_str[i]);
ASSERT_OK(Put(1, "k" + std::to_string(i), rnd.RandomString(1000)));
if (i % 100 == 0) {
Flush(1);
}
}
Flush(1);
MoveFilesToLevel(2, 1);
for (int i = 501; i < 1000; ++i) {
keys_str[i] = "k" + std::to_string(i);
keys[i] = Slice(keys_str[i]);
ASSERT_OK(Put(1, "k" + std::to_string(i), rnd.RandomString(1000)));
if (i % 100 == 0) {
Flush(1);
}
}
Flush(1);
MoveFilesToLevel(2, 1);
for (int i = 1001; i < total_keys; ++i) {
keys_str[i] = "k" + std::to_string(i);
keys[i] = Slice(keys_str[i]);
ASSERT_OK(Put(1, "k" + std::to_string(i), rnd.RandomString(1000)));
if (i % 100 == 0) {
Flush(1);
}
}
Flush(1);
Close();
ReopenWithColumnFamilies({"default", "pikachu"}, options);
ASSERT_OK(options.statistics->Reset());
db_->MultiGet(read_opts, handles_[1], total_keys, keys.data(), values.data(),
s.data(), false);
ASSERT_EQ(values.size(), total_keys);
HistogramData hist_data_blocks;
HistogramData hist_index_and_filter_blocks;
HistogramData hist_sst;
options.statistics->histogramData(NUM_DATA_BLOCKS_READ_PER_LEVEL,
&hist_data_blocks);
options.statistics->histogramData(NUM_INDEX_AND_FILTER_BLOCKS_READ_PER_LEVEL,
&hist_index_and_filter_blocks);
options.statistics->histogramData(NUM_SST_READ_PER_LEVEL, &hist_sst);
// Maximum number of blocks read from a file system in a level.
ASSERT_GT(hist_data_blocks.max, 0);
ASSERT_GT(hist_index_and_filter_blocks.max, 0);
// Maximum number of sst files read from file system in a level.
ASSERT_GT(hist_sst.max, 0);
// Minimun number of blocks read in a level.
ASSERT_EQ(hist_data_blocks.min, 0);
ASSERT_GT(hist_index_and_filter_blocks.min, 0);
// Minimun number of sst files read in a level.
ASSERT_GT(hist_sst.max, 0);
}
// Test class for batched MultiGet with prefix extractor
// Param bool - If true, use partitioned filters
// If false, use full filter block
@@ -2322,14 +2231,12 @@ TEST_F(DBBasicTest, RecoverWithMissingFiles) {
std::unique_ptr<Iterator> iter(db_->NewIterator(read_opts, handles_[0]));
iter->SeekToFirst();
ASSERT_FALSE(iter->Valid());
ASSERT_OK(iter->status());
iter.reset(db_->NewIterator(read_opts, handles_[1]));
iter->SeekToFirst();
ASSERT_TRUE(iter->Valid());
ASSERT_EQ("a", iter->key());
iter->Next();
ASSERT_FALSE(iter->Valid());
ASSERT_OK(iter->status());
iter.reset(db_->NewIterator(read_opts, handles_[2]));
iter->SeekToFirst();
ASSERT_TRUE(iter->Valid());
@@ -2339,7 +2246,6 @@ TEST_F(DBBasicTest, RecoverWithMissingFiles) {
ASSERT_EQ("b", iter->key());
iter->Next();
ASSERT_FALSE(iter->Valid());
ASSERT_OK(iter->status());
}
}
@@ -2452,14 +2358,12 @@ TEST_F(DBBasicTest, SkipWALIfMissingTableFiles) {
std::unique_ptr<Iterator> iter(db_->NewIterator(read_opts, handles_[0]));
iter->SeekToFirst();
ASSERT_FALSE(iter->Valid());
ASSERT_OK(iter->status());
iter.reset(db_->NewIterator(read_opts, handles_[1]));
iter->SeekToFirst();
ASSERT_TRUE(iter->Valid());
ASSERT_EQ("a", iter->key());
iter->Next();
ASSERT_FALSE(iter->Valid());
ASSERT_OK(iter->status());
}
#endif // !ROCKSDB_LITE
@@ -2495,7 +2399,7 @@ class DBBasicTestMultiGet : public DBTestBase {
DBBasicTestMultiGet(std::string test_dir, int num_cfs, bool compressed_cache,
bool uncompressed_cache, bool _compression_enabled,
bool _fill_cache, uint32_t compression_parallel_threads)
: DBTestBase(test_dir, /*env_do_fsync=*/false) {
: DBTestBase(test_dir, /*env_do_fsync=*/true) {
compression_enabled_ = _compression_enabled;
fill_cache_ = _fill_cache;
@@ -2989,8 +2893,9 @@ class DeadlineFS : public FileSystemWrapper {
std::unique_ptr<FSRandomAccessFile>* result,
IODebugContext* dbg) override {
std::unique_ptr<FSRandomAccessFile> file;
IOStatus s = target()->NewRandomAccessFile(fname, opts, &file, dbg);
EXPECT_OK(s);
IOStatus s;
s = target()->NewRandomAccessFile(fname, opts, &file, dbg);
result->reset(new DeadlineRandomAccessFile(*this, file));
const std::chrono::microseconds deadline = GetDeadline();
@@ -3353,15 +3258,18 @@ TEST_P(DBBasicTestDeadline, PointLookupDeadline) {
SetTimeElapseOnlySleepOnReopen(&options);
Reopen(options);
if (options.table_factory) {
block_cache = options.table_factory->GetOptions<Cache>(
TableFactory::kBlockCacheOpts());
if (options.table_factory &&
!strcmp(options.table_factory->Name(),
BlockBasedTableFactory::kName.c_str())) {
BlockBasedTableFactory* bbtf =
static_cast<BlockBasedTableFactory*>(options.table_factory.get());
block_cache = bbtf->table_options().block_cache.get();
}
Random rnd(301);
for (int i = 0; i < 400; ++i) {
std::string key = "k" + ToString(i);
ASSERT_OK(Put(key, rnd.RandomString(100)));
Put(key, rnd.RandomString(100));
}
Flush();
@@ -3436,17 +3344,20 @@ TEST_P(DBBasicTestDeadline, IteratorDeadline) {
SetTimeElapseOnlySleepOnReopen(&options);
Reopen(options);
if (options.table_factory) {
block_cache = options.table_factory->GetOptions<Cache>(
TableFactory::kBlockCacheOpts());
if (options.table_factory &&
!strcmp(options.table_factory->Name(),
BlockBasedTableFactory::kName.c_str())) {
BlockBasedTableFactory* bbtf =
static_cast<BlockBasedTableFactory*>(options.table_factory.get());
block_cache = bbtf->table_options().block_cache.get();
}
Random rnd(301);
for (int i = 0; i < 400; ++i) {
std::string key = "k" + ToString(i);
ASSERT_OK(Put(key, rnd.RandomString(100)));
Put(key, rnd.RandomString(100));
}
ASSERT_OK(Flush());
Flush();
bool timedout = true;
// A timeout will be forced when the IO counter reaches this value
+11 -168
View File
@@ -50,7 +50,7 @@ class DBBlockCacheTest : public DBTestBase {
options.avoid_flush_during_recovery = false;
// options.compression = kNoCompression;
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
return options;
}
@@ -158,7 +158,7 @@ TEST_F(DBBlockCacheTest, IteratorBlockCacheUsage) {
std::shared_ptr<Cache> cache = NewLRUCache(0, 0, false);
table_options.block_cache = cache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
Reopen(options);
RecordCacheCounters(options);
@@ -182,7 +182,7 @@ TEST_F(DBBlockCacheTest, TestWithoutCompressedBlockCache) {
std::shared_ptr<Cache> cache = NewLRUCache(0, 0, false);
table_options.block_cache = cache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
Reopen(options);
RecordCacheCounters(options);
@@ -238,7 +238,7 @@ TEST_F(DBBlockCacheTest, TestWithCompressedBlockCache) {
std::shared_ptr<Cache> compressed_cache = NewLRUCache(1 << 25, 0, false);
table_options.block_cache = cache;
table_options.block_cache_compressed = compressed_cache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
Reopen(options);
RecordCacheCounters(options);
@@ -299,7 +299,7 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksOfNewTableAddedToCache) {
BlockBasedTableOptions table_options;
table_options.cache_index_and_filter_blocks = true;
table_options.filter_policy.reset(NewBloomFilterPolicy(20));
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "key", "val"));
@@ -355,7 +355,7 @@ TEST_F(DBBlockCacheTest, FillCacheAndIterateDB) {
std::shared_ptr<Cache> cache = NewLRUCache(10, 0, true);
table_options.block_cache = cache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
Reopen(options);
ASSERT_OK(Put("key1", "val1"));
ASSERT_OK(Put("key2", "val2"));
@@ -393,7 +393,7 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksStats) {
std::shared_ptr<Cache> cache = NewLRUCache(co);
table_options.block_cache = cache;
table_options.filter_policy.reset(NewBloomFilterPolicy(20, true));
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "longer_key", "val"));
@@ -474,7 +474,7 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksCachePriority) {
table_options.filter_policy.reset(NewBloomFilterPolicy(20));
table_options.cache_index_and_filter_blocks_with_high_priority =
priority == Cache::Priority::HIGH ? true : false;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
DestroyAndReopen(options);
MockCache::high_pri_insert_count = 0;
@@ -573,7 +573,7 @@ TEST_F(DBBlockCacheTest, AddRedundantStats) {
table_options.cache_index_and_filter_blocks = true;
table_options.block_cache = cache;
table_options.filter_policy.reset(NewBloomFilterPolicy(50));
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
DestroyAndReopen(options);
// Create a new table.
@@ -662,7 +662,7 @@ TEST_F(DBBlockCacheTest, ParanoidFileChecks) {
BlockBasedTableOptions table_options;
table_options.cache_index_and_filter_blocks = false;
table_options.filter_policy.reset(NewBloomFilterPolicy(20));
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "1_key", "val"));
@@ -846,7 +846,7 @@ TEST_F(DBBlockCacheTest, CacheCompressionDict) {
BlockBasedTableOptions table_options;
table_options.cache_index_and_filter_blocks = true;
table_options.block_cache.reset(new MockCache());
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
DestroyAndReopen(options);
RecordCacheCountersForCompressionDict(options);
@@ -888,163 +888,6 @@ TEST_F(DBBlockCacheTest, CacheCompressionDict) {
#endif // ROCKSDB_LITE
class DBBlockCachePinningTest
: public DBTestBase,
public testing::WithParamInterface<
std::tuple<bool, PinningTier, PinningTier, PinningTier>> {
public:
DBBlockCachePinningTest()
: DBTestBase("/db_block_cache_test", /*env_do_fsync=*/false) {}
void SetUp() override {
partition_index_and_filters_ = std::get<0>(GetParam());
top_level_index_pinning_ = std::get<1>(GetParam());
partition_pinning_ = std::get<2>(GetParam());
unpartitioned_pinning_ = std::get<3>(GetParam());
}
bool partition_index_and_filters_;
PinningTier top_level_index_pinning_;
PinningTier partition_pinning_;
PinningTier unpartitioned_pinning_;
};
TEST_P(DBBlockCachePinningTest, TwoLevelDB) {
// Creates one file in L0 and one file in L1. Both files have enough data that
// their index and filter blocks are partitioned. The L1 file will also have
// a compression dictionary (those are trained only during compaction), which
// must be unpartitioned.
const int kKeySize = 32;
const int kBlockSize = 128;
const int kNumBlocksPerFile = 128;
const int kNumKeysPerFile = kBlockSize * kNumBlocksPerFile / kKeySize;
Options options = CurrentOptions();
// `kNoCompression` makes the unit test more portable. But it relies on the
// current behavior of persisting/accessing dictionary even when there's no
// (de)compression happening, which seems fairly likely to change over time.
options.compression = kNoCompression;
options.compression_opts.max_dict_bytes = 4 << 10;
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
table_options.block_cache = NewLRUCache(1 << 20 /* capacity */);
table_options.block_size = kBlockSize;
table_options.metadata_block_size = kBlockSize;
table_options.cache_index_and_filter_blocks = true;
table_options.metadata_cache_options.top_level_index_pinning =
top_level_index_pinning_;
table_options.metadata_cache_options.partition_pinning = partition_pinning_;
table_options.metadata_cache_options.unpartitioned_pinning =
unpartitioned_pinning_;
table_options.filter_policy.reset(
NewBloomFilterPolicy(10 /* bits_per_key */));
if (partition_index_and_filters_) {
table_options.index_type =
BlockBasedTableOptions::IndexType::kTwoLevelIndexSearch;
table_options.partition_filters = true;
}
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
Reopen(options);
Random rnd(301);
for (int i = 0; i < 2; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kKeySize)));
}
ASSERT_OK(Flush());
if (i == 0) {
// Prevent trivial move so file will be rewritten with dictionary and
// reopened with L1's pinning settings.
CompactRangeOptions cro;
cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
}
}
// Clear all unpinned blocks so unpinned blocks will show up as cache misses
// when reading a key from a file.
table_options.block_cache->EraseUnRefEntries();
// Get base cache values
uint64_t filter_misses = TestGetTickerCount(options, BLOCK_CACHE_FILTER_MISS);
uint64_t index_misses = TestGetTickerCount(options, BLOCK_CACHE_INDEX_MISS);
uint64_t compression_dict_misses =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS);
// Read a key from the L0 file
Get(Key(kNumKeysPerFile));
uint64_t expected_filter_misses = filter_misses;
uint64_t expected_index_misses = index_misses;
uint64_t expected_compression_dict_misses = compression_dict_misses;
if (partition_index_and_filters_) {
if (top_level_index_pinning_ == PinningTier::kNone) {
++expected_filter_misses;
++expected_index_misses;
}
if (partition_pinning_ == PinningTier::kNone) {
++expected_filter_misses;
++expected_index_misses;
}
} else {
if (unpartitioned_pinning_ == PinningTier::kNone) {
++expected_filter_misses;
++expected_index_misses;
}
}
ASSERT_EQ(expected_filter_misses,
TestGetTickerCount(options, BLOCK_CACHE_FILTER_MISS));
ASSERT_EQ(expected_index_misses,
TestGetTickerCount(options, BLOCK_CACHE_INDEX_MISS));
ASSERT_EQ(expected_compression_dict_misses,
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS));
// Clear all unpinned blocks so unpinned blocks will show up as cache misses
// when reading a key from a file.
table_options.block_cache->EraseUnRefEntries();
// Read a key from the L1 file
Get(Key(0));
if (partition_index_and_filters_) {
if (top_level_index_pinning_ == PinningTier::kNone ||
top_level_index_pinning_ == PinningTier::kFlushedAndSimilar) {
++expected_filter_misses;
++expected_index_misses;
}
if (partition_pinning_ == PinningTier::kNone ||
partition_pinning_ == PinningTier::kFlushedAndSimilar) {
++expected_filter_misses;
++expected_index_misses;
}
} else {
if (unpartitioned_pinning_ == PinningTier::kNone ||
unpartitioned_pinning_ == PinningTier::kFlushedAndSimilar) {
++expected_filter_misses;
++expected_index_misses;
}
}
if (unpartitioned_pinning_ == PinningTier::kNone ||
unpartitioned_pinning_ == PinningTier::kFlushedAndSimilar) {
++expected_compression_dict_misses;
}
ASSERT_EQ(expected_filter_misses,
TestGetTickerCount(options, BLOCK_CACHE_FILTER_MISS));
ASSERT_EQ(expected_index_misses,
TestGetTickerCount(options, BLOCK_CACHE_INDEX_MISS));
ASSERT_EQ(expected_compression_dict_misses,
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS));
}
INSTANTIATE_TEST_CASE_P(
DBBlockCachePinningTest, DBBlockCachePinningTest,
::testing::Combine(
::testing::Bool(),
::testing::Values(PinningTier::kNone, PinningTier::kFlushedAndSimilar,
PinningTier::kAll),
::testing::Values(PinningTier::kNone, PinningTier::kFlushedAndSimilar,
PinningTier::kAll),
::testing::Values(PinningTier::kNone, PinningTier::kFlushedAndSimilar,
PinningTier::kAll)));
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+7 -10
View File
@@ -83,16 +83,13 @@ TEST_P(DBBloomFilterTestDefFormatVersion, KeyMayExist) {
options_override.partition_filters = partition_filters_;
options_override.metadata_block_size = 32;
Options options = CurrentOptions(options_override);
if (partition_filters_) {
auto* table_options =
options.table_factory->GetOptions<BlockBasedTableOptions>();
if (table_options != nullptr &&
table_options->index_type !=
BlockBasedTableOptions::kTwoLevelIndexSearch) {
// In the current implementation partitioned filters depend on
// partitioned indexes
continue;
}
if (partition_filters_ &&
static_cast<BlockBasedTableOptions*>(
options.table_factory->GetOptions())
->index_type != BlockBasedTableOptions::kTwoLevelIndexSearch) {
// In the current implementation partitioned filters depend on partitioned
// indexes
continue;
}
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
CreateAndReopenWithCF({"pikachu"}, options);
+3 -3
View File
@@ -314,7 +314,7 @@ TEST_F(DBTestCompactionFilter, CompactionFilter) {
ASSERT_OK(iter->status());
while (iter->Valid()) {
ParsedInternalKey ikey(Slice(), 0, kTypeValue);
ASSERT_OK(ParseInternalKey(iter->key(), &ikey));
ASSERT_EQ(ParseInternalKey(iter->key(), &ikey), true);
total++;
if (ikey.sequence != 0) {
count++;
@@ -405,7 +405,7 @@ TEST_F(DBTestCompactionFilter, CompactionFilter) {
ASSERT_OK(iter->status());
while (iter->Valid()) {
ParsedInternalKey ikey(Slice(), 0, kTypeValue);
ASSERT_OK(ParseInternalKey(iter->key(), &ikey));
ASSERT_EQ(ParseInternalKey(iter->key(), &ikey), true);
ASSERT_NE(ikey.sequence, (unsigned)0);
count++;
iter->Next();
@@ -624,7 +624,7 @@ TEST_F(DBTestCompactionFilter, CompactionFilterContextManual) {
ASSERT_OK(iter->status());
while (iter->Valid()) {
ParsedInternalKey ikey(Slice(), 0, kTypeValue);
ASSERT_OK(ParseInternalKey(iter->key(), &ikey));
ASSERT_EQ(ParseInternalKey(iter->key(), &ikey), true);
total++;
if (ikey.sequence != 0) {
count++;
+9 -168
View File
@@ -48,18 +48,6 @@ class DBCompactionTestWithParam
bool exclusive_manual_compaction_;
};
class DBCompactionTestWithBottommostParam
: public DBTestBase,
public testing::WithParamInterface<BottommostLevelCompaction> {
public:
DBCompactionTestWithBottommostParam()
: DBTestBase("/db_compaction_test", /*env_do_fsync=*/true) {
bottommost_level_compaction_ = GetParam();
}
BottommostLevelCompaction bottommost_level_compaction_;
};
class DBCompactionDirectIOTest : public DBCompactionTest,
public ::testing::WithParamInterface<bool> {
public:
@@ -3278,13 +3266,8 @@ TEST_P(DBCompactionTestWithParam, IntraL0Compaction) {
Random rnd(301);
std::string value(rnd.RandomString(kValueSize));
// The L0->L1 must be picked before we begin flushing files to trigger
// intra-L0 compaction, and must not finish until after an intra-L0
// compaction has been picked.
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"LevelCompactionPicker::PickCompaction:Return",
"DBCompactionTest::IntraL0Compaction:L0ToL1Ready"},
{"LevelCompactionPicker::PickCompactionBySize:0",
{{"LevelCompactionPicker::PickCompactionBySize:0",
"CompactionJob::Run():Start"}});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
@@ -3302,7 +3285,6 @@ TEST_P(DBCompactionTestWithParam, IntraL0Compaction) {
for (int i = 0; i < 10; ++i) {
ASSERT_OK(Put(Key(0), "")); // prevents trivial move
if (i == 5) {
TEST_SYNC_POINT("DBCompactionTest::IntraL0Compaction:L0ToL1Ready");
ASSERT_OK(Put(Key(i + 1), value + value));
} else {
ASSERT_OK(Put(Key(i + 1), value));
@@ -3348,14 +3330,8 @@ TEST_P(DBCompactionTestWithParam, IntraL0CompactionDoesNotObsoleteDeletions) {
Random rnd(301);
std::string value(rnd.RandomString(kValueSize));
// The L0->L1 must be picked before we begin flushing files to trigger
// intra-L0 compaction, and must not finish until after an intra-L0
// compaction has been picked.
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"LevelCompactionPicker::PickCompaction:Return",
"DBCompactionTest::IntraL0CompactionDoesNotObsoleteDeletions:"
"L0ToL1Ready"},
{"LevelCompactionPicker::PickCompactionBySize:0",
{{"LevelCompactionPicker::PickCompactionBySize:0",
"CompactionJob::Run():Start"}});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
@@ -3379,11 +3355,6 @@ TEST_P(DBCompactionTestWithParam, IntraL0CompactionDoesNotObsoleteDeletions) {
} else {
ASSERT_OK(Delete(Key(0)));
}
if (i == 5) {
TEST_SYNC_POINT(
"DBCompactionTest::IntraL0CompactionDoesNotObsoleteDeletions:"
"L0ToL1Ready");
}
ASSERT_OK(Put(Key(i + 1), value));
ASSERT_OK(Flush());
}
@@ -5316,14 +5287,8 @@ TEST_P(DBCompactionTestWithParam,
std::atomic<int> pick_intra_l0_count(0);
std::string value(rnd.RandomString(kValueSize));
// The L0->L1 must be picked before we begin ingesting files to trigger
// intra-L0 compaction, and must not finish until after an intra-L0
// compaction has been picked.
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"LevelCompactionPicker::PickCompaction:Return",
"DBCompactionTestWithParam::"
"FlushAfterIntraL0CompactionCheckConsistencyFail:L0ToL1Ready"},
{"LevelCompactionPicker::PickCompactionBySize:0",
{{"DBCompactionTestWithParam::FlushAfterIntraL0:1",
"CompactionJob::Run():Start"}});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"FindIntraL0Compaction",
@@ -5351,16 +5316,14 @@ TEST_P(DBCompactionTestWithParam,
ASSERT_OK(Put(Key(0), "a"));
ASSERT_EQ(5, NumTableFilesAtLevel(0));
TEST_SYNC_POINT(
"DBCompactionTestWithParam::"
"FlushAfterIntraL0CompactionCheckConsistencyFail:L0ToL1Ready");
// Ingest 5 L0 sst. And this files would trigger PickIntraL0Compaction.
for (int i = 5; i < 10; i++) {
ASSERT_EQ(i, NumTableFilesAtLevel(0));
IngestOneKeyValue(dbfull(), Key(i), value, options);
ASSERT_EQ(i + 1, NumTableFilesAtLevel(0));
}
TEST_SYNC_POINT("DBCompactionTestWithParam::FlushAfterIntraL0:1");
// Put one key, to make biggest log sequence number in this memtable is bigger
// than sst which would be ingested in next step.
ASSERT_OK(Put(Key(2), "b"));
@@ -5402,14 +5365,8 @@ TEST_P(DBCompactionTestWithParam,
ASSERT_EQ(0, NumTableFilesAtLevel(0));
std::atomic<int> pick_intra_l0_count(0);
// The L0->L1 must be picked before we begin ingesting files to trigger
// intra-L0 compaction, and must not finish until after an intra-L0
// compaction has been picked.
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"LevelCompactionPicker::PickCompaction:Return",
"DBCompactionTestWithParam::"
"IntraL0CompactionAfterFlushCheckConsistencyFail:L0ToL1Ready"},
{"LevelCompactionPicker::PickCompactionBySize:0",
{{"DBCompactionTestWithParam::IntraL0CompactionAfterFlush:1",
"CompactionJob::Run():Start"}});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"FindIntraL0Compaction",
@@ -5440,18 +5397,17 @@ TEST_P(DBCompactionTestWithParam,
}
ASSERT_EQ(6, NumTableFilesAtLevel(0));
TEST_SYNC_POINT(
"DBCompactionTestWithParam::"
"IntraL0CompactionAfterFlushCheckConsistencyFail:L0ToL1Ready");
// ingest file to trigger IntraL0Compaction
for (int i = 6; i < 10; ++i) {
ASSERT_EQ(i, NumTableFilesAtLevel(0));
IngestOneKeyValue(dbfull(), Key(i), value2, options);
}
ASSERT_EQ(10, NumTableFilesAtLevel(0));
// Wake up flush job
sleeping_tasks.WakeUp();
sleeping_tasks.WaitUntilDone();
TEST_SYNC_POINT("DBCompactionTestWithParam::IntraL0CompactionAfterFlush:1");
dbfull()->TEST_WaitForCompact();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
@@ -5467,47 +5423,6 @@ TEST_P(DBCompactionTestWithParam,
}
}
TEST_P(DBCompactionTestWithBottommostParam, SequenceKeysManualCompaction) {
constexpr int kSstNum = 10;
Options options = CurrentOptions();
options.disable_auto_compactions = true;
DestroyAndReopen(options);
// Generate some sst files on level 0 with sequence keys (no overlap)
for (int i = 0; i < kSstNum; i++) {
for (int j = 1; j < UCHAR_MAX; j++) {
auto key = std::string(kSstNum, '\0');
key[kSstNum - i] += static_cast<char>(j);
Put(key, std::string(i % 1000, 'A'));
}
ASSERT_OK(Flush());
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_EQ(ToString(kSstNum), FilesPerLevel(0));
auto cro = CompactRangeOptions();
cro.bottommost_level_compaction = bottommost_level_compaction_;
db_->CompactRange(cro, nullptr, nullptr);
if (bottommost_level_compaction_ == BottommostLevelCompaction::kForce ||
bottommost_level_compaction_ ==
BottommostLevelCompaction::kForceOptimized) {
// Real compaction to compact all sst files from level 0 to 1 file on level
// 1
ASSERT_EQ("0,1", FilesPerLevel(0));
} else {
// Just trivial move from level 0 -> 1
ASSERT_EQ("0," + ToString(kSstNum), FilesPerLevel(0));
}
}
INSTANTIATE_TEST_CASE_P(
DBCompactionTestWithBottommostParam, DBCompactionTestWithBottommostParam,
::testing::Values(BottommostLevelCompaction::kSkip,
BottommostLevelCompaction::kIfHaveCompactionFilter,
BottommostLevelCompaction::kForce,
BottommostLevelCompaction::kForceOptimized));
TEST_F(DBCompactionTest, UpdateLevelSubCompactionTest) {
Options options = CurrentOptions();
options.max_subcompactions = 10;
@@ -5707,6 +5622,7 @@ TEST_F(DBCompactionTest, ChangeLevelCompactRangeConflictsWithManual) {
GenerateNewFile(&rnd, &key_idx);
GenerateNewFile(&rnd, &key_idx);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ("1,1,2", FilesPerLevel(0));
// The background thread will refit L2->L1 while the
@@ -5770,81 +5686,6 @@ TEST_F(DBCompactionTest, ChangeLevelCompactRangeConflictsWithManual) {
refit_level_thread.join();
}
TEST_F(DBCompactionTest, ChangeLevelErrorPathTest) {
// This test is added to ensure that RefitLevel() error paths are clearing
// internal flags and to test that subsequent valid RefitLevel() calls
// succeeds
Options options = CurrentOptions();
options.memtable_factory.reset(
new SpecialSkipListFactory(KNumKeysByGenerateNewFile - 1));
options.level0_file_num_compaction_trigger = 2;
options.num_levels = 3;
Reopen(options);
ASSERT_EQ("", FilesPerLevel(0));
// Setup an LSM with three levels populated.
Random rnd(301);
int key_idx = 0;
GenerateNewFile(&rnd, &key_idx);
ASSERT_EQ("1", FilesPerLevel(0));
{
CompactRangeOptions cro;
cro.change_level = true;
cro.target_level = 2;
ASSERT_OK(dbfull()->CompactRange(cro, nullptr, nullptr));
}
ASSERT_EQ("0,0,2", FilesPerLevel(0));
auto start_idx = key_idx;
GenerateNewFile(&rnd, &key_idx);
GenerateNewFile(&rnd, &key_idx);
auto end_idx = key_idx - 1;
ASSERT_EQ("1,1,2", FilesPerLevel(0));
// Next two CompactRange() calls are used to test exercise error paths within
// RefitLevel() before triggering a valid RefitLevel() call
// Trigger a refit to L1 first
{
std::string begin_string = Key(start_idx);
std::string end_string = Key(end_idx);
Slice begin(begin_string);
Slice end(end_string);
CompactRangeOptions cro;
cro.change_level = true;
cro.target_level = 1;
ASSERT_OK(dbfull()->CompactRange(cro, &begin, &end));
}
ASSERT_EQ("0,3,2", FilesPerLevel(0));
// Try a refit from L2->L1 - this should fail and exercise error paths in
// RefitLevel()
{
// Select key range that matches the bottom most level (L2)
std::string begin_string = Key(0);
std::string end_string = Key(start_idx - 1);
Slice begin(begin_string);
Slice end(end_string);
CompactRangeOptions cro;
cro.change_level = true;
cro.target_level = 1;
ASSERT_NOK(dbfull()->CompactRange(cro, &begin, &end));
}
ASSERT_EQ("0,3,2", FilesPerLevel(0));
// Try a valid Refit request to ensure, the path is still working
{
CompactRangeOptions cro;
cro.change_level = true;
cro.target_level = 1;
ASSERT_OK(dbfull()->CompactRange(cro, nullptr, nullptr));
}
ASSERT_EQ("0,5", FilesPerLevel(0));
}
#endif // !defined(ROCKSDB_LITE)
} // namespace ROCKSDB_NAMESPACE
+9 -176
View File
@@ -11,7 +11,6 @@
#include "db/db_impl/db_impl.h"
#include "db/db_test_util.h"
#include "file/filename.h"
#include "port/port.h"
#include "port/stack_trace.h"
#include "test_util/sync_point.h"
@@ -63,7 +62,7 @@ TEST_F(DBFlushTest, FlushWhileWritingManifest) {
ASSERT_OK(Put("bar", "v"));
ASSERT_OK(dbfull()->Flush(no_wait));
// If the issue is hit we will wait here forever.
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
#ifndef ROCKSDB_LITE
ASSERT_EQ(2, TotalTableFiles());
#endif // ROCKSDB_LITE
@@ -88,7 +87,7 @@ TEST_F(DBFlushTest, SyncFail) {
SyncPoint::GetInstance()->EnableProcessing();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put("key", "value"));
Put("key", "value");
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(db_->DefaultColumnFamily())
->cfd();
@@ -107,8 +106,7 @@ TEST_F(DBFlushTest, SyncFail) {
TEST_SYNC_POINT("DBFlushTest::SyncFail:2");
fault_injection_env->SetFilesystemActive(true);
// Now the background job will do the flush; wait for it.
// Returns the IO error happend during flush.
ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
#ifndef ROCKSDB_LITE
ASSERT_EQ("", FilesPerLevel()); // flush failed.
#endif // ROCKSDB_LITE
@@ -127,7 +125,7 @@ TEST_F(DBFlushTest, SyncSkip) {
SyncPoint::GetInstance()->EnableProcessing();
Reopen(options);
ASSERT_OK(Put("key", "value"));
Put("key", "value");
FlushOptions flush_options;
flush_options.wait = false;
@@ -137,7 +135,7 @@ TEST_F(DBFlushTest, SyncSkip) {
TEST_SYNC_POINT("DBFlushTest::SyncSkip:2");
// Now the background job will do the flush; wait for it.
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
Destroy(options);
}
@@ -172,9 +170,9 @@ TEST_F(DBFlushTest, FlushInLowPriThreadPool) {
ASSERT_OK(Put("key", "val"));
for (int i = 0; i < 4; ++i) {
ASSERT_OK(Put("key", "val"));
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(4, num_flushes);
ASSERT_EQ(1, num_compactions);
}
@@ -307,8 +305,7 @@ TEST_F(DBFlushTest, ManualFlushFailsInReadOnlyMode) {
// mode.
fault_injection_env->SetFilesystemActive(false);
ASSERT_OK(db_->ContinueBackgroundWork());
// We ingested the error to env, so the returned status is not OK.
ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
#ifndef ROCKSDB_LITE
uint64_t num_bg_errors;
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kBackgroundErrors,
@@ -446,169 +443,6 @@ TEST_F(DBFlushTest, FireOnFlushCompletedAfterCommittedResult) {
}
#endif // !ROCKSDB_LITE
TEST_F(DBFlushTest, FlushWithBlob) {
constexpr uint64_t min_blob_size = 10;
Options options;
options.enable_blob_files = true;
options.min_blob_size = min_blob_size;
options.disable_auto_compactions = true;
Reopen(options);
constexpr char short_value[] = "short";
static_assert(sizeof(short_value) - 1 < min_blob_size,
"short_value too long");
constexpr char long_value[] = "long_value";
static_assert(sizeof(long_value) - 1 >= min_blob_size,
"long_value too short");
ASSERT_OK(Put("key1", short_value));
ASSERT_OK(Put("key2", long_value));
ASSERT_OK(Flush());
ASSERT_EQ(Get("key1"), short_value);
// TODO: enable once Get support is implemented for blobs
// ASSERT_EQ(Get("key2"), long_value);
VersionSet* const versions = dbfull()->TEST_GetVersionSet();
assert(versions);
ColumnFamilyData* const cfd = versions->GetColumnFamilySet()->GetDefault();
assert(cfd);
Version* const current = cfd->current();
assert(current);
const VersionStorageInfo* const storage_info = current->storage_info();
assert(storage_info);
const auto& l0_files = storage_info->LevelFiles(0);
ASSERT_EQ(l0_files.size(), 1);
const FileMetaData* const table_file = l0_files[0];
assert(table_file);
const auto& blob_files = storage_info->GetBlobFiles();
ASSERT_EQ(blob_files.size(), 1);
const auto& blob_file = blob_files.begin()->second;
assert(blob_file);
ASSERT_EQ(table_file->smallest.user_key(), "key1");
ASSERT_EQ(table_file->largest.user_key(), "key2");
ASSERT_EQ(table_file->fd.smallest_seqno, 1);
ASSERT_EQ(table_file->fd.largest_seqno, 2);
ASSERT_EQ(table_file->oldest_blob_file_number,
blob_file->GetBlobFileNumber());
ASSERT_EQ(blob_file->GetTotalBlobCount(), 1);
#ifndef ROCKSDB_LITE
const InternalStats* const internal_stats = cfd->internal_stats();
assert(internal_stats);
const uint64_t expected_bytes =
table_file->fd.GetFileSize() + blob_file->GetTotalBlobBytes();
const auto& compaction_stats = internal_stats->TEST_GetCompactionStats();
ASSERT_FALSE(compaction_stats.empty());
ASSERT_EQ(compaction_stats[0].bytes_written, expected_bytes);
ASSERT_EQ(compaction_stats[0].num_output_files, 2);
const uint64_t* const cf_stats_value = internal_stats->TEST_GetCFStatsValue();
ASSERT_EQ(cf_stats_value[InternalStats::BYTES_FLUSHED], expected_bytes);
#endif // ROCKSDB_LITE
}
class DBFlushTestBlobError : public DBFlushTest,
public testing::WithParamInterface<std::string> {
public:
DBFlushTestBlobError() : fault_injection_env_(env_) {}
~DBFlushTestBlobError() { Close(); }
FaultInjectionTestEnv fault_injection_env_;
};
INSTANTIATE_TEST_CASE_P(DBFlushTestBlobError, DBFlushTestBlobError,
::testing::ValuesIn(std::vector<std::string>{
"BlobFileBuilder::WriteBlobToFile:AddRecord",
"BlobFileBuilder::WriteBlobToFile:AppendFooter"}));
TEST_P(DBFlushTestBlobError, FlushError) {
Options options;
options.enable_blob_files = true;
options.disable_auto_compactions = true;
options.env = &fault_injection_env_;
Reopen(options);
ASSERT_OK(Put("key", "blob"));
SyncPoint::GetInstance()->SetCallBack(GetParam(), [this](void* /* arg */) {
fault_injection_env_.SetFilesystemActive(false, Status::IOError());
});
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable", [this](void* /* arg */) {
fault_injection_env_.SetFilesystemActive(true);
});
SyncPoint::GetInstance()->EnableProcessing();
ASSERT_NOK(Flush());
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
VersionSet* const versions = dbfull()->TEST_GetVersionSet();
assert(versions);
ColumnFamilyData* const cfd = versions->GetColumnFamilySet()->GetDefault();
assert(cfd);
Version* const current = cfd->current();
assert(current);
const VersionStorageInfo* const storage_info = current->storage_info();
assert(storage_info);
const auto& l0_files = storage_info->LevelFiles(0);
ASSERT_TRUE(l0_files.empty());
const auto& blob_files = storage_info->GetBlobFiles();
ASSERT_TRUE(blob_files.empty());
// Make sure the files generated by the failed job have been deleted
std::vector<std::string> files;
ASSERT_OK(env_->GetChildren(dbname_, &files));
for (const auto& file : files) {
uint64_t number = 0;
FileType type = kTableFile;
if (!ParseFileName(file, &number, &type)) {
continue;
}
ASSERT_NE(type, kTableFile);
ASSERT_NE(type, kBlobFile);
}
#ifndef ROCKSDB_LITE
const InternalStats* const internal_stats = cfd->internal_stats();
assert(internal_stats);
const auto& compaction_stats = internal_stats->TEST_GetCompactionStats();
ASSERT_FALSE(compaction_stats.empty());
ASSERT_EQ(compaction_stats[0].bytes_written, 0);
ASSERT_EQ(compaction_stats[0].num_output_files, 0);
const uint64_t* const cf_stats_value = internal_stats->TEST_GetCFStatsValue();
ASSERT_EQ(cf_stats_value[InternalStats::BYTES_FLUSHED], 0);
#endif // ROCKSDB_LITE
}
TEST_P(DBAtomicFlushTest, ManualAtomicFlush) {
Options options = CurrentOptions();
options.create_if_missing = true;
@@ -715,8 +549,7 @@ TEST_P(DBAtomicFlushTest, AtomicFlushRollbackSomeJobs) {
fault_injection_env->SetFilesystemActive(false);
TEST_SYNC_POINT("DBAtomicFlushTest::AtomicFlushRollbackSomeJobs:2");
for (auto* cfh : handles_) {
// Returns the IO error happend during flush.
ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable(cfh));
dbfull()->TEST_WaitForFlushMemTable(cfh);
}
for (size_t i = 0; i != num_cfs; ++i) {
auto cfh = static_cast<ColumnFamilyHandleImpl*>(handles_[i]);
+79 -145
View File
@@ -44,7 +44,6 @@
#include "db/memtable_list.h"
#include "db/merge_context.h"
#include "db/merge_helper.h"
#include "db/periodic_work_scheduler.h"
#include "db/range_tombstone_fragmenter.h"
#include "db/table_cache.h"
#include "db/table_properties_collector.h"
@@ -66,6 +65,7 @@
#include "monitoring/iostats_context_imp.h"
#include "monitoring/perf_context_imp.h"
#include "monitoring/persistent_stats_history.h"
#include "monitoring/stats_dump_scheduler.h"
#include "monitoring/thread_status_updater.h"
#include "monitoring/thread_status_util.h"
#include "options/cf_options.h"
@@ -207,7 +207,7 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
refitting_level_(false),
opened_successfully_(false),
#ifndef ROCKSDB_LITE
periodic_work_scheduler_(nullptr),
stats_dump_scheduler_(nullptr),
#endif // ROCKSDB_LITE
two_write_queues_(options.two_write_queues),
manual_wal_flush_(options.manual_wal_flush),
@@ -407,7 +407,7 @@ Status DBImpl::ResumeImpl(DBRecoverContext context) {
// during previous error handling.
if (file_deletion_disabled) {
// Always return ok
s = EnableFileDeletions(/*force=*/true);
EnableFileDeletions(/*force=*/true);
}
ROCKS_LOG_INFO(immutable_db_options_.info_log, "Successfully resumed DB");
}
@@ -446,8 +446,8 @@ void DBImpl::CancelAllBackgroundWork(bool wait) {
"Shutdown: canceling all background work");
#ifndef ROCKSDB_LITE
if (periodic_work_scheduler_ != nullptr) {
periodic_work_scheduler_->Unregister(this);
if (stats_dump_scheduler_ != nullptr) {
stats_dump_scheduler_->Unregister(this);
}
#endif // !ROCKSDB_LITE
@@ -459,17 +459,14 @@ void DBImpl::CancelAllBackgroundWork(bool wait) {
autovector<ColumnFamilyData*> cfds;
SelectColumnFamiliesForAtomicFlush(&cfds);
mutex_.Unlock();
Status s =
AtomicFlushMemTables(cfds, FlushOptions(), FlushReason::kShutDown);
s.PermitUncheckedError(); //**TODO: What to do on error?
AtomicFlushMemTables(cfds, FlushOptions(), FlushReason::kShutDown);
mutex_.Lock();
} else {
for (auto cfd : *versions_->GetColumnFamilySet()) {
if (!cfd->IsDropped() && cfd->initialized() && !cfd->mem()->IsEmpty()) {
cfd->Ref();
mutex_.Unlock();
Status s = FlushMemTable(cfd, FlushOptions(), FlushReason::kShutDown);
s.PermitUncheckedError(); //**TODO: What to do on error?
FlushMemTable(cfd, FlushOptions(), FlushReason::kShutDown);
mutex_.Lock();
cfd->UnrefAndTryDelete();
}
@@ -687,18 +684,18 @@ void DBImpl::PrintStatistics() {
}
}
void DBImpl::StartPeriodicWorkScheduler() {
void DBImpl::StartStatsDumpScheduler() {
#ifndef ROCKSDB_LITE
{
InstrumentedMutexLock l(&mutex_);
periodic_work_scheduler_ = PeriodicWorkScheduler::Default();
TEST_SYNC_POINT_CALLBACK("DBImpl::StartPeriodicWorkScheduler:Init",
&periodic_work_scheduler_);
stats_dump_scheduler_ = StatsDumpScheduler::Default();
TEST_SYNC_POINT_CALLBACK("DBImpl::StartStatsDumpScheduler:Init",
&stats_dump_scheduler_);
}
periodic_work_scheduler_->Register(
this, mutable_db_options_.stats_dump_period_sec,
mutable_db_options_.stats_persist_period_sec);
stats_dump_scheduler_->Register(this,
mutable_db_options_.stats_dump_period_sec,
mutable_db_options_.stats_persist_period_sec);
#endif // !ROCKSDB_LITE
}
@@ -747,34 +744,29 @@ void DBImpl::PersistStats() {
if (immutable_db_options_.persist_stats_to_disk) {
WriteBatch batch;
Status s = Status::OK();
if (stats_slice_initialized_) {
ROCKS_LOG_INFO(immutable_db_options_.info_log,
"Reading %" ROCKSDB_PRIszt " stats from statistics\n",
stats_slice_.size());
for (const auto& stat : stats_map) {
if (s.ok()) {
char key[100];
int length =
EncodePersistentStatsKey(now_seconds, stat.first, 100, key);
// calculate the delta from last time
if (stats_slice_.find(stat.first) != stats_slice_.end()) {
uint64_t delta = stat.second - stats_slice_[stat.first];
s = batch.Put(persist_stats_cf_handle_,
Slice(key, std::min(100, length)), ToString(delta));
}
char key[100];
int length =
EncodePersistentStatsKey(now_seconds, stat.first, 100, key);
// calculate the delta from last time
if (stats_slice_.find(stat.first) != stats_slice_.end()) {
uint64_t delta = stat.second - stats_slice_[stat.first];
batch.Put(persist_stats_cf_handle_, Slice(key, std::min(100, length)),
ToString(delta));
}
}
}
stats_slice_initialized_ = true;
std::swap(stats_slice_, stats_map);
if (s.ok()) {
WriteOptions wo;
wo.low_pri = true;
wo.no_slowdown = true;
wo.sync = false;
s = Write(wo, &batch);
}
WriteOptions wo;
wo.low_pri = true;
wo.no_slowdown = true;
wo.sync = false;
Status s = Write(wo, &batch);
if (!s.ok()) {
ROCKS_LOG_INFO(immutable_db_options_.info_log,
"Writing to persistent stats CF failed -- %s",
@@ -914,14 +906,6 @@ void DBImpl::DumpStats() {
PrintStatistics();
}
void DBImpl::FlushInfoLog() {
if (shutdown_initiated_) {
return;
}
TEST_SYNC_POINT("DBImpl::FlushInfoLog:StartRunning");
LogFlush(immutable_db_options_.info_log);
}
Status DBImpl::TablesRangeTombstoneSummary(ColumnFamilyHandle* column_family,
int max_entries_to_print,
std::string* out_str) {
@@ -985,8 +969,8 @@ Status DBImpl::SetOptions(
new_options = *cfd->GetLatestMutableCFOptions();
// Append new version to recompute compaction score.
VersionEdit dummy_edit;
s = versions_->LogAndApply(cfd, new_options, &dummy_edit, &mutex_,
directories_.GetDbDir());
versions_->LogAndApply(cfd, new_options, &dummy_edit, &mutex_,
directories_.GetDbDir());
// Trigger possible flush/compactions. This has to be before we persist
// options to file, otherwise there will be a deadlock with writer
// thread.
@@ -1036,7 +1020,7 @@ Status DBImpl::SetDBOptions(
MutableDBOptions new_options;
Status s;
Status persist_options_status = Status::OK();
Status persist_options_status;
bool wal_changed = false;
WriteContext write_context;
{
@@ -1097,12 +1081,19 @@ Status DBImpl::SetDBOptions(
mutable_db_options_.stats_dump_period_sec ||
new_options.stats_persist_period_sec !=
mutable_db_options_.stats_persist_period_sec) {
mutex_.Unlock();
periodic_work_scheduler_->Unregister(this);
periodic_work_scheduler_->Register(
this, new_options.stats_dump_period_sec,
new_options.stats_persist_period_sec);
mutex_.Lock();
if (stats_dump_scheduler_) {
mutex_.Unlock();
stats_dump_scheduler_->Unregister(this);
mutex_.Lock();
}
if (new_options.stats_dump_period_sec > 0 ||
new_options.stats_persist_period_sec > 0) {
mutex_.Unlock();
stats_dump_scheduler_->Register(this,
new_options.stats_dump_period_sec,
new_options.stats_persist_period_sec);
mutex_.Lock();
}
}
write_controller_.set_max_delayed_write_rate(
new_options.delayed_write_rate);
@@ -1134,10 +1125,6 @@ Status DBImpl::SetDBOptions(
persist_options_status = WriteOptionsFile(
false /*need_mutex_lock*/, false /*need_enter_write_thread*/);
write_thread_.ExitUnbatched(&w);
} else {
// To get here, we must have had invalid options and will not attempt to
// persist the options, which means the status is "OK/Uninitialized.
persist_options_status.PermitUncheckedError();
}
}
ROCKS_LOG_INFO(immutable_db_options_.info_log, "SetDBOptions(), inputs:");
@@ -1608,8 +1595,7 @@ Status DBImpl::GetImpl(const ReadOptions& read_options, const Slice& key,
// tracing is enabled.
InstrumentedMutexLock lock(&trace_mutex_);
if (tracer_) {
// TODO: maybe handle the tracing status?
tracer_->Get(get_impl_options.column_family, key).PermitUncheckedError();
tracer_->Get(get_impl_options.column_family, key);
}
}
@@ -1634,11 +1620,9 @@ Status DBImpl::GetImpl(const ReadOptions& read_options, const Slice& key,
// data for the snapshot, so the reader would see neither data that was be
// visible to the snapshot before compaction nor the newer data inserted
// afterwards.
if (last_seq_same_as_publish_seq_) {
snapshot = versions_->LastSequence();
} else {
snapshot = versions_->LastPublishedSequence();
}
snapshot = last_seq_same_as_publish_seq_
? versions_->LastSequence()
: versions_->LastPublishedSequence();
if (get_impl_options.callback) {
// The unprep_seqs are not published for write unprepared, so it could be
// that max_visible_seq is larger. Seek to the std::max of the two.
@@ -1826,9 +1810,6 @@ std::vector<Status> DBImpl::MultiGet(
read_options, nullptr, iter_deref_lambda, &multiget_cf_data,
&consistent_seqnum);
TEST_SYNC_POINT("DBImpl::MultiGet:AfterGetSeqNum1");
TEST_SYNC_POINT("DBImpl::MultiGet:AfterGetSeqNum2");
// Contain a list of merge operations if merge occurs.
MergeContext merge_context;
@@ -1851,14 +1832,6 @@ std::vector<Status> DBImpl::MultiGet(
size_t num_found = 0;
size_t keys_read;
uint64_t curr_value_size = 0;
GetWithTimestampReadCallback timestamp_read_callback(0);
ReadCallback* read_callback = nullptr;
if (read_options.timestamp && read_options.timestamp->size() > 0) {
timestamp_read_callback.Refresh(consistent_seqnum);
read_callback = &timestamp_read_callback;
}
for (keys_read = 0; keys_read < num_keys; ++keys_read) {
merge_context.Clear();
Status& s = stat_list[keys_read];
@@ -1879,14 +1852,12 @@ std::vector<Status> DBImpl::MultiGet(
bool done = false;
if (!skip_memtable) {
if (super_version->mem->Get(lkey, value, timestamp, &s, &merge_context,
&max_covering_tombstone_seq, read_options,
read_callback)) {
&max_covering_tombstone_seq, read_options)) {
done = true;
RecordTick(stats_, MEMTABLE_HIT);
} else if (super_version->imm->Get(lkey, value, timestamp, &s,
&merge_context,
&max_covering_tombstone_seq,
read_options, read_callback)) {
} else if (super_version->imm->Get(
lkey, value, timestamp, &s, &merge_context,
&max_covering_tombstone_seq, read_options)) {
done = true;
RecordTick(stats_, MEMTABLE_HIT);
}
@@ -1894,11 +1865,9 @@ std::vector<Status> DBImpl::MultiGet(
if (!done) {
PinnableSlice pinnable_val;
PERF_TIMER_GUARD(get_from_output_files_time);
super_version->current->Get(
read_options, lkey, &pinnable_val, timestamp, &s, &merge_context,
&max_covering_tombstone_seq, /*value_found=*/nullptr,
/*key_exists=*/nullptr,
/*seq=*/nullptr, read_callback);
super_version->current->Get(read_options, lkey, &pinnable_val, timestamp,
&s, &merge_context,
&max_covering_tombstone_seq);
value->assign(pinnable_val.data(), pinnable_val.size());
RecordTick(stats_, MEMTABLE_MISS);
}
@@ -1995,11 +1964,9 @@ bool DBImpl::MultiCFSnapshot(
// version because a flush happening in between may compact away data for
// the snapshot, but the snapshot is earlier than the data overwriting it,
// so users may see wrong results.
if (last_seq_same_as_publish_seq_) {
*snapshot = versions_->LastSequence();
} else {
*snapshot = versions_->LastPublishedSequence();
}
*snapshot = last_seq_same_as_publish_seq_
? versions_->LastSequence()
: versions_->LastPublishedSequence();
}
} else {
// If we end up with the same issue of memtable geting sealed during 2
@@ -2030,11 +1997,9 @@ bool DBImpl::MultiCFSnapshot(
// acquire the lock so we're sure to succeed
mutex_.Lock();
}
if (last_seq_same_as_publish_seq_) {
*snapshot = versions_->LastSequence();
} else {
*snapshot = versions_->LastPublishedSequence();
}
*snapshot = last_seq_same_as_publish_seq_
? versions_->LastSequence()
: versions_->LastPublishedSequence();
} else {
*snapshot = reinterpret_cast<const SnapshotImpl*>(read_options.snapshot)
->number_;
@@ -2160,19 +2125,12 @@ void DBImpl::MultiGet(const ReadOptions& read_options, const size_t num_keys,
read_options, nullptr, iter_deref_lambda, &multiget_cf_data,
&consistent_seqnum);
GetWithTimestampReadCallback timestamp_read_callback(0);
ReadCallback* read_callback = nullptr;
if (read_options.timestamp && read_options.timestamp->size() > 0) {
timestamp_read_callback.Refresh(consistent_seqnum);
read_callback = &timestamp_read_callback;
}
Status s;
auto cf_iter = multiget_cf_data.begin();
for (; cf_iter != multiget_cf_data.end(); ++cf_iter) {
s = MultiGetImpl(read_options, cf_iter->start, cf_iter->num_keys,
&sorted_keys, cf_iter->super_version, consistent_seqnum,
read_callback, nullptr);
nullptr, nullptr);
if (!s.ok()) {
break;
}
@@ -2335,16 +2293,9 @@ void DBImpl::MultiGetWithCallback(
consistent_seqnum = callback->max_visible_seq();
}
GetWithTimestampReadCallback timestamp_read_callback(0);
ReadCallback* read_callback = nullptr;
if (read_options.timestamp && read_options.timestamp->size() > 0) {
timestamp_read_callback.Refresh(consistent_seqnum);
read_callback = &timestamp_read_callback;
}
Status s = MultiGetImpl(read_options, 0, num_keys, sorted_keys,
multiget_cf_data[0].super_version, consistent_seqnum,
read_callback, nullptr);
nullptr, nullptr);
assert(s.ok() || s.IsTimedOut() || s.IsAborted());
ReturnAndCleanupSuperVersion(multiget_cf_data[0].cfd,
multiget_cf_data[0].super_version);
@@ -2529,6 +2480,7 @@ Status DBImpl::CreateColumnFamilyImpl(const ColumnFamilyOptions& cf_options,
const std::string& column_family_name,
ColumnFamilyHandle** handle) {
Status s;
Status persist_options_status;
*handle = nullptr;
DBOptions db_options =
@@ -2932,7 +2884,7 @@ const Snapshot* DBImpl::GetSnapshotForWriteConflictBoundary() {
SnapshotImpl* DBImpl::GetSnapshotImpl(bool is_write_conflict_boundary,
bool lock) {
int64_t unix_time = 0;
env_->GetCurrentTime(&unix_time).PermitUncheckedError(); // Ignore error
env_->GetCurrentTime(&unix_time); // Ignore error
SnapshotImpl* s = new SnapshotImpl;
if (lock) {
@@ -2976,11 +2928,9 @@ void DBImpl::ReleaseSnapshot(const Snapshot* s) {
snapshots_.Delete(casted_s);
uint64_t oldest_snapshot;
if (snapshots_.empty()) {
if (last_seq_same_as_publish_seq_) {
oldest_snapshot = versions_->LastSequence();
} else {
oldest_snapshot = versions_->LastPublishedSequence();
}
oldest_snapshot = last_seq_same_as_publish_seq_
? versions_->LastSequence()
: versions_->LastPublishedSequence();
} else {
oldest_snapshot = snapshots_.oldest()->number_;
}
@@ -3081,7 +3031,6 @@ FileSystem* DBImpl::GetFileSystem() const {
Status DBImpl::StartIOTrace(Env* env, const TraceOptions& trace_options,
std::unique_ptr<TraceWriter>&& trace_writer) {
assert(trace_writer != nullptr);
return io_tracer_->StartIOTrace(env, trace_options, std::move(trace_writer));
}
@@ -3182,17 +3131,17 @@ bool DBImpl::GetIntPropertyInternal(ColumnFamilyData* cfd,
}
} else {
SuperVersion* sv = nullptr;
if (is_locked) {
mutex_.Unlock();
if (!is_locked) {
sv = GetAndRefSuperVersion(cfd);
} else {
sv = cfd->GetSuperVersion();
}
sv = GetAndRefSuperVersion(cfd);
bool ret = cfd->internal_stats()->GetIntPropertyOutOfMutex(
property_info, sv->current, value);
ReturnAndCleanupSuperVersion(cfd, sv);
if (is_locked) {
mutex_.Lock();
if (!is_locked) {
ReturnAndCleanupSuperVersion(cfd, sv);
}
return ret;
@@ -3229,7 +3178,6 @@ bool DBImpl::GetAggregatedIntProperty(const Slice& property,
}
uint64_t sum = 0;
bool ret = true;
{
// Needs mutex to protect the list of column families.
InstrumentedMutexLock l(&mutex_);
@@ -3238,21 +3186,15 @@ bool DBImpl::GetAggregatedIntProperty(const Slice& property,
if (!cfd->initialized()) {
continue;
}
cfd->Ref();
ret = GetIntPropertyInternal(cfd, *property_info, true, &value);
// GetIntPropertyInternal may release db mutex and re-acquire it.
mutex_.AssertHeld();
cfd->UnrefAndTryDelete();
if (ret) {
if (GetIntPropertyInternal(cfd, *property_info, true, &value)) {
sum += value;
} else {
ret = false;
break;
return false;
}
}
}
*aggregated_value = sum;
return ret;
return true;
}
SuperVersion* DBImpl::GetAndRefSuperVersion(ColumnFamilyData* cfd) {
@@ -4555,9 +4497,7 @@ Status DBImpl::IngestExternalFiles(
// be pessimistic and try write to a new MANIFEST.
// TODO: distinguish between MANIFEST write and CURRENT renaming
const IOStatus& io_s = versions_->io_status();
// Should handle return error?
error_handler_.SetBGError(io_s, BackgroundErrorReason::kManifestWrite)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kManifestWrite);
}
// Resume writes to the DB
@@ -4712,14 +4652,8 @@ Status DBImpl::CreateColumnFamilyWithImport(
import_job.Cleanup(status);
if (!status.ok()) {
Status temp_s = DropColumnFamily(*handle);
if (!temp_s.ok()) {
ROCKS_LOG_ERROR(immutable_db_options_.info_log,
"DropColumnFamily failed with error %s",
temp_s.ToString().c_str());
}
// Always returns Status::OK()
assert(DestroyColumnFamilyHandle(*handle).ok());
DropColumnFamily(*handle);
DestroyColumnFamilyHandle(*handle);
*handle = nullptr;
}
return status;
+10 -14
View File
@@ -70,9 +70,9 @@ class ArenaWrappedDBIter;
class InMemoryStatsHistoryIterator;
class MemTable;
class PersistentStatsHistoryIterator;
class PeriodicWorkScheduler;
class StatsDumpScheduler;
#ifndef NDEBUG
class PeriodicWorkTestScheduler;
class StatsDumpTestScheduler;
#endif // !NDEBUG
class TableCache;
class TaskLimiterToken;
@@ -1002,7 +1002,7 @@ class DBImpl : public DB {
}
#ifndef ROCKSDB_LITE
PeriodicWorkTestScheduler* TEST_GetPeriodicWorkScheduler() const;
StatsDumpTestScheduler* TEST_GetStatsDumpScheduler() const;
#endif // !ROCKSDB_LITE
#endif // NDEBUG
@@ -1013,9 +1013,6 @@ class DBImpl : public DB {
// dump rocksdb.stats to LOG
void DumpStats();
// flush LOG out of application buffer
void FlushInfoLog();
protected:
const std::string dbname_;
std::string db_id_;
@@ -1189,8 +1186,8 @@ class DBImpl : public DB {
// skipped.
virtual Status Recover(
const std::vector<ColumnFamilyDescriptor>& column_families,
bool read_only = false, bool error_if_wal_file_exists = false,
bool error_if_data_exists_in_wals = false,
bool read_only = false, bool error_if_log_file_exist = false,
bool error_if_data_exists_in_logs = false,
uint64_t* recovered_seq = nullptr);
virtual bool OwnTablesAndLogs() const { return true; }
@@ -1655,7 +1652,7 @@ class DBImpl : public DB {
LogBuffer* log_buffer);
// Schedule background tasks
void StartPeriodicWorkScheduler();
void StartStatsDumpScheduler();
void PrintStatistics();
@@ -2114,11 +2111,10 @@ class DBImpl : public DB {
std::unique_ptr<PreReleaseCallback> recoverable_state_pre_release_callback_;
#ifndef ROCKSDB_LITE
// Scheduler to run DumpStats(), PersistStats(), and FlushInfoLog().
// Currently, it always use a global instance from
// PeriodicWorkScheduler::Default(). Only in unittest, it can be overrided by
// PeriodicWorkTestScheduler.
PeriodicWorkScheduler* periodic_work_scheduler_;
// Scheduler to run DumpStats() and PersistStats(). Currently, it always use
// a global instance from StatsDumpScheduler::Default(). Only in unittest, it
// can be overrided by StatsDumpTestSchduler.
StatsDumpScheduler* stats_dump_scheduler_;
#endif
// When set, we use a separate queue for writes that don't write to memtable.
+26 -113
View File
@@ -126,12 +126,10 @@ IOStatus DBImpl::SyncClosedLogs(JobContext* job_context) {
MarkLogsSynced(current_log_number - 1, true, io_s);
if (!io_s.ok()) {
if (total_log_size_ > 0) {
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush);
} else {
// If the WAL is empty, we use different error reason
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlushNoWAL)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlushNoWAL);
}
TEST_SYNC_POINT("DBImpl::SyncClosedLogs:Failed");
return io_s;
@@ -149,7 +147,6 @@ Status DBImpl::FlushMemTableToOutputFile(
SnapshotChecker* snapshot_checker, LogBuffer* log_buffer,
Env::Priority thread_pri) {
mutex_.AssertHeld();
assert(cfd);
assert(cfd->imm()->NumNotFlushed() != 0);
assert(cfd->imm()->IsFlushPending());
@@ -188,8 +185,7 @@ Status DBImpl::FlushMemTableToOutputFile(
s = io_s;
if (!io_s.ok() && !io_s.IsShutdownInProgress() &&
!io_s.IsColumnFamilyDropped()) {
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush);
}
} else {
TEST_SYNC_POINT("DBImpl::SyncClosedLogs:Skip");
@@ -216,28 +212,10 @@ Status DBImpl::FlushMemTableToOutputFile(
if (made_progress) {
*made_progress = true;
}
const std::string& column_family_name = cfd->GetName();
Version* const current = cfd->current();
assert(current);
const VersionStorageInfo* const storage_info = current->storage_info();
assert(storage_info);
VersionStorageInfo::LevelSummaryStorage tmp;
ROCKS_LOG_BUFFER(log_buffer, "[%s] Level summary: %s\n",
column_family_name.c_str(),
storage_info->LevelSummary(&tmp));
const auto& blob_files = storage_info->GetBlobFiles();
if (!blob_files.empty()) {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Blob file summary: head=%" PRIu64 ", tail=%" PRIu64
"\n",
column_family_name.c_str(), blob_files.begin()->first,
blob_files.rbegin()->first);
}
cfd->GetName().c_str(),
cfd->current()->storage_info()->LevelSummary(&tmp));
}
if (!s.ok() && !s.IsShutdownInProgress() && !s.IsColumnFamilyDropped()) {
@@ -249,24 +227,16 @@ Status DBImpl::FlushMemTableToOutputFile(
// be pessimistic and try write to a new MANIFEST.
// TODO: distinguish between MANIFEST write and CURRENT renaming
if (!versions_->io_status().ok()) {
// Should handle return error?
error_handler_.SetBGError(io_s, BackgroundErrorReason::kManifestWrite)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kManifestWrite);
} else if (total_log_size_ > 0) {
// Should handle return error?
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush);
} else {
// If the WAL is empty, we use different error reason
// Should handle return error?
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlushNoWAL)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlushNoWAL);
}
} else {
Status new_bg_error = s;
// Should handle return error?
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush)
.PermitUncheckedError();
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush);
}
} else {
// If we got here, then we decided not to care about the i_os status (either
@@ -291,9 +261,7 @@ Status DBImpl::FlushMemTableToOutputFile(
TEST_SYNC_POINT_CALLBACK(
"DBImpl::FlushMemTableToOutputFile:MaxAllowedSpaceReached",
&new_bg_error);
// Should handle this error?
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush)
.PermitUncheckedError();
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush);
}
}
#endif // ROCKSDB_LITE
@@ -589,36 +557,16 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
assert(num_cfs ==
static_cast<int>(job_context->superversion_contexts.size()));
for (int i = 0; i != num_cfs; ++i) {
assert(cfds[i]);
if (cfds[i]->IsDropped()) {
continue;
}
InstallSuperVersionAndScheduleWork(cfds[i],
&job_context->superversion_contexts[i],
all_mutable_cf_options[i]);
const std::string& column_family_name = cfds[i]->GetName();
Version* const current = cfds[i]->current();
assert(current);
const VersionStorageInfo* const storage_info = current->storage_info();
assert(storage_info);
VersionStorageInfo::LevelSummaryStorage tmp;
ROCKS_LOG_BUFFER(log_buffer, "[%s] Level summary: %s\n",
column_family_name.c_str(),
storage_info->LevelSummary(&tmp));
const auto& blob_files = storage_info->GetBlobFiles();
if (!blob_files.empty()) {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Blob file summary: head=%" PRIu64
", tail=%" PRIu64 "\n",
column_family_name.c_str(), blob_files.begin()->first,
blob_files.rbegin()->first);
}
cfds[i]->GetName().c_str(),
cfds[i]->current()->storage_info()->LevelSummary(&tmp));
}
if (made_progress) {
*made_progress = true;
@@ -641,9 +589,8 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
error_handler_.GetBGError().ok()) {
Status new_bg_error =
Status::SpaceLimit("Max allowed space was reached");
// Should Handle this error?
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush)
.PermitUncheckedError();
error_handler_.SetBGError(new_bg_error,
BackgroundErrorReason::kFlush);
}
}
}
@@ -660,24 +607,16 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
// be pessimistic and try write to a new MANIFEST.
// TODO: distinguish between MANIFEST write and CURRENT renaming
if (!versions_->io_status().ok()) {
// Should Handle this error?
error_handler_.SetBGError(io_s, BackgroundErrorReason::kManifestWrite)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kManifestWrite);
} else if (total_log_size_ > 0) {
// Should Handle this error?
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush);
} else {
// If the WAL is empty, we use different error reason
// Should Handle this error?
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlushNoWAL)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlushNoWAL);
}
} else {
Status new_bg_error = s;
// Should Handle this error?
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush)
.PermitUncheckedError();
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush);
}
}
@@ -873,7 +812,6 @@ Status DBImpl::CompactRange(const CompactRangeOptions& options,
int output_level;
for (int level = first_overlapped_level; level <= max_overlapped_level;
level++) {
bool disallow_trivial_move = false;
// in case the compaction is universal or if we're compacting the
// bottom-most level, the output level will be the same as input one.
// level 0 can never be the bottommost level (i.e. if all files are in
@@ -906,19 +844,9 @@ Status DBImpl::CompactRange(const CompactRangeOptions& options,
level == 0) {
output_level = ColumnFamilyData::kCompactToBaseLevel;
}
// if it's a BottommostLevel compaction and `kForce*` compaction is
// set, disallow trivial move
if (level == max_overlapped_level &&
(options.bottommost_level_compaction ==
BottommostLevelCompaction::kForce ||
options.bottommost_level_compaction ==
BottommostLevelCompaction::kForceOptimized)) {
disallow_trivial_move = true;
}
}
s = RunManualCompaction(cfd, level, output_level, options, begin, end,
exclusive, disallow_trivial_move,
max_file_num_to_ignore);
exclusive, false, max_file_num_to_ignore);
if (!s.ok()) {
break;
}
@@ -949,8 +877,7 @@ Status DBImpl::CompactRange(const CompactRangeOptions& options,
TEST_SYNC_POINT("DBImpl::CompactRange:PreRefitLevel");
s = ReFitLevel(cfd, final_output_level, options.target_level);
TEST_SYNC_POINT("DBImpl::CompactRange:PostRefitLevel");
// ContinueBackgroundWork always return Status::OK().
assert(ContinueBackgroundWork().ok());
ContinueBackgroundWork();
}
EnableManualCompaction();
}
@@ -1173,15 +1100,10 @@ Status DBImpl::CompactFilesImpl(
Status status = compaction_job.Install(*c->mutable_cf_options());
if (status.ok()) {
assert(compaction_job.io_status().ok());
InstallSuperVersionAndScheduleWork(c->column_family_data(),
&job_context->superversion_contexts[0],
*c->mutable_cf_options());
}
// status above captures any error during compaction_job.Install, so its ok
// not check compaction_job.io_status() explicitly if we're not calling
// SetBGError
compaction_job.io_status().PermitUncheckedError();
c->ReleaseCompactionFiles(s);
#ifndef ROCKSDB_LITE
// Need to make sure SstFileManager does its bookkeeping
@@ -1216,11 +1138,9 @@ Status DBImpl::CompactFilesImpl(
job_context->job_id, status.ToString().c_str());
IOStatus io_s = compaction_job.io_status();
if (!io_s.ok()) {
error_handler_.SetBGError(io_s, BackgroundErrorReason::kCompaction)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kCompaction);
} else {
error_handler_.SetBGError(status, BackgroundErrorReason::kCompaction)
.PermitUncheckedError();
error_handler_.SetBGError(status, BackgroundErrorReason::kCompaction);
}
}
@@ -1301,7 +1221,6 @@ void DBImpl::NotifyOnCompactionBegin(ColumnFamilyData* cfd, Compaction* c,
for (auto listener : immutable_db_options_.listeners) {
listener->OnCompactionBegin(this, info);
}
info.status.PermitUncheckedError();
}
mutex_.Lock();
current->Unref();
@@ -1388,14 +1307,12 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
if (to_level != level) {
if (to_level > level) {
if (level == 0) {
refitting_level_ = false;
return Status::NotSupported(
"Cannot change from level 0 to other levels.");
}
// Check levels are empty for a trivial move
for (int l = level + 1; l <= to_level; l++) {
if (vstorage->NumLevelFiles(l) > 0) {
refitting_level_ = false;
return Status::NotSupported(
"Levels between source and target are not empty for a move.");
}
@@ -1405,7 +1322,6 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
// Check levels are empty for a trivial move
for (int l = to_level; l < level; l++) {
if (vstorage->NumLevelFiles(l) > 0) {
refitting_level_ = false;
return Status::NotSupported(
"Levels between source and target are not empty for a move.");
}
@@ -2965,11 +2881,11 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
NotifyOnCompactionBegin(c->column_family_data(), c.get(), status,
compaction_job_stats, job_context->job_id);
mutex_.Unlock();
TEST_SYNC_POINT_CALLBACK(
"DBImpl::BackgroundCompaction:NonTrivial:BeforeRun", nullptr);
// Should handle erorr?
compaction_job.Run().PermitUncheckedError();
compaction_job.Run();
TEST_SYNC_POINT("DBImpl::BackgroundCompaction:NonTrivial:AfterRun");
mutex_.Lock();
@@ -2987,8 +2903,6 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
if (status.ok() && !io_s.ok()) {
status = io_s;
} else {
io_s.PermitUncheckedError();
}
if (c != nullptr) {
@@ -3025,10 +2939,9 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
auto err_reason = versions_->io_status().ok()
? BackgroundErrorReason::kCompaction
: BackgroundErrorReason::kManifestWrite;
error_handler_.SetBGError(io_s, err_reason).PermitUncheckedError();
error_handler_.SetBGError(io_s, err_reason);
} else {
error_handler_.SetBGError(status, BackgroundErrorReason::kCompaction)
.PermitUncheckedError();
error_handler_.SetBGError(status, BackgroundErrorReason::kCompaction);
}
if (c != nullptr && !is_manual && !error_handler_.IsBGWorkStopped()) {
// Put this cfd back in the compaction queue so we can retry after some
+5 -5
View File
@@ -12,7 +12,7 @@
#include "db/column_family.h"
#include "db/db_impl/db_impl.h"
#include "db/error_handler.h"
#include "db/periodic_work_scheduler.h"
#include "monitoring/stats_dump_scheduler.h"
#include "monitoring/thread_status_updater.h"
#include "util/cast_util.h"
@@ -274,14 +274,14 @@ size_t DBImpl::TEST_GetWalPreallocateBlockSize(
#ifndef ROCKSDB_LITE
void DBImpl::TEST_WaitForStatsDumpRun(std::function<void()> callback) const {
if (periodic_work_scheduler_ != nullptr) {
static_cast<PeriodicWorkTestScheduler*>(periodic_work_scheduler_)
if (stats_dump_scheduler_ != nullptr) {
static_cast<StatsDumpTestScheduler*>(stats_dump_scheduler_)
->TEST_WaitForRun(callback);
}
}
PeriodicWorkTestScheduler* DBImpl::TEST_GetPeriodicWorkScheduler() const {
return static_cast<PeriodicWorkTestScheduler*>(periodic_work_scheduler_);
StatsDumpTestScheduler* DBImpl::TEST_GetStatsDumpScheduler() const {
return static_cast<StatsDumpTestScheduler*>(stats_dump_scheduler_);
}
#endif // !ROCKSDB_LITE
+9 -19
View File
@@ -58,7 +58,7 @@ Status DBImpl::EnableFileDeletions(bool force) {
// Job id == 0 means that this is not our background process, but rather
// user thread
JobContext job_context(0);
int saved_counter; // initialize on all paths
bool file_deletion_enabled = false;
{
InstrumentedMutexLock l(&mutex_);
if (force) {
@@ -67,13 +67,13 @@ Status DBImpl::EnableFileDeletions(bool force) {
} else if (disable_delete_obsolete_files_ > 0) {
--disable_delete_obsolete_files_;
}
saved_counter = disable_delete_obsolete_files_;
if (saved_counter == 0) {
if (disable_delete_obsolete_files_ == 0) {
file_deletion_enabled = true;
FindObsoleteFiles(&job_context, true);
bg_cv_.SignalAll();
}
}
if (saved_counter == 0) {
if (file_deletion_enabled) {
ROCKS_LOG_INFO(immutable_db_options_.info_log, "File Deletions Enabled");
if (job_context.HaveSomethingToDelete()) {
PurgeObsoleteFiles(job_context);
@@ -81,7 +81,7 @@ Status DBImpl::EnableFileDeletions(bool force) {
} else {
ROCKS_LOG_WARN(immutable_db_options_.info_log,
"File Deletions Enable, but not really enabled. Counter: %d",
saved_counter);
disable_delete_obsolete_files_);
}
job_context.Clean();
LogFlush(immutable_db_options_.info_log);
@@ -215,8 +215,7 @@ void DBImpl::FindObsoleteFiles(JobContext* job_context, bool force,
if (immutable_db_options_.wal_dir != dbname_) {
std::vector<std::string> log_files;
env_->GetChildren(immutable_db_options_.wal_dir,
&log_files)
.PermitUncheckedError(); // Ignore errors
&log_files); // Ignore errors
for (const std::string& log_file : log_files) {
job_context->full_scan_candidate_files.emplace_back(
log_file, immutable_db_options_.wal_dir);
@@ -227,8 +226,7 @@ void DBImpl::FindObsoleteFiles(JobContext* job_context, bool force,
immutable_db_options_.db_log_dir != dbname_) {
std::vector<std::string> info_log_files;
// Ignore errors
env_->GetChildren(immutable_db_options_.db_log_dir, &info_log_files)
.PermitUncheckedError();
env_->GetChildren(immutable_db_options_.db_log_dir, &info_log_files);
for (std::string& log_file : info_log_files) {
job_context->full_scan_candidate_files.emplace_back(
log_file, immutable_db_options_.db_log_dir);
@@ -762,13 +760,9 @@ Status DBImpl::FinishBestEffortsRecovery() {
uint64_t next_file_number = versions_->current_next_file_number();
uint64_t largest_file_number = next_file_number;
std::set<std::string> files_to_delete;
Status s;
for (const auto& path : paths) {
std::vector<std::string> files;
s = env_->GetChildren(path, &files);
if (!s.ok()) {
break;
}
env_->GetChildren(path, &files);
for (const auto& fname : files) {
uint64_t number = 0;
FileType type;
@@ -784,10 +778,6 @@ Status DBImpl::FinishBestEffortsRecovery() {
}
}
}
if (!s.ok()) {
return s;
}
if (largest_file_number > next_file_number) {
versions_->next_file_number_.store(largest_file_number + 1);
}
@@ -799,7 +789,7 @@ Status DBImpl::FinishBestEffortsRecovery() {
assert(default_cfd);
// Even if new_descriptor_log is false, we will still switch to a new
// MANIFEST and update CURRENT file, since this is in recovery.
s = versions_->LogAndApply(
Status s = versions_->LogAndApply(
default_cfd, *default_cfd->GetLatestMutableCFOptions(), &edit, &mutex_,
directories_.GetDbDir(), /*new_descriptor_log*/ false);
if (!s.ok()) {
+48 -90
View File
@@ -11,15 +11,15 @@
#include "db/builder.h"
#include "db/db_impl/db_impl.h"
#include "db/error_handler.h"
#include "db/periodic_work_scheduler.h"
#include "env/composite_env_wrapper.h"
#include "file/read_write_util.h"
#include "file/sst_file_manager_impl.h"
#include "file/writable_file_writer.h"
#include "monitoring/persistent_stats_history.h"
#include "monitoring/stats_dump_scheduler.h"
#include "options/options_helper.h"
#include "rocksdb/table.h"
#include "rocksdb/wal_filter.h"
#include "table/block_based/block_based_table_factory.h"
#include "test_util/sync_point.h"
#include "util/rate_limiter.h"
@@ -185,12 +185,12 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src) {
}
namespace {
Status ValidateOptionsByTable(
Status SanitizeOptionsByTable(
const DBOptions& db_opts,
const std::vector<ColumnFamilyDescriptor>& column_families) {
Status s;
for (auto cf : column_families) {
s = ValidateOptions(db_opts, cf.options);
s = cf.options.table_factory->SanitizeOptions(db_opts, cf.options);
if (!s.ok()) {
return s;
}
@@ -364,7 +364,7 @@ IOStatus Directories::SetDirectories(FileSystem* fs, const std::string& dbname,
Status DBImpl::Recover(
const std::vector<ColumnFamilyDescriptor>& column_families, bool read_only,
bool error_if_wal_file_exists, bool error_if_data_exists_in_wals,
bool error_if_log_file_exist, bool error_if_data_exists_in_logs,
uint64_t* recovered_seq) {
mutex_.AssertHeld();
@@ -588,11 +588,11 @@ Status DBImpl::Recover(
}
if (logs.size() > 0) {
if (error_if_wal_file_exists) {
if (error_if_log_file_exist) {
return Status::Corruption(
"The db was opened in readonly mode with error_if_wal_file_exists"
"flag but a WAL file already exists");
} else if (error_if_data_exists_in_wals) {
"The db was opened in readonly mode with error_if_log_file_exist"
"flag but a log file already exists");
} else if (error_if_data_exists_in_logs) {
for (auto& log : logs) {
std::string fname = LogFileName(immutable_db_options_.wal_dir, log);
uint64_t bytes;
@@ -600,8 +600,8 @@ Status DBImpl::Recover(
if (s.ok()) {
if (bytes > 0) {
return Status::Corruption(
"error_if_data_exists_in_wals is set but there are data "
" in WAL files.");
"error_if_data_exists_in_logs is set but there are data "
" in log files.");
}
}
}
@@ -685,56 +685,41 @@ Status DBImpl::PersistentStatsProcessFormatVersion() {
(kStatsCFCurrentFormatVersion < format_version_recovered &&
kStatsCFCompatibleFormatVersion < compatible_version_recovered)) {
if (!s_format.ok() || !s_compatible.ok()) {
ROCKS_LOG_WARN(
ROCKS_LOG_INFO(
immutable_db_options_.info_log,
"Recreating persistent stats column family since reading "
"persistent stats version key failed. Format key: %s, compatible "
"key: %s",
"Reading persistent stats version key failed. Format key: %s, "
"compatible key: %s",
s_format.ToString().c_str(), s_compatible.ToString().c_str());
} else {
ROCKS_LOG_WARN(
ROCKS_LOG_INFO(
immutable_db_options_.info_log,
"Recreating persistent stats column family due to corrupted or "
"incompatible format version. Recovered format: %" PRIu64
"; recovered format compatible since: %" PRIu64 "\n",
format_version_recovered, compatible_version_recovered);
}
s = DropColumnFamily(persist_stats_cf_handle_);
if (s.ok()) {
s = DestroyColumnFamilyHandle(persist_stats_cf_handle_);
"Disable persistent stats due to corrupted or incompatible format "
"version\n");
}
DropColumnFamily(persist_stats_cf_handle_);
DestroyColumnFamilyHandle(persist_stats_cf_handle_);
ColumnFamilyHandle* handle = nullptr;
if (s.ok()) {
ColumnFamilyOptions cfo;
OptimizeForPersistentStats(&cfo);
s = CreateColumnFamily(cfo, kPersistentStatsColumnFamilyName, &handle);
}
if (s.ok()) {
persist_stats_cf_handle_ = static_cast<ColumnFamilyHandleImpl*>(handle);
// should also persist version here because old stats CF is discarded
should_persist_format_version = true;
}
ColumnFamilyOptions cfo;
OptimizeForPersistentStats(&cfo);
s = CreateColumnFamily(cfo, kPersistentStatsColumnFamilyName, &handle);
persist_stats_cf_handle_ = static_cast<ColumnFamilyHandleImpl*>(handle);
// should also persist version here because old stats CF is discarded
should_persist_format_version = true;
}
}
if (should_persist_format_version) {
if (s.ok() && should_persist_format_version) {
// Persistent stats CF being created for the first time, need to write
// format version key
WriteBatch batch;
if (s.ok()) {
s = batch.Put(persist_stats_cf_handle_, kFormatVersionKeyString,
ToString(kStatsCFCurrentFormatVersion));
}
if (s.ok()) {
s = batch.Put(persist_stats_cf_handle_, kCompatibleVersionKeyString,
ToString(kStatsCFCompatibleFormatVersion));
}
if (s.ok()) {
WriteOptions wo;
wo.low_pri = true;
wo.no_slowdown = true;
wo.sync = false;
s = Write(wo, &batch);
}
batch.Put(persist_stats_cf_handle_, kFormatVersionKeyString,
ToString(kStatsCFCurrentFormatVersion));
batch.Put(persist_stats_cf_handle_, kCompatibleVersionKeyString,
ToString(kStatsCFCompatibleFormatVersion));
WriteOptions wo;
wo.low_pri = true;
wo.no_slowdown = true;
wo.sync = false;
s = Write(wo, &batch);
}
mutex_.Lock();
return s;
@@ -1287,10 +1272,7 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
MemTable* mem, VersionEdit* edit) {
mutex_.AssertHeld();
const uint64_t start_micros = env_->NowMicros();
FileMetaData meta;
std::vector<BlobFileAddition> blob_file_additions;
std::unique_ptr<std::list<uint64_t>::iterator> pending_outputs_inserted_elem(
new std::list<uint64_t>::iterator(
CaptureCurrentFileNumberInPendingOutputs()));
@@ -1337,15 +1319,13 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
if (range_del_iter != nullptr) {
range_del_iters.emplace_back(range_del_iter);
}
IOStatus io_s;
s = BuildTable(
dbname_, versions_.get(), env_, fs_.get(), *cfd->ioptions(),
mutable_cf_options, file_options_for_compaction_, cfd->table_cache(),
iter.get(), std::move(range_del_iters), &meta, &blob_file_additions,
cfd->internal_comparator(), cfd->int_tbl_prop_collector_factories(),
cfd->GetID(), cfd->GetName(), snapshot_seqs,
earliest_write_conflict_snapshot, snapshot_checker,
dbname_, env_, fs_.get(), *cfd->ioptions(), mutable_cf_options,
file_options_for_compaction_, cfd->table_cache(), iter.get(),
std::move(range_del_iters), &meta, cfd->internal_comparator(),
cfd->int_tbl_prop_collector_factories(), cfd->GetID(), cfd->GetName(),
snapshot_seqs, earliest_write_conflict_snapshot, snapshot_checker,
GetCompressionFlush(*cfd->ioptions(), mutable_cf_options),
mutable_cf_options.sample_for_compression,
mutable_cf_options.compression_opts, paranoid_file_checks,
@@ -1367,39 +1347,23 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
// Note that if file_size is zero, the file has been deleted and
// should not be added to the manifest.
const bool has_output = meta.fd.GetFileSize() > 0;
assert(has_output || blob_file_additions.empty());
constexpr int level = 0;
if (s.ok() && has_output) {
int level = 0;
if (s.ok() && meta.fd.GetFileSize() > 0) {
edit->AddFile(level, meta.fd.GetNumber(), meta.fd.GetPathId(),
meta.fd.GetFileSize(), meta.smallest, meta.largest,
meta.fd.smallest_seqno, meta.fd.largest_seqno,
meta.marked_for_compaction, meta.oldest_blob_file_number,
meta.oldest_ancester_time, meta.file_creation_time,
meta.file_checksum, meta.file_checksum_func_name);
edit->SetBlobFileAdditions(std::move(blob_file_additions));
}
InternalStats::CompactionStats stats(CompactionReason::kFlush, 1);
stats.micros = env_->NowMicros() - start_micros;
if (has_output) {
stats.bytes_written = meta.fd.GetFileSize();
const auto& blobs = edit->GetBlobFileAdditions();
for (const auto& blob : blobs) {
stats.bytes_written += blob.GetTotalBlobBytes();
}
stats.num_output_files = static_cast<int>(blobs.size()) + 1;
}
stats.bytes_written = meta.fd.GetFileSize();
stats.num_output_files = 1;
cfd->internal_stats()->AddCompactionStats(level, Env::Priority::USER, stats);
cfd->internal_stats()->AddCFStats(InternalStats::BYTES_FLUSHED,
stats.bytes_written);
meta.fd.GetFileSize());
RecordTick(stats_, COMPACT_WRITE_BYTES, meta.fd.GetFileSize());
return s;
}
@@ -1487,7 +1451,7 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
const std::vector<ColumnFamilyDescriptor>& column_families,
std::vector<ColumnFamilyHandle*>* handles, DB** dbptr,
const bool seq_per_batch, const bool batch_per_txn) {
Status s = ValidateOptionsByTable(db_options, column_families);
Status s = SanitizeOptionsByTable(db_options, column_families);
if (!s.ok()) {
return s;
}
@@ -1634,7 +1598,7 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
}
}
if (s.ok() && impl->immutable_db_options_.persist_stats_to_disk) {
// try to read format version
// try to read format version but no need to fail Open() even if it fails
s = impl->PersistentStatsProcessFormatVersion();
}
@@ -1678,8 +1642,6 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
*dbptr = impl;
impl->opened_successfully_ = true;
impl->MaybeScheduleFlushOrCompaction();
} else {
persist_options_status.PermitUncheckedError();
}
impl->mutex_.Unlock();
@@ -1774,13 +1736,9 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
"DB::Open() failed --- Unable to persist Options file",
persist_options_status.ToString());
}
} else {
ROCKS_LOG_WARN(impl->immutable_db_options_.info_log,
"Persisting Option File error: %s",
persist_options_status.ToString().c_str());
}
if (s.ok()) {
impl->StartPeriodicWorkScheduler();
impl->StartStatsDumpScheduler();
} else {
for (auto* h : *handles) {
delete h;
+7 -7
View File
@@ -151,7 +151,7 @@ Status OpenForReadOnlyCheckExistence(const DBOptions& db_options,
} // namespace
Status DB::OpenForReadOnly(const Options& options, const std::string& dbname,
DB** dbptr, bool /*error_if_wal_file_exists*/) {
DB** dbptr, bool /*error_if_log_file_exist*/) {
// If dbname does not exist in the file system, should not do anything
Status s = OpenForReadOnlyCheckExistence(options, dbname);
if (!s.ok()) {
@@ -188,7 +188,7 @@ Status DB::OpenForReadOnly(
const DBOptions& db_options, const std::string& dbname,
const std::vector<ColumnFamilyDescriptor>& column_families,
std::vector<ColumnFamilyHandle*>* handles, DB** dbptr,
bool error_if_wal_file_exists) {
bool error_if_log_file_exist) {
// If dbname does not exist in the file system, should not do anything
Status s = OpenForReadOnlyCheckExistence(db_options, dbname);
if (!s.ok()) {
@@ -197,14 +197,14 @@ Status DB::OpenForReadOnly(
return DBImplReadOnly::OpenForReadOnlyWithoutCheck(
db_options, dbname, column_families, handles, dbptr,
error_if_wal_file_exists);
error_if_log_file_exist);
}
Status DBImplReadOnly::OpenForReadOnlyWithoutCheck(
const DBOptions& db_options, const std::string& dbname,
const std::vector<ColumnFamilyDescriptor>& column_families,
std::vector<ColumnFamilyHandle*>* handles, DB** dbptr,
bool error_if_wal_file_exists) {
bool error_if_log_file_exist) {
*dbptr = nullptr;
handles->clear();
@@ -212,7 +212,7 @@ Status DBImplReadOnly::OpenForReadOnlyWithoutCheck(
DBImplReadOnly* impl = new DBImplReadOnly(db_options, dbname);
impl->mutex_.Lock();
Status s = impl->Recover(column_families, true /* read only */,
error_if_wal_file_exists);
error_if_log_file_exist);
if (s.ok()) {
// set column family handles
for (auto cf : column_families) {
@@ -253,7 +253,7 @@ Status DBImplReadOnly::OpenForReadOnlyWithoutCheck(
Status DB::OpenForReadOnly(const Options& /*options*/,
const std::string& /*dbname*/, DB** /*dbptr*/,
bool /*error_if_wal_file_exists*/) {
bool /*error_if_log_file_exist*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
@@ -261,7 +261,7 @@ Status DB::OpenForReadOnly(
const DBOptions& /*db_options*/, const std::string& /*dbname*/,
const std::vector<ColumnFamilyDescriptor>& /*column_families*/,
std::vector<ColumnFamilyHandle*>* /*handles*/, DB** /*dbptr*/,
bool /*error_if_wal_file_exists*/) {
bool /*error_if_log_file_exist*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
#endif // !ROCKSDB_LITE
+1 -1
View File
@@ -138,7 +138,7 @@ class DBImplReadOnly : public DBImpl {
const DBOptions& db_options, const std::string& dbname,
const std::vector<ColumnFamilyDescriptor>& column_families,
std::vector<ColumnFamilyHandle*>* handles, DB** dbptr,
bool error_if_wal_file_exists = false);
bool error_if_log_file_exist = false);
friend class DB;
};
} // namespace ROCKSDB_NAMESPACE
+4 -12
View File
@@ -28,8 +28,8 @@ DBImplSecondary::~DBImplSecondary() {}
Status DBImplSecondary::Recover(
const std::vector<ColumnFamilyDescriptor>& column_families,
bool /*readonly*/, bool /*error_if_wal_file_exists*/,
bool /*error_if_data_exists_in_wals*/, uint64_t*) {
bool /*readonly*/, bool /*error_if_log_file_exist*/,
bool /*error_if_data_exists_in_logs*/, uint64_t*) {
mutex_.AssertHeld();
JobContext job_context(0);
@@ -192,8 +192,6 @@ Status DBImplSecondary::RecoverLogFiles(
auto it = log_readers_.find(log_number);
assert(it != log_readers_.end());
log::FragmentBufferedReader* reader = it->second->reader_;
Status* wal_read_status = it->second->status_;
assert(wal_read_status);
// Manually update the file number allocation counter in VersionSet.
versions_->MarkFileNumberUsed(log_number);
@@ -205,16 +203,13 @@ Status DBImplSecondary::RecoverLogFiles(
while (reader->ReadRecord(&record, &scratch,
immutable_db_options_.wal_recovery_mode) &&
wal_read_status->ok() && status.ok()) {
status.ok()) {
if (record.size() < WriteBatchInternal::kHeader) {
reader->GetReporter()->Corruption(
record.size(), Status::Corruption("log record too small"));
continue;
}
status = WriteBatchInternal::SetContents(&batch, record);
if (!status.ok()) {
break;
}
WriteBatchInternal::SetContents(&batch, record);
SequenceNumber seq_of_batch = WriteBatchInternal::Sequence(&batch);
std::vector<uint32_t> column_family_ids;
status = CollectColumnFamilyIdsFromWriteBatch(batch, &column_family_ids);
@@ -300,9 +295,6 @@ Status DBImplSecondary::RecoverLogFiles(
reader->GetReporter()->Corruption(record.size(), status);
}
}
if (status.ok() && !wal_read_status->ok()) {
status = *wal_read_status;
}
if (!status.ok()) {
return status;
}
+2 -2
View File
@@ -77,8 +77,8 @@ class DBImplSecondary : public DBImpl {
// Recover by replaying MANIFEST and WAL. Also initialize manifest_reader_
// and log_readers_ to facilitate future operations.
Status Recover(const std::vector<ColumnFamilyDescriptor>& column_families,
bool read_only, bool error_if_wal_file_exists,
bool error_if_data_exists_in_wals,
bool read_only, bool error_if_log_file_exist,
bool error_if_data_exists_in_logs,
uint64_t* = nullptr) override;
// Implementations of the DB interface
+21 -49
View File
@@ -77,8 +77,7 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
if (tracer_) {
InstrumentedMutexLock lock(&trace_mutex_);
if (tracer_) {
// TODO: maybe handle the tracing status?
tracer_->Write(my_batch).PermitUncheckedError();
tracer_->Write(my_batch);
}
}
if (write_options.sync && write_options.disableWAL) {
@@ -102,10 +101,12 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
assert(!WriteBatchInternal::IsLatestPersistentState(my_batch) ||
disable_memtable);
Status status;
IOStatus io_s;
if (write_options.low_pri) {
Status s = ThrottleLowPriWritesIfNeeded(write_options, my_batch);
if (!s.ok()) {
return s;
status = ThrottleLowPriWritesIfNeeded(write_options, my_batch);
if (!status.ok()) {
return status;
}
}
@@ -125,13 +126,13 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
? batch_cnt
// every key is a sub-batch consuming a seq
: WriteBatchInternal::Count(my_batch);
uint64_t seq = 0;
uint64_t seq;
// Use a write thread to i) optimize for WAL write, ii) publish last
// sequence in in increasing order, iii) call pre_release_callback serially
Status status = WriteImplWALOnly(
&write_thread_, write_options, my_batch, callback, log_used, log_ref,
&seq, sub_batch_cnt, pre_release_callback, kDoAssignOrder,
kDoPublishLastSeq, disable_memtable);
status = WriteImplWALOnly(&write_thread_, write_options, my_batch, callback,
log_used, log_ref, &seq, sub_batch_cnt,
pre_release_callback, kDoAssignOrder,
kDoPublishLastSeq, disable_memtable);
TEST_SYNC_POINT("DBImpl::WriteImpl:UnorderedWriteAfterWriteWAL");
if (!status.ok()) {
return status;
@@ -163,7 +164,6 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
StopWatch write_sw(env_, immutable_db_options_.statistics.get(), DB_WRITE);
write_thread_.JoinBatchGroup(&w);
Status status;
if (w.state == WriteThread::STATE_PARALLEL_MEMTABLE_WRITER) {
// we are a non-leader in a parallel group
@@ -204,8 +204,6 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
*seq_used = w.sequence;
}
// write is complete and leader has updated sequence
// Should we handle it?
status.PermitUncheckedError();
return w.FinalStatus();
}
// else we are the leader of the write batch group
@@ -254,7 +252,6 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
last_batch_group_size_ =
write_thread_.EnterAsBatchGroupLeader(&w, &write_group);
IOStatus io_s;
if (status.ok()) {
// Rules for when we can update the memtable concurrently
// 1. supported by memtable
@@ -841,9 +838,7 @@ void DBImpl::WriteStatusCheckOnLocked(const Status& status) {
mutex_.AssertHeld();
if (immutable_db_options_.paranoid_checks && !status.ok() &&
!status.IsBusy() && !status.IsIncomplete()) {
// Maybe change the return status to void?
error_handler_.SetBGError(status, BackgroundErrorReason::kWriteCallback)
.PermitUncheckedError();
error_handler_.SetBGError(status, BackgroundErrorReason::kWriteCallback);
}
}
@@ -854,9 +849,7 @@ void DBImpl::WriteStatusCheck(const Status& status) {
if (immutable_db_options_.paranoid_checks && !status.ok() &&
!status.IsBusy() && !status.IsIncomplete()) {
mutex_.Lock();
// Maybe change the return status to void?
error_handler_.SetBGError(status, BackgroundErrorReason::kWriteCallback)
.PermitUncheckedError();
error_handler_.SetBGError(status, BackgroundErrorReason::kWriteCallback);
mutex_.Unlock();
}
}
@@ -868,9 +861,7 @@ void DBImpl::IOStatusCheck(const IOStatus& io_status) {
!io_status.IsBusy() && !io_status.IsIncomplete()) ||
io_status.IsIOFenced()) {
mutex_.Lock();
// Maybe change the return status to void?
error_handler_.SetBGError(io_status, BackgroundErrorReason::kWriteCallback)
.PermitUncheckedError();
error_handler_.SetBGError(io_status, BackgroundErrorReason::kWriteCallback);
mutex_.Unlock();
}
}
@@ -884,9 +875,7 @@ void DBImpl::MemTableInsertStatusCheck(const Status& status) {
if (!status.ok()) {
mutex_.Lock();
assert(!error_handler_.IsBGWorkStopped());
// Maybe change the return status to void?
error_handler_.SetBGError(status, BackgroundErrorReason::kMemTable)
.PermitUncheckedError();
error_handler_.SetBGError(status, BackgroundErrorReason::kMemTable);
mutex_.Unlock();
}
}
@@ -1000,10 +989,8 @@ WriteBatch* DBImpl::MergeBatch(const WriteThread::WriteGroup& write_group,
merged_batch = tmp_batch;
for (auto writer : write_group) {
if (!writer->CallbackFailed()) {
Status s = WriteBatchInternal::Append(merged_batch, writer->batch,
/*WAL_only*/ true);
// Always returns Status::OK.
assert(s.ok());
WriteBatchInternal::Append(merged_batch, writer->batch,
/*WAL_only*/ true);
if (WriteBatchInternal::IsLatestPersistentState(writer->batch)) {
// We only need to cache the last of such write batch
*to_be_cached_state = writer->batch;
@@ -1782,15 +1769,10 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
}
// We may have lost data from the WritableFileBuffer in-memory buffer for
// the current log, so treat it as a fatal error and set bg_error
// Should handle return error?
if (!io_s.ok()) {
// Should handle return error?
error_handler_.SetBGError(io_s, BackgroundErrorReason::kMemTable)
.PermitUncheckedError();
error_handler_.SetBGError(io_s, BackgroundErrorReason::kMemTable);
} else {
// Should handle return error?
error_handler_.SetBGError(s, BackgroundErrorReason::kMemTable)
.PermitUncheckedError();
error_handler_.SetBGError(s, BackgroundErrorReason::kMemTable);
}
// Read back bg_error in order to get the right severity
s = error_handler_.GetBGError();
@@ -1824,10 +1806,6 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
NotifyOnMemTableSealed(cfd, memtable_info);
mutex_.Lock();
#endif // ROCKSDB_LITE
// It is possible that we got here without checking the value of i_os, but
// that is okay. If we did, it most likely means that s was already an error.
// In any case, ignore any unchecked error for i_os here.
io_s.PermitUncheckedError();
return s;
}
@@ -1918,10 +1896,7 @@ Status DB::Delete(const WriteOptions& opt, ColumnFamilyHandle* column_family,
Status DB::SingleDelete(const WriteOptions& opt,
ColumnFamilyHandle* column_family, const Slice& key) {
WriteBatch batch;
Status s = batch.SingleDelete(column_family, key);
if (!s.ok()) {
return s;
}
batch.SingleDelete(column_family, key);
return Write(opt, &batch);
}
@@ -1929,10 +1904,7 @@ Status DB::DeleteRange(const WriteOptions& opt,
ColumnFamilyHandle* column_family,
const Slice& begin_key, const Slice& end_key) {
WriteBatch batch;
Status s = batch.DeleteRange(column_family, begin_key, end_key);
if (!s.ok()) {
return s;
}
batch.DeleteRange(column_family, begin_key, end_key);
return Write(opt, &batch);
}
+8 -11
View File
@@ -52,7 +52,7 @@ class DBSecondaryTest : public DBTestBase {
void CloseSecondary() {
for (auto h : handles_secondary_) {
ASSERT_OK(db_secondary_->DestroyColumnFamilyHandle(h));
db_secondary_->DestroyColumnFamilyHandle(h);
}
handles_secondary_.clear();
delete db_secondary_;
@@ -97,7 +97,7 @@ void DBSecondaryTest::CheckFileTypeCounts(const std::string& dir,
int expected_log, int expected_sst,
int expected_manifest) const {
std::vector<std::string> filenames;
ASSERT_OK(env_->GetChildren(dir, &filenames));
env_->GetChildren(dir, &filenames);
int log_cnt = 0, sst_cnt = 0, manifest_cnt = 0;
for (auto file : filenames) {
@@ -777,8 +777,8 @@ TEST_F(DBSecondaryTest, CatchUpAfterFlush) {
WriteOptions write_opts;
WriteBatch wb;
ASSERT_OK(wb.Put("key0", "value0"));
ASSERT_OK(wb.Put("key1", "value1"));
wb.Put("key0", "value0");
wb.Put("key1", "value1");
ASSERT_OK(dbfull()->Write(write_opts, &wb));
ReadOptions read_opts;
std::unique_ptr<Iterator> iter1(db_secondary_->NewIterator(read_opts));
@@ -791,27 +791,25 @@ TEST_F(DBSecondaryTest, CatchUpAfterFlush) {
ASSERT_FALSE(iter1->Valid());
iter1->Seek("key1");
ASSERT_FALSE(iter1->Valid());
ASSERT_OK(iter1->status());
std::unique_ptr<Iterator> iter2(db_secondary_->NewIterator(read_opts));
iter2->Seek("key0");
ASSERT_TRUE(iter2->Valid());
ASSERT_EQ("value0", iter2->value());
iter2->Seek("key1");
ASSERT_TRUE(iter2->Valid());
ASSERT_OK(iter2->status());
ASSERT_EQ("value1", iter2->value());
{
WriteBatch wb1;
ASSERT_OK(wb1.Put("key0", "value01"));
ASSERT_OK(wb1.Put("key1", "value11"));
wb1.Put("key0", "value01");
wb1.Put("key1", "value11");
ASSERT_OK(dbfull()->Write(write_opts, &wb1));
}
{
WriteBatch wb2;
ASSERT_OK(wb2.Put("key0", "new_value0"));
ASSERT_OK(wb2.Delete("key1"));
wb2.Put("key0", "new_value0");
wb2.Delete("key1");
ASSERT_OK(dbfull()->Write(write_opts, &wb2));
}
@@ -825,7 +823,6 @@ TEST_F(DBSecondaryTest, CatchUpAfterFlush) {
ASSERT_EQ("new_value0", iter3->value());
iter3->Seek("key1");
ASSERT_FALSE(iter3->Valid());
ASSERT_OK(iter3->status());
}
TEST_F(DBSecondaryTest, CheckConsistencyWhenOpen) {
+12 -31
View File
@@ -107,7 +107,7 @@ Status DBIter::GetProperty(std::string prop_name, std::string* prop) {
}
bool DBIter::ParseKey(ParsedInternalKey* ikey) {
if (ParseInternalKey(iter_.key(), ikey) != Status::OK()) {
if (!ParseInternalKey(iter_.key(), ikey)) {
status_ = Status::Corruption("corrupted internal key in DBIter");
valid_ = false;
ROCKS_LOG_ERROR(logger_, "corrupted internal key in DBIter: %s",
@@ -147,13 +147,13 @@ void DBIter::Next() {
local_stats_.next_count_++;
if (ok && iter_.Valid()) {
Slice prefix;
if (prefix_same_as_start_) {
assert(prefix_extractor_ != nullptr);
const Slice prefix = prefix_.GetUserKey();
FindNextUserEntry(true /* skipping the current user key */, &prefix);
} else {
FindNextUserEntry(true /* skipping the current user key */, nullptr);
prefix = prefix_.GetUserKey();
}
FindNextUserEntry(true /* skipping the current user key */,
prefix_same_as_start_ ? &prefix : nullptr);
} else {
is_key_seqnum_zero_ = false;
valid_ = false;
@@ -384,11 +384,8 @@ bool DBIter::FindNextUserEntryInternal(bool skipping_saved_key,
}
break;
default:
valid_ = false;
status_ = Status::Corruption(
"Unknown value type: " +
std::to_string(static_cast<unsigned int>(ikey_.type)));
return false;
assert(false);
break;
}
}
} else {
@@ -559,11 +556,7 @@ bool DBIter::MergeValuesNewToOld() {
valid_ = false;
return false;
} else {
valid_ = false;
status_ = Status::Corruption(
"Unrecognized value type: " +
std::to_string(static_cast<unsigned int>(ikey.type)));
return false;
assert(false);
}
}
@@ -839,11 +832,7 @@ bool DBIter::FindValueForCurrentKey() {
}
break;
default:
valid_ = false;
status_ = Status::Corruption(
"Unknown value type: " +
std::to_string(static_cast<unsigned int>(last_key_entry_type)));
return false;
assert(false);
}
PERF_COUNTER_ADD(internal_key_skipped_count, 1);
@@ -857,7 +846,6 @@ bool DBIter::FindValueForCurrentKey() {
}
Status s;
s.PermitUncheckedError();
is_blob_ = false;
switch (last_key_entry_type) {
case kTypeDeletion:
@@ -909,11 +897,8 @@ bool DBIter::FindValueForCurrentKey() {
is_blob_ = true;
break;
default:
valid_ = false;
status_ = Status::Corruption(
"Unknown value type: " +
std::to_string(static_cast<unsigned int>(last_key_entry_type)));
return false;
assert(false);
break;
}
if (!s.ok()) {
valid_ = false;
@@ -1062,11 +1047,7 @@ bool DBIter::FindValueForCurrentKeyUsingSeek() {
valid_ = false;
return false;
} else {
valid_ = false;
status_ = Status::Corruption(
"Unknown value type: " +
std::to_string(static_cast<unsigned int>(ikey.type)));
return false;
assert(false);
}
}
+1 -8
View File
@@ -140,14 +140,7 @@ class DBIter final : public Iterator {
}
ReadRangeDelAggregator* GetRangeDelAggregator() { return &range_del_agg_; }
bool Valid() const override {
#ifdef ROCKSDB_ASSERT_STATUS_CHECKED
if (valid_) {
status_.PermitUncheckedError();
}
#endif // ROCKSDB_ASSERT_STATUS_CHECKED
return valid_;
}
bool Valid() const override { return valid_; }
Slice key() const override {
assert(valid_);
if (start_seqnum_ > 0 || timestamp_lb_) {
+3 -4
View File
@@ -99,10 +99,9 @@ class TestIterator : public InternalIterator {
}
for (auto it = data_.begin(); it != data_.end(); ++it) {
ParsedInternalKey ikey;
Status pikStatus = ParseInternalKey(it->first, &ikey);
pikStatus.PermitUncheckedError();
assert(pikStatus.ok());
if (!pikStatus.ok() || ikey.user_key != _key) {
bool ok __attribute__((__unused__)) = ParseInternalKey(it->first, &ikey);
assert(ok);
if (ikey.user_key != _key) {
continue;
}
if (valid_ && data_.begin() + iter_ > it) {
+1 -1
View File
@@ -2146,7 +2146,7 @@ TEST_P(DBIteratorTest, ReadAhead) {
BlockBasedTableOptions table_options;
table_options.block_size = 1024;
table_options.no_block_cache = true;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.table_factory.reset(new BlockBasedTableFactory(table_options));
Reopen(options);
std::string value(1024, 'a');
+51 -58
View File
@@ -33,14 +33,17 @@ class DBOptionsTest : public DBTestBase {
std::unordered_map<std::string, std::string> GetMutableDBOptionsMap(
const DBOptions& options) {
std::string options_str;
std::unordered_map<std::string, std::string> mutable_map;
ConfigOptions config_options;
config_options.delimiter = "; ";
EXPECT_OK(GetStringFromMutableDBOptions(
config_options, MutableDBOptions(options), &options_str));
EXPECT_OK(StringToMap(options_str, &mutable_map));
GetStringFromDBOptions(config_options, options, &options_str);
std::unordered_map<std::string, std::string> options_map;
StringToMap(options_str, &options_map);
std::unordered_map<std::string, std::string> mutable_map;
for (const auto& opt : db_options_type_info) {
if (opt.second.IsMutable() && opt.second.ShouldSerialize()) {
mutable_map[opt.first] = options_map[opt.first];
}
}
return mutable_map;
}
@@ -50,10 +53,15 @@ class DBOptionsTest : public DBTestBase {
ConfigOptions config_options;
config_options.delimiter = "; ";
GetStringFromColumnFamilyOptions(config_options, options, &options_str);
std::unordered_map<std::string, std::string> options_map;
StringToMap(options_str, &options_map);
std::unordered_map<std::string, std::string> mutable_map;
EXPECT_OK(GetStringFromMutableCFOptions(
config_options, MutableCFOptions(options), &options_str));
EXPECT_OK(StringToMap(options_str, &mutable_map));
for (const auto& opt : cf_options_type_info) {
if (opt.second.IsMutable() && opt.second.ShouldSerialize()) {
mutable_map[opt.first] = options_map[opt.first];
}
}
return mutable_map;
}
@@ -79,21 +87,6 @@ class DBOptionsTest : public DBTestBase {
#endif // ROCKSDB_LITE
};
TEST_F(DBOptionsTest, ImmutableTrackAndVerifyWalsInManifest) {
Options options;
options.track_and_verify_wals_in_manifest = true;
ImmutableDBOptions db_options(options);
ASSERT_TRUE(db_options.track_and_verify_wals_in_manifest);
Reopen(options);
ASSERT_TRUE(dbfull()->GetDBOptions().track_and_verify_wals_in_manifest);
Status s =
dbfull()->SetDBOptions({{"track_and_verify_wals_in_manifest", "false"}});
ASSERT_FALSE(s.ok());
}
// RocksDB lite don't support dynamic options.
#ifndef ROCKSDB_LITE
@@ -150,7 +143,7 @@ TEST_F(DBOptionsTest, SetBytesPerSync) {
WriteOptions write_opts;
// should sync approximately 40MB/1MB ~= 40 times.
for (i = 0; i < 40; i++) {
ASSERT_OK(Put(Key(i), kValue, write_opts));
Put(Key(i), kValue, write_opts);
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
@@ -166,7 +159,7 @@ TEST_F(DBOptionsTest, SetBytesPerSync) {
// should sync approximately 40MB*2/8MB ~= 10 times.
// data will be 40*2MB because of previous Puts too.
for (i = 0; i < 40; i++) {
ASSERT_OK(Put(Key(i), kValue, write_opts));
Put(Key(i), kValue, write_opts);
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
@@ -197,7 +190,7 @@ TEST_F(DBOptionsTest, SetWalBytesPerSync) {
const std::string kValue(kValueSize, 'v');
int i = 0;
for (; i < 10; i++) {
ASSERT_OK(Put(Key(i), kValue));
Put(Key(i), kValue);
}
// Do not flush. If we flush here, SwitchWAL will reuse old WAL file since its
// empty and will not get the new wal_bytes_per_sync value.
@@ -208,7 +201,7 @@ TEST_F(DBOptionsTest, SetWalBytesPerSync) {
counter = 0;
i = 0;
for (; i < 10; i++) {
ASSERT_OK(Put(Key(i), kValue));
Put(Key(i), kValue);
}
ASSERT_GT(counter, 0);
ASSERT_GT(low_bytes_per_sync, 0);
@@ -243,9 +236,9 @@ TEST_F(DBOptionsTest, WritableFileMaxBufferSize) {
for (; i < 3; i++) {
ASSERT_OK(Put("foo", ToString(i)));
ASSERT_OK(Put("bar", ToString(i)));
ASSERT_OK(Flush());
Flush();
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(unmatch_cnt, 0);
ASSERT_GE(match_cnt, 11);
@@ -261,9 +254,9 @@ TEST_F(DBOptionsTest, WritableFileMaxBufferSize) {
for (; i < 3; i++) {
ASSERT_OK(Put("foo", ToString(i)));
ASSERT_OK(Put("bar", ToString(i)));
ASSERT_OK(Flush());
Flush();
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(unmatch_cnt, 0);
ASSERT_GE(match_cnt, 11);
}
@@ -299,14 +292,14 @@ TEST_F(DBOptionsTest, EnableAutoCompactionAndTriggerStall) {
DestroyAndReopen(options);
int i = 0;
for (; i < 1024; i++) {
ASSERT_OK(Put(Key(i), kValue));
Put(Key(i), kValue);
}
ASSERT_OK(Flush());
Flush();
for (; i < 1024 * 2; i++) {
ASSERT_OK(Put(Key(i), kValue));
Put(Key(i), kValue);
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
Flush();
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_EQ(2, NumTableFilesAtLevel(0));
uint64_t l0_size = SizeAtLevel(0);
@@ -328,7 +321,7 @@ TEST_F(DBOptionsTest, EnableAutoCompactionAndTriggerStall) {
break;
}
Reopen(options);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_FALSE(dbfull()->TEST_write_controler().IsStopped());
ASSERT_FALSE(dbfull()->TEST_write_controler().NeedsDelay());
@@ -375,7 +368,7 @@ TEST_F(DBOptionsTest, EnableAutoCompactionAndTriggerStall) {
TEST_SYNC_POINT("DBOptionsTest::EnableAutoCompactionAndTriggerStall:3");
// Background compaction executed.
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_FALSE(dbfull()->TEST_write_controler().IsStopped());
ASSERT_FALSE(dbfull()->TEST_write_controler().NeedsDelay());
}
@@ -392,12 +385,12 @@ TEST_F(DBOptionsTest, SetOptionsMayTriggerCompaction) {
// Need to insert two keys to avoid trivial move.
ASSERT_OK(Put("foo", ToString(i)));
ASSERT_OK(Put("bar", ToString(i)));
ASSERT_OK(Flush());
Flush();
}
ASSERT_EQ("3", FilesPerLevel());
ASSERT_OK(
dbfull()->SetOptions({{"level0_file_num_compaction_trigger", "3"}}));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ("0,1", FilesPerLevel());
}
@@ -519,7 +512,7 @@ TEST_F(DBOptionsTest, MaxTotalWalSizeChange) {
ASSERT_OK(dbfull()->SetDBOptions({{"max_total_wal_size", "10"}}));
for (size_t cf = 0; cf < handles_.size(); ++cf) {
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[cf]));
dbfull()->TEST_WaitForFlushMemTable(handles_[cf]);
ASSERT_EQ("1", FilesPerLevel(static_cast<int>(cf)));
}
}
@@ -732,13 +725,13 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
// No files should be compacted as ttl is set to 1 hour.
ASSERT_EQ(dbfull()->GetOptions().ttl, 3600);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_EQ(NumTableFilesAtLevel(0), 10);
// Set ttl to 1 minute. So all files should get deleted.
ASSERT_OK(dbfull()->SetOptions({{"ttl", "60"}}));
ASSERT_EQ(dbfull()->GetOptions().ttl, 60);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(NumTableFilesAtLevel(0), 0);
@@ -754,7 +747,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
}
ASSERT_OK(Flush());
Flush();
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(NumTableFilesAtLevel(0), 10);
@@ -762,7 +755,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
// No files should be compacted as max_table_files_size is set to 500 KB.
ASSERT_EQ(dbfull()->GetOptions().compaction_options_fifo.max_table_files_size,
500 << 10);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_EQ(NumTableFilesAtLevel(0), 10);
// Set max_table_files_size to 12 KB. So only 1 file should remain now.
@@ -770,7 +763,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
{{"compaction_options_fifo", "{max_table_files_size=12288;}"}}));
ASSERT_EQ(dbfull()->GetOptions().compaction_options_fifo.max_table_files_size,
12 << 10);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(NumTableFilesAtLevel(0), 1);
@@ -786,7 +779,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
}
ASSERT_OK(Flush());
Flush();
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(NumTableFilesAtLevel(0), 10);
@@ -795,7 +788,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
// allow_compaction is false
ASSERT_EQ(dbfull()->GetOptions().compaction_options_fifo.allow_compaction,
false);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_EQ(NumTableFilesAtLevel(0), 10);
// Set allow_compaction to true. So number of files should be between 1 and 5.
@@ -803,7 +796,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
{{"compaction_options_fifo", "{allow_compaction=true;}"}}));
ASSERT_EQ(dbfull()->GetOptions().compaction_options_fifo.allow_compaction,
true);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_GE(NumTableFilesAtLevel(0), 1);
ASSERT_LE(NumTableFilesAtLevel(0), 5);
@@ -824,14 +817,14 @@ TEST_F(DBOptionsTest, CompactionReadaheadSizeChange) {
ASSERT_OK(dbfull()->SetDBOptions({{"compaction_readahead_size", "256"}}));
ASSERT_EQ(256, dbfull()->GetDBOptions().compaction_readahead_size);
for (int i = 0; i < 1024; i++) {
ASSERT_OK(Put(Key(i), kValue));
Put(Key(i), kValue);
}
ASSERT_OK(Flush());
Flush();
for (int i = 0; i < 1024 * 2; i++) {
ASSERT_OK(Put(Key(i), kValue));
Put(Key(i), kValue);
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
Flush();
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(256, env_->compaction_readahead_size_);
Close();
}
@@ -850,7 +843,7 @@ TEST_F(DBOptionsTest, FIFOTtlBackwardCompatible) {
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
}
ASSERT_OK(Flush());
Flush();
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(NumTableFilesAtLevel(0), 10);
@@ -915,7 +908,7 @@ TEST_F(DBOptionsTest, ChangeCompression) {
ASSERT_OK(Put("foo", "foofoofoo"));
ASSERT_OK(Put("bar", "foofoofoo"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_TRUE(compacted);
ASSERT_EQ(CompressionType::kNoCompression, compression_used);
ASSERT_EQ(options.compression_opts.level, compression_opt_used.level);
@@ -933,7 +926,7 @@ TEST_F(DBOptionsTest, ChangeCompression) {
ASSERT_OK(Put("foo", "foofoofoo"));
ASSERT_OK(Put("bar", "foofoofoo"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_TRUE(compacted);
ASSERT_EQ(CompressionType::kSnappyCompression, compression_used);
ASSERT_EQ(6, compression_opt_used.level);
+88 -98
View File
@@ -27,7 +27,7 @@ namespace ROCKSDB_NAMESPACE {
class DBPropertiesTest : public DBTestBase {
public:
DBPropertiesTest()
: DBTestBase("/db_properties_test", /*env_do_fsync=*/false) {}
: DBTestBase("/db_properties_test", /*env_do_fsync=*/true) {}
};
#ifndef ROCKSDB_LITE
@@ -53,12 +53,12 @@ TEST_F(DBPropertiesTest, Empty) {
// Block sync calls
env_->delay_sstable_sync_.store(true, std::memory_order_release);
ASSERT_OK(Put(1, "k1", std::string(100000, 'x'))); // Fill memtable
Put(1, "k1", std::string(100000, 'x')); // Fill memtable
ASSERT_TRUE(dbfull()->GetProperty(
handles_[1], "rocksdb.num-entries-active-mem-table", &num));
ASSERT_EQ("2", num);
ASSERT_OK(Put(1, "k2", std::string(100000, 'y'))); // Trigger compaction
Put(1, "k2", std::string(100000, 'y')); // Trigger compaction
ASSERT_TRUE(dbfull()->GetProperty(
handles_[1], "rocksdb.num-entries-active-mem-table", &num));
ASSERT_EQ("1", num);
@@ -98,10 +98,10 @@ TEST_F(DBPropertiesTest, CurrentVersionNumber) {
uint64_t v1, v2, v3;
ASSERT_TRUE(
dbfull()->GetIntProperty("rocksdb.current-super-version-number", &v1));
ASSERT_OK(Put("12345678", ""));
Put("12345678", "");
ASSERT_TRUE(
dbfull()->GetIntProperty("rocksdb.current-super-version-number", &v2));
ASSERT_OK(Flush());
Flush();
ASSERT_TRUE(
dbfull()->GetIntProperty("rocksdb.current-super-version-number", &v3));
@@ -127,8 +127,8 @@ TEST_F(DBPropertiesTest, GetAggregatedIntPropertyTest) {
Random rnd(301);
for (auto* handle : handles_) {
for (int i = 0; i < kKeyNum; ++i) {
ASSERT_OK(db_->Put(WriteOptions(), handle, rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize)));
db_->Put(WriteOptions(), handle, rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
}
}
@@ -154,7 +154,7 @@ TEST_F(DBPropertiesTest, GetAggregatedIntPropertyTest) {
DB::Properties::kEstimateTableReadersMem, &before_flush_trm));
// Issue flush and expect larger memory usage of table readers.
ASSERT_OK(db_->Flush(FlushOptions(), handle));
db_->Flush(FlushOptions(), handle);
ASSERT_TRUE(db_->GetAggregatedIntProperty(
DB::Properties::kEstimateTableReadersMem, &after_flush_trm));
@@ -300,9 +300,9 @@ TEST_F(DBPropertiesTest, ValidateSampleNumber) {
for (int i = 0; i < files; i++) {
int rows = files / 10;
for (int j = 0; j < rows; j++) {
ASSERT_OK(db_->Put(WriteOptions(), std::to_string(++key), "foo"));
db_->Put(WriteOptions(), std::to_string(++key), "foo");
}
ASSERT_OK(db_->Flush(FlushOptions()));
db_->Flush(FlushOptions());
}
}
std::string num;
@@ -347,24 +347,23 @@ TEST_F(DBPropertiesTest, AggregatedTableProperties) {
Random rnd(5632);
for (int table = 1; table <= kTableCount; ++table) {
for (int i = 0; i < kPutsPerTable; ++i) {
ASSERT_OK(db_->Put(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize)));
db_->Put(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
}
for (int i = 0; i < kDeletionsPerTable; i++) {
ASSERT_OK(db_->Delete(WriteOptions(), rnd.RandomString(kKeySize)));
db_->Delete(WriteOptions(), rnd.RandomString(kKeySize));
}
for (int i = 0; i < kMergeOperandsPerTable; i++) {
ASSERT_OK(db_->Merge(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize)));
db_->Merge(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
}
for (int i = 0; i < kRangeDeletionsPerTable; i++) {
std::string start = rnd.RandomString(kKeySize);
std::string end = start;
end.resize(kValueSize);
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
start, end));
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), start, end);
}
ASSERT_OK(db_->Flush(FlushOptions()));
db_->Flush(FlushOptions());
}
std::string property;
db_->GetProperty(DB::Properties::kAggregatedTableProperties, &property);
@@ -411,11 +410,11 @@ TEST_F(DBPropertiesTest, ReadLatencyHistogramByLevel) {
int key_index = 0;
Random rnd(301);
for (int num = 0; num < 8; num++) {
ASSERT_OK(Put("foo", "bar"));
Put("foo", "bar");
GenerateNewFile(&rnd, &key_index);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
std::string prop;
ASSERT_TRUE(dbfull()->GetProperty("rocksdb.dbstats", &prop));
@@ -431,7 +430,7 @@ TEST_F(DBPropertiesTest, ReadLatencyHistogramByLevel) {
// Reopen and issue Get(). See thee latency tracked
ReopenWithColumnFamilies({"default", "pikachu"}, options);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
for (int key = 0; key < key_index; key++) {
Get(Key(key));
}
@@ -459,7 +458,6 @@ TEST_F(DBPropertiesTest, ReadLatencyHistogramByLevel) {
std::unique_ptr<Iterator> iter(db_->NewIterator(ReadOptions()));
for (iter->Seek(Key(0)); iter->Valid(); iter->Next()) {
}
ASSERT_OK(iter->status());
}
ASSERT_TRUE(dbfull()->GetProperty("rocksdb.cf-file-histogram", &prop));
ASSERT_NE(std::string::npos, prop.find("** Level 0 read latency histogram"));
@@ -473,9 +471,9 @@ TEST_F(DBPropertiesTest, ReadLatencyHistogramByLevel) {
ASSERT_EQ(std::string::npos, prop.find("** Level 1 read latency histogram"));
ASSERT_EQ(std::string::npos, prop.find("** Level 2 read latency histogram"));
// put something and read it back , CF 1 should show histogram.
ASSERT_OK(Put(1, "foo", "bar"));
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
Put(1, "foo", "bar");
Flush(1);
dbfull()->TEST_WaitForCompact();
ASSERT_EQ("bar", Get(1, "foo"));
ASSERT_TRUE(
@@ -501,7 +499,7 @@ TEST_F(DBPropertiesTest, ReadLatencyHistogramByLevel) {
ASSERT_EQ(std::string::npos, prop.find("** Level 2 read latency histogram"));
// Clear internal stats
ASSERT_OK(dbfull()->ResetStats());
dbfull()->ResetStats();
ASSERT_TRUE(dbfull()->GetProperty("rocksdb.cfstats", &prop));
ASSERT_EQ(std::string::npos, prop.find("** Level 0 read latency histogram"));
ASSERT_EQ(std::string::npos, prop.find("** Level 1 read latency histogram"));
@@ -549,25 +547,24 @@ TEST_F(DBPropertiesTest, AggregatedTablePropertiesAtLevel) {
TableProperties tp, sum_tp, expected_tp;
for (int table = 1; table <= kTableCount; ++table) {
for (int i = 0; i < kPutsPerTable; ++i) {
ASSERT_OK(db_->Put(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize)));
db_->Put(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
}
for (int i = 0; i < kDeletionsPerTable; i++) {
ASSERT_OK(db_->Delete(WriteOptions(), rnd.RandomString(kKeySize)));
db_->Delete(WriteOptions(), rnd.RandomString(kKeySize));
}
for (int i = 0; i < kMergeOperandsPerTable; i++) {
ASSERT_OK(db_->Merge(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize)));
db_->Merge(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
}
for (int i = 0; i < kRangeDeletionsPerTable; i++) {
std::string start = rnd.RandomString(kKeySize);
std::string end = start;
end.resize(kValueSize);
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
start, end));
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), start, end);
}
ASSERT_OK(db_->Flush(FlushOptions()));
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
db_->Flush(FlushOptions());
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ResetTableProperties(&sum_tp);
for (int level = 0; level < kMaxLevel; ++level) {
db_->GetProperty(
@@ -832,7 +829,7 @@ TEST_F(DBPropertiesTest, DISABLED_GetProperty) {
// Wait for compaction to be done. This is important because otherwise RocksDB
// might schedule a compaction when reopening the database, failing assertion
// (A) as a result.
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
options.max_open_files = 10;
Reopen(options);
// After reopening, no table reader is loaded, so no memory for table readers
@@ -860,7 +857,7 @@ TEST_F(DBPropertiesTest, DISABLED_GetProperty) {
std::unique_ptr<Iterator> iter1(dbfull()->NewIterator(ReadOptions()));
ASSERT_OK(dbfull()->Put(writeOpt, "k6", big_value));
ASSERT_OK(Flush());
Flush();
ASSERT_TRUE(
dbfull()->GetIntProperty("rocksdb.num-live-versions", &int_num));
ASSERT_EQ(int_num, 2U);
@@ -869,7 +866,7 @@ TEST_F(DBPropertiesTest, DISABLED_GetProperty) {
std::unique_ptr<Iterator> iter2(dbfull()->NewIterator(ReadOptions()));
ASSERT_OK(dbfull()->Put(writeOpt, "k7", big_value));
ASSERT_OK(Flush());
Flush();
ASSERT_TRUE(
dbfull()->GetIntProperty("rocksdb.num-live-versions", &int_num));
ASSERT_EQ(int_num, 3U);
@@ -924,12 +921,11 @@ TEST_F(DBPropertiesTest, ApproximateMemoryUsage) {
for (int r = 0; r < kNumRounds; ++r) {
for (int f = 0; f < kFlushesPerRound; ++f) {
for (int w = 0; w < kWritesPerFlush; ++w) {
ASSERT_OK(
Put(rnd.RandomString(kKeySize), rnd.RandomString(kValueSize)));
Put(rnd.RandomString(kKeySize), rnd.RandomString(kValueSize));
}
}
// Make sure that there is no flush between getting the two properties.
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->GetIntProperty("rocksdb.cur-size-all-mem-tables", &unflushed_mem);
dbfull()->GetIntProperty("rocksdb.size-all-mem-tables", &all_mem);
// in no iterator case, these two number should be the same.
@@ -943,13 +939,12 @@ TEST_F(DBPropertiesTest, ApproximateMemoryUsage) {
iters.push_back(db_->NewIterator(ReadOptions()));
for (int f = 0; f < kFlushesPerRound; ++f) {
for (int w = 0; w < kWritesPerFlush; ++w) {
ASSERT_OK(
Put(rnd.RandomString(kKeySize), rnd.RandomString(kValueSize)));
Put(rnd.RandomString(kKeySize), rnd.RandomString(kValueSize));
}
}
// Force flush to prevent flush from happening between getting the
// properties or after getting the properties and before the new round.
ASSERT_OK(Flush());
Flush();
// In the second round, add iterators.
dbfull()->GetIntProperty("rocksdb.cur-size-active-mem-table", &active_mem);
@@ -964,7 +959,6 @@ TEST_F(DBPropertiesTest, ApproximateMemoryUsage) {
// Phase 3. Delete iterators and expect "size-all-mem-tables" shrinks
// whenever we release an iterator.
for (auto* iter : iters) {
ASSERT_OK(iter->status());
delete iter;
dbfull()->GetIntProperty("rocksdb.size-all-mem-tables", &all_mem);
// Expect the size shrinking
@@ -1014,19 +1008,19 @@ TEST_F(DBPropertiesTest, EstimatePendingCompBytes) {
uint64_t int_num;
ASSERT_OK(dbfull()->Put(writeOpt, "k1", big_value));
ASSERT_OK(Flush());
Flush();
ASSERT_TRUE(dbfull()->GetIntProperty(
"rocksdb.estimate-pending-compaction-bytes", &int_num));
ASSERT_EQ(int_num, 0U);
ASSERT_OK(dbfull()->Put(writeOpt, "k2", big_value));
ASSERT_OK(Flush());
Flush();
ASSERT_TRUE(dbfull()->GetIntProperty(
"rocksdb.estimate-pending-compaction-bytes", &int_num));
ASSERT_GT(int_num, 0U);
ASSERT_OK(dbfull()->Put(writeOpt, "k3", big_value));
ASSERT_OK(Flush());
Flush();
ASSERT_TRUE(dbfull()->GetIntProperty(
"rocksdb.estimate-pending-compaction-bytes", &int_num));
ASSERT_GT(int_num, 0U);
@@ -1034,7 +1028,7 @@ TEST_F(DBPropertiesTest, EstimatePendingCompBytes) {
sleeping_task_low.WakeUp();
sleeping_task_low.WaitUntilDone();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_TRUE(dbfull()->GetIntProperty(
"rocksdb.estimate-pending-compaction-bytes", &int_num));
ASSERT_EQ(int_num, 0U);
@@ -1064,7 +1058,7 @@ TEST_F(DBPropertiesTest, EstimateCompressionRatio) {
std::string key = ToString(i) + ToString(j) + "key";
ASSERT_OK(dbfull()->Put(WriteOptions(), key, kVal));
}
ASSERT_OK(Flush());
Flush();
}
// no compression at L0, so ratio is less than one
@@ -1072,7 +1066,7 @@ TEST_F(DBPropertiesTest, EstimateCompressionRatio) {
ASSERT_GT(CompressionRatioAtLevel(0), 0.0);
ASSERT_EQ(CompressionRatioAtLevel(1), -1.0);
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
dbfull()->TEST_CompactRange(0, nullptr, nullptr);
ASSERT_EQ(CompressionRatioAtLevel(0), -1.0);
// Data at L1 should be highly compressed thanks to Snappy and redundant data
@@ -1187,9 +1181,9 @@ TEST_F(DBPropertiesTest, GetUserDefinedTableProperties) {
// Create 4 tables
for (int table = 0; table < 4; ++table) {
for (int i = 0; i < 10 + table; ++i) {
ASSERT_OK(db_->Put(WriteOptions(), ToString(table * 100 + i), "val"));
db_->Put(WriteOptions(), ToString(table * 100 + i), "val");
}
ASSERT_OK(db_->Flush(FlushOptions()));
db_->Flush(FlushOptions());
}
TablePropertiesCollection props;
@@ -1211,7 +1205,7 @@ TEST_F(DBPropertiesTest, GetUserDefinedTableProperties) {
ASSERT_GT(collector_factory->num_created_, 0U);
collector_factory->num_created_ = 0;
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
dbfull()->TEST_CompactRange(0, nullptr, nullptr);
ASSERT_GT(collector_factory->num_created_, 0U);
}
#endif // ROCKSDB_LITE
@@ -1227,9 +1221,9 @@ TEST_F(DBPropertiesTest, UserDefinedTablePropertiesContext) {
// Create 2 files
for (int table = 0; table < 2; ++table) {
for (int i = 0; i < 10 + table; ++i) {
ASSERT_OK(Put(1, ToString(table * 100 + i), "val"));
Put(1, ToString(table * 100 + i), "val");
}
ASSERT_OK(Flush(1));
Flush(1);
}
ASSERT_GT(collector_factory->num_created_, 0U);
@@ -1237,15 +1231,15 @@ TEST_F(DBPropertiesTest, UserDefinedTablePropertiesContext) {
// Trigger automatic compactions.
for (int table = 0; table < 3; ++table) {
for (int i = 0; i < 10 + table; ++i) {
ASSERT_OK(Put(1, ToString(table * 100 + i), "val"));
Put(1, ToString(table * 100 + i), "val");
}
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
Flush(1);
dbfull()->TEST_WaitForCompact();
}
ASSERT_GT(collector_factory->num_created_, 0U);
collector_factory->num_created_ = 0;
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr, handles_[1]));
dbfull()->TEST_CompactRange(0, nullptr, nullptr, handles_[1]);
ASSERT_GT(collector_factory->num_created_, 0U);
// Come back to write to default column family
@@ -1254,9 +1248,9 @@ TEST_F(DBPropertiesTest, UserDefinedTablePropertiesContext) {
// Create 4 tables in default column family
for (int table = 0; table < 2; ++table) {
for (int i = 0; i < 10 + table; ++i) {
ASSERT_OK(Put(ToString(table * 100 + i), "val"));
Put(ToString(table * 100 + i), "val");
}
ASSERT_OK(Flush());
Flush();
}
ASSERT_GT(collector_factory->num_created_, 0U);
@@ -1264,15 +1258,15 @@ TEST_F(DBPropertiesTest, UserDefinedTablePropertiesContext) {
// Trigger automatic compactions.
for (int table = 0; table < 3; ++table) {
for (int i = 0; i < 10 + table; ++i) {
ASSERT_OK(Put(ToString(table * 100 + i), "val"));
Put(ToString(table * 100 + i), "val");
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
Flush();
dbfull()->TEST_WaitForCompact();
}
ASSERT_GT(collector_factory->num_created_, 0U);
collector_factory->num_created_ = 0;
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
dbfull()->TEST_CompactRange(0, nullptr, nullptr);
ASSERT_GT(collector_factory->num_created_, 0U);
}
@@ -1306,15 +1300,15 @@ TEST_F(DBPropertiesTest, TablePropertiesNeedCompactTest) {
ASSERT_OK(Put(Key(i), rnd.RandomString(102)));
ASSERT_OK(Put(Key(kMaxKey + i), rnd.RandomString(102)));
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
Flush();
dbfull()->TEST_WaitForCompact();
if (NumTableFilesAtLevel(0) == 1) {
// Clear Level 0 so that when later flush a file with deletions,
// we don't trigger an organic compaction.
ASSERT_OK(Put(Key(0), ""));
ASSERT_OK(Put(Key(kMaxKey * 2), ""));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
Flush();
dbfull()->TEST_WaitForCompact();
}
ASSERT_EQ(NumTableFilesAtLevel(0), 0);
@@ -1326,18 +1320,17 @@ TEST_F(DBPropertiesTest, TablePropertiesNeedCompactTest) {
iter->Next();
++c;
}
ASSERT_OK(iter->status());
ASSERT_EQ(c, 200);
}
ASSERT_OK(Delete(Key(0)));
Delete(Key(0));
for (int i = kMaxKey - 100; i < kMaxKey + 100; i++) {
ASSERT_OK(Delete(Key(i)));
Delete(Key(i));
}
ASSERT_OK(Delete(Key(kMaxKey * 2)));
Delete(Key(kMaxKey * 2));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
Flush();
dbfull()->TEST_WaitForCompact();
{
SetPerfLevel(kEnableCount);
@@ -1348,7 +1341,6 @@ TEST_F(DBPropertiesTest, TablePropertiesNeedCompactTest) {
while (iter->Valid() && iter->key().compare(Key(kMaxKey + 100)) < 0) {
iter->Next();
}
ASSERT_OK(iter->status());
ASSERT_EQ(c, 0);
ASSERT_LT(get_perf_context()->internal_delete_skipped_count, 30u);
ASSERT_LT(get_perf_context()->internal_key_skipped_count, 30u);
@@ -1379,14 +1371,14 @@ TEST_F(DBPropertiesTest, NeedCompactHintPersistentTest) {
for (int i = 0; i < kMaxKey; i++) {
ASSERT_OK(Put(Key(i), ""));
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
Flush();
dbfull()->TEST_WaitForFlushMemTable();
for (int i = 1; i < kMaxKey - 1; i++) {
ASSERT_OK(Delete(Key(i)));
Delete(Key(i));
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
Flush();
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_EQ(NumTableFilesAtLevel(0), 2);
// Restart the DB. Although number of files didn't reach
@@ -1394,7 +1386,7 @@ TEST_F(DBPropertiesTest, NeedCompactHintPersistentTest) {
// still be triggered because of the need-compaction hint.
options.disable_auto_compactions = false;
Reopen(options);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(NumTableFilesAtLevel(0), 0);
{
SetPerfLevel(kEnableCount);
@@ -1404,7 +1396,6 @@ TEST_F(DBPropertiesTest, NeedCompactHintPersistentTest) {
for (iter->Seek(Key(0)); iter->Valid(); iter->Next()) {
c++;
}
ASSERT_OK(iter->status());
ASSERT_EQ(c, 2);
ASSERT_EQ(get_perf_context()->internal_delete_skipped_count, 0);
// We iterate every key twice. Is it a bug?
@@ -1416,9 +1407,9 @@ TEST_F(DBPropertiesTest, NeedCompactHintPersistentTest) {
TEST_F(DBPropertiesTest, EstimateNumKeysUnderflow) {
Options options;
Reopen(options);
ASSERT_OK(Put("foo", "bar"));
ASSERT_OK(Delete("foo"));
ASSERT_OK(Delete("foo"));
Put("foo", "bar");
Delete("foo");
Delete("foo");
uint64_t num_keys = 0;
ASSERT_TRUE(dbfull()->GetIntProperty("rocksdb.estimate-num-keys", &num_keys));
ASSERT_EQ(0, num_keys);
@@ -1596,11 +1587,11 @@ TEST_F(DBPropertiesTest, MinObsoleteSstNumberToKeep) {
for (int i = 0; i < kNumL0Files; ++i) {
// Make sure they overlap in keyspace to prevent trivial move
ASSERT_OK(Put("key1", "val"));
ASSERT_OK(Put("key2", "val"));
ASSERT_OK(Flush());
Put("key1", "val");
Put("key2", "val");
Flush();
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_TRUE(listener->Validated());
}
@@ -1658,8 +1649,7 @@ TEST_F(DBPropertiesTest, BlockCacheProperties) {
// Insert unpinned item to the cache and check size.
constexpr size_t kSize1 = 50;
ASSERT_OK(block_cache->Insert("item1", nullptr /*value*/, kSize1,
nullptr /*deleter*/));
block_cache->Insert("item1", nullptr /*value*/, kSize1, nullptr /*deleter*/);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kBlockCacheCapacity, &value));
ASSERT_EQ(kCapacity, value);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kBlockCacheUsage, &value));
@@ -1671,8 +1661,8 @@ TEST_F(DBPropertiesTest, BlockCacheProperties) {
// Insert pinned item to the cache and check size.
constexpr size_t kSize2 = 30;
Cache::Handle* item2 = nullptr;
ASSERT_OK(block_cache->Insert("item2", nullptr /*value*/, kSize2,
nullptr /*deleter*/, &item2));
block_cache->Insert("item2", nullptr /*value*/, kSize2, nullptr /*deleter*/,
&item2);
ASSERT_NE(nullptr, item2);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kBlockCacheCapacity, &value));
ASSERT_EQ(kCapacity, value);
@@ -1685,8 +1675,8 @@ TEST_F(DBPropertiesTest, BlockCacheProperties) {
// Insert another pinned item to make the cache over-sized.
constexpr size_t kSize3 = 80;
Cache::Handle* item3 = nullptr;
ASSERT_OK(block_cache->Insert("item3", nullptr /*value*/, kSize3,
nullptr /*deleter*/, &item3));
block_cache->Insert("item3", nullptr /*value*/, kSize3, nullptr /*deleter*/,
&item3);
ASSERT_NE(nullptr, item2);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kBlockCacheCapacity, &value));
ASSERT_EQ(kCapacity, value);
+6 -13
View File
@@ -304,14 +304,11 @@ TEST_F(DBSSTTest, DBWithSstFileManager) {
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
// Verify that we are tracking all sst files in dbname_
std::unordered_map<std::string, uint64_t> files_in_db;
ASSERT_OK(GetAllSSTFiles(&files_in_db));
ASSERT_EQ(sfm->GetTrackedFiles(), files_in_db);
ASSERT_EQ(sfm->GetTrackedFiles(), GetAllSSTFiles());
}
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
std::unordered_map<std::string, uint64_t> files_in_db;
ASSERT_OK(GetAllSSTFiles(&files_in_db));
auto files_in_db = GetAllSSTFiles();
// Verify that we are tracking all sst files in dbname_
ASSERT_EQ(sfm->GetTrackedFiles(), files_in_db);
// Verify the total files size
@@ -765,8 +762,7 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowed) {
ASSERT_OK(Flush());
uint64_t first_file_size = 0;
std::unordered_map<std::string, uint64_t> files_in_db;
ASSERT_OK(GetAllSSTFiles(&files_in_db, &first_file_size));
auto files_in_db = GetAllSSTFiles(&first_file_size);
ASSERT_EQ(sfm->GetTotalSize(), first_file_size);
// Set the maximum allowed space usage to the current total size
@@ -806,8 +802,7 @@ TEST_F(DBSSTTest, CancellingCompactionsWorks) {
}
ASSERT_OK(Flush());
uint64_t total_file_size = 0;
std::unordered_map<std::string, uint64_t> files_in_db;
ASSERT_OK(GetAllSSTFiles(&files_in_db, &total_file_size));
auto files_in_db = GetAllSSTFiles(&total_file_size);
// Set the maximum allowed space usage to the current total size
sfm->SetMaxAllowedSpaceUsage(2 * total_file_size + 1);
@@ -854,8 +849,7 @@ TEST_F(DBSSTTest, CancellingManualCompactionsWorks) {
}
ASSERT_OK(Flush());
uint64_t total_file_size = 0;
std::unordered_map<std::string, uint64_t> files_in_db;
ASSERT_OK(GetAllSSTFiles(&files_in_db, &total_file_size));
auto files_in_db = GetAllSSTFiles(&total_file_size);
// Set the maximum allowed space usage to the current total size
sfm->SetMaxAllowedSpaceUsage(2 * total_file_size + 1);
@@ -965,8 +959,7 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowedRandomized) {
}
ASSERT_TRUE(bg_error_set);
uint64_t total_sst_files_size = 0;
std::unordered_map<std::string, uint64_t> files_in_db;
ASSERT_OK(GetAllSSTFiles(&files_in_db, &total_sst_files_size));
GetAllSSTFiles(&total_sst_files_size);
ASSERT_GE(total_sst_files_size, limit_mb * 1024 * 1024);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
+7 -13
View File
@@ -55,7 +55,9 @@
#include "rocksdb/utilities/checkpoint.h"
#include "rocksdb/utilities/optimistic_transaction_db.h"
#include "rocksdb/utilities/write_batch_with_index.h"
#include "table/block_based/block_based_table_factory.h"
#include "table/mock_table.h"
#include "table/plain/plain_table_factory.h"
#include "table/scoped_arena_iterator.h"
#include "test_util/sync_point.h"
#include "test_util/testharness.h"
@@ -3840,7 +3842,7 @@ TEST_F(DBTest, TableOptionsSanitizeTest) {
DestroyAndReopen(options);
ASSERT_EQ(db_->GetOptions().allow_mmap_reads, false);
options.table_factory.reset(NewPlainTableFactory());
options.table_factory.reset(new PlainTableFactory());
options.prefix_extractor.reset(NewNoopTransform());
Destroy(options);
ASSERT_TRUE(!TryReopen(options).IsNotSupported());
@@ -5156,13 +5158,12 @@ TEST_F(DBTest, FileCreationRandomFailure) {
DestroyAndReopen(options);
Random rnd(301);
constexpr int kCDTKeysPerBuffer = 4;
constexpr int kTestSize = kCDTKeysPerBuffer * 4096;
constexpr int kTotalIteration = 20;
const int kCDTKeysPerBuffer = 4;
const int kTestSize = kCDTKeysPerBuffer * 4096;
const int kTotalIteration = 100;
// the second half of the test involves in random failure
// of file creation.
constexpr int kRandomFailureTest = kTotalIteration / 2;
const int kRandomFailureTest = kTotalIteration / 2;
std::vector<std::string> values;
for (int i = 0; i < kTestSize; ++i) {
values.push_back("NOT_FOUND");
@@ -5292,13 +5293,6 @@ TEST_F(DBTest, DynamicMiscOptions) {
ASSERT_OK(dbfull()->TEST_GetLatestMutableCFOptions(handles_[1],
&mutable_cf_options));
ASSERT_TRUE(mutable_cf_options.report_bg_io_stats);
ASSERT_TRUE(mutable_cf_options.check_flush_compaction_key_order);
ASSERT_OK(dbfull()->SetOptions(
handles_[1], {{"check_flush_compaction_key_order", "false"}}));
ASSERT_OK(dbfull()->TEST_GetLatestMutableCFOptions(handles_[1],
&mutable_cf_options));
ASSERT_FALSE(mutable_cf_options.check_flush_compaction_key_order);
}
#endif // ROCKSDB_LITE
+7 -7
View File
@@ -2967,11 +2967,11 @@ TEST_F(DBTest2, OptimizeForSmallDB) {
options.OptimizeForSmallDb();
// Find the cache object
ASSERT_TRUE(options.table_factory->IsInstanceOf(
TableFactory::kBlockBasedTableName()));
auto table_options =
options.table_factory->GetOptions<BlockBasedTableOptions>();
ASSERT_EQ(std::string(BlockBasedTableFactory::kName),
std::string(options.table_factory->Name()));
BlockBasedTableOptions* table_options =
reinterpret_cast<BlockBasedTableOptions*>(
options.table_factory->GetOptions());
ASSERT_TRUE(table_options != nullptr);
std::shared_ptr<Cache> cache = table_options->block_cache;
@@ -4376,8 +4376,8 @@ class DummyOldStats : public Statistics {
}
bool HistEnabledForType(uint32_t /*type*/) const override { return false; }
std::string ToString() const override { return ""; }
std::atomic<int> num_rt{0};
std::atomic<int> num_mt{0};
int num_rt = 0;
int num_mt = 0;
};
} // namespace
+35 -40
View File
@@ -10,7 +10,6 @@
#include "db/db_test_util.h"
#include "db/forward_iterator.h"
#include "rocksdb/convenience.h"
#include "rocksdb/env_encryption.h"
#include "rocksdb/utilities/object_registry.h"
#include "util/random.h"
@@ -54,6 +53,10 @@ SpecialEnv::SpecialEnv(Env* base, bool time_elapse_only_sleep)
non_writable_count_ = 0;
table_write_callback_ = nullptr;
}
#ifndef ROCKSDB_LITE
ROT13BlockCipher rot13Cipher_(16);
#endif // ROCKSDB_LITE
DBTestBase::DBTestBase(const std::string path, bool env_do_fsync)
: mem_env_(nullptr), encrypted_env_(nullptr), option_config_(kDefault) {
Env* base_env = Env::Default();
@@ -73,11 +76,8 @@ DBTestBase::DBTestBase(const std::string path, bool env_do_fsync)
}
#ifndef ROCKSDB_LITE
if (getenv("ENCRYPTED_ENV")) {
std::shared_ptr<EncryptionProvider> provider;
Status s = EncryptionProvider::CreateFromString(
ConfigOptions(), std::string("test://") + getenv("ENCRYPTED_ENV"),
&provider);
encrypted_env_ = NewEncryptedEnv(mem_env_ ? mem_env_ : base_env, provider);
encrypted_env_ = NewEncryptedEnv(mem_env_ ? mem_env_ : base_env,
new CTREncryptionProvider(rot13Cipher_));
}
#endif // !ROCKSDB_LITE
env_ = new SpecialEnv(encrypted_env_ ? encrypted_env_
@@ -666,7 +666,7 @@ void DBTestBase::Reopen(const Options& options) {
void DBTestBase::Close() {
for (auto h : handles_) {
EXPECT_OK(db_->DestroyColumnFamilyHandle(h));
db_->DestroyColumnFamilyHandle(h);
}
handles_.clear();
delete db_;
@@ -684,8 +684,7 @@ void DBTestBase::Destroy(const Options& options, bool delete_cf_paths) {
if (delete_cf_paths) {
for (size_t i = 0; i < handles_.size(); ++i) {
ColumnFamilyDescriptor cfdescriptor;
// GetDescriptor is not implemented for ROCKSDB_LITE
handles_[i]->GetDescriptor(&cfdescriptor).PermitUncheckedError();
handles_[i]->GetDescriptor(&cfdescriptor);
column_families.push_back(cfdescriptor);
}
}
@@ -963,7 +962,7 @@ std::string DBTestBase::AllEntriesFor(const Slice& user_key, int cf) {
bool first = true;
while (iter->Valid()) {
ParsedInternalKey ikey(Slice(), 0, kTypeValue);
if (ParseInternalKey(iter->key(), &ikey) != Status::OK()) {
if (!ParseInternalKey(iter->key(), &ikey)) {
result += "CORRUPTED";
} else {
if (!last_options_.comparator->Equal(ikey.user_key, user_key)) {
@@ -1183,9 +1182,9 @@ void DBTestBase::FillLevels(const std::string& smallest,
void DBTestBase::MoveFilesToLevel(int level, int cf) {
for (int l = 0; l < level; ++l) {
if (cf > 0) {
EXPECT_OK(dbfull()->TEST_CompactRange(l, nullptr, nullptr, handles_[cf]));
dbfull()->TEST_CompactRange(l, nullptr, nullptr, handles_[cf]);
} else {
EXPECT_OK(dbfull()->TEST_CompactRange(l, nullptr, nullptr));
dbfull()->TEST_CompactRange(l, nullptr, nullptr);
}
}
}
@@ -1212,7 +1211,7 @@ std::string DBTestBase::DumpSSTableList() {
void DBTestBase::GetSstFiles(Env* env, std::string path,
std::vector<std::string>* files) {
EXPECT_OK(env->GetChildren(path, files));
env->GetChildren(path, files);
files->erase(
std::remove_if(files->begin(), files->end(), [](std::string name) {
@@ -1236,8 +1235,8 @@ void DBTestBase::GenerateNewFile(int cf, Random* rnd, int* key_idx,
(*key_idx)++;
}
if (!nowait) {
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
}
}
@@ -1248,8 +1247,8 @@ void DBTestBase::GenerateNewFile(Random* rnd, int* key_idx, bool nowait) {
(*key_idx)++;
}
if (!nowait) {
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
}
}
@@ -1375,7 +1374,7 @@ void DBTestBase::validateNumberOfEntries(int numValues, int cf) {
while (iter->Valid()) {
ParsedInternalKey ikey;
ikey.clear();
ASSERT_OK(ParseInternalKey(iter->key(), &ikey));
ASSERT_EQ(ParseInternalKey(iter->key(), &ikey), true);
// checks sequence number for updates
ASSERT_EQ(ikey.sequence, (unsigned)seq--);
@@ -1408,40 +1407,36 @@ void DBTestBase::CopyFile(const std::string& source,
ASSERT_OK(destfile->Close());
}
Status DBTestBase::GetAllSSTFiles(
std::unordered_map<std::string, uint64_t>* sst_files,
uint64_t* total_size /* = nullptr */) {
std::unordered_map<std::string, uint64_t> DBTestBase::GetAllSSTFiles(
uint64_t* total_size) {
std::unordered_map<std::string, uint64_t> res;
if (total_size) {
*total_size = 0;
}
std::vector<std::string> files;
Status s = env_->GetChildren(dbname_, &files);
if (s.ok()) {
for (auto& file_name : files) {
uint64_t number;
FileType type;
if (ParseFileName(file_name, &number, &type) && type == kTableFile) {
std::string file_path = dbname_ + "/" + file_name;
uint64_t file_size = 0;
s = env_->GetFileSize(file_path, &file_size);
if (!s.ok()) {
break;
}
(*sst_files)[file_path] = file_size;
if (total_size) {
*total_size += file_size;
}
env_->GetChildren(dbname_, &files);
for (auto& file_name : files) {
uint64_t number;
FileType type;
std::string file_path = dbname_ + "/" + file_name;
if (ParseFileName(file_name, &number, &type) && type == kTableFile) {
uint64_t file_size = 0;
env_->GetFileSize(file_path, &file_size);
res[file_path] = file_size;
if (total_size) {
*total_size += file_size;
}
}
}
return s;
return res;
}
std::vector<std::uint64_t> DBTestBase::ListTableFiles(Env* env,
const std::string& path) {
std::vector<std::string> files;
std::vector<uint64_t> file_numbers;
EXPECT_OK(env->GetChildren(path, &files));
env->GetChildren(path, &files);
uint64_t number;
FileType type;
for (size_t i = 0; i < files.size(); ++i) {
@@ -1580,7 +1575,7 @@ void DBTestBase::VerifyDBInternal(
for (auto p : true_data) {
ASSERT_TRUE(iter->Valid());
ParsedInternalKey ikey;
ASSERT_OK(ParseInternalKey(iter->key(), &ikey));
ASSERT_TRUE(ParseInternalKey(iter->key(), &ikey));
ASSERT_EQ(p.first, ikey.user_key);
ASSERT_EQ(p.second, iter->value());
iter->Next();
+4 -2
View File
@@ -38,7 +38,9 @@
#include "rocksdb/statistics.h"
#include "rocksdb/table.h"
#include "rocksdb/utilities/checkpoint.h"
#include "table/block_based/block_based_table_factory.h"
#include "table/mock_table.h"
#include "table/plain/plain_table_factory.h"
#include "table/scoped_arena_iterator.h"
#include "test_util/mock_time_env.h"
#include "test_util/sync_point.h"
@@ -1145,8 +1147,8 @@ class DBTestBase : public testing::Test {
void CopyFile(const std::string& source, const std::string& destination,
uint64_t size = 0);
Status GetAllSSTFiles(std::unordered_map<std::string, uint64_t>* sst_files,
uint64_t* total_size = nullptr);
std::unordered_map<std::string, uint64_t> GetAllSSTFiles(
uint64_t* total_size = nullptr);
std::vector<std::uint64_t> ListTableFiles(Env* env, const std::string& path);
+111 -116
View File
@@ -27,7 +27,7 @@ class DBTestUniversalCompactionBase
public ::testing::WithParamInterface<std::tuple<int, bool>> {
public:
explicit DBTestUniversalCompactionBase(const std::string& path)
: DBTestBase(path, /*env_do_fsync=*/false) {}
: DBTestBase(path, /*env_do_fsync=*/true) {}
void SetUp() override {
num_levels_ = std::get<0>(GetParam());
exclusive_manual_compaction_ = std::get<1>(GetParam());
@@ -45,7 +45,7 @@ class DBTestUniversalCompaction : public DBTestUniversalCompactionBase {
class DBTestUniversalCompaction2 : public DBTestBase {
public:
DBTestUniversalCompaction2()
: DBTestBase("/db_universal_compaction_test2", /*env_do_fsync=*/false) {}
: DBTestBase("/db_universal_compaction_test2", /*env_do_fsync=*/true) {}
};
namespace {
@@ -126,11 +126,11 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionSingleSortedRun) {
for (int num = 0; num < 16; num++) {
// Write 100KB file. And immediately it should be compacted to one file.
GenerateNewFile(&rnd, &key_idx);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(NumSortedRuns(0), 1);
}
ASSERT_OK(Put(Key(key_idx), ""));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(NumSortedRuns(0), 1);
}
@@ -162,15 +162,15 @@ TEST_P(DBTestUniversalCompaction, OptimizeFiltersForHits) {
Env::Priority::LOW);
for (int num = 0; num < options.level0_file_num_compaction_trigger; num++) {
ASSERT_OK(Put(Key(num * 10), "val"));
Put(Key(num * 10), "val");
if (num) {
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
}
ASSERT_OK(Put(Key(30 + num * 10), "val"));
ASSERT_OK(Put(Key(60 + num * 10), "val"));
Put(Key(30 + num * 10), "val");
Put(Key(60 + num * 10), "val");
}
ASSERT_OK(Put("", ""));
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
Put("", "");
dbfull()->TEST_WaitForFlushMemTable();
// Query set of non existing keys
for (int i = 5; i < 90; i += 10) {
@@ -190,7 +190,7 @@ TEST_P(DBTestUniversalCompaction, OptimizeFiltersForHits) {
// Unblock compaction and wait it for happening.
sleeping_task_low.WakeUp();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// The same queries will not trigger bloom filter
for (int i = 5; i < 90; i += 10) {
@@ -294,7 +294,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionTrigger) {
// Now we have 3 files at level 0, with size 4, 2.4, 2. Let's generate a
// new file of size 1.
GenerateNewFile(1, &rnd, &key_idx);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Level-0 compaction is triggered, but no file will be picked up.
ASSERT_EQ(NumSortedRuns(1), 4);
@@ -303,7 +303,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionTrigger) {
// a new file of size 1.
filter->expect_full_compaction_.store(true);
GenerateNewFile(1, &rnd, &key_idx);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// All files at level 0 will be compacted into a single one.
ASSERT_EQ(NumSortedRuns(1), 1);
@@ -336,7 +336,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionSizeAmplification) {
ASSERT_OK(Put(1, Key(key_idx), rnd.RandomString(10000)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[1]));
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
ASSERT_EQ(NumSortedRuns(1), num + 1);
}
ASSERT_EQ(NumSortedRuns(1), 2);
@@ -346,7 +346,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionSizeAmplification) {
// but will instead trigger size amplification.
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Verify that size amplification did occur
ASSERT_EQ(NumSortedRuns(1), 1);
@@ -394,7 +394,7 @@ TEST_P(DBTestUniversalCompaction, DynamicUniversalCompactionSizeAmplification) {
ASSERT_OK(Put(1, Key(key_idx), rnd.RandomString(10000)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[1]));
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
ASSERT_EQ(NumSortedRuns(1), num + 1);
}
ASSERT_EQ(NumSortedRuns(1), 2);
@@ -404,7 +404,7 @@ TEST_P(DBTestUniversalCompaction, DynamicUniversalCompactionSizeAmplification) {
// but could instead trigger size amplification if it's set
// to 110.
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Verify compaction did not happen
ASSERT_EQ(NumSortedRuns(1), 3);
@@ -425,7 +425,7 @@ TEST_P(DBTestUniversalCompaction, DynamicUniversalCompactionSizeAmplification) {
ASSERT_EQ(110u, mutable_cf_options.compaction_options_universal
.max_size_amplification_percent);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Verify that size amplification did happen
ASSERT_EQ(NumSortedRuns(1), 1);
ASSERT_EQ(total_picked_compactions, 1);
@@ -473,7 +473,7 @@ TEST_P(DBTestUniversalCompaction, DynamicUniversalCompactionReadAmplification) {
ASSERT_OK(Put(1, Key(key_idx), rnd.RandomString(10000)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[1]));
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
ASSERT_EQ(NumSortedRuns(1), num + 1);
}
ASSERT_EQ(NumSortedRuns(1), options.level0_file_num_compaction_trigger);
@@ -481,7 +481,7 @@ TEST_P(DBTestUniversalCompaction, DynamicUniversalCompactionReadAmplification) {
// Flush whatever is remaining in memtable. This is typically small, about
// 30KB.
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Verify compaction did not happen
ASSERT_EQ(NumSortedRuns(1), options.level0_file_num_compaction_trigger + 1);
ASSERT_EQ(total_picked_compactions, 0);
@@ -510,7 +510,7 @@ TEST_P(DBTestUniversalCompaction, DynamicUniversalCompactionReadAmplification) {
ASSERT_EQ(mutable_cf_options.compaction_options_universal.max_merge_width,
2u);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Files in L0 are approx: 0.3 (30KB), 1, 1, 1.
// On compaction: the files are below the size amp threshold, so we
@@ -550,8 +550,8 @@ TEST_P(DBTestUniversalCompaction, CompactFilesOnUniversalCompaction) {
for (int key = 1024 * kEntriesPerBuffer; key >= 0; --key) {
ASSERT_OK(Put(1, ToString(key), rnd.RandomString(kTestValueSize)));
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[1]));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
dbfull()->TEST_WaitForCompact();
ColumnFamilyMetaData cf_meta;
dbfull()->GetColumnFamilyMetaData(handles_[1], &cf_meta);
std::vector<std::string> compaction_input_file_names;
@@ -631,7 +631,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionTargetLevel) {
compact_options.change_level = true;
compact_options.target_level = 4;
compact_options.exclusive_manual_compaction = exclusive_manual_compaction_;
ASSERT_OK(db_->CompactRange(compact_options, nullptr, nullptr));
db_->CompactRange(compact_options, nullptr, nullptr);
ASSERT_EQ("0,0,0,0,1", FilesPerLevel(0));
}
@@ -665,7 +665,7 @@ TEST_P(DBTestUniversalCompactionMultiLevels, UniversalCompactionMultiLevels) {
ASSERT_OK(Put(1, Key(i % num_keys), Key(i)));
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
for (int i = num_keys; i < num_keys * 2; i++) {
ASSERT_EQ(Get(1, Key(i % num_keys)), Key(i));
@@ -712,7 +712,7 @@ TEST_P(DBTestUniversalCompactionMultiLevels, UniversalCompactionTrivialMove) {
std::vector<std::string> values;
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_GT(trivial_move, 0);
ASSERT_GT(non_trivial_move, 0);
@@ -775,7 +775,7 @@ TEST_P(DBTestUniversalCompactionParallel, UniversalCompactionParallel) {
for (int i = 0; i < num_keys * 2; i++) {
ASSERT_OK(Put(1, Key(i % num_keys), Key(i)));
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_EQ(num_compactions_running.load(), 0);
@@ -853,7 +853,7 @@ TEST_P(DBTestUniversalCompactionParallel, PickByFileNumberBug) {
// Hold the 1st compaction from finishing
TEST_SYNC_POINT("DBTestUniversalCompactionParallel::PickByFileNumberBug:2");
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// There should only be one picked compaction as the score drops below one
// after the first one is picked.
@@ -901,7 +901,7 @@ TEST_P(DBTestUniversalCompactionParallel, PickByFileNumberBug) {
// Hold the 1st and 2nd compaction from finishing
TEST_SYNC_POINT("DBTestUniversalCompactionParallel::PickByFileNumberBug:2");
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// This time we will trigger a compaction because of size ratio and
// another compaction because of number of files that are not compacted
@@ -935,14 +935,14 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionOptions) {
ASSERT_OK(Put(1, Key(key_idx), rnd.RandomString(990)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[1]));
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
if (num < options.level0_file_num_compaction_trigger - 1) {
ASSERT_EQ(NumSortedRuns(1), num + 1);
}
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(NumSortedRuns(1), 1);
}
@@ -973,7 +973,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_EQ(NumSortedRuns(), num + 1);
}
@@ -983,7 +983,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Suppose each file flushed from mem table has size 1. Now we compact
// (level0_file_num_compaction_trigger+1)=4 files and should have a big
// file of size 4.
@@ -1004,7 +1004,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_EQ(NumSortedRuns(), num + 3);
}
@@ -1014,7 +1014,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Before compaction, we have 4 files at level 0, with size 4, 0.4, 1, 1.
// After compaction, we should have 3 files, with size 4, 0.4, 2.
ASSERT_EQ(NumSortedRuns(), 3);
@@ -1025,7 +1025,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Level-0 compaction is triggered, but no file will be picked up.
ASSERT_EQ(NumSortedRuns(), 4);
}
@@ -1054,8 +1054,8 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionCompressRatio1) {
ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
}
ASSERT_LT(TotalSize(), 110000U * 2 * 0.9);
@@ -1066,8 +1066,8 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionCompressRatio1) {
ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
}
ASSERT_LT(TotalSize(), 110000 * 4 * 0.9);
@@ -1079,8 +1079,8 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionCompressRatio1) {
ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
}
ASSERT_LT(TotalSize(), 110000 * 6 * 0.9);
@@ -1092,8 +1092,8 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionCompressRatio1) {
ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
}
ASSERT_GT(TotalSize(), 110000 * 11 * 0.8 + 110000 * 2);
}
@@ -1122,8 +1122,8 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionCompressRatio2) {
ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
key_idx++;
}
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
}
ASSERT_LT(TotalSize(), 120000U * 12 * 0.82 + 120000 * 2);
}
@@ -1169,7 +1169,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionTrivialMoveTest1) {
std::vector<std::string> values;
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_GT(trivial_move, 0);
ASSERT_GT(non_trivial_move, 0);
@@ -1215,7 +1215,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionTrivialMoveTest2) {
std::vector<std::string> values;
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_GT(trivial_move, 0);
@@ -1239,14 +1239,12 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionFourPaths) {
options.num_levels = 1;
std::vector<std::string> filenames;
if (env_->GetChildren(options.db_paths[1].path, &filenames).ok()) {
// Delete archival files.
for (size_t i = 0; i < filenames.size(); ++i) {
ASSERT_OK(
env_->DeleteFile(options.db_paths[1].path + "/" + filenames[i]));
}
ASSERT_OK(env_->DeleteDir(options.db_paths[1].path));
env_->GetChildren(options.db_paths[1].path, &filenames);
// Delete archival files.
for (size_t i = 0; i < filenames.size(); ++i) {
env_->DeleteFile(options.db_paths[1].path + "/" + filenames[i]);
}
env_->DeleteDir(options.db_paths[1].path);
Reopen(options);
Random rnd(301);
@@ -1505,11 +1503,11 @@ TEST_P(DBTestUniversalCompaction, IncreaseUniversalCompactionNumLevels) {
for (int i = 0; i <= max_key1; i++) {
// each value is 10K
ASSERT_OK(Put(1, Key(i), rnd.RandomString(10000)));
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[1]));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
dbfull()->TEST_WaitForCompact();
}
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Stage 2: reopen with universal compaction, num_levels=4
options.compaction_style = kCompactionStyleUniversal;
@@ -1523,11 +1521,11 @@ TEST_P(DBTestUniversalCompaction, IncreaseUniversalCompactionNumLevels) {
for (int i = max_key1 + 1; i <= max_key2; i++) {
// each value is 10K
ASSERT_OK(Put(1, Key(i), rnd.RandomString(10000)));
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[1]));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
dbfull()->TEST_WaitForCompact();
}
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
verify_func(max_key2);
// Compaction to non-L0 has happened.
@@ -1542,8 +1540,7 @@ TEST_P(DBTestUniversalCompaction, IncreaseUniversalCompactionNumLevels) {
compact_options.change_level = true;
compact_options.target_level = 0;
compact_options.exclusive_manual_compaction = exclusive_manual_compaction_;
ASSERT_OK(
dbfull()->CompactRange(compact_options, handles_[1], nullptr, nullptr));
dbfull()->CompactRange(compact_options, handles_[1], nullptr, nullptr);
// Need to restart it once to remove higher level records in manifest.
ReopenWithColumnFamilies({"default", "pikachu"}, options);
// Final reopen
@@ -1556,11 +1553,11 @@ TEST_P(DBTestUniversalCompaction, IncreaseUniversalCompactionNumLevels) {
for (int i = max_key2 + 1; i <= max_key3; i++) {
// each value is 10K
ASSERT_OK(Put(1, Key(i), rnd.RandomString(10000)));
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[1]));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
dbfull()->TEST_WaitForCompact();
}
ASSERT_OK(Flush(1));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
verify_func(max_key3);
}
@@ -1582,14 +1579,12 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionSecondPathRatio) {
new SpecialSkipListFactory(KNumKeysByGenerateNewFile - 1));
std::vector<std::string> filenames;
if (env_->GetChildren(options.db_paths[1].path, &filenames).ok()) {
// Delete archival files.
for (size_t i = 0; i < filenames.size(); ++i) {
ASSERT_OK(
env_->DeleteFile(options.db_paths[1].path + "/" + filenames[i]));
}
ASSERT_OK(env_->DeleteDir(options.db_paths[1].path));
env_->GetChildren(options.db_paths[1].path, &filenames);
// Delete archival files.
for (size_t i = 0; i < filenames.size(); ++i) {
env_->DeleteFile(options.db_paths[1].path + "/" + filenames[i]);
}
env_->DeleteDir(options.db_paths[1].path);
Reopen(options);
Random rnd(301);
@@ -1704,14 +1699,14 @@ TEST_P(DBTestUniversalCompaction, ConcurrentBottomPriLowPriCompactions) {
// use no_wait above because that one waits for flush and compaction. We
// don't want to wait for compaction because the full compaction is
// intentionally blocked while more files are flushed.
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
dbfull()->TEST_WaitForFlushMemTable();
}
if (i == 0) {
TEST_SYNC_POINT(
"DBTestUniversalCompaction:ConcurrentBottomPriLowPriCompactions:0");
}
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// First compaction should output to bottom level. Second should output to L0
// since older L0 files pending compaction prevent it from being placed lower.
@@ -1750,7 +1745,7 @@ TEST_P(DBTestUniversalCompaction, RecalculateScoreAfterPicking) {
int key_idx = 0;
GenerateNewFile(&rnd, &key_idx);
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
// Compacting the first four files was enough to bring the score below one so
// there's no need to schedule any more compactions.
ASSERT_EQ(1, num_compactions_attempted);
@@ -1780,9 +1775,9 @@ TEST_P(DBTestUniversalCompaction, FinalSortedRunCompactFilesConflict) {
auto stop_token =
dbfull()->TEST_write_controler().GetCompactionPressureToken();
ASSERT_OK(Put("key", "val"));
Put("key", "val");
Flush();
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_EQ(NumTableFilesAtLevel(num_levels_ - 1), 1);
ColumnFamilyMetaData cf_meta;
ColumnFamilyHandle* default_cfh = db_->DefaultColumnFamily();
@@ -1806,10 +1801,10 @@ TEST_P(DBTestUniversalCompaction, FinalSortedRunCompactFilesConflict) {
TEST_SYNC_POINT(
"DBTestUniversalCompaction:FinalSortedRunCompactFilesConflict:0");
for (int i = 0; i < 2; ++i) {
ASSERT_OK(Put("key", "val"));
Put("key", "val");
Flush();
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
compact_files_thread.join();
}
@@ -1840,7 +1835,7 @@ TEST_P(DBTestUniversalManualCompactionOutputPathId,
DestroyAndReopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
MakeTables(3, "p", "q", 1);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(2, TotalLiveFiles(1));
ASSERT_EQ(2, GetSstFileCount(options.db_paths[0].path));
ASSERT_EQ(0, GetSstFileCount(options.db_paths[1].path));
@@ -1849,7 +1844,7 @@ TEST_P(DBTestUniversalManualCompactionOutputPathId,
CompactRangeOptions compact_options;
compact_options.target_path_id = 1;
compact_options.exclusive_manual_compaction = exclusive_manual_compaction_;
ASSERT_OK(db_->CompactRange(compact_options, handles_[1], nullptr, nullptr));
db_->CompactRange(compact_options, handles_[1], nullptr, nullptr);
ASSERT_EQ(1, TotalLiveFiles(1));
ASSERT_EQ(0, GetSstFileCount(options.db_paths[0].path));
ASSERT_EQ(1, GetSstFileCount(options.db_paths[1].path));
@@ -1872,7 +1867,7 @@ TEST_P(DBTestUniversalManualCompactionOutputPathId,
// Full compaction to DB path 0
compact_options.target_path_id = 0;
compact_options.exclusive_manual_compaction = exclusive_manual_compaction_;
ASSERT_OK(db_->CompactRange(compact_options, handles_[1], nullptr, nullptr));
db_->CompactRange(compact_options, handles_[1], nullptr, nullptr);
ASSERT_EQ(1, TotalLiveFiles(1));
ASSERT_EQ(1, GetSstFileCount(options.db_paths[0].path));
ASSERT_EQ(0, GetSstFileCount(options.db_paths[1].path));
@@ -1909,23 +1904,23 @@ TEST_F(DBTestUniversalCompaction2, BasicL0toL1) {
// during flush
int i;
for (i = 0; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
// MoveFilesToLevel(6);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
for (i = 1999; i < kNumKeys; ++i) {
if (i >= kNumKeys - kWindowSize &&
i < kNumKeys - kWindowSize + kNumDelsTrigger) {
ASSERT_OK(Delete(Key(i)));
Delete(Key(i));
} else {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_GT(NumTableFilesAtLevel(6), 0);
}
@@ -1952,21 +1947,21 @@ TEST_F(DBTestUniversalCompaction2, SingleLevel) {
// during flush
int i;
for (i = 0; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
for (i = 1999; i < kNumKeys; ++i) {
if (i >= kNumKeys - kWindowSize &&
i < kNumKeys - kWindowSize + kNumDelsTrigger) {
ASSERT_OK(Delete(Key(i)));
Delete(Key(i));
} else {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(1, NumTableFilesAtLevel(0));
}
#endif // ENABLE_SINGLE_LEVEL_DTC
@@ -1990,50 +1985,50 @@ TEST_F(DBTestUniversalCompaction2, MultipleLevels) {
// during flush
int i;
for (i = 0; i < 500; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
for (i = 500; i < 1000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
for (i = 1000; i < 1500; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
for (i = 1500; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_GT(NumTableFilesAtLevel(6), 0);
for (i = 1999; i < 2333; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
for (i = 2333; i < 2666; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
for (i = 2666; i < 2999; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_GT(NumTableFilesAtLevel(6), 0);
ASSERT_GT(NumTableFilesAtLevel(5), 0);
for (i = 1900; i < 2100; ++i) {
ASSERT_OK(Delete(Key(i)));
Delete(Key(i));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_EQ(0, NumTableFilesAtLevel(1));
ASSERT_EQ(0, NumTableFilesAtLevel(2));
@@ -2062,23 +2057,23 @@ TEST_F(DBTestUniversalCompaction2, OverlappingL0) {
// during flush
int i;
for (i = 0; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
for (i = 2000; i < 3000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
for (i = 3500; i < 4000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
for (i = 2900; i < 3100; ++i) {
ASSERT_OK(Delete(Key(i)));
Delete(Key(i));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(2, NumTableFilesAtLevel(0));
ASSERT_GT(NumTableFilesAtLevel(6), 0);
}
@@ -2104,23 +2099,23 @@ TEST_F(DBTestUniversalCompaction2, IngestBehind) {
// during flush
int i;
for (i = 0; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
Flush();
// MoveFilesToLevel(6);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
for (i = 1999; i < kNumKeys; ++i) {
if (i >= kNumKeys - kWindowSize &&
i < kNumKeys - kWindowSize + kNumDelsTrigger) {
ASSERT_OK(Delete(Key(i)));
Delete(Key(i));
} else {
ASSERT_OK(Put(Key(i), "val"));
Put(Key(i), "val");
}
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_EQ(0, NumTableFilesAtLevel(6));
ASSERT_GT(NumTableFilesAtLevel(5), 0);
@@ -2193,7 +2188,7 @@ TEST_F(DBTestUniversalCompaction2, PeriodicCompaction) {
// Another flush would trigger compaction the oldest file.
ASSERT_OK(Put("foo", "bar2"));
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(1, periodic_compactions);
ASSERT_EQ(0, start_level);
@@ -2204,7 +2199,7 @@ TEST_F(DBTestUniversalCompaction2, PeriodicCompaction) {
// A flush doesn't trigger a periodic compaction when threshold not hit
ASSERT_OK(Put("foo", "bar2"));
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(0, periodic_compactions);
// After periodic compaction threshold hits, a flush will trigger
@@ -2212,7 +2207,7 @@ TEST_F(DBTestUniversalCompaction2, PeriodicCompaction) {
ASSERT_OK(Put("foo", "bar2"));
env_->MockSleepForSeconds(48 * 60 * 60 + 100);
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(1, periodic_compactions);
ASSERT_EQ(0, start_level);
ASSERT_EQ(4, output_level);
-157
View File
@@ -338,163 +338,6 @@ TEST_F(DBWALTest, RecoverWithTableHandle) {
} while (ChangeWalOptions());
}
TEST_F(DBWALTest, RecoverWithBlob) {
// Write a value that's below the prospective size limit for blobs and another
// one that's above. Note that blob files are not actually enabled at this
// point.
constexpr uint64_t min_blob_size = 10;
constexpr char short_value[] = "short";
static_assert(sizeof(short_value) - 1 < min_blob_size,
"short_value too long");
constexpr char long_value[] = "long_value";
static_assert(sizeof(long_value) - 1 >= min_blob_size,
"long_value too short");
ASSERT_OK(Put("key1", short_value));
ASSERT_OK(Put("key2", long_value));
// There should be no files just yet since we haven't flushed.
{
VersionSet* const versions = dbfull()->TEST_GetVersionSet();
assert(versions);
ColumnFamilyData* const cfd = versions->GetColumnFamilySet()->GetDefault();
assert(cfd);
Version* const current = cfd->current();
assert(current);
const VersionStorageInfo* const storage_info = current->storage_info();
assert(storage_info);
ASSERT_EQ(storage_info->num_non_empty_levels(), 0);
ASSERT_TRUE(storage_info->GetBlobFiles().empty());
}
// Reopen the database with blob files enabled. A new table file/blob file
// pair should be written during recovery.
Options options;
options.enable_blob_files = true;
options.min_blob_size = min_blob_size;
options.avoid_flush_during_recovery = false;
options.disable_auto_compactions = true;
Reopen(options);
ASSERT_EQ(Get("key1"), short_value);
// TODO: enable once Get support is implemented for blobs
// ASSERT_EQ(Get("key2"), long_value);
VersionSet* const versions = dbfull()->TEST_GetVersionSet();
assert(versions);
ColumnFamilyData* const cfd = versions->GetColumnFamilySet()->GetDefault();
assert(cfd);
Version* const current = cfd->current();
assert(current);
const VersionStorageInfo* const storage_info = current->storage_info();
assert(storage_info);
const auto& l0_files = storage_info->LevelFiles(0);
ASSERT_EQ(l0_files.size(), 1);
const FileMetaData* const table_file = l0_files[0];
assert(table_file);
const auto& blob_files = storage_info->GetBlobFiles();
ASSERT_EQ(blob_files.size(), 1);
const auto& blob_file = blob_files.begin()->second;
assert(blob_file);
ASSERT_EQ(table_file->smallest.user_key(), "key1");
ASSERT_EQ(table_file->largest.user_key(), "key2");
ASSERT_EQ(table_file->fd.smallest_seqno, 1);
ASSERT_EQ(table_file->fd.largest_seqno, 2);
ASSERT_EQ(table_file->oldest_blob_file_number,
blob_file->GetBlobFileNumber());
ASSERT_EQ(blob_file->GetTotalBlobCount(), 1);
#ifndef ROCKSDB_LITE
const InternalStats* const internal_stats = cfd->internal_stats();
assert(internal_stats);
const uint64_t expected_bytes =
table_file->fd.GetFileSize() + blob_file->GetTotalBlobBytes();
const auto& compaction_stats = internal_stats->TEST_GetCompactionStats();
ASSERT_FALSE(compaction_stats.empty());
ASSERT_EQ(compaction_stats[0].bytes_written, expected_bytes);
ASSERT_EQ(compaction_stats[0].num_output_files, 2);
const uint64_t* const cf_stats_value = internal_stats->TEST_GetCFStatsValue();
ASSERT_EQ(cf_stats_value[InternalStats::BYTES_FLUSHED], expected_bytes);
#endif // ROCKSDB_LITE
}
class DBRecoveryTestBlobError
: public DBWALTest,
public testing::WithParamInterface<std::string> {
public:
DBRecoveryTestBlobError() : fault_injection_env_(env_) {}
~DBRecoveryTestBlobError() { Close(); }
FaultInjectionTestEnv fault_injection_env_;
};
INSTANTIATE_TEST_CASE_P(DBRecoveryTestBlobError, DBRecoveryTestBlobError,
::testing::ValuesIn(std::vector<std::string>{
"BlobFileBuilder::WriteBlobToFile:AddRecord",
"BlobFileBuilder::WriteBlobToFile:AppendFooter"}));
TEST_P(DBRecoveryTestBlobError, RecoverWithBlobError) {
// Write a value. Note that blob files are not actually enabled at this point.
ASSERT_OK(Put("key", "blob"));
// Reopen with blob files enabled but make blob file writing fail during
// recovery.
SyncPoint::GetInstance()->SetCallBack(GetParam(), [this](void* /* arg */) {
fault_injection_env_.SetFilesystemActive(false, Status::IOError());
});
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable", [this](void* /* arg */) {
fault_injection_env_.SetFilesystemActive(true);
});
SyncPoint::GetInstance()->EnableProcessing();
Options options;
options.enable_blob_files = true;
options.avoid_flush_during_recovery = false;
options.disable_auto_compactions = true;
options.env = &fault_injection_env_;
ASSERT_NOK(TryReopen(options));
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
// Make sure the files generated by the failed recovery have been deleted.
std::vector<std::string> files;
ASSERT_OK(env_->GetChildren(dbname_, &files));
for (const auto& file : files) {
uint64_t number = 0;
FileType type = kTableFile;
if (!ParseFileName(file, &number, &type)) {
continue;
}
ASSERT_NE(type, kTableFile);
ASSERT_NE(type, kBlobFile);
}
}
TEST_F(DBWALTest, IgnoreRecoveredLog) {
std::string backup_logs = dbname_ + "/backup_logs";
+9 -280
View File
@@ -141,7 +141,7 @@ class DBBasicTestWithTimestampBase : public DBTestBase {
ukey_and_ts.assign(expected_ukey.data(), expected_ukey.size());
ukey_and_ts.append(expected_ts.data(), expected_ts.size());
ParsedInternalKey parsed_ikey;
ASSERT_OK(ParseInternalKey(it->key(), &parsed_ikey));
ASSERT_TRUE(ParseInternalKey(it->key(), &parsed_ikey));
ASSERT_EQ(ukey_and_ts, parsed_ikey.user_key);
ASSERT_EQ(expected_val_type, parsed_ikey.type);
ASSERT_EQ(expected_seq, parsed_ikey.sequence);
@@ -161,7 +161,7 @@ class DBBasicTestWithTimestampBase : public DBTestBase {
ukey_and_ts.append(expected_ts.data(), expected_ts.size());
ParsedInternalKey parsed_ikey;
ASSERT_OK(ParseInternalKey(it->key(), &parsed_ikey));
ASSERT_TRUE(ParseInternalKey(it->key(), &parsed_ikey));
ASSERT_EQ(expected_val_type, parsed_ikey.type);
ASSERT_EQ(Slice(ukey_and_ts), parsed_ikey.user_key);
if (expected_val_type == kTypeValue) {
@@ -468,8 +468,8 @@ TEST_F(DBBasicTestWithTimestamp, ReseekToNextUserKey) {
{
std::string ts_str = Timestamp(static_cast<uint64_t>(kNumKeys + 1), 0);
WriteBatch batch(0, 0, kTimestampSize);
ASSERT_OK(batch.Put("a", "new_value"));
ASSERT_OK(batch.Put("b", "new_value"));
batch.Put("a", "new_value");
batch.Put("b", "new_value");
s = batch.AssignTimestamp(ts_str);
ASSERT_OK(s);
s = db_->Write(write_opts, &batch);
@@ -590,132 +590,8 @@ TEST_F(DBBasicTestWithTimestamp, CompactDeletionWithTimestampMarkerToBottom) {
class DataVisibilityTest : public DBBasicTestWithTimestampBase {
public:
DataVisibilityTest() : DBBasicTestWithTimestampBase("data_visibility_test") {
// Initialize test data
for (int i = 0; i < kTestDataSize; i++) {
test_data_[i].key = "key" + ToString(i);
test_data_[i].value = "value" + ToString(i);
test_data_[i].timestamp = Timestamp(i, 0);
test_data_[i].ts = i;
test_data_[i].seq_num = kMaxSequenceNumber;
}
}
protected:
struct TestData {
std::string key;
std::string value;
int ts;
std::string timestamp;
SequenceNumber seq_num;
};
constexpr static int kTestDataSize = 3;
TestData test_data_[kTestDataSize];
void PutTestData(int index, ColumnFamilyHandle* cfh = nullptr) {
ASSERT_LE(index, kTestDataSize);
WriteOptions write_opts;
Slice ts_slice = test_data_[index].timestamp;
write_opts.timestamp = &ts_slice;
if (cfh == nullptr) {
ASSERT_OK(
db_->Put(write_opts, test_data_[index].key, test_data_[index].value));
const Snapshot* snap = db_->GetSnapshot();
test_data_[index].seq_num = snap->GetSequenceNumber();
if (index > 0) {
ASSERT_GT(test_data_[index].seq_num, test_data_[index - 1].seq_num);
}
db_->ReleaseSnapshot(snap);
} else {
ASSERT_OK(db_->Put(write_opts, cfh, test_data_[index].key,
test_data_[index].value));
}
}
void AssertVisibility(int ts, SequenceNumber seq,
std::vector<Status> statuses) {
ASSERT_EQ(kTestDataSize, statuses.size());
for (int i = 0; i < kTestDataSize; i++) {
if (test_data_[i].seq_num <= seq && test_data_[i].ts <= ts) {
ASSERT_OK(statuses[i]);
} else {
ASSERT_TRUE(statuses[i].IsNotFound());
}
}
}
std::vector<Slice> GetKeys() {
std::vector<Slice> ret(kTestDataSize);
for (int i = 0; i < kTestDataSize; i++) {
ret[i] = test_data_[i].key;
}
return ret;
}
void VerifyDefaultCF(int ts, const Snapshot* snap = nullptr) {
ReadOptions read_opts;
std::string read_ts = Timestamp(ts, 0);
Slice read_ts_slice = read_ts;
read_opts.timestamp = &read_ts_slice;
read_opts.snapshot = snap;
ColumnFamilyHandle* cfh = db_->DefaultColumnFamily();
std::vector<ColumnFamilyHandle*> cfs(kTestDataSize, cfh);
SequenceNumber seq =
snap ? snap->GetSequenceNumber() : kMaxSequenceNumber - 1;
// There're several MultiGet interfaces with not exactly the same
// implementations, query data with all of them.
auto keys = GetKeys();
std::vector<std::string> values;
auto s1 = db_->MultiGet(read_opts, cfs, keys, &values);
AssertVisibility(ts, seq, s1);
auto s2 = db_->MultiGet(read_opts, keys, &values);
AssertVisibility(ts, seq, s2);
std::vector<std::string> timestamps;
auto s3 = db_->MultiGet(read_opts, cfs, keys, &values, &timestamps);
AssertVisibility(ts, seq, s3);
auto s4 = db_->MultiGet(read_opts, keys, &values, &timestamps);
AssertVisibility(ts, seq, s4);
std::vector<PinnableSlice> values_ps5(kTestDataSize);
std::vector<Status> s5(kTestDataSize);
db_->MultiGet(read_opts, cfh, kTestDataSize, keys.data(), values_ps5.data(),
s5.data());
AssertVisibility(ts, seq, s5);
std::vector<PinnableSlice> values_ps6(kTestDataSize);
std::vector<Status> s6(kTestDataSize);
std::vector<std::string> timestamps_array(kTestDataSize);
db_->MultiGet(read_opts, cfh, kTestDataSize, keys.data(), values_ps6.data(),
timestamps_array.data(), s6.data());
AssertVisibility(ts, seq, s6);
std::vector<PinnableSlice> values_ps7(kTestDataSize);
std::vector<Status> s7(kTestDataSize);
db_->MultiGet(read_opts, kTestDataSize, cfs.data(), keys.data(),
values_ps7.data(), s7.data());
AssertVisibility(ts, seq, s7);
std::vector<PinnableSlice> values_ps8(kTestDataSize);
std::vector<Status> s8(kTestDataSize);
db_->MultiGet(read_opts, kTestDataSize, cfs.data(), keys.data(),
values_ps8.data(), timestamps_array.data(), s8.data());
AssertVisibility(ts, seq, s8);
}
void VerifyDefaultCF(const Snapshot* snap = nullptr) {
for (int i = 0; i <= kTestDataSize; i++) {
VerifyDefaultCF(i, snap);
}
}
DataVisibilityTest() : DBBasicTestWithTimestampBase("data_visibility_test") {}
};
constexpr int DataVisibilityTest::kTestDataSize;
// Application specifies timestamp but not snapshot.
// reader writer
@@ -1030,153 +906,6 @@ TEST_F(DataVisibilityTest, RangeScanWithSnapshot) {
Close();
}
// Application specifies both timestamp and snapshot.
// Query each combination and make sure for MultiGet key <k, t1, s1>, only
// return keys that ts>=t1 AND seq>=s1.
TEST_F(DataVisibilityTest, MultiGetWithTimestamp) {
Options options = CurrentOptions();
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
const Snapshot* snap0 = db_->GetSnapshot();
PutTestData(0);
VerifyDefaultCF();
VerifyDefaultCF(snap0);
const Snapshot* snap1 = db_->GetSnapshot();
PutTestData(1);
VerifyDefaultCF();
VerifyDefaultCF(snap0);
VerifyDefaultCF(snap1);
Flush();
const Snapshot* snap2 = db_->GetSnapshot();
PutTestData(2);
VerifyDefaultCF();
VerifyDefaultCF(snap0);
VerifyDefaultCF(snap1);
VerifyDefaultCF(snap2);
db_->ReleaseSnapshot(snap0);
db_->ReleaseSnapshot(snap1);
db_->ReleaseSnapshot(snap2);
Close();
}
// Application specifies timestamp but not snapshot.
// reader writer
// ts'=0, 1
// ts=3
// seq=10
// seq'=11, 12
// write finishes
// MultiGet(ts,seq)
// For MultiGet <k, t1, s1>, only return keys that ts>=t1 AND seq>=s1.
TEST_F(DataVisibilityTest, MultiGetWithoutSnapshot) {
Options options = CurrentOptions();
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->LoadDependency({
{"DBImpl::MultiGet:AfterGetSeqNum1",
"DataVisibilityTest::MultiGetWithoutSnapshot:BeforePut"},
{"DataVisibilityTest::MultiGetWithoutSnapshot:AfterPut",
"DBImpl::MultiGet:AfterGetSeqNum2"},
});
SyncPoint::GetInstance()->EnableProcessing();
port::Thread writer_thread([this]() {
TEST_SYNC_POINT("DataVisibilityTest::MultiGetWithoutSnapshot:BeforePut");
PutTestData(0);
PutTestData(1);
TEST_SYNC_POINT("DataVisibilityTest::MultiGetWithoutSnapshot:AfterPut");
});
ReadOptions read_opts;
std::string read_ts = Timestamp(kTestDataSize, 0);
Slice read_ts_slice = read_ts;
read_opts.timestamp = &read_ts_slice;
auto keys = GetKeys();
std::vector<std::string> values;
auto ss = db_->MultiGet(read_opts, keys, &values);
writer_thread.join();
for (auto s : ss) {
ASSERT_TRUE(s.IsNotFound());
}
VerifyDefaultCF();
Close();
}
TEST_F(DataVisibilityTest, MultiGetCrossCF) {
Options options = CurrentOptions();
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
CreateAndReopenWithCF({"second"}, options);
ColumnFamilyHandle* second_cf = handles_[1];
const Snapshot* snap0 = db_->GetSnapshot();
PutTestData(0);
PutTestData(0, second_cf);
VerifyDefaultCF();
VerifyDefaultCF(snap0);
const Snapshot* snap1 = db_->GetSnapshot();
PutTestData(1);
PutTestData(1, second_cf);
VerifyDefaultCF();
VerifyDefaultCF(snap0);
VerifyDefaultCF(snap1);
Flush();
const Snapshot* snap2 = db_->GetSnapshot();
PutTestData(2);
PutTestData(2, second_cf);
VerifyDefaultCF();
VerifyDefaultCF(snap0);
VerifyDefaultCF(snap1);
VerifyDefaultCF(snap2);
ReadOptions read_opts;
std::string read_ts = Timestamp(kTestDataSize, 0);
Slice read_ts_slice = read_ts;
read_opts.timestamp = &read_ts_slice;
read_opts.snapshot = snap1;
auto keys = GetKeys();
auto keys2 = GetKeys();
keys.insert(keys.end(), keys2.begin(), keys2.end());
std::vector<ColumnFamilyHandle*> cfs(kTestDataSize,
db_->DefaultColumnFamily());
std::vector<ColumnFamilyHandle*> cfs2(kTestDataSize, second_cf);
cfs.insert(cfs.end(), cfs2.begin(), cfs2.end());
std::vector<std::string> values;
auto ss = db_->MultiGet(read_opts, cfs, keys, &values);
for (int i = 0; i < 2 * kTestDataSize; i++) {
if (i % 3 == 0) {
// only the first key for each column family should be returned
ASSERT_OK(ss[i]);
} else {
ASSERT_TRUE(ss[i].IsNotFound());
}
}
db_->ReleaseSnapshot(snap0);
db_->ReleaseSnapshot(snap1);
db_->ReleaseSnapshot(snap2);
Close();
}
class DBBasicTestWithTimestampCompressionSettings
: public DBBasicTestWithTimestampBase,
public testing::WithParamInterface<
@@ -1485,9 +1214,9 @@ TEST_P(DBBasicTestWithTimestampCompressionSettings, PutAndGetWithCompaction) {
// at higher levels.
CompactionOptions compact_opt;
compact_opt.compression = kNoCompression;
ASSERT_OK(db_->CompactFiles(compact_opt, handles_[cf],
collector->GetFlushedFiles(),
static_cast<int>(kNumTimestamps - i)));
db_->CompactFiles(compact_opt, handles_[cf],
collector->GetFlushedFiles(),
static_cast<int>(kNumTimestamps - i));
collector->ClearFlushedFiles();
}
}
@@ -1576,7 +1305,7 @@ TEST_F(DBBasicTestWithTimestamp, BatchWriteAndMultiGet) {
batch.Put(handles_[cf], Key1(j),
"value_" + std::to_string(j) + "_" + std::to_string(i)));
}
ASSERT_OK(batch.AssignTimestamp(write_ts));
batch.AssignTimestamp(write_ts);
ASSERT_OK(db_->Write(wopts, &batch));
verify_records_func(i, handles_[cf]);
+6 -11
View File
@@ -168,8 +168,6 @@ TEST_P(DBWriteTest, WriteThreadHangOnWriteStall) {
std::atomic<int> thread_num(0);
port::Mutex mutex;
port::CondVar cv(&mutex);
// Guarded by mutex
int writers = 0;
Reopen(options);
@@ -189,7 +187,6 @@ TEST_P(DBWriteTest, WriteThreadHangOnWriteStall) {
};
std::function<void(void *)> unblock_main_thread_func = [&](void *) {
mutex.Lock();
++writers;
cv.SignalAll();
mutex.Unlock();
};
@@ -228,18 +225,18 @@ TEST_P(DBWriteTest, WriteThreadHangOnWriteStall) {
mutex.Lock();
// First leader
threads.emplace_back(write_slowdown_func);
while (writers != 1) {
cv.Wait();
}
cv.Wait();
// Second leader. Will stall writes
threads.emplace_back(write_slowdown_func);
cv.Wait();
threads.emplace_back(write_no_slowdown_func);
cv.Wait();
threads.emplace_back(write_slowdown_func);
cv.Wait();
threads.emplace_back(write_no_slowdown_func);
cv.Wait();
threads.emplace_back(write_slowdown_func);
while (writers != 6) {
cv.Wait();
}
cv.Wait();
mutex.Unlock();
TEST_SYNC_POINT("DBWriteTest::WriteThreadHangOnWriteStall:1");
@@ -253,8 +250,6 @@ TEST_P(DBWriteTest, WriteThreadHangOnWriteStall) {
for (auto& t : threads) {
t.join();
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_P(DBWriteTest, IOErrorOnWALWritePropagateToWriteThreadFollower) {
+2 -2
View File
@@ -49,7 +49,7 @@ EntryType GetEntryType(ValueType value_type) {
bool ParseFullKey(const Slice& internal_key, FullKey* fkey) {
ParsedInternalKey ikey;
if (ParseInternalKey(internal_key, &ikey) != Status::OK()) {
if (!ParseInternalKey(internal_key, &ikey)) {
return false;
}
fkey->user_key = ikey.user_key;
@@ -90,7 +90,7 @@ std::string ParsedInternalKey::DebugString(bool hex) const {
std::string InternalKey::DebugString(bool hex) const {
std::string result;
ParsedInternalKey parsed;
if (ParseInternalKey(rep_, &parsed) == Status::OK()) {
if (ParseInternalKey(rep_, &parsed)) {
result = parsed.DebugString(hex);
} else {
result = "(bad)";
+32 -45
View File
@@ -96,8 +96,6 @@ static const SequenceNumber kMaxSequenceNumber = ((0x1ull << 56) - 1);
static const SequenceNumber kDisableGlobalSequenceNumber = port::kMaxUint64;
constexpr uint64_t kNumInternalBytes = 8;
// The data structure that represents an internal key in the way that user_key,
// sequence number and type are stored in separated forms.
struct ParsedInternalKey {
@@ -106,9 +104,8 @@ struct ParsedInternalKey {
ValueType type;
ParsedInternalKey()
: sequence(kMaxSequenceNumber),
type(kTypeDeletion) // Make code analyzer happy
{} // Intentionally left uninitialized (for speed)
: sequence(kMaxSequenceNumber) // Make code analyzer happy
{} // Intentionally left uninitialized (for speed)
// u contains timestamp if user timestamp feature is enabled.
ParsedInternalKey(const Slice& u, const SequenceNumber& seq, ValueType t)
: user_key(u), sequence(seq), type(t) {}
@@ -123,7 +120,7 @@ struct ParsedInternalKey {
// Return the length of the encoding of "key".
inline size_t InternalKeyEncodingLength(const ParsedInternalKey& key) {
return key.user_key.size() + kNumInternalBytes;
return key.user_key.size() + 8;
}
// Pack a sequence number and a ValueType into a uint64_t
@@ -165,20 +162,19 @@ extern void AppendInternalKeyFooter(std::string* result, SequenceNumber s,
// stores the parsed data in "*result", and returns true.
//
// On error, returns false, leaves "*result" in an undefined state.
extern Status ParseInternalKey(const Slice& internal_key,
ParsedInternalKey* result);
extern bool ParseInternalKey(const Slice& internal_key,
ParsedInternalKey* result);
// Returns the user key portion of an internal key.
inline Slice ExtractUserKey(const Slice& internal_key) {
assert(internal_key.size() >= kNumInternalBytes);
return Slice(internal_key.data(), internal_key.size() - kNumInternalBytes);
assert(internal_key.size() >= 8);
return Slice(internal_key.data(), internal_key.size() - 8);
}
inline Slice ExtractUserKeyAndStripTimestamp(const Slice& internal_key,
size_t ts_sz) {
assert(internal_key.size() >= kNumInternalBytes + ts_sz);
return Slice(internal_key.data(),
internal_key.size() - kNumInternalBytes - ts_sz);
assert(internal_key.size() >= 8 + ts_sz);
return Slice(internal_key.data(), internal_key.size() - 8 - ts_sz);
}
inline Slice StripTimestampFromUserKey(const Slice& user_key, size_t ts_sz) {
@@ -192,9 +188,9 @@ inline Slice ExtractTimestampFromUserKey(const Slice& user_key, size_t ts_sz) {
}
inline uint64_t ExtractInternalKeyFooter(const Slice& internal_key) {
assert(internal_key.size() >= kNumInternalBytes);
assert(internal_key.size() >= 8);
const size_t n = internal_key.size();
return DecodeFixed64(internal_key.data() + n - kNumInternalBytes);
return DecodeFixed64(internal_key.data() + n - 8);
}
inline ValueType ExtractValueType(const Slice& internal_key) {
@@ -285,8 +281,7 @@ class InternalKey {
bool Valid() const {
ParsedInternalKey parsed;
return (ParseInternalKey(Slice(rep_), &parsed) == Status::OK()) ? true
: false;
return ParseInternalKey(Slice(rep_), &parsed);
}
void DecodeFrom(const Slice& s) { rep_.assign(s.data(), s.size()); }
@@ -327,40 +322,36 @@ inline int InternalKeyComparator::Compare(const InternalKey& a,
return Compare(a.Encode(), b.Encode());
}
inline Status ParseInternalKey(const Slice& internal_key,
ParsedInternalKey* result) {
inline bool ParseInternalKey(const Slice& internal_key,
ParsedInternalKey* result) {
const size_t n = internal_key.size();
if (n < kNumInternalBytes) {
return Status::Corruption("Internal Key too small");
}
uint64_t num = DecodeFixed64(internal_key.data() + n - kNumInternalBytes);
if (n < 8) return false;
uint64_t num = DecodeFixed64(internal_key.data() + n - 8);
unsigned char c = num & 0xff;
result->sequence = num >> 8;
result->type = static_cast<ValueType>(c);
assert(result->type <= ValueType::kMaxValue);
result->user_key = Slice(internal_key.data(), n - kNumInternalBytes);
return IsExtendedValueType(result->type)
? Status::OK()
: Status::Corruption("Invalid Key Type");
result->user_key = Slice(internal_key.data(), n - 8);
return IsExtendedValueType(result->type);
}
// Update the sequence number in the internal key.
// Guarantees not to invalidate ikey.data().
inline void UpdateInternalKey(std::string* ikey, uint64_t seq, ValueType t) {
size_t ikey_sz = ikey->size();
assert(ikey_sz >= kNumInternalBytes);
assert(ikey_sz >= 8);
uint64_t newval = (seq << 8) | t;
// Note: Since C++11, strings are guaranteed to be stored contiguously and
// string::operator[]() is guaranteed not to change ikey.data().
EncodeFixed64(&(*ikey)[ikey_sz - kNumInternalBytes], newval);
EncodeFixed64(&(*ikey)[ikey_sz - 8], newval);
}
// Get the sequence number from the internal key
inline uint64_t GetInternalKeySeqno(const Slice& internal_key) {
const size_t n = internal_key.size();
assert(n >= kNumInternalBytes);
uint64_t num = DecodeFixed64(internal_key.data() + n - kNumInternalBytes);
assert(n >= 8);
uint64_t num = DecodeFixed64(internal_key.data() + n - 8);
return num >> 8;
}
@@ -399,8 +390,8 @@ class IterKey {
if (IsUserKey()) {
return Slice(key_, key_size_);
} else {
assert(key_size_ >= kNumInternalBytes);
return Slice(key_, key_size_ - kNumInternalBytes);
assert(key_size_ >= 8);
return Slice(key_, key_size_ - 8);
}
}
@@ -458,9 +449,9 @@ class IterKey {
// and returns a Slice referencing the new copy.
Slice SetInternalKey(const Slice& key, ParsedInternalKey* ikey) {
size_t key_n = key.size();
assert(key_n >= kNumInternalBytes);
assert(key_n >= 8);
SetInternalKey(key);
ikey->user_key = Slice(key_, key_n - kNumInternalBytes);
ikey->user_key = Slice(key_, key_n - 8);
return Slice(key_, key_n);
}
@@ -477,9 +468,9 @@ class IterKey {
// invalidate slices to the key (and the user key).
void UpdateInternalKey(uint64_t seq, ValueType t) {
assert(!IsKeyPinned());
assert(key_size_ >= kNumInternalBytes);
assert(key_size_ >= 8);
uint64_t newval = (seq << 8) | t;
EncodeFixed64(&buf_[key_size_ - kNumInternalBytes], newval);
EncodeFixed64(&buf_[key_size_ - 8], newval);
}
bool IsKeyPinned() const { return (key_ != buf_); }
@@ -684,10 +675,8 @@ inline int InternalKeyComparator::Compare(const Slice& akey,
// decreasing type (though sequence# should be enough to disambiguate)
int r = user_comparator_.Compare(ExtractUserKey(akey), ExtractUserKey(bkey));
if (r == 0) {
const uint64_t anum =
DecodeFixed64(akey.data() + akey.size() - kNumInternalBytes);
const uint64_t bnum =
DecodeFixed64(bkey.data() + bkey.size() - kNumInternalBytes);
const uint64_t anum = DecodeFixed64(akey.data() + akey.size() - 8);
const uint64_t bnum = DecodeFixed64(bkey.data() + bkey.size() - 8);
if (anum > bnum) {
r = -1;
} else if (anum < bnum) {
@@ -705,10 +694,8 @@ inline int InternalKeyComparator::CompareKeySeq(const Slice& akey,
int r = user_comparator_.Compare(ExtractUserKey(akey), ExtractUserKey(bkey));
if (r == 0) {
// Shift the number to exclude the last byte which contains the value type
const uint64_t anum =
DecodeFixed64(akey.data() + akey.size() - kNumInternalBytes) >> 8;
const uint64_t bnum =
DecodeFixed64(bkey.data() + bkey.size() - kNumInternalBytes) >> 8;
const uint64_t anum = DecodeFixed64(akey.data() + akey.size() - 8) >> 8;
const uint64_t bnum = DecodeFixed64(bkey.data() + bkey.size() - 8) >> 8;
if (anum > bnum) {
r = -1;
} else if (anum < bnum) {
+3 -3
View File
@@ -41,12 +41,12 @@ static void TestKey(const std::string& key,
Slice in(encoded);
ParsedInternalKey decoded("", 0, kTypeValue);
ASSERT_OK(ParseInternalKey(in, &decoded));
ASSERT_TRUE(ParseInternalKey(in, &decoded));
ASSERT_EQ(key, decoded.user_key.ToString());
ASSERT_EQ(seq, decoded.sequence);
ASSERT_EQ(vt, decoded.type);
ASSERT_NOK(ParseInternalKey(Slice("bar"), &decoded));
ASSERT_TRUE(!ParseInternalKey(Slice("bar"), &decoded));
}
class FormatTest : public testing::Test {};
@@ -186,7 +186,7 @@ TEST_F(FormatTest, UpdateInternalKey) {
Slice in(ikey);
ParsedInternalKey decoded;
ASSERT_OK(ParseInternalKey(in, &decoded));
ASSERT_TRUE(ParseInternalKey(in, &decoded));
ASSERT_EQ(user_key, decoded.user_key.ToString());
ASSERT_EQ(new_seq, decoded.sequence);
ASSERT_EQ(new_val_type, decoded.type);
+2 -2
View File
@@ -338,14 +338,14 @@ Status ErrorHandler::SetBGError(const IOStatus& bg_io_err,
}
if (BackgroundErrorReason::kManifestWrite == reason) {
// Always returns ok
db_->DisableFileDeletionsWithLock().PermitUncheckedError();
db_->DisableFileDeletionsWithLock();
}
Status new_bg_io_err = bg_io_err;
Status s;
DBRecoverContext context;
if (bg_io_err.GetDataLoss()) {
// First, data loss is treated as unrecoverable error. So it can directly
// FIrst, data loss is treated as unrecoverable error. So it can directly
// overwrite any existing bg_error_.
bool auto_recovery = false;
Status bg_err(new_bg_io_err, Status::Severity::kUnrecoverableError);
+100 -105
View File
@@ -31,10 +31,7 @@ class DBErrorHandlingFSTest : public DBTestBase {
std::vector<std::string> live_files;
uint64_t manifest_size;
Status s = dbfull()->GetLiveFiles(live_files, &manifest_size, false);
if (!s.ok()) {
return "";
}
dbfull()->GetLiveFiles(live_files, &manifest_size, false);
for (auto& file : live_files) {
uint64_t num = 0;
FileType type;
@@ -72,10 +69,6 @@ class ErrorHandlerFSListener : public EventListener {
override_bg_error_(false),
file_count_(0),
fault_fs_(nullptr) {}
~ErrorHandlerFSListener() {
file_creation_error_.PermitUncheckedError();
bg_error_.PermitUncheckedError();
}
void OnTableFileCreationStarted(
const TableFileCreationBriefInfo& /*ti*/) override {
@@ -90,19 +83,17 @@ class ErrorHandlerFSListener : public EventListener {
cv_.SignalAll();
}
void OnErrorRecoveryBegin(BackgroundErrorReason /*reason*/, Status bg_error,
bool* auto_recovery) override {
bg_error.PermitUncheckedError();
void OnErrorRecoveryBegin(BackgroundErrorReason /*reason*/,
Status /*bg_error*/, bool* auto_recovery) override {
if (*auto_recovery && no_auto_recovery_) {
*auto_recovery = false;
}
}
void OnErrorRecoveryCompleted(Status old_bg_error) override {
void OnErrorRecoveryCompleted(Status /*old_bg_error*/) override {
InstrumentedMutexLock l(&mutex_);
recovery_complete_ = true;
cv_.SignalAll();
old_bg_error.PermitUncheckedError();
}
bool WaitForRecovery(uint64_t /*abs_time_us*/) {
@@ -175,7 +166,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWriteError) {
listener->EnableAutoRecovery(false);
DestroyAndReopen(options);
ASSERT_OK(Put(Key(0), "val"));
Put(Key(0), "val");
SyncPoint::GetInstance()->SetCallBack("FlushJob::Start", [&](void*) {
fault_fs->SetFilesystemActive(false, IOStatus::NoSpace("Out of space"));
});
@@ -211,7 +202,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableError) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(1), "val1"));
Put(Key(1), "val1");
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
@@ -225,7 +216,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableError) {
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
Put(Key(2), "val2");
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeSyncTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
@@ -239,7 +230,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableError) {
Reopen(options);
ASSERT_EQ("val2", Get(Key(2)));
ASSERT_OK(Put(Key(3), "val3"));
Put(Key(3), "val3");
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeCloseTableFile",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
@@ -277,13 +268,13 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableError1) {
WriteOptions wo = WriteOptions();
wo.disableWAL = true;
ASSERT_OK(Put(Key(1), "val1", wo));
Put(Key(1), "val1", wo);
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_OK(Put(Key(2), "val2", wo));
Put(Key(2), "val2", wo);
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
ASSERT_EQ("val2", Get(Key(2)));
SyncPoint::GetInstance()->DisableProcessing();
@@ -292,7 +283,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableError1) {
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_EQ("val2", Get(Key(2)));
ASSERT_OK(Put(Key(3), "val3", wo));
Put(Key(3), "val3", wo);
ASSERT_EQ("val3", Get(Key(3)));
s = Flush();
ASSERT_OK(s);
@@ -323,13 +314,13 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableError2) {
WriteOptions wo = WriteOptions();
wo.disableWAL = true;
ASSERT_OK(Put(Key(1), "val1", wo));
Put(Key(1), "val1", wo);
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeSyncTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_OK(Put(Key(2), "val2", wo));
Put(Key(2), "val2", wo);
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
ASSERT_EQ("val2", Get(Key(2)));
SyncPoint::GetInstance()->DisableProcessing();
@@ -338,7 +329,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableError2) {
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_EQ("val2", Get(Key(2)));
ASSERT_OK(Put(Key(3), "val3", wo));
Put(Key(3), "val3", wo);
ASSERT_EQ("val3", Get(Key(3)));
s = Flush();
ASSERT_OK(s);
@@ -369,13 +360,13 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableError3) {
WriteOptions wo = WriteOptions();
wo.disableWAL = true;
ASSERT_OK(Put(Key(1), "val1", wo));
Put(Key(1), "val1", wo);
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeCloseTableFile",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_OK(Put(Key(2), "val2", wo));
Put(Key(2), "val2", wo);
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
ASSERT_EQ("val2", Get(Key(2)));
SyncPoint::GetInstance()->DisableProcessing();
@@ -384,7 +375,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableError3) {
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_EQ("val2", Get(Key(2)));
ASSERT_OK(Put(Key(3), "val3", wo));
Put(Key(3), "val3", wo);
ASSERT_EQ("val3", Get(Key(3)));
s = Flush();
ASSERT_OK(s);
@@ -411,9 +402,9 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteError) {
DestroyAndReopen(options);
old_manifest = GetManifestNameFromLiveFiles();
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put(Key(1), "val"));
Put(Key(0), "val");
Flush();
Put(Key(1), "val");
SyncPoint::GetInstance()->SetCallBack(
"VersionSet::LogAndApply:WriteManifest", [&](void*) {
fault_fs->SetFilesystemActive(false, IOStatus::NoSpace("Out of space"));
@@ -458,9 +449,9 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteRetryableError) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put(Key(1), "val"));
Put(Key(0), "val");
Flush();
Put(Key(1), "val");
SyncPoint::GetInstance()->SetCallBack(
"VersionSet::LogAndApply:WriteManifest",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
@@ -500,9 +491,9 @@ TEST_F(DBErrorHandlingFSTest, DoubleManifestWriteError) {
DestroyAndReopen(options);
old_manifest = GetManifestNameFromLiveFiles();
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put(Key(1), "val"));
Put(Key(0), "val");
Flush();
Put(Key(1), "val");
SyncPoint::GetInstance()->SetCallBack(
"VersionSet::LogAndApply:WriteManifest", [&](void*) {
fault_fs->SetFilesystemActive(false, IOStatus::NoSpace("Out of space"));
@@ -550,8 +541,8 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteError) {
DestroyAndReopen(options);
old_manifest = GetManifestNameFromLiveFiles();
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Put(Key(2), "val"));
Put(Key(0), "val");
Put(Key(2), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -579,7 +570,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteError) {
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(Key(1), "val"));
Put(Key(1), "val");
// This Flush will trigger a compaction, which will fail when appending to
// the manifest
s = Flush();
@@ -627,8 +618,8 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteRetryableError) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Put(Key(2), "val"));
Put(Key(0), "val");
Put(Key(2), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -654,7 +645,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteRetryableError) {
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(Key(1), "val"));
Put(Key(1), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -694,8 +685,8 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteError) {
Status s;
DestroyAndReopen(options);
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
Put(Key(0), "va;");
Put(Key(2), "va;");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -711,7 +702,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteError) {
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(Key(1), "val"));
Put(Key(1), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -742,8 +733,8 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableError) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
Put(Key(0), "va;");
Put(Key(2), "va;");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -757,7 +748,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableError) {
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(Key(1), "val"));
Put(Key(1), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -783,8 +774,8 @@ TEST_F(DBErrorHandlingFSTest, CorruptionError) {
Status s;
DestroyAndReopen(options);
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
Put(Key(0), "va;");
Put(Key(2), "va;");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -798,7 +789,7 @@ TEST_F(DBErrorHandlingFSTest, CorruptionError) {
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(Key(1), "val"));
Put(Key(1), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -827,7 +818,7 @@ TEST_F(DBErrorHandlingFSTest, AutoRecoverFlushError) {
listener->EnableAutoRecovery();
DestroyAndReopen(options);
ASSERT_OK(Put(Key(0), "val"));
Put(Key(0), "val");
SyncPoint::GetInstance()->SetCallBack("FlushJob::Start", [&](void*) {
fault_fs->SetFilesystemActive(false, IOStatus::NoSpace("Out of space"));
});
@@ -862,7 +853,7 @@ TEST_F(DBErrorHandlingFSTest, FailRecoverFlushError) {
listener->EnableAutoRecovery();
DestroyAndReopen(options);
ASSERT_OK(Put(Key(0), "val"));
Put(Key(0), "val");
SyncPoint::GetInstance()->SetCallBack("FlushJob::Start", [&](void*) {
fault_fs->SetFilesystemActive(false, IOStatus::NoSpace("Out of space"));
});
@@ -896,7 +887,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteError) {
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -909,7 +900,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteError) {
int write_error = 0;
for (auto i = 100; i < 199; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
SyncPoint::GetInstance()->SetCallBack(
@@ -973,7 +964,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -988,7 +979,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
int write_error = 0;
for (auto i = 100; i < 200; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
SyncPoint::GetInstance()->SetCallBack(
@@ -1024,7 +1015,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
WriteBatch batch;
for (auto i = 200; i < 300; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -1065,7 +1056,7 @@ TEST_F(DBErrorHandlingFSTest, MultiCFWALWriteError) {
for (auto i = 1; i < 4; ++i) {
for (auto j = 0; j < 100; ++j) {
ASSERT_OK(batch.Put(handles_[i], Key(j), rnd.RandomString(1024)));
batch.Put(handles_[i], Key(j), rnd.RandomString(1024));
}
}
@@ -1080,7 +1071,7 @@ TEST_F(DBErrorHandlingFSTest, MultiCFWALWriteError) {
// Write to one CF
for (auto i = 100; i < 199; ++i) {
ASSERT_OK(batch.Put(handles_[2], Key(i), rnd.RandomString(1024)));
batch.Put(handles_[2], Key(i), rnd.RandomString(1024));
}
SyncPoint::GetInstance()->SetCallBack(
@@ -1169,7 +1160,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBCompactionError) {
WriteBatch batch;
for (auto j = 0; j <= 100; ++j) {
ASSERT_OK(batch.Put(Key(j), rnd.RandomString(1024)));
batch.Put(Key(j), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -1184,7 +1175,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBCompactionError) {
// Write to one CF
for (auto j = 100; j < 199; ++j) {
ASSERT_OK(batch.Put(Key(j), rnd.RandomString(1024)));
batch.Put(Key(j), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -1282,7 +1273,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
WriteBatch batch;
for (auto j = 0; j <= 100; ++j) {
ASSERT_OK(batch.Put(Key(j), rnd.RandomString(1024)));
batch.Put(Key(j), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -1297,7 +1288,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
// Write to one CF
for (auto j = 100; j < 199; ++j) {
ASSERT_OK(batch.Put(Key(j), rnd.RandomString(1024)));
batch.Put(Key(j), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -1387,7 +1378,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover1) {
WriteOptions wo = WriteOptions();
wo.disableWAL = true;
ASSERT_OK(Put(Key(1), "val1", wo));
Put(Key(1), "val1", wo);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:LoopOut",
"FLushWritNoWALRetryableeErrorAutoRecover1:1"}});
@@ -1404,7 +1395,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover1) {
ASSERT_EQ("val1", Get(Key(1)));
SyncPoint::GetInstance()->DisableProcessing();
fault_fs->SetFilesystemActive(true);
ASSERT_OK(Put(Key(2), "val2", wo));
Put(Key(2), "val2", wo);
s = Flush();
// Since auto resume fails, the bg error is not cleand, flush will
// return the bg_error set before.
@@ -1414,7 +1405,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover1) {
// call auto resume
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(Put(Key(3), "val3", wo));
Put(Key(3), "val3", wo);
s = Flush();
// After resume is successful, the flush should be ok.
ASSERT_EQ(s, Status::OK());
@@ -1445,7 +1436,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover2) {
WriteOptions wo = WriteOptions();
wo.disableWAL = true;
ASSERT_OK(Put(Key(1), "val1", wo));
Put(Key(1), "val1", wo);
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
@@ -1458,7 +1449,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover2) {
fault_fs->SetFilesystemActive(true);
ASSERT_EQ(listener->WaitForRecovery(5000000), true);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2", wo));
Put(Key(2), "val2", wo);
s = Flush();
// Since auto resume is successful, the bg error is cleaned, flush will
// be successful.
@@ -1488,7 +1479,7 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover1) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(1), "val1"));
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeWait0",
"FLushWritRetryableeErrorAutoRecover1:0"},
@@ -1512,7 +1503,7 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover1) {
ASSERT_EQ("val1", Get(Key(1)));
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
@@ -1541,7 +1532,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover2) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(1), "val1"));
Put(Key(1), "val1");
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
@@ -1556,7 +1547,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover2) {
ASSERT_EQ("val1", Get(Key(1)));
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
@@ -1585,7 +1576,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover3) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(1), "val1"));
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"FLushWritRetryableeErrorAutoRecover3:0",
"RecoverFromRetryableBGIOError:BeforeStart"},
@@ -1609,7 +1600,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover3) {
s = dbfull()->Resume();
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_EQ(s, Status::OK());
ASSERT_OK(Put(Key(2), "val2"));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
@@ -1641,7 +1632,7 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover4) {
IOStatus nr_msg = IOStatus::IOError("No Retryable Fatal IO Error");
nr_msg.SetRetryable(false);
ASSERT_OK(Put(Key(1), "val1"));
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeStart",
"FLushWritRetryableeErrorAutoRecover4:0"},
@@ -1668,14 +1659,14 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover4) {
s = dbfull()->Resume();
ASSERT_NE(s, Status::OK());
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
Put(Key(2), "val2");
s = Flush();
ASSERT_NE(s, Status::OK());
ASSERT_EQ("NOT_FOUND", Get(Key(2)));
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
@@ -1705,8 +1696,10 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover5) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
IOStatus nr_msg = IOStatus::IOError("No Retryable Fatal IO Error");
nr_msg.SetRetryable(false);
ASSERT_OK(Put(Key(1), "val1"));
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeStart",
"FLushWritRetryableeErrorAutoRecover5:0"}});
@@ -1727,7 +1720,7 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover5) {
Reopen(options);
ASSERT_NE("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
@@ -1757,8 +1750,10 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover6) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
IOStatus nr_msg = IOStatus::IOError("No Retryable Fatal IO Error");
nr_msg.SetRetryable(false);
ASSERT_OK(Put(Key(1), "val1"));
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"FLushWritRetryableeErrorAutoRecover6:0",
"RecoverFromRetryableBGIOError:BeforeStart"},
@@ -1788,7 +1783,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover6) {
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
@@ -1820,9 +1815,9 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteRetryableErrorAutoRecover) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put(Key(1), "val"));
Put(Key(0), "val");
Flush();
Put(Key(1), "val");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeStart",
"ManifestWriteRetryableErrorAutoRecover:0"},
@@ -1876,8 +1871,8 @@ TEST_F(DBErrorHandlingFSTest,
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Put(Key(2), "val"));
Put(Key(0), "val");
Put(Key(2), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -1914,7 +1909,7 @@ TEST_F(DBErrorHandlingFSTest,
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(Key(1), "val"));
Put(Key(1), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -1966,8 +1961,8 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableErrorAutoRecover) {
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
Put(Key(0), "va;");
Put(Key(2), "va;");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -1992,7 +1987,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableErrorAutoRecover) {
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(Key(1), "val"));
Put(Key(1), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -2032,7 +2027,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover1) {
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -2047,7 +2042,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover1) {
int write_error = 0;
for (auto i = 100; i < 200; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeResume0", "WALWriteError1:0"},
@@ -2089,7 +2084,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover1) {
WriteBatch batch;
for (auto i = 200; i < 300; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -2136,7 +2131,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover2) {
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -2151,7 +2146,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover2) {
int write_error = 0;
for (auto i = 100; i < 200; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeWait0", "WALWriteError2:0"},
@@ -2193,7 +2188,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover2) {
WriteBatch batch;
for (auto i = 200; i < 300; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -2231,7 +2226,7 @@ TEST_P(DBErrorHandlingFencingTest, FLushWriteFenced) {
listener->EnableAutoRecovery(true);
DestroyAndReopen(options);
ASSERT_OK(Put(Key(0), "val"));
Put(Key(0), "val");
SyncPoint::GetInstance()->SetCallBack("FlushJob::Start", [&](void*) {
fault_fs->SetFilesystemActive(false, IOStatus::IOFenced("IO fenced"));
});
@@ -2265,9 +2260,9 @@ TEST_P(DBErrorHandlingFencingTest, ManifestWriteFenced) {
DestroyAndReopen(options);
old_manifest = GetManifestNameFromLiveFiles();
ASSERT_OK(Put(Key(0), "val"));
Put(Key(0), "val");
Flush();
ASSERT_OK(Put(Key(1), "val"));
Put(Key(1), "val");
SyncPoint::GetInstance()->SetCallBack(
"VersionSet::LogAndApply:WriteManifest", [&](void*) {
fault_fs->SetFilesystemActive(false, IOStatus::IOFenced("IO fenced"));
@@ -2299,8 +2294,8 @@ TEST_P(DBErrorHandlingFencingTest, CompactionWriteFenced) {
Status s;
DestroyAndReopen(options);
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
Put(Key(0), "va;");
Put(Key(2), "va;");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -2314,7 +2309,7 @@ TEST_P(DBErrorHandlingFencingTest, CompactionWriteFenced) {
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(Key(1), "val"));
Put(Key(1), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
@@ -2350,7 +2345,7 @@ TEST_P(DBErrorHandlingFencingTest, WALWriteFenced) {
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
@@ -2363,7 +2358,7 @@ TEST_P(DBErrorHandlingFencingTest, WALWriteFenced) {
int write_error = 0;
for (auto i = 100; i < 199; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
SyncPoint::GetInstance()->SetCallBack(
@@ -2386,7 +2381,7 @@ TEST_P(DBErrorHandlingFencingTest, WALWriteFenced) {
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
ASSERT_OK(batch.Put(Key(i), rnd.RandomString(1024)));
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
-4
View File
@@ -51,7 +51,6 @@ void EventHelpers::NotifyOnBackgroundError(
db_mutex->Unlock();
for (auto& listener : listeners) {
listener->OnBackgroundError(reason, bg_error);
bg_error->PermitUncheckedError();
if (*auto_recovery) {
listener->OnErrorRecoveryBegin(reason, *bg_error, auto_recovery);
}
@@ -163,7 +162,6 @@ void EventHelpers::LogAndNotifyTableFileCreationFinished(
for (auto& listener : listeners) {
listener->OnTableFileCreated(info);
}
info.status.PermitUncheckedError();
#else
(void)listeners;
(void)db_name;
@@ -201,7 +199,6 @@ void EventHelpers::LogAndNotifyTableFileDeletion(
for (auto& listener : listeners) {
listener->OnTableFileDeleted(info);
}
info.status.PermitUncheckedError();
#else
(void)file_path;
(void)dbname;
@@ -222,7 +219,6 @@ void EventHelpers::NotifyOnErrorRecoveryCompleted(
for (auto& listener : listeners) {
listener->OnErrorRecoveryCompleted(old_bg_error);
}
old_bg_error.PermitUncheckedError();
db_mutex->Lock();
#else
(void)listeners;
+3 -3
View File
@@ -601,7 +601,7 @@ Status ExternalSstFileIngestionJob::GetIngestedFileInfo(
bool bounds_set = false;
iter->SeekToFirst();
if (iter->Valid()) {
if (ParseInternalKey(iter->key(), &key) != Status::OK()) {
if (!ParseInternalKey(iter->key(), &key)) {
return Status::Corruption("external file have corrupted keys");
}
if (key.sequence != 0) {
@@ -610,7 +610,7 @@ Status ExternalSstFileIngestionJob::GetIngestedFileInfo(
file_to_ingest->smallest_internal_key.SetFrom(key);
iter->SeekToLast();
if (ParseInternalKey(iter->key(), &key) != Status::OK()) {
if (!ParseInternalKey(iter->key(), &key)) {
return Status::Corruption("external file have corrupted keys");
}
if (key.sequence != 0) {
@@ -627,7 +627,7 @@ Status ExternalSstFileIngestionJob::GetIngestedFileInfo(
if (range_del_iter != nullptr) {
for (range_del_iter->SeekToFirst(); range_del_iter->Valid();
range_del_iter->Next()) {
if (ParseInternalKey(range_del_iter->key(), &key) != Status::OK()) {
if (!ParseInternalKey(range_del_iter->key(), &key)) {
return Status::Corruption("external file have corrupted keys");
}
RangeTombstone tombstone(key, range_del_iter->value());
+8 -34
View File
@@ -39,6 +39,8 @@
#include "rocksdb/statistics.h"
#include "rocksdb/status.h"
#include "rocksdb/table.h"
#include "table/block_based/block.h"
#include "table/block_based/block_based_table_factory.h"
#include "table/merging_iterator.h"
#include "table/table_builder.h"
#include "table/two_level_iterator.h"
@@ -130,7 +132,6 @@ FlushJob::FlushJob(
}
FlushJob::~FlushJob() {
io_status_.PermitUncheckedError();
ThreadStatusUtil::ResetThreadStatus();
}
@@ -267,13 +268,6 @@ Status FlushJob::Run(LogsWithPrepTracker* prep_tracker,
stream << vstorage->NumLevelFiles(level);
}
stream.EndArray();
const auto& blob_files = vstorage->GetBlobFiles();
if (!blob_files.empty()) {
stream << "blob_file_head" << blob_files.begin()->first;
stream << "blob_file_tail" << blob_files.rbegin()->first;
}
stream << "immutable_memtables" << cfd_->imm()->NumNotFlushed();
if (measure_io_stats_) {
@@ -308,9 +302,6 @@ Status FlushJob::WriteLevel0Table() {
const uint64_t start_micros = db_options_.env->NowMicros();
const uint64_t start_cpu_micros = db_options_.env->NowCPUNanos() / 1000;
Status s;
std::vector<BlobFileAddition> blob_file_additions;
{
auto write_hint = cfd_->CalculateSSTWriteHint(0);
db_mutex_->Unlock();
@@ -399,10 +390,9 @@ Status FlushJob::WriteLevel0Table() {
IOStatus io_s;
s = BuildTable(
dbname_, versions_, db_options_.env, db_options_.fs.get(),
*cfd_->ioptions(), mutable_cf_options_, file_options_,
cfd_->table_cache(), iter.get(), std::move(range_del_iters), &meta_,
&blob_file_additions, cfd_->internal_comparator(),
dbname_, db_options_.env, db_options_.fs.get(), *cfd_->ioptions(),
mutable_cf_options_, file_options_, cfd_->table_cache(), iter.get(),
std::move(range_del_iters), &meta_, cfd_->internal_comparator(),
cfd_->int_tbl_prop_collector_factories(), cfd_->GetID(),
cfd_->GetName(), existing_snapshots_,
earliest_write_conflict_snapshot_, snapshot_checker_,
@@ -437,10 +427,7 @@ Status FlushJob::WriteLevel0Table() {
// Note that if file_size is zero, the file has been deleted and
// should not be added to the manifest.
const bool has_output = meta_.fd.GetFileSize() > 0;
assert(has_output || blob_file_additions.empty());
if (s.ok() && has_output) {
if (s.ok() && meta_.fd.GetFileSize() > 0) {
// if we have more than 1 background thread, then we cannot
// insert files directly into higher levels because some other
// threads could be concurrently producing compacted files for
@@ -452,8 +439,6 @@ Status FlushJob::WriteLevel0Table() {
meta_.marked_for_compaction, meta_.oldest_blob_file_number,
meta_.oldest_ancester_time, meta_.file_creation_time,
meta_.file_checksum, meta_.file_checksum_func_name);
edit_->SetBlobFileAdditions(std::move(blob_file_additions));
}
#ifndef ROCKSDB_LITE
// Piggyback FlushJobInfo on the first first flushed memtable.
@@ -464,22 +449,11 @@ Status FlushJob::WriteLevel0Table() {
InternalStats::CompactionStats stats(CompactionReason::kFlush, 1);
stats.micros = db_options_.env->NowMicros() - start_micros;
stats.cpu_micros = db_options_.env->NowCPUNanos() / 1000 - start_cpu_micros;
if (has_output) {
stats.bytes_written = meta_.fd.GetFileSize();
const auto& blobs = edit_->GetBlobFileAdditions();
for (const auto& blob : blobs) {
stats.bytes_written += blob.GetTotalBlobBytes();
}
stats.num_output_files = static_cast<int>(blobs.size()) + 1;
}
stats.bytes_written = meta_.fd.GetFileSize();
RecordTimeToHistogram(stats_, FLUSH_TIME, stats.micros);
cfd_->internal_stats()->AddCompactionStats(0 /* level */, thread_pri_, stats);
cfd_->internal_stats()->AddCFStats(InternalStats::BYTES_FLUSHED,
stats.bytes_written);
meta_.fd.GetFileSize());
RecordFlushIOStats();
return s;
}
+5 -7
View File
@@ -111,7 +111,6 @@ class FlushJobTest : public testing::Test {
ASSERT_OK(s);
// Make "CURRENT" file that points to the new manifest file.
s = SetCurrentFile(fs_.get(), dbname_, 1, nullptr);
ASSERT_OK(s);
}
Env* env_;
@@ -170,14 +169,14 @@ TEST_F(FlushJobTest, NonEmpty) {
new_mem->Add(SequenceNumber(i), kTypeValue, key, value);
if ((i + 1000) % 10000 < 9995) {
InternalKey internal_key(key, SequenceNumber(i), kTypeValue);
inserted_keys.push_back({internal_key.Encode().ToString(), value});
inserted_keys.insert({internal_key.Encode().ToString(), value});
}
}
{
new_mem->Add(SequenceNumber(10000), kTypeRangeDeletion, "9995", "9999a");
InternalKey internal_key("9995", SequenceNumber(10000), kTypeRangeDeletion);
inserted_keys.push_back({internal_key.Encode().ToString(), "9999a"});
inserted_keys.insert({internal_key.Encode().ToString(), "9999a"});
}
// Note: the first two blob references will not be considered when resolving
@@ -205,9 +204,9 @@ TEST_F(FlushJobTest, NonEmpty) {
new_mem->Add(seq, kTypeBlobIndex, key, blob_index);
InternalKey internal_key(key, seq, kTypeBlobIndex);
inserted_keys.push_back({internal_key.Encode().ToString(), blob_index});
inserted_keys.emplace_hint(inserted_keys.end(),
internal_key.Encode().ToString(), blob_index);
}
mock::SortKVVector(&inserted_keys);
autovector<MemTable*> to_delete;
cfd->imm()->Add(new_mem, &to_delete);
@@ -454,11 +453,10 @@ TEST_F(FlushJobTest, Snapshots) {
(snapshots_set.find(seqno) != snapshots_set.end());
if (visible) {
InternalKey internal_key(key, seqno, kTypeValue);
inserted_keys.push_back({internal_key.Encode().ToString(), value});
inserted_keys.insert({internal_key.Encode().ToString(), value});
}
}
}
mock::SortKVVector(&inserted_keys);
autovector<MemTable*> to_delete;
cfd->imm()->Add(new_mem, &to_delete);
+4 -10
View File
@@ -661,11 +661,8 @@ void ForwardIterator::RebuildIterators(bool refresh_sv) {
sv_->mem->NewRangeTombstoneIterator(
read_options_, sv_->current->version_set()->LastSequence()));
range_del_agg.AddTombstones(std::move(range_del_iter));
// Always return Status::OK().
assert(
sv_->imm
->AddRangeTombstoneIterators(read_options_, &arena_, &range_del_agg)
.ok());
sv_->imm->AddRangeTombstoneIterators(read_options_, &arena_,
&range_del_agg);
}
has_iter_trimmed_for_upper_bound_ = false;
@@ -727,11 +724,8 @@ void ForwardIterator::RenewIterators() {
svnew->mem->NewRangeTombstoneIterator(
read_options_, sv_->current->version_set()->LastSequence()));
range_del_agg.AddTombstones(std::move(range_del_iter));
// Always return Status::OK().
assert(
svnew->imm
->AddRangeTombstoneIterators(read_options_, &arena_, &range_del_agg)
.ok());
svnew->imm->AddRangeTombstoneIterators(read_options_, &arena_,
&range_del_agg);
}
const auto* vstorage = sv_->current->storage_info();
+2 -2
View File
@@ -252,14 +252,14 @@ Status ImportColumnFamilyJob::GetIngestedFileInfo(
// Get first (smallest) key from file
iter->SeekToFirst();
if (ParseInternalKey(iter->key(), &key) != Status::OK()) {
if (!ParseInternalKey(iter->key(), &key)) {
return Status::Corruption("external file have corrupted keys");
}
file_to_import->smallest_internal_key.SetFrom(key);
// Get last (largest) key from file
iter->SeekToLast();
if (ParseInternalKey(iter->key(), &key) != Status::OK()) {
if (!ParseInternalKey(iter->key(), &key)) {
return Status::Corruption("external file have corrupted keys");
}
file_to_import->largest_internal_key.SetFrom(key);
+29 -29
View File
@@ -25,27 +25,27 @@ class ImportColumnFamilyTest : public DBTestBase {
~ImportColumnFamilyTest() {
if (import_cfh_) {
EXPECT_OK(db_->DropColumnFamily(import_cfh_));
EXPECT_OK(db_->DestroyColumnFamilyHandle(import_cfh_));
db_->DropColumnFamily(import_cfh_);
db_->DestroyColumnFamilyHandle(import_cfh_);
import_cfh_ = nullptr;
}
if (import_cfh2_) {
EXPECT_OK(db_->DropColumnFamily(import_cfh2_));
EXPECT_OK(db_->DestroyColumnFamilyHandle(import_cfh2_));
db_->DropColumnFamily(import_cfh2_);
db_->DestroyColumnFamilyHandle(import_cfh2_);
import_cfh2_ = nullptr;
}
if (metadata_ptr_) {
delete metadata_ptr_;
metadata_ptr_ = nullptr;
}
EXPECT_OK(DestroyDir(env_, sst_files_dir_));
EXPECT_OK(DestroyDir(env_, export_files_dir_));
DestroyDir(env_, sst_files_dir_);
DestroyDir(env_, export_files_dir_);
}
void DestroyAndRecreateExternalSSTFilesDir() {
EXPECT_OK(DestroyDir(env_, sst_files_dir_));
EXPECT_OK(env_->CreateDir(sst_files_dir_));
EXPECT_OK(DestroyDir(env_, export_files_dir_));
DestroyDir(env_, sst_files_dir_);
env_->CreateDir(sst_files_dir_);
DestroyDir(env_, export_files_dir_);
}
LiveFileMetaData LiveFileMetaDataInit(std::string name, std::string path,
@@ -143,7 +143,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
const std::string file3_sst = sst_files_dir_ + file3_sst_name;
ASSERT_OK(sfw_cf1.Open(file3_sst));
for (int i = 0; i < 100; ++i) {
ASSERT_OK(sfw_cf1.Put(Key(i), Key(i) + "_val"));
sfw_cf1.Put(Key(i), Key(i) + "_val");
}
ASSERT_OK(sfw_cf1.Finish());
@@ -152,7 +152,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
const std::string file2_sst = sst_files_dir_ + file2_sst_name;
ASSERT_OK(sfw_cf1.Open(file2_sst));
for (int i = 0; i < 100; i += 2) {
ASSERT_OK(sfw_cf1.Put(Key(i), Key(i) + "_overwrite1"));
sfw_cf1.Put(Key(i), Key(i) + "_overwrite1");
}
ASSERT_OK(sfw_cf1.Finish());
@@ -161,7 +161,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
const std::string file1a_sst = sst_files_dir_ + file1a_sst_name;
ASSERT_OK(sfw_cf1.Open(file1a_sst));
for (int i = 0; i < 52; i += 4) {
ASSERT_OK(sfw_cf1.Put(Key(i), Key(i) + "_overwrite2"));
sfw_cf1.Put(Key(i), Key(i) + "_overwrite2");
}
ASSERT_OK(sfw_cf1.Finish());
@@ -170,7 +170,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
const std::string file1b_sst = sst_files_dir_ + file1b_sst_name;
ASSERT_OK(sfw_cf1.Open(file1b_sst));
for (int i = 52; i < 100; i += 4) {
ASSERT_OK(sfw_cf1.Put(Key(i), Key(i) + "_overwrite2"));
sfw_cf1.Put(Key(i), Key(i) + "_overwrite2");
}
ASSERT_OK(sfw_cf1.Finish());
@@ -179,7 +179,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
const std::string file0a_sst = sst_files_dir_ + file0a_sst_name;
ASSERT_OK(sfw_cf1.Open(file0a_sst));
for (int i = 0; i < 100; i += 16) {
ASSERT_OK(sfw_cf1.Put(Key(i), Key(i) + "_overwrite3"));
sfw_cf1.Put(Key(i), Key(i) + "_overwrite3");
}
ASSERT_OK(sfw_cf1.Finish());
@@ -188,7 +188,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
const std::string file0b_sst = sst_files_dir_ + file0b_sst_name;
ASSERT_OK(sfw_cf1.Open(file0b_sst));
for (int i = 0; i < 100; i += 16) {
ASSERT_OK(sfw_cf1.Put(Key(i), Key(i) + "_overwrite4"));
sfw_cf1.Put(Key(i), Key(i) + "_overwrite4");
}
ASSERT_OK(sfw_cf1.Finish());
@@ -274,7 +274,7 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherCF) {
CreateAndReopenWithCF({"koko"}, options);
for (int i = 0; i < 100; ++i) {
ASSERT_OK(Put(1, Key(i), Key(i) + "_val"));
Put(1, Key(i), Key(i) + "_val");
}
ASSERT_OK(Flush(1));
@@ -283,13 +283,13 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherCF) {
// Overwrite the value in the same set of keys.
for (int i = 0; i < 100; ++i) {
ASSERT_OK(Put(1, Key(i), Key(i) + "_overwrite"));
Put(1, Key(i), Key(i) + "_overwrite");
}
// Flush to create L0 file.
ASSERT_OK(Flush(1));
for (int i = 0; i < 100; ++i) {
ASSERT_OK(Put(1, Key(i), Key(i) + "_overwrite2"));
Put(1, Key(i), Key(i) + "_overwrite2");
}
// Flush again to create another L0 file. It should have higher sequencer.
@@ -382,7 +382,7 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherDB) {
CreateAndReopenWithCF({"koko"}, options);
for (int i = 0; i < 100; ++i) {
ASSERT_OK(Put(1, Key(i), Key(i) + "_val"));
Put(1, Key(i), Key(i) + "_val");
}
ASSERT_OK(Flush(1));
@@ -392,14 +392,14 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherDB) {
// Overwrite the value in the same set of keys.
for (int i = 0; i < 50; ++i) {
ASSERT_OK(Put(1, Key(i), Key(i) + "_overwrite"));
Put(1, Key(i), Key(i) + "_overwrite");
}
// Flush to create L0 file.
ASSERT_OK(Flush(1));
for (int i = 0; i < 25; ++i) {
ASSERT_OK(Put(1, Key(i), Key(i) + "_overwrite2"));
Put(1, Key(i), Key(i) + "_overwrite2");
}
// Flush again to create another L0 file. It should have higher sequencer.
@@ -414,7 +414,7 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherDB) {
// Create a new db and import the files.
DB* db_copy;
ASSERT_OK(DestroyDir(env_, dbname_ + "/db_copy"));
DestroyDir(env_, dbname_ + "/db_copy");
ASSERT_OK(DB::Open(options, dbname_ + "/db_copy", &db_copy));
ColumnFamilyHandle* cfh = nullptr;
ASSERT_OK(db_copy->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
@@ -427,10 +427,10 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherDB) {
db_copy->Get(ReadOptions(), cfh, Key(i), &value);
ASSERT_EQ(Get(1, Key(i)), value);
}
ASSERT_OK(db_copy->DropColumnFamily(cfh));
ASSERT_OK(db_copy->DestroyColumnFamilyHandle(cfh));
db_copy->DropColumnFamily(cfh);
db_copy->DestroyColumnFamilyHandle(cfh);
delete db_copy;
ASSERT_OK(DestroyDir(env_, dbname_ + "/db_copy"));
DestroyDir(env_, dbname_ + "/db_copy");
}
TEST_F(ImportColumnFamilyTest, LevelFilesOverlappingAtEndpoints) {
@@ -474,7 +474,7 @@ TEST_F(ImportColumnFamilyTest, LevelFilesOverlappingAtEndpoints) {
// Create a new db and import the files.
DB* db_copy;
ASSERT_OK(DestroyDir(env_, dbname_ + "/db_copy"));
DestroyDir(env_, dbname_ + "/db_copy");
ASSERT_OK(DB::Open(options, dbname_ + "/db_copy", &db_copy));
ColumnFamilyHandle* cfh = nullptr;
ASSERT_OK(db_copy->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
@@ -486,10 +486,10 @@ TEST_F(ImportColumnFamilyTest, LevelFilesOverlappingAtEndpoints) {
std::string value;
ASSERT_OK(db_copy->Get(ReadOptions(), cfh, "key", &value));
}
ASSERT_OK(db_copy->DropColumnFamily(cfh));
ASSERT_OK(db_copy->DestroyColumnFamilyHandle(cfh));
db_copy->DropColumnFamily(cfh);
db_copy->DestroyColumnFamilyHandle(cfh);
delete db_copy;
ASSERT_OK(DestroyDir(env_, dbname_ + "/db_copy"));
DestroyDir(env_, dbname_ + "/db_copy");
for (const Snapshot* snapshot : snapshots) {
db_->ReleaseSnapshot(snapshot);
}
+24 -20
View File
@@ -13,14 +13,13 @@
#include <algorithm>
#include <cinttypes>
#include <limits>
#include <sstream>
#include <string>
#include <utility>
#include <vector>
#include "db/column_family.h"
#include "db/db_impl/db_impl.h"
#include "rocksdb/table.h"
#include "table/block_based/block_based_table_factory.h"
#include "util/string_util.h"
namespace ROCKSDB_NAMESPACE {
@@ -908,9 +907,19 @@ bool InternalStats::HandleBlockCacheStat(Cache** block_cache) {
assert(block_cache != nullptr);
auto* table_factory = cfd_->ioptions()->table_factory;
assert(table_factory != nullptr);
*block_cache =
table_factory->GetOptions<Cache>(TableFactory::kBlockCacheOpts());
return *block_cache != nullptr;
if (BlockBasedTableFactory::kName != table_factory->Name()) {
return false;
}
auto* table_options =
reinterpret_cast<BlockBasedTableOptions*>(table_factory->GetOptions());
if (table_options == nullptr) {
return false;
}
*block_cache = table_options->block_cache.get();
if (table_options->no_block_cache || *block_cache == nullptr) {
return false;
}
return true;
}
bool InternalStats::HandleBlockCacheCapacity(uint64_t* value, DBImpl* /*db*/,
@@ -1387,26 +1396,21 @@ void InternalStats::DumpCFStatsNoFileHistogram(std::string* value) {
}
void InternalStats::DumpCFFileHistogram(std::string* value) {
assert(value);
assert(cfd_);
std::ostringstream oss;
oss << "\n** File Read Latency Histogram By Level [" << cfd_->GetName()
<< "] **\n";
char buf[2000];
snprintf(buf, sizeof(buf),
"\n** File Read Latency Histogram By Level [%s] **\n",
cfd_->GetName().c_str());
value->append(buf);
for (int level = 0; level < number_levels_; level++) {
if (!file_read_latency_[level].Empty()) {
oss << "** Level " << level << " read latency histogram (micros):\n"
<< file_read_latency_[level].ToString() << '\n';
char buf2[5000];
snprintf(buf2, sizeof(buf2),
"** Level %d read latency histogram (micros):\n%s\n", level,
file_read_latency_[level].ToString().c_str());
value->append(buf2);
}
}
if (!blob_file_read_latency_.Empty()) {
oss << "** Blob file read latency histogram (micros):\n"
<< blob_file_read_latency_.ToString() << '\n';
}
*value = oss.str();
}
#else
-6
View File
@@ -335,7 +335,6 @@ class InternalStats {
for (auto& h : file_read_latency_) {
h.Clear();
}
blob_file_read_latency_.Clear();
cf_stats_snapshot_.Clear();
db_stats_snapshot_.Clear();
bg_error_count_ = 0;
@@ -376,8 +375,6 @@ class InternalStats {
return &file_read_latency_[level];
}
HistogramImpl* GetBlobFileReadHist() { return &blob_file_read_latency_; }
uint64_t GetBackgroundErrorCount() const { return bg_error_count_; }
uint64_t BumpAndGetBackgroundErrorCount() { return ++bg_error_count_; }
@@ -395,8 +392,6 @@ class InternalStats {
bool GetIntPropertyOutOfMutex(const DBPropertyInfo& property_info,
Version* version, uint64_t* value);
const uint64_t* TEST_GetCFStatsValue() const { return cf_stats_value_; }
const std::vector<CompactionStats>& TEST_GetCompactionStats() const {
return comp_stats_;
}
@@ -429,7 +424,6 @@ class InternalStats {
std::vector<CompactionStats> comp_stats_;
std::vector<CompactionStats> comp_stats_by_pri_;
std::vector<HistogramImpl> file_read_latency_;
HistogramImpl blob_file_read_latency_;
// Used to compute per-interval statistics
struct CFStatsSnapshot {
+2
View File
@@ -24,6 +24,8 @@
#include "rocksdb/slice_transform.h"
#include "rocksdb/table.h"
#include "rocksdb/table_properties.h"
#include "table/block_based/block_based_table_factory.h"
#include "table/plain/plain_table_factory.h"
#include "test_util/sync_point.h"
#include "test_util/testharness.h"
#include "test_util/testutil.h"
+9 -15
View File
@@ -58,8 +58,7 @@ ImmutableMemTableOptions::ImmutableMemTableOptions(
max_successive_merges(mutable_cf_options.max_successive_merges),
statistics(ioptions.statistics),
merge_operator(ioptions.merge_operator),
info_log(ioptions.info_log),
allow_data_in_errors(ioptions.allow_data_in_errors) {}
info_log(ioptions.info_log) {}
MemTable::MemTable(const InternalKeyComparator& cmp,
const ImmutableCFOptions& ioptions,
@@ -376,7 +375,6 @@ class MemTableIterator : public InternalIterator {
PERF_COUNTER_ADD(next_on_memtable_count, 1);
assert(Valid());
iter_->Next();
TEST_SYNC_POINT_CALLBACK("MemTableIterator::Next:0", iter_);
valid_ = iter_->Valid();
}
bool NextAndGetResult(IterateResult* result) override {
@@ -456,7 +454,7 @@ FragmentedRangeTombstoneIterator* MemTable::NewRangeTombstoneIterator(
}
port::RWMutex* MemTable::GetLock(const Slice& key) {
return &locks_[GetSliceRangedNPHash(key, locks_.size())];
return &locks_[fastrange64(GetSliceNPHash64(key), locks_.size())];
}
MemTable::MemTableStats MemTable::ApproximateStats(const Slice& start_ikey,
@@ -625,7 +623,7 @@ struct Saver {
Env* env_;
ReadCallback* callback_;
bool* is_blob_index;
bool allow_data_in_errors;
bool CheckCallback(SequenceNumber _seq) {
if (callback_) {
return callback_->IsVisible(_seq);
@@ -780,17 +778,14 @@ static bool SaveValue(void* arg, const char* entry) {
return true;
}
default: {
std::string msg("Corrupted value not expected.");
if (s->allow_data_in_errors) {
msg.append("Unrecognized value type: " +
std::to_string(static_cast<int>(type)) + ". ");
msg.append("User key: " + user_key_slice.ToString(/*hex=*/true) +
". ");
msg.append("seq: " + std::to_string(seq) + ".");
}
std::string msg("Unrecognized value type: " +
std::to_string(static_cast<int>(type)) + ". ");
msg.append("User key: " + user_key_slice.ToString(/*hex=*/true) + ". ");
msg.append("seq: " + std::to_string(seq) + ".");
*(s->status) = Status::Corruption(msg.c_str());
return false;
}
assert(false);
return true;
}
}
@@ -886,7 +881,6 @@ void MemTable::GetFromTable(const LookupKey& key,
saver.callback_ = callback;
saver.is_blob_index = is_blob_index;
saver.do_merge = do_merge;
saver.allow_data_in_errors = moptions_.allow_data_in_errors;
table_->Get(key, &saver, SaveValue);
*seq = saver.seq;
}
-1
View File
@@ -54,7 +54,6 @@ struct ImmutableMemTableOptions {
Statistics* statistics;
MergeOperator* merge_operator;
Logger* info_log;
bool allow_data_in_errors;
};
// Batched counters to updated when inserting keys in one write batch.
+19 -78
View File
@@ -445,18 +445,9 @@ Status MemTableList::TryInstallMemtableFlushResults(
}
if (it == memlist.rbegin() || batch_file_number != m->file_number_) {
batch_file_number = m->file_number_;
if (m->edit_.GetBlobFileAdditions().empty()) {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64 " started",
cfd->GetName().c_str(), m->file_number_);
} else {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64
" (+%zu blob files) started",
cfd->GetName().c_str(), m->file_number_,
m->edit_.GetBlobFileAdditions().size());
}
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64 " started",
cfd->GetName().c_str(), m->file_number_);
edit_list.push_back(&m->edit_);
memtables_to_flush.push_back(m);
#ifndef ROCKSDB_LITE
@@ -511,20 +502,9 @@ Status MemTableList::TryInstallMemtableFlushResults(
if (s.ok() && !cfd->IsDropped()) { // commit new state
while (batch_count-- > 0) {
MemTable* m = current_->memlist_.back();
if (m->edit_.GetBlobFileAdditions().empty()) {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64
": memtable #%" PRIu64 " done",
cfd->GetName().c_str(), m->file_number_, mem_id);
} else {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64
" (+%zu blob files)"
": memtable #%" PRIu64 " done",
cfd->GetName().c_str(), m->file_number_,
m->edit_.GetBlobFileAdditions().size(), mem_id);
}
ROCKS_LOG_BUFFER(log_buffer, "[%s] Level-0 commit table #%" PRIu64
": memtable #%" PRIu64 " done",
cfd->GetName().c_str(), m->file_number_, mem_id);
assert(m->file_number_ > 0);
current_->Remove(m, to_delete);
UpdateCachedValuesFromMemTableListVersion();
@@ -535,20 +515,9 @@ Status MemTableList::TryInstallMemtableFlushResults(
for (auto it = current_->memlist_.rbegin(); batch_count-- > 0; ++it) {
MemTable* m = *it;
// commit failed. setup state so that we can flush again.
if (m->edit_.GetBlobFileAdditions().empty()) {
ROCKS_LOG_BUFFER(log_buffer,
"Level-0 commit table #%" PRIu64
": memtable #%" PRIu64 " failed",
m->file_number_, mem_id);
} else {
ROCKS_LOG_BUFFER(log_buffer,
"Level-0 commit table #%" PRIu64
" (+%zu blob files)"
": memtable #%" PRIu64 " failed",
m->file_number_,
m->edit_.GetBlobFileAdditions().size(), mem_id);
}
ROCKS_LOG_BUFFER(log_buffer, "Level-0 commit table #%" PRIu64
": memtable #%" PRIu64 " failed",
m->file_number_, mem_id);
m->flush_completed_ = false;
m->flush_in_progress_ = false;
m->edit_.Clear();
@@ -744,25 +713,11 @@ Status InstallMemtableAtomicFlushResults(
for (auto m : *mems_list[i]) {
assert(m->GetFileNumber() > 0);
uint64_t mem_id = m->GetID();
const VersionEdit* const edit = m->GetEdits();
assert(edit);
if (edit->GetBlobFileAdditions().empty()) {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64
": memtable #%" PRIu64 " done",
cfds[i]->GetName().c_str(), m->GetFileNumber(),
mem_id);
} else {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64
" (+%zu blob files)"
": memtable #%" PRIu64 " done",
cfds[i]->GetName().c_str(), m->GetFileNumber(),
edit->GetBlobFileAdditions().size(), mem_id);
}
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64
": memtable #%" PRIu64 " done",
cfds[i]->GetName().c_str(), m->GetFileNumber(),
mem_id);
imm->current_->Remove(m, to_delete);
imm->UpdateCachedValuesFromMemTableListVersion();
imm->ResetTrimHistoryNeeded();
@@ -773,25 +728,11 @@ Status InstallMemtableAtomicFlushResults(
auto* imm = (imm_lists == nullptr) ? cfds[i]->imm() : imm_lists->at(i);
for (auto m : *mems_list[i]) {
uint64_t mem_id = m->GetID();
const VersionEdit* const edit = m->GetEdits();
assert(edit);
if (edit->GetBlobFileAdditions().empty()) {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64
": memtable #%" PRIu64 " failed",
cfds[i]->GetName().c_str(), m->GetFileNumber(),
mem_id);
} else {
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64
" (+%zu blob files)"
": memtable #%" PRIu64 " failed",
cfds[i]->GetName().c_str(), m->GetFileNumber(),
edit->GetBlobFileAdditions().size(), mem_id);
}
ROCKS_LOG_BUFFER(log_buffer,
"[%s] Level-0 commit table #%" PRIu64
": memtable #%" PRIu64 " failed",
cfds[i]->GetName().c_str(), m->GetFileNumber(),
mem_id);
m->SetFlushCompleted(false);
m->SetFlushInProgress(false);
m->GetEdits()->Clear();
+2 -5
View File
@@ -65,14 +65,12 @@ class MemTableListTest : public testing::Test {
~MemTableListTest() override {
if (db) {
std::vector<ColumnFamilyDescriptor> cf_descs(handles.size());
#ifndef ROCKSDB_LITE
for (int i = 0; i != static_cast<int>(handles.size()); ++i) {
EXPECT_OK(handles[i]->GetDescriptor(&cf_descs[i]));
handles[i]->GetDescriptor(&cf_descs[i]);
}
#endif // !ROCKSDB_LITE
for (auto h : handles) {
if (h) {
EXPECT_OK(db->DestroyColumnFamilyHandle(h));
db->DestroyColumnFamilyHandle(h);
}
}
handles.clear();
@@ -125,7 +123,6 @@ class MemTableListTest : public testing::Test {
Status s = list->TryInstallMemtableFlushResults(
cfd, mutable_cf_options, m, &dummy_prep_tracker, &versions, &mutex,
file_num, to_delete, nullptr, &log_buffer, &flush_jobs_info, &io_s);
EXPECT_OK(io_s);
return s;
}
+13 -15
View File
@@ -138,27 +138,27 @@ Status MergeHelper::MergeUntil(InternalIterator* iter,
// orig_ikey is backed by original_key if keys_.empty()
// orig_ikey is backed by keys_.back() if !keys_.empty()
ParsedInternalKey orig_ikey;
bool succ = ParseInternalKey(original_key, &orig_ikey);
assert(succ);
if (!succ) {
return Status::Corruption("Cannot parse key in MergeUntil");
}
Status s = ParseInternalKey(original_key, &orig_ikey);
assert(s.ok());
if (!s.ok()) return s;
Status s;
bool hit_the_next_user_key = false;
for (; iter->Valid(); iter->Next(), original_key_is_iter = false) {
if (IsShuttingDown()) {
s = Status::ShutdownInProgress();
return s;
return Status::ShutdownInProgress();
}
ParsedInternalKey ikey;
assert(keys_.size() == merge_context_.GetNumOperands());
if (ParseInternalKey(iter->key(), &ikey) != Status::OK()) {
if (!ParseInternalKey(iter->key(), &ikey)) {
// stop at corrupted key
if (assert_valid_internal_key_) {
assert(!"Corrupted internal key not expected.");
s = Status::Corruption("Corrupted internal key not expected.");
return s;
return Status::Corruption("Corrupted internal key not expected.");
}
break;
} else if (first_key) {
@@ -193,7 +193,7 @@ Status MergeHelper::MergeUntil(InternalIterator* iter,
// the compaction iterator to write out the key we're currently at, which
// is the put/delete we just encountered.
if (keys_.empty()) {
return s;
return Status::OK();
}
// TODO(noetzli) If the merge operator returns false, we are currently
@@ -268,9 +268,7 @@ Status MergeHelper::MergeUntil(InternalIterator* iter,
if (keys_.size() == 1) {
// we need to re-anchor the orig_ikey because it was anchored by
// original_key before
Status pikStatus = ParseInternalKey(keys_.back(), &orig_ikey);
pikStatus.PermitUncheckedError();
assert(pikStatus.ok());
ParseInternalKey(keys_.back(), &orig_ikey);
}
if (filter == CompactionFilter::Decision::kKeep) {
merge_context_.PushOperand(
@@ -286,14 +284,14 @@ Status MergeHelper::MergeUntil(InternalIterator* iter,
keys_.clear();
merge_context_.Clear();
has_compaction_filter_skip_until_ = true;
return s;
return Status::OK();
}
}
}
if (merge_context_.GetNumOperands() == 0) {
// we filtered out all the merge operands
return s;
return Status::OK();
}
// We are sure we have seen this key's entire history if:
+2 -2
View File
@@ -25,7 +25,7 @@ void UpdateOptionsFiles(DB* db,
std::unordered_set<std::string>* filename_history,
int* options_files_count) {
std::vector<std::string> filenames;
EXPECT_OK(db->GetEnv()->GetChildren(db->GetName(), &filenames));
db->GetEnv()->GetChildren(db->GetName(), &filenames);
uint64_t number;
FileType type;
*options_files_count = 0;
@@ -42,7 +42,7 @@ void VerifyOptionsFileName(
DB* db, const std::unordered_set<std::string>& past_filenames) {
std::vector<std::string> filenames;
std::unordered_set<std::string> current_filenames;
EXPECT_OK(db->GetEnv()->GetChildren(db->GetName(), &filenames));
db->GetEnv()->GetChildren(db->GetName(), &filenames);
uint64_t number;
FileType type;
for (auto filename : filenames) {
-30
View File
@@ -1,30 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
#include "db/output_validator.h"
namespace ROCKSDB_NAMESPACE {
Status OutputValidator::Add(const Slice& key, const Slice& value) {
if (enable_hash_) {
// Generate a rolling 64-bit hash of the key and values
paranoid_hash_ = Hash64(key.data(), key.size(), paranoid_hash_);
paranoid_hash_ = Hash64(value.data(), value.size(), paranoid_hash_);
}
if (enable_order_check_) {
TEST_SYNC_POINT_CALLBACK("OutputValidator::Add:order_check",
/*arg=*/nullptr);
if (key.size() < kNumInternalBytes) {
return Status::Corruption(
"Compaction tries to write a key without internal bytes.");
}
// prev_key_ starts with empty.
if (!prev_key_.empty() && icmp_.Compare(key, prev_key_) < 0) {
return Status::Corruption("Compaction sees out-of-order keys.");
}
prev_key_.assign(key.data(), key.size());
}
return Status::OK();
}
} // namespace ROCKSDB_NAMESPACE
-45
View File
@@ -1,45 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
#pragma once
#include "db/dbformat.h"
#include "rocksdb/status.h"
#include "test_util/sync_point.h"
#include "util/hash.h"
namespace ROCKSDB_NAMESPACE {
// A class that validates key/value that is inserted to an SST file.
// Pass every key/value of the file using OutputValidator::Add()
// and the class validates key order and optionally calculate a hash
// of all the key and value.
class OutputValidator {
public:
explicit OutputValidator(const InternalKeyComparator& icmp,
bool enable_order_check, bool enable_hash)
: icmp_(icmp),
enable_order_check_(enable_order_check),
enable_hash_(enable_hash) {}
// Add a key to the KV sequence, and return whether the key follows
// criteria, e.g. key is ordered.
Status Add(const Slice& key, const Slice& value);
// Compare result of two key orders are the same. It can be used
// to compare the keys inserted into a file, and what is read back.
// Return true if the validation passes.
bool CompareValidator(const OutputValidator& other_validator) {
return GetHash() == other_validator.GetHash();
}
private:
uint64_t GetHash() const { return paranoid_hash_; }
const InternalKeyComparator& icmp_;
std::string prev_key_;
uint64_t paranoid_hash_ = 0;
bool enable_order_check_;
bool enable_hash_;
};
} // namespace ROCKSDB_NAMESPACE
-7
View File
@@ -817,13 +817,6 @@ TEST_F(PerfContextTest, PerfContextByLevelGetSet) {
}
TEST_F(PerfContextTest, CPUTimer) {
if (Env::Default()->NowCPUNanos() == 0) {
// TODO: This should be a GTEST_SKIP when the embedded gtest is updated
// to 1.10 or higher.
GTEST_SUCCESS_("Skipped on target without NowCPUNanos support");
return;
}
DestroyDB(kDbName, Options());
auto db = OpenDb();
WriteOptions write_options;
-70
View File
@@ -1,70 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#ifndef ROCKSDB_LITE
#include "db/db_impl/db_impl.h"
#include "util/timer.h"
namespace ROCKSDB_NAMESPACE {
// PeriodicWorkScheduler is a singleton object, which is scheduling/running
// DumpStats(), PersistStats(), and FlushInfoLog() for all DB instances. All DB
// instances use the same object from `Default()`.
//
// Internally, it uses a single threaded timer object to run the periodic work
// functions. Timer thread will always be started since the info log flushing
// cannot be disabled.
class PeriodicWorkScheduler {
public:
static PeriodicWorkScheduler* Default();
PeriodicWorkScheduler() = delete;
PeriodicWorkScheduler(const PeriodicWorkScheduler&) = delete;
PeriodicWorkScheduler(PeriodicWorkScheduler&&) = delete;
PeriodicWorkScheduler& operator=(const PeriodicWorkScheduler&) = delete;
PeriodicWorkScheduler& operator=(PeriodicWorkScheduler&&) = delete;
void Register(DBImpl* dbi, unsigned int stats_dump_period_sec,
unsigned int stats_persist_period_sec);
void Unregister(DBImpl* dbi);
// Periodically flush info log out of application buffer at a low frequency.
// This improves debuggability in case of RocksDB hanging since it ensures the
// log messages leading up to the hang will eventually become visible in the
// log.
static const uint64_t kDefaultFlushInfoLogPeriodSec = 10;
protected:
std::unique_ptr<Timer> timer;
explicit PeriodicWorkScheduler(Env* env);
private:
std::string GetTaskName(DBImpl* dbi, const std::string& func_name);
};
#ifndef NDEBUG
// PeriodicWorkTestScheduler is for unittest, which can specify the Env like
// SafeMockTimeEnv. It also contains functions for unittest.
class PeriodicWorkTestScheduler : public PeriodicWorkScheduler {
public:
static PeriodicWorkTestScheduler* Default(Env* env);
void TEST_WaitForRun(std::function<void()> callback) const;
size_t TEST_GetValidTaskNum() const;
private:
explicit PeriodicWorkTestScheduler(Env* env);
};
#endif // !NDEBUG
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+17 -31
View File
@@ -371,11 +371,9 @@ TEST_F(PrefixTest, TestResult) {
ASSERT_TRUE(v16 == iter->value());
iter->Next();
ASSERT_TRUE(!iter->Valid());
ASSERT_OK(iter->status());
SeekIterator(iter.get(), 2, 0);
ASSERT_TRUE(!iter->Valid());
ASSERT_OK(iter->status());
ASSERT_EQ(v16.ToString(), Get(db.get(), read_options, 1, 6));
ASSERT_EQ(kNotFoundResult, Get(db.get(), read_options, 1, 5));
@@ -399,11 +397,9 @@ TEST_F(PrefixTest, TestResult) {
ASSERT_TRUE(v17 == iter->value());
iter->Next();
ASSERT_TRUE(!iter->Valid());
ASSERT_OK(iter->status());
SeekIterator(iter.get(), 2, 0);
ASSERT_TRUE(!iter->Valid());
ASSERT_OK(iter->status());
// 3. Insert an entry for the same prefix as the head of the bucket.
Slice v15("v15");
@@ -546,8 +542,8 @@ TEST_F(PrefixTest, PrefixValid) {
db->Flush(FlushOptions());
TestKey test_key(12346, 8);
std::string s;
ASSERT_OK(db->Delete(write_options, TestKeyToSlice(s, test_key)));
ASSERT_OK(db->Flush(FlushOptions()));
db->Delete(write_options, TestKeyToSlice(s, test_key));
db->Flush(FlushOptions());
read_options.prefix_same_as_start = true;
std::unique_ptr<Iterator> iter(db->NewIterator(read_options));
SeekIterator(iter.get(), 12345, 6);
@@ -572,7 +568,6 @@ TEST_F(PrefixTest, PrefixValid) {
// Verify seeking past the prefix won't return a result.
SeekIterator(iter.get(), 12345, 10);
ASSERT_TRUE(!iter->Valid());
ASSERT_OK(iter->status());
}
}
}
@@ -670,7 +665,6 @@ TEST_F(PrefixTest, DynamicPrefixIterator) {
hist_no_seek_time.Add(timer.ElapsedNanos());
hist_no_seek_comparison.Add(get_perf_context()->user_key_comparison_count);
ASSERT_TRUE(!iter->Valid());
ASSERT_OK(iter->status());
}
std::cout << "non-existing Seek key comparison: \n"
@@ -774,7 +768,6 @@ TEST_F(PrefixTest, PrefixSeekModePrev) {
SliceToTestKey(iter->key()).prefix != stored_prefix) {
break;
}
ASSERT_OK(iter->status());
stored_prefix = SliceToTestKey(iter->key()).prefix;
ASSERT_TRUE(iter->Valid());
ASSERT_NE(it, whole_map.end());
@@ -806,7 +799,7 @@ TEST_F(PrefixTest, PrefixSeekModePrev2) {
options.memtable_factory.reset(new SkipListFactory);
options.write_buffer_size = 1024 * 1024;
std::string v13("v13");
ASSERT_OK(DestroyDB(kDbName, Options()));
DestroyDB(kDbName, Options());
auto db = OpenDb();
WriteOptions write_options;
ReadOptions read_options;
@@ -814,20 +807,17 @@ TEST_F(PrefixTest, PrefixSeekModePrev2) {
PutKey(db.get(), write_options, TestKey(1, 4), "v14");
PutKey(db.get(), write_options, TestKey(3, 3), "v33");
PutKey(db.get(), write_options, TestKey(3, 4), "v34");
ASSERT_OK(db->Flush(FlushOptions()));
ASSERT_OK(
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable());
db->Flush(FlushOptions());
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable();
PutKey(db.get(), write_options, TestKey(1, 1), "v11");
PutKey(db.get(), write_options, TestKey(1, 3), "v13");
PutKey(db.get(), write_options, TestKey(2, 1), "v21");
PutKey(db.get(), write_options, TestKey(2, 2), "v22");
ASSERT_OK(db->Flush(FlushOptions()));
ASSERT_OK(
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable());
db->Flush(FlushOptions());
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable();
std::unique_ptr<Iterator> iter(db->NewIterator(read_options));
SeekIterator(iter.get(), 1, 5);
iter->Prev();
ASSERT_TRUE(iter->Valid());
ASSERT_EQ(iter->value(), v13);
}
@@ -842,29 +832,27 @@ TEST_F(PrefixTest, PrefixSeekModePrev3) {
Slice upper_bound = TestKeyToSlice(s, upper_bound_key);
{
ASSERT_OK(DestroyDB(kDbName, Options()));
DestroyDB(kDbName, Options());
auto db = OpenDb();
WriteOptions write_options;
ReadOptions read_options;
read_options.iterate_upper_bound = &upper_bound;
PutKey(db.get(), write_options, TestKey(1, 2), "v12");
PutKey(db.get(), write_options, TestKey(1, 4), "v14");
ASSERT_OK(db->Flush(FlushOptions()));
ASSERT_OK(
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable());
db->Flush(FlushOptions());
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable();
PutKey(db.get(), write_options, TestKey(1, 1), "v11");
PutKey(db.get(), write_options, TestKey(1, 3), "v13");
PutKey(db.get(), write_options, TestKey(2, 1), "v21");
PutKey(db.get(), write_options, TestKey(2, 2), "v22");
ASSERT_OK(db->Flush(FlushOptions()));
ASSERT_OK(
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable());
db->Flush(FlushOptions());
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable();
std::unique_ptr<Iterator> iter(db->NewIterator(read_options));
iter->SeekToLast();
ASSERT_EQ(iter->value(), v14);
}
{
ASSERT_OK(DestroyDB(kDbName, Options()));
DestroyDB(kDbName, Options());
auto db = OpenDb();
WriteOptions write_options;
ReadOptions read_options;
@@ -873,14 +861,12 @@ TEST_F(PrefixTest, PrefixSeekModePrev3) {
PutKey(db.get(), write_options, TestKey(1, 4), "v14");
PutKey(db.get(), write_options, TestKey(3, 3), "v33");
PutKey(db.get(), write_options, TestKey(3, 4), "v34");
ASSERT_OK(db->Flush(FlushOptions()));
ASSERT_OK(
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable());
db->Flush(FlushOptions());
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable();
PutKey(db.get(), write_options, TestKey(1, 1), "v11");
PutKey(db.get(), write_options, TestKey(1, 3), "v13");
ASSERT_OK(db->Flush(FlushOptions()));
ASSERT_OK(
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable());
db->Flush(FlushOptions());
static_cast_with_check<DBImpl>(db.get())->TEST_WaitForFlushMemTable();
std::unique_ptr<Iterator> iter(db->NewIterator(read_options));
iter->SeekToLast();
ASSERT_EQ(iter->value(), v14);
+6 -9
View File
@@ -33,20 +33,17 @@ TruncatedRangeDelIterator::TruncatedRangeDelIterator(
if (smallest != nullptr) {
pinned_bounds_.emplace_back();
auto& parsed_smallest = pinned_bounds_.back();
Status pikStatus = ParseInternalKey(smallest->Encode(), &parsed_smallest);
pikStatus.PermitUncheckedError();
assert(pikStatus.ok());
if (!ParseInternalKey(smallest->Encode(), &parsed_smallest)) {
assert(false);
}
smallest_ = &parsed_smallest;
}
if (largest != nullptr) {
pinned_bounds_.emplace_back();
auto& parsed_largest = pinned_bounds_.back();
Status pikStatus = ParseInternalKey(largest->Encode(), &parsed_largest);
pikStatus.PermitUncheckedError();
assert(pikStatus.ok());
if (!ParseInternalKey(largest->Encode(), &parsed_largest)) {
assert(false);
}
if (parsed_largest.type == kTypeRangeDeletion &&
parsed_largest.sequence == kMaxSequenceNumber) {
// The file boundary has been artificially extended by a range tombstone.
+1 -5
View File
@@ -283,13 +283,9 @@ class RangeDelAggregator {
bool ShouldDelete(const Slice& key, RangeDelPositioningMode mode) {
ParsedInternalKey parsed;
Status pikStatus = ParseInternalKey(key, &parsed);
assert(pikStatus.ok());
if (!pikStatus.ok()) {
if (!ParseInternalKey(key, &parsed)) {
return false;
}
return ShouldDelete(parsed, mode);
}
virtual bool ShouldDelete(const ParsedInternalKey& parsed,
+39 -65
View File
@@ -119,8 +119,7 @@ class Repairer {
raw_table_cache_.get(), &wb_, &wc_,
/*block_cache_tracer=*/nullptr, /*io_tracer=*/nullptr),
next_file_number_(1),
db_lock_(nullptr),
closed_(false) {
db_lock_(nullptr) {
for (const auto& cfd : column_families) {
cf_name_to_opts_[cfd.name] = cfd.options;
}
@@ -164,41 +163,31 @@ class Repairer {
return status;
}
Status Close() {
Status s = Status::OK();
if (!closed_) {
if (db_lock_ != nullptr) {
s = env_->UnlockFile(db_lock_);
db_lock_ = nullptr;
}
closed_ = true;
~Repairer() {
if (db_lock_ != nullptr) {
env_->UnlockFile(db_lock_);
}
return s;
delete table_cache_;
}
~Repairer() { Close().PermitUncheckedError(); }
Status Run() {
Status status = env_->LockFile(LockFileName(dbname_), &db_lock_);
if (!status.ok()) {
return status;
}
status = FindFiles();
DBImpl* db_impl = nullptr;
if (status.ok()) {
// Discard older manifests and start a fresh one
for (size_t i = 0; i < manifests_.size(); i++) {
ArchiveFile(dbname_ + "/" + manifests_[i]);
}
// Just create a DBImpl temporarily so we can reuse NewDB()
db_impl = new DBImpl(db_options_, dbname_);
DBImpl* db_impl = new DBImpl(db_options_, dbname_);
// Also use this temp DBImpl to get a session id
status = db_impl->GetDbSessionId(db_session_id_);
}
if (status.ok()) {
db_impl->GetDbSessionId(db_session_id_);
status = db_impl->NewDB(/*new_filenames=*/nullptr);
delete db_impl;
}
delete db_impl;
if (status.ok()) {
// Recover using the fresh manifest created by NewDB()
@@ -253,7 +242,7 @@ class Repairer {
const ColumnFamilyOptions unknown_cf_opts_;
const bool create_unknown_cfs_;
std::shared_ptr<Cache> raw_table_cache_;
std::unique_ptr<TableCache> table_cache_;
TableCache* table_cache_;
WriteBufferManager wb_;
WriteController wc_;
VersionSet vset_;
@@ -268,7 +257,6 @@ class Repairer {
// Lock over the persistent DB state. Non-nullptr iff successfully
// acquired.
FileLock* db_lock_;
bool closed_;
Status FindFiles() {
std::vector<std::string> filenames;
@@ -397,16 +385,15 @@ class Repairer {
record.size(), Status::Corruption("log record too small"));
continue;
}
Status record_status = WriteBatchInternal::SetContents(&batch, record);
if (record_status.ok()) {
record_status =
WriteBatchInternal::InsertInto(&batch, cf_mems, nullptr, nullptr);
}
if (record_status.ok()) {
WriteBatchInternal::SetContents(&batch, record);
status =
WriteBatchInternal::InsertInto(&batch, cf_mems, nullptr, nullptr);
if (status.ok()) {
counter += WriteBatchInternal::Count(&batch);
} else {
ROCKS_LOG_WARN(db_options_.info_log, "Log #%" PRIu64 ": ignoring %s",
log, record_status.ToString().c_str());
log, status.ToString().c_str());
status = Status::OK(); // Keep going with rest of file
}
}
@@ -442,19 +429,18 @@ class Repairer {
LegacyFileSystemWrapper fs(env_);
IOStatus io_s;
status = BuildTable(
dbname_, /* versions */ nullptr, env_, &fs, *cfd->ioptions(),
*cfd->GetLatestMutableCFOptions(), env_options_, table_cache_.get(),
dbname_, env_, &fs, *cfd->ioptions(),
*cfd->GetLatestMutableCFOptions(), env_options_, table_cache_,
iter.get(), std::move(range_del_iters), &meta,
nullptr /* blob_file_additions */, cfd->internal_comparator(),
cfd->int_tbl_prop_collector_factories(), cfd->GetID(), cfd->GetName(),
{}, kMaxSequenceNumber, snapshot_checker, kNoCompression,
0 /* sample_for_compression */, CompressionOptions(), false,
nullptr /* internal_stats */, TableFileCreationReason::kRecovery,
&io_s, nullptr /*IOTracer*/, nullptr /* event_logger */,
0 /* job_id */, Env::IO_HIGH, nullptr /* table_properties */,
-1 /* level */, current_time, 0 /* oldest_key_time */, write_hint,
0 /* file_creation_time */, "DB Repairer" /* db_id */,
db_session_id_);
cfd->internal_comparator(), cfd->int_tbl_prop_collector_factories(),
cfd->GetID(), cfd->GetName(), {}, kMaxSequenceNumber,
snapshot_checker, kNoCompression, 0 /* sample_for_compression */,
CompressionOptions(), false, nullptr /* internal_stats */,
TableFileCreationReason::kRecovery, &io_s, nullptr /*IOTracer*/,
nullptr /* event_logger */, 0 /* job_id */, Env::IO_HIGH,
nullptr /* table_properties */, -1 /* level */, current_time,
0 /* oldest_key_time */, write_hint, 0 /* file_creation_time */,
"DB Repairer" /* db_id */, db_session_id_);
ROCKS_LOG_INFO(db_options_.info_log,
"Log #%" PRIu64 ": %d ops saved to Table #%" PRIu64 " %s",
log, counter, meta.fd.GetNumber(),
@@ -554,7 +540,7 @@ class Repairer {
ParsedInternalKey parsed;
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
Slice key = iter->key();
if (ParseInternalKey(key, &parsed) != Status::OK()) {
if (!ParseInternalKey(key, &parsed)) {
ROCKS_LOG_ERROR(db_options_.info_log,
"Table #%" PRIu64 ": unparsable key %s",
t->meta.fd.GetNumber(), EscapeString(key).c_str());
@@ -636,7 +622,7 @@ class Repairer {
new_dir.assign(fname.data(), slash - fname.data());
}
new_dir.append("/lost");
env_->CreateDir(new_dir).PermitUncheckedError(); // Ignore error
env_->CreateDir(new_dir); // Ignore error
std::string new_file = new_dir;
new_file.append("/");
new_file.append((slash == nullptr) ? fname.c_str() : slash + 1);
@@ -668,16 +654,12 @@ Status RepairDB(const std::string& dbname, const DBOptions& db_options,
) {
ColumnFamilyOptions default_cf_opts;
Status status = GetDefaultCFOptions(column_families, &default_cf_opts);
if (!status.ok()) {
return status;
}
Repairer repairer(dbname, db_options, column_families, default_cf_opts,
ColumnFamilyOptions() /* unknown_cf_opts */,
false /* create_unknown_cfs */);
status = repairer.Run();
if (status.ok()) {
status = repairer.Close();
Repairer repairer(dbname, db_options, column_families,
default_cf_opts,
ColumnFamilyOptions() /* unknown_cf_opts */,
false /* create_unknown_cfs */);
status = repairer.Run();
}
return status;
}
@@ -687,33 +669,25 @@ Status RepairDB(const std::string& dbname, const DBOptions& db_options,
const ColumnFamilyOptions& unknown_cf_opts) {
ColumnFamilyOptions default_cf_opts;
Status status = GetDefaultCFOptions(column_families, &default_cf_opts);
if (!status.ok()) {
return status;
}
Repairer repairer(dbname, db_options, column_families, default_cf_opts,
unknown_cf_opts, true /* create_unknown_cfs */);
status = repairer.Run();
if (status.ok()) {
status = repairer.Close();
Repairer repairer(dbname, db_options,
column_families, default_cf_opts,
unknown_cf_opts, true /* create_unknown_cfs */);
status = repairer.Run();
}
return status;
}
Status RepairDB(const std::string& dbname, const Options& options) {
Options opts(options);
DBOptions db_options(opts);
ColumnFamilyOptions cf_options(opts);
Repairer repairer(dbname, db_options,
{}, cf_options /* default_cf_opts */,
cf_options /* unknown_cf_opts */,
true /* create_unknown_cfs */);
Status status = repairer.Run();
if (status.ok()) {
status = repairer.Close();
}
return status;
return repairer.Run();
}
} // namespace ROCKSDB_NAMESPACE
+58 -79
View File
@@ -24,33 +24,28 @@ class RepairTest : public DBTestBase {
public:
RepairTest() : DBTestBase("/repair_test", /*env_do_fsync=*/true) {}
Status GetFirstSstPath(std::string* first_sst_path) {
assert(first_sst_path != nullptr);
first_sst_path->clear();
std::string GetFirstSstPath() {
uint64_t manifest_size;
std::vector<std::string> files;
Status s = db_->GetLiveFiles(files, &manifest_size);
if (s.ok()) {
auto sst_iter =
std::find_if(files.begin(), files.end(), [](const std::string& file) {
uint64_t number;
FileType type;
bool ok = ParseFileName(file, &number, &type);
return ok && type == kTableFile;
});
*first_sst_path = sst_iter == files.end() ? "" : dbname_ + *sst_iter;
}
return s;
db_->GetLiveFiles(files, &manifest_size);
auto sst_iter =
std::find_if(files.begin(), files.end(), [](const std::string& file) {
uint64_t number;
FileType type;
bool ok = ParseFileName(file, &number, &type);
return ok && type == kTableFile;
});
return sst_iter == files.end() ? "" : dbname_ + *sst_iter;
}
};
TEST_F(RepairTest, LostManifest) {
// Add a couple SST files, delete the manifest, and verify RepairDB() saves
// the day.
ASSERT_OK(Put("key", "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put("key2", "val2"));
ASSERT_OK(Flush());
Put("key", "val");
Flush();
Put("key2", "val2");
Flush();
// Need to get path before Close() deletes db_, but delete it after Close() to
// ensure Close() didn't change the manifest.
std::string manifest_path =
@@ -68,10 +63,10 @@ TEST_F(RepairTest, LostManifest) {
TEST_F(RepairTest, CorruptManifest) {
// Manifest is in an invalid format. Expect a full recovery.
ASSERT_OK(Put("key", "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put("key2", "val2"));
ASSERT_OK(Flush());
Put("key", "val");
Flush();
Put("key2", "val2");
Flush();
// Need to get path before Close() deletes db_, but overwrite it after Close()
// to ensure Close() didn't change the manifest.
std::string manifest_path =
@@ -81,7 +76,7 @@ TEST_F(RepairTest, CorruptManifest) {
ASSERT_OK(env_->FileExists(manifest_path));
LegacyFileSystemWrapper fs(env_);
ASSERT_OK(CreateFile(&fs, manifest_path, "blah", false /* use_fsync */));
CreateFile(&fs, manifest_path, "blah", false /* use_fsync */);
ASSERT_OK(RepairDB(dbname_, CurrentOptions()));
Reopen(CurrentOptions());
@@ -92,13 +87,13 @@ TEST_F(RepairTest, CorruptManifest) {
TEST_F(RepairTest, IncompleteManifest) {
// In this case, the manifest is valid but does not reference all of the SST
// files. Expect a full recovery.
ASSERT_OK(Put("key", "val"));
ASSERT_OK(Flush());
Put("key", "val");
Flush();
std::string orig_manifest_path =
DescriptorFileName(dbname_, dbfull()->TEST_Current_Manifest_FileNo());
CopyFile(orig_manifest_path, orig_manifest_path + ".tmp");
ASSERT_OK(Put("key2", "val2"));
ASSERT_OK(Flush());
Put("key2", "val2");
Flush();
// Need to get path before Close() deletes db_, but overwrite it after Close()
// to ensure Close() didn't change the manifest.
std::string new_manifest_path =
@@ -118,10 +113,10 @@ TEST_F(RepairTest, IncompleteManifest) {
TEST_F(RepairTest, PostRepairSstFileNumbering) {
// Verify after a DB is repaired, new files will be assigned higher numbers
// than old files.
ASSERT_OK(Put("key", "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put("key2", "val2"));
ASSERT_OK(Flush());
Put("key", "val");
Flush();
Put("key2", "val2");
Flush();
uint64_t pre_repair_file_num = dbfull()->TEST_Current_Next_FileNo();
Close();
@@ -135,12 +130,11 @@ TEST_F(RepairTest, PostRepairSstFileNumbering) {
TEST_F(RepairTest, LostSst) {
// Delete one of the SST files but preserve the manifest that refers to it,
// then verify the DB is still usable for the intact SST.
ASSERT_OK(Put("key", "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put("key2", "val2"));
ASSERT_OK(Flush());
std::string sst_path;
ASSERT_OK(GetFirstSstPath(&sst_path));
Put("key", "val");
Flush();
Put("key2", "val2");
Flush();
auto sst_path = GetFirstSstPath();
ASSERT_FALSE(sst_path.empty());
ASSERT_OK(env_->DeleteFile(sst_path));
@@ -155,16 +149,15 @@ TEST_F(RepairTest, LostSst) {
TEST_F(RepairTest, CorruptSst) {
// Corrupt one of the SST files but preserve the manifest that refers to it,
// then verify the DB is still usable for the intact SST.
ASSERT_OK(Put("key", "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put("key2", "val2"));
ASSERT_OK(Flush());
std::string sst_path;
ASSERT_OK(GetFirstSstPath(&sst_path));
Put("key", "val");
Flush();
Put("key2", "val2");
Flush();
auto sst_path = GetFirstSstPath();
ASSERT_FALSE(sst_path.empty());
LegacyFileSystemWrapper fs(env_);
ASSERT_OK(CreateFile(&fs, sst_path, "blah", false /* use_fsync */));
CreateFile(&fs, sst_path, "blah", false /* use_fsync */);
Close();
ASSERT_OK(RepairDB(dbname_, CurrentOptions()));
@@ -177,16 +170,13 @@ TEST_F(RepairTest, CorruptSst) {
TEST_F(RepairTest, UnflushedSst) {
// This test case invokes repair while some data is unflushed, then verifies
// that data is in the db.
ASSERT_OK(Put("key", "val"));
Put("key", "val");
VectorLogPtr wal_files;
ASSERT_OK(dbfull()->GetSortedWalFiles(wal_files));
ASSERT_EQ(wal_files.size(), 1);
{
uint64_t total_ssts_size;
std::unordered_map<std::string, uint64_t> sst_files;
ASSERT_OK(GetAllSSTFiles(&sst_files, &total_ssts_size));
ASSERT_EQ(total_ssts_size, 0);
}
uint64_t total_ssts_size;
GetAllSSTFiles(&total_ssts_size);
ASSERT_EQ(total_ssts_size, 0);
// Need to get path before Close() deletes db_, but delete it after Close() to
// ensure Close() didn't change the manifest.
std::string manifest_path =
@@ -200,12 +190,8 @@ TEST_F(RepairTest, UnflushedSst) {
ASSERT_OK(dbfull()->GetSortedWalFiles(wal_files));
ASSERT_EQ(wal_files.size(), 0);
{
uint64_t total_ssts_size;
std::unordered_map<std::string, uint64_t> sst_files;
ASSERT_OK(GetAllSSTFiles(&sst_files, &total_ssts_size));
ASSERT_GT(total_ssts_size, 0);
}
GetAllSSTFiles(&total_ssts_size);
ASSERT_GT(total_ssts_size, 0);
ASSERT_EQ(Get("key"), "val");
}
@@ -213,17 +199,14 @@ TEST_F(RepairTest, SeparateWalDir) {
do {
Options options = CurrentOptions();
DestroyAndReopen(options);
ASSERT_OK(Put("key", "val"));
ASSERT_OK(Put("foo", "bar"));
Put("key", "val");
Put("foo", "bar");
VectorLogPtr wal_files;
ASSERT_OK(dbfull()->GetSortedWalFiles(wal_files));
ASSERT_EQ(wal_files.size(), 1);
{
uint64_t total_ssts_size;
std::unordered_map<std::string, uint64_t> sst_files;
ASSERT_OK(GetAllSSTFiles(&sst_files, &total_ssts_size));
ASSERT_EQ(total_ssts_size, 0);
}
uint64_t total_ssts_size;
GetAllSSTFiles(&total_ssts_size);
ASSERT_EQ(total_ssts_size, 0);
std::string manifest_path =
DescriptorFileName(dbname_, dbfull()->TEST_Current_Manifest_FileNo());
@@ -238,12 +221,8 @@ TEST_F(RepairTest, SeparateWalDir) {
Reopen(options);
ASSERT_OK(dbfull()->GetSortedWalFiles(wal_files));
ASSERT_EQ(wal_files.size(), 0);
{
uint64_t total_ssts_size;
std::unordered_map<std::string, uint64_t> sst_files;
ASSERT_OK(GetAllSSTFiles(&sst_files, &total_ssts_size));
ASSERT_GT(total_ssts_size, 0);
}
GetAllSSTFiles(&total_ssts_size);
ASSERT_GT(total_ssts_size, 0);
ASSERT_EQ(Get("key"), "val");
ASSERT_EQ(Get("foo"), "bar");
@@ -259,13 +238,13 @@ TEST_F(RepairTest, RepairMultipleColumnFamilies) {
CreateAndReopenWithCF({"pikachu1", "pikachu2"}, CurrentOptions());
for (int i = 0; i < kNumCfs; ++i) {
for (int j = 0; j < kEntriesPerCf; ++j) {
ASSERT_OK(Put(i, "key" + ToString(j), "val" + ToString(j)));
Put(i, "key" + ToString(j), "val" + ToString(j));
if (j == kEntriesPerCf - 1 && i == kNumCfs - 1) {
// Leave one unflushed so we can verify WAL entries are properly
// associated with column families.
continue;
}
ASSERT_OK(Flush(i));
Flush(i);
}
}
@@ -304,12 +283,12 @@ TEST_F(RepairTest, RepairColumnFamilyOptions) {
std::vector<Options>{opts, rev_opts});
for (int i = 0; i < kNumCfs; ++i) {
for (int j = 0; j < kEntriesPerCf; ++j) {
ASSERT_OK(Put(i, "key" + ToString(j), "val" + ToString(j)));
Put(i, "key" + ToString(j), "val" + ToString(j));
if (i == kNumCfs - 1 && j == kEntriesPerCf - 1) {
// Leave one unflushed so we can verify RepairDB's flush logic
continue;
}
ASSERT_OK(Flush(i));
Flush(i);
}
}
Close();
@@ -329,7 +308,7 @@ TEST_F(RepairTest, RepairColumnFamilyOptions) {
// Examine table properties to verify RepairDB() used the right options when
// converting WAL->SST
TablePropertiesCollection fname_to_props;
ASSERT_OK(db_->GetPropertiesOfAllTables(handles_[1], &fname_to_props));
db_->GetPropertiesOfAllTables(handles_[1], &fname_to_props);
ASSERT_EQ(fname_to_props.size(), 2U);
for (const auto& fname_and_props : fname_to_props) {
std::string comparator_name (
@@ -363,8 +342,8 @@ TEST_F(RepairTest, DbNameContainsTrailingSlash) {
}
}
ASSERT_OK(Put("key", "val"));
ASSERT_OK(Flush());
Put("key", "val");
Flush();
Close();
ASSERT_OK(RepairDB(dbname_ + "/", CurrentOptions()));
+21 -26
View File
@@ -162,6 +162,7 @@ Status TableCache::FindTable(const ReadOptions& ro,
int level, bool prefetch_index_and_filter_in_cache,
size_t max_file_size_for_l0_meta_pin) {
PERF_TIMER_GUARD_WITH_ENV(find_table_nanos, ioptions_.env);
Status s;
uint64_t number = fd.GetNumber();
Slice key = GetSliceForFileNumber(&number);
*handle = cache_->Lookup(key);
@@ -169,22 +170,22 @@ Status TableCache::FindTable(const ReadOptions& ro,
const_cast<bool*>(&no_io));
if (*handle == nullptr) {
if (no_io) {
if (no_io) { // Don't do IO and return a not-found status
return Status::Incomplete("Table not found in table_cache, no_io is set");
}
MutexLock load_lock(loader_mutex_.get(key));
// We check the cache again under loading mutex
*handle = cache_->Lookup(key);
if (*handle != nullptr) {
return Status::OK();
return s;
}
std::unique_ptr<TableReader> table_reader;
Status s = GetTableReader(
ro, file_options, internal_comparator, fd, false /* sequential mode */,
record_read_stats, file_read_hist, &table_reader, prefix_extractor,
skip_filters, level, prefetch_index_and_filter_in_cache,
max_file_size_for_l0_meta_pin);
s = GetTableReader(ro, file_options, internal_comparator, fd,
false /* sequential mode */, record_read_stats,
file_read_hist, &table_reader, prefix_extractor,
skip_filters, level, prefetch_index_and_filter_in_cache,
max_file_size_for_l0_meta_pin);
if (!s.ok()) {
assert(table_reader == nullptr);
RecordTick(ioptions_.statistics, NO_FILE_ERRORS);
@@ -198,9 +199,8 @@ Status TableCache::FindTable(const ReadOptions& ro,
table_reader.release();
}
}
return s;
}
return Status::OK();
return s;
}
InternalIterator* TableCache::NewIterator(
@@ -413,8 +413,7 @@ Status TableCache::Get(const ReadOptions& options,
Status s;
TableReader* t = fd.table_reader;
Cache::Handle* handle = nullptr;
if (!done) {
assert(s.ok());
if (!done && s.ok()) {
if (t == nullptr) {
s = FindTable(options, file_options_, internal_comparator, fd, &handle,
prefix_extractor,
@@ -456,11 +455,8 @@ Status TableCache::Get(const ReadOptions& options,
size_t charge =
row_cache_key.Size() + row_cache_entry->size() + sizeof(std::string);
void* row_ptr = new std::string(std::move(*row_cache_entry));
// If row cache is full, it's OK to continue.
ioptions_.row_cache
->Insert(row_cache_key.GetUserKey(), row_ptr, charge,
&DeleteEntry<std::string>)
.PermitUncheckedError();
ioptions_.row_cache->Insert(row_cache_key.GetUserKey(), row_ptr, charge,
&DeleteEntry<std::string>);
}
#endif // ROCKSDB_LITE
@@ -579,11 +575,8 @@ Status TableCache::MultiGet(const ReadOptions& options,
size_t charge =
row_cache_key.Size() + row_cache_entry.size() + sizeof(std::string);
void* row_ptr = new std::string(std::move(row_cache_entry));
// If row cache is full, it's OK.
ioptions_.row_cache
->Insert(row_cache_key.GetUserKey(), row_ptr, charge,
&DeleteEntry<std::string>)
.PermitUncheckedError();
ioptions_.row_cache->Insert(row_cache_key.GetUserKey(), row_ptr, charge,
&DeleteEntry<std::string>);
}
}
}
@@ -600,17 +593,18 @@ Status TableCache::GetTableProperties(
const InternalKeyComparator& internal_comparator, const FileDescriptor& fd,
std::shared_ptr<const TableProperties>* properties,
const SliceTransform* prefix_extractor, bool no_io) {
Status s;
auto table_reader = fd.table_reader;
// table already been pre-loaded?
if (table_reader) {
*properties = table_reader->GetTableProperties();
return Status::OK();
return s;
}
Cache::Handle* table_handle = nullptr;
Status s = FindTable(ReadOptions(), file_options, internal_comparator, fd,
&table_handle, prefix_extractor, no_io);
s = FindTable(ReadOptions(), file_options, internal_comparator, fd,
&table_handle, prefix_extractor, no_io);
if (!s.ok()) {
return s;
}
@@ -625,6 +619,7 @@ size_t TableCache::GetMemoryUsageByTableReader(
const FileOptions& file_options,
const InternalKeyComparator& internal_comparator, const FileDescriptor& fd,
const SliceTransform* prefix_extractor) {
Status s;
auto table_reader = fd.table_reader;
// table already been pre-loaded?
if (table_reader) {
@@ -632,8 +627,8 @@ size_t TableCache::GetMemoryUsageByTableReader(
}
Cache::Handle* table_handle = nullptr;
Status s = FindTable(ReadOptions(), file_options, internal_comparator, fd,
&table_handle, prefix_extractor, true);
s = FindTable(ReadOptions(), file_options, internal_comparator, fd,
&table_handle, prefix_extractor, true);
if (!s.ok()) {
return 0;
}

Some files were not shown because too many files have changed in this diff Show More