Summary:
while debugging stress test failure, I noticed that sst_dump and ldb do not work if custom db_stress compression manager is used. This PR adds support for it.
```
./sst_dump --command=raw --show_properties --file=/tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst
options.env is 0x7f2b1f4b9000
Process /tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst
Sst file format: block-based
/tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst: Not implemented: Could not load CompressionManager: DbStressCustom1
/tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst is not a valid SST file
./ldb idump --db=/tmp/rocksdb_crashtest_whiteboxy_emah11 --ignore_unknown_options --hex >> /tmp/i_dump
Failed: Not implemented: Could not load CompressionManager: DbStressCustom1
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13827
Test Plan: manually tested that ldb and sst_dump work with DbStressCustomCompressionManager after this PR
Reviewed By: pdillinger
Differential Revision: D79461175
Pulled By: cbi42
fbshipit-source-id: c8c092b10b4fde3a295b00751057749e8f0cf095
Summary:
* Make new format_version=7 a supported setting.
* Fix a bug in compressed_secondary_cache.cc that is newly exercised by custom compression types and showing up in crash test with tiered secondary cache
* Small change to handling of disabled compression in fv=7: use empty compression manager compatibility name.
* Get rid of GetDefaultBuiltinCompressionManager() in public API because it could cause unexpected+unsafe schema change on a user's CompressionManager if built upon the default built-in manager and we add a new built-in schema. Now must be referenced by explicit compression schema version in the public API. (That notion was already exposed in compressed secondary cache API, for better or worse.)
* Improve some error messages for compression misconfiguration
* Improve testing with ObjectLibrary and CompressionManagers
* Improve testing of compression_name table property in BlockBasedTableTest.BlockBasedTableProperties2
* Improve some comments
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13713
Test Plan: existing and updated tests. Notably, the crash test has already been running with (unpublished) format_version=7
Reviewed By: mszeszko-meta, hx235
Differential Revision: D77035482
Pulled By: pdillinger
fbshipit-source-id: 95278de8734a79706a22361bff2184b1edb230ca
Summary:
Some pieces of follow-up to https://github.com/facebook/rocksdb/issues/13659.
_Recommend hiding whitespace for review_
* Add support for instantiating CompressionManagers through CreateFromString/ObjectLibrary.
* Pull CompressorCustomAlg and DecompressorCustomAlg out of db_test2, refactor/improvement them a bit, and put them in testutil.h for sharing with db_stress. Switched it from being built on snappy to being built on lz4 so that it can properly test dictionary compression.
* Add a custom compression manager for db_stress that uses these, and add to crash test. This depends on the ObjectLibrary stuff because some invocations of db_stress will not be configured with the custom compression manager but will need to access it to read some existing SST files.
* Remove some pieces where the concern of setting compression=kZSTD for compatibility purposes had leaked into configuring some tests and compression managers. After https://github.com/facebook/rocksdb/issues/13659 this compatibility concern is contained in the SST building code.
* Fix BuiltinDecompressorV2SnappyOnly hiding the (ignored) compression dictionary. SST read logic expects the serialized dictionary to be returned by the decompressor even if it's effectively ignored. Updated DBBlockCacheTest.CacheCompressionDict to cover this case.
For follow-up:
* Combine custom compression and mixed compression types in a file (not clean/easy without duplicating or majorly refactoring the mixed/random compressor)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13710
Test Plan: unit tests updated
Reviewed By: hx235
Differential Revision: D76928974
Pulled By: pdillinger
fbshipit-source-id: 772cf9cb048d737699b0e2887c624fb64a68aa8c
Changyu Bi