Compare commits

..

24 Commits

Author SHA1 Message Date
sdong 1ad2267284 Disable error as warning 2019-11-05 11:01:40 -08:00
sdong 6d353bc011 Add one more #include<functional> 2019-11-05 11:01:40 -08:00
sdong 622456584d Add some include<functional> 2019-10-31 14:32:48 -07:00
sdong 4887a3f680 [FB Internal] Point to the latest tool chain. 2019-10-31 14:32:48 -07:00
sdong bcf926b664 [fb only] revert unintended change of USE_SSE
The previuos change that use gcc-5 set USE_SSE to wrong flag by mistake. Fix it.
2017-07-17 22:21:24 -07:00
sdong e0ff529d5d [FB Only] use gcc-5 2017-07-17 21:39:43 -07:00
Islam AbdelRahman bcaf392114 backport fbcode gcc path fix 2016-06-08 12:50:09 -07:00
agiardullo b8c78da6ed update fb-specific dependencies by running update_dependencies.sh 2016-02-18 15:13:33 -08:00
Islam AbdelRahman 5dac6ba2dc Update fbcode_config4.8.1.sh to use update_dependencies.sh
Summary:
This patch update fbcode_config4.8.1.sh to get it's dependencies the same way we updated fbcode_config.sh in D53037

as a result zstd is upgraded to 0.4.7 instead of 0.4.5

Test Plan:
make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 make check -j64
make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 USE_CLANG=1 make check -j64

Reviewers: yhchiang, andrewkr, rven, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53355

(cherry picked from commit 77ef87ccb0)
2016-02-18 15:12:29 -08:00
Islam AbdelRahman 61d6cab888 Include rest of dependencies in dependencies.sh
Summary:
This diff

  - Include the rest of the dependencies (kernel-headers, binutils, valgrind) in dependencies.sh
  - updtade zst to 0.4.7
  - It also fix a problem in clang scan build

Test Plan:
make check
USE_CLANG=1 make check
USE_CLANG=1 make analyze
coverage_test.sh

Reviewers: sdong, yhchiang, rven, andrewkr, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53301

(cherry picked from commit f7ef1a6132)
2016-02-18 15:11:14 -08:00
Islam AbdelRahman 26a2fb1cea Update fbcode_config.sh to use latest versions automatically
Summary:
Update fbcode_config.sh so that It try to use the latest version for dependencies that we are using, after updating the code these libraries where updated

```
Snappy: 1.0.3 => 1.1.3
GFLAGS: 1.6 => 2.1.1
JEMALLOC: 3.6.0 => 4.0.3
```

I have also updated clang from 3.7 to 3.7.1

```
Clang 3.7 => 3.7.1
```

Another change is that we use the same tp2 directory as fbcode, so we dont need to keep changing commit hash every time we need to change a version of a compiler or a library

Test Plan:
make check -j64
USE_CLANG=1 make check -j64

DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32 (make sure it's running)

Reviewers: yhchiang, anthony, rven, kradhakrishnan, andrewkr, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53037

(cherry picked from commit 538eec0661)
2016-02-18 15:11:01 -08:00
agiardullo 41eb08a960 update build version to 4.4.1
Summary: prepare to release 4.4.1

Test Plan: unit tests pass

Reviewers: kradhakrishnan, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54393
2016-02-18 14:40:13 -08:00
agiardullo e7708343f3 Fix transaction locking
Summary: Broke transaction locking in 4.4 in D52197.  Will cherry-pick this change into 4.4 (which hasn't yet been fully released).  Repro'd using db_bench.

Test Plan: unit tests and db_Bench

Reviewers: sdong, yhchiang, kradhakrishnan, ngbronson

Reviewed By: ngbronson

Subscribers: ngbronson, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54021

(cherry picked from commit d08d50295c)
2016-02-18 13:37:45 -08:00
Yueh-Hsuan Chiang dbffb828fa Remove the SyncPoint usage in the destructor of PosixEnv
Summary:
Remove the SyncPoint usage in the destructor of PosixEnv as none
of any active tests is using it.

SyncPoint is a test-only utility class, and it's a static varible.
As a result, using SyncPoint in the destructor of PosixEnv will
make default Env depends on SyncPoint.  Removing such dependency
could solve the problem crash issue only reproducable in Mac
environment.

Test Plan: OPT=-DTRAVIS V=1 make -j4 check on Mac environment

Reviewers: sdong, anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54333
2016-02-18 00:47:36 -08:00
Yueh-Hsuan Chiang 2ed9b9206b Add missing SyncPoint::DisableProcessing in compact_file_test
Summary: Add missing SyncPoint::DisableProcessing in compact_file_test

Test Plan: compact_file_test

Reviewers: anthony, sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54327
2016-02-17 23:21:58 -08:00
Yueh-Hsuan Chiang 4860bfdeab Fixed a dependency issue of ThreadLocalPtr
Summary:
When a child thread that uses ThreadLocalPtr, ThreadLocalPtr::OnThreadExit
will be called when that child thread is destroyed.  However,
OnThreadExit will try to access a static singleton of ThreadLocalPtr,
which will be destroyed when the main thread exit.  As a result,
when a child thread that uses ThreadLocalPtr exits AFTER the main thread
exits, illegal memory access will occur.

This diff includes a test that reproduce this legacy bug.

    ==2095206==ERROR: AddressSanitizer: heap-use-after-free on address
    0x608000007fa0 at pc 0x959b79 bp 0x7f5fa7426b60 sp 0x7f5fa7426b58
    READ of size 8 at 0x608000007fa0 thread T1

This patch fix this issue by having the thread local mutex never be deleted
(but will leak small piece of memory at the end.)   The patch also describe
a better solution (thread_local) in the comment that requires gcc 4.8.1 and
in latest clang as a future work once we agree to move toward gcc 4.8.

Test Plan:
COMPILE_WITH_ASAN=1 make thread_local_test -j32
./thread_local_test --gtest_filter="*MainThreadDiesFirst"

Reviewers: anthony, hermanlee4, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53013
2016-02-10 16:56:21 -08:00
sdong 74bcf18772 Fix LITE db_test build broken by previous commit
Summary: Previous commit introduces a test that is not supported in LITE. Fix it.

Test Plan: Build the test with ROCKSDB_LITE.

Reviewers: kradhakrishnan, IslamAbdelRahman, anthony, yhchiang, andrewkr

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53901
2016-02-05 15:36:25 -08:00
sdong 18a4f6a1de Explictly fail when memtable doesn't support concurrent insert
Summary: If users turn on concurrent insert but the memtable doesn't support it, they might see unexcepted crash. Fix it by explicitly fail.

Test Plan:
Run different setting of stress_test and make sure it fails correctly.
Will add a unit test too.

Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, andrewkr, ngbronson

Reviewed By: ngbronson

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53895
2016-02-05 15:35:37 -08:00
sdong f300da40d0 Fix merge erorr in previous cherry-pick 2016-02-05 09:03:59 -08:00
Nathan Bronson b8f656767a always invalidate sequential-insertion cache for concurrent skiplist adds
Summary:
InlineSkipList::InsertConcurrently should invalidate the
sequential-insertion cache prev_[] for all inserts of multi-level nodes,
not just those that increase the height of the skip list.  The invariant
for prev_ is that prev_[i] (i > 0) is supposed to be the predecessor of
prev_[0] at level i.  Before this diff InsertConcurrently could violate
this constraint when inserting a multi-level node after prev_[i] but
before prev_[0].

This diff also reenables kConcurrentSkipList as db_test's
MultiThreaded/MultiThreadedDBTest.MultiThreaded/29.

Test Plan:
1. unit tests
2. temporarily hack kConcurrentSkipList timing so that it is fast but has a 1.5% failure rate on my dev box (1ms stagger on thread launch, 1s test duration, failure rate baseline over 1000 runs)
3. observe 1000 passes post-fix

Reviewers: igor, sdong

Reviewed By: sdong

Subscribers: MarkCallaghan, dhruba

Differential Revision: https://reviews.facebook.net/D53751
2016-02-04 14:21:27 -08:00
Nathan Bronson fab086daa1 Fix for --allow_concurrent_memtable_write with batching
Summary:
Concurrent memtable adds were incorrectly computing
the last sequence number for a write batch group when the
write batches were not solitary.  This is the cause of
https://github.com/facebook/mysql-5.6/issues/155

Test Plan:
1. unit tests
2. new unit test
3. parallel db_bench stress tests with batch size of 10 and asserts enabled

Reviewers: igor, sdong

Reviewed By: sdong

Subscribers: IslamAbdelRahman, MarkCallaghan, dhruba

Differential Revision: https://reviews.facebook.net/D53595
2016-02-04 14:19:34 -08:00
sdong f9a3440b76 Should not skip bloom filter for L0 during the query.
Summary: It's a regression bug caused by e089db40f9. With the change, if options.optimize_filters_for_hits=true and there are only L0 files (like single level universal compaction), we skip all the files in L0, which is more than necessary. Fix it by always trying to query bloom filter for files in level 0.

Test Plan: Add a unit test for it.

Reviewers: anthony, rven, yhchiang, IslamAbdelRahman, kradhakrishnan, andrewkr

Reviewed By: andrewkr

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53493
2016-01-27 16:17:02 -08:00
Islam AbdelRahman 3bbf0e1e4f Fix issue in Iterator::Seek when using Block based filter block with prefix_extractor
Summary: Similar to D53385 we need to check InDomain before checking the filter block.

Test Plan: unit tests

Reviewers: yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53421
2016-01-27 15:55:22 -08:00
Islam AbdelRahman 0be02649d8 Fix bug in block based tables with full filter block and prefix_extractor
Summary:
Right now when we are creating a BlockBasedTable with fill filter block
we add to the filter all the prefixes that are InDomain() based on the prefix_extractor

the problem is that when we read a key from the file, we check the filter block for the prefix whether or not it's InDomain()

Test Plan: unit tests

Reviewers: yhchiang, rven, anthony, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53385
2016-01-27 15:55:02 -08:00
2533 changed files with 144221 additions and 905568 deletions
+17
View File
@@ -0,0 +1,17 @@
{
"project_id" : "rocksdb",
"conduit_uri" : "https://reviews.facebook.net/",
"copyright_holder" : "Facebook",
"load" : [
"arcanist_util"
],
"lint.engine" : "FacebookFbcodeLintEngine",
"lint.engine.single.linter" : "FbcodeCppLinter",
"unit.engine" : "FacebookFbcodeUnitTestEngine",
"arcanist_configuration" : "FacebookArcanistConfiguration",
"base" : "git:HEAD^, hg:.^",
"git.default-relative-commit" : "HEAD^",
"git:arc.feature.start.default" : "origin/master",
"arc.feature.start.default" : "master",
"history.immutable" : false
}
-86
View File
@@ -1,86 +0,0 @@
# When making changes, verify the output of:
# clang-tidy -list-checks
---
Checks: "-*,\
bugprone-argument-comment,\
bugprone-dangling-handle,\
bugprone-fold-init-type,\
bugprone-forward-declaration-namespace,\
bugprone-forwarding-reference-overload,\
bugprone-shadow,\
bugprone-sizeof-*,\
bugprone-string-constructor,\
bugprone-undefined-memory-manipulation,\
bugprone-unused-return-value,\
bugprone-use-after-move,\
cert-env33-c,\
cert-err58-cpp,\
cert-msc30-c,\
cert-msc50-cpp,\
clang-analyzer-*,\
clang-diagnostic-*,\
-clang-diagnostic-missing-designated-field-initializers,\
concurrency-mt-unsafe,\
cppcoreguidelines-avoid-non-const-global-variables,\
cppcoreguidelines-missing-std-forward,\
cppcoreguidelines-pro-type-member-init,\
cppcoreguidelines-special-member-functions,\
cppcoreguidelines-virtual-class-destructor,\
google-build-using-namespace,\
google-explicit-constructor,\
google-readability-avoid-underscore-in-googletest-name,\
misc-definitions-in-headers,\
misc-redundant-expression,\
modernize-make-shared,\
modernize-use-emplace,\
modernize-use-noexcept,\
modernize-use-override,\
modernize-use-using,\
performance-faster-string-find,\
performance-for-range-copy,\
performance-implicit-conversion-in-loop,\
performance-inefficient-algorithm,\
performance-inefficient-string-concatenation,\
performance-inefficient-vector-operation,\
performance-move-const-arg,\
performance-move-constructor-init,\
performance-no-automatic-move,\
performance-no-int-to-ptr,\
performance-noexcept-move-constructor,\
performance-noexcept-swap,\
performance-trivially-destructible,\
performance-type-promotion-in-math-fn,\
performance-unnecessary-copy-initialization,\
performance-unnecessary-value-param,\
readability-braces-around-statements,\
readability-duplicate-include,\
readability-isolate-declaration,\
readability-operators-representation,\
readability-redundant-string-init"
WarningsAsErrors: "bugprone-use-after-move"
CheckOptions:
- key: bugprone-easily-swappable-parameters.MinimumLength
value: 4
- key: cppcoreguidelines-avoid-non-const-global-variables.AllowThreadLocal
value: true
- key: cppcoreguidelines-special-member-functions.AllowSoleDefaultDtor
value: true
- key: cppcoreguidelines-special-member-functions.AllowImplicitlyDeletedCopyOrMove
value: true
- key: modernize-use-using.IgnoreExternC
value: true
- key: performance-move-const-arg.CheckTriviallyCopyableMove
value: false
- key: performance-unnecessary-value-param.AllowedTypes
value: '[Pp]ointer$;[Pp]tr$;[Rr]ef(erence)?$'
- key: performance-unnecessary-copy-initialization.AllowedTypes
value: '[Pp]ointer$;[Pp]tr$;[Rr]ef(erence)?$'
- key: readability-operators-representation.BinaryOperators
value: '&&;&=;&;|;~;!;!=;||;|=;^;^='
- key: readability-redundant-string-init.StringNames
value: '::std::basic_string'
- key: readability-named-parameter.InsertPlainNamesInForwardDecls
value: true
...
-26
View File
@@ -1,26 +0,0 @@
name: build-folly
description: Build folly and dependencies (skipped if cache hit)
inputs:
cache-hit:
description: Whether the folly cache was hit
required: true
runs:
using: composite
steps:
- name: Build folly and dependencies
if: ${{ inputs.cache-hit != 'true' }}
run: |
clean_path=()
IFS=: read -ra path_entries <<< "$PATH"
for entry in "${path_entries[@]}"; do
if [[ "$entry" != "/usr/lib/ccache" ]]; then
clean_path+=("$entry")
fi
done
export PATH="$(IFS=:; echo "${clean_path[*]}")"
make build_folly
shell: bash
- name: Skip folly build (using cached version)
if: ${{ inputs.cache-hit == 'true' }}
run: echo "Folly build skipped - using cached version"
shell: bash
@@ -1,8 +0,0 @@
name: build-for-benchmarks
runs:
using: composite
steps:
- uses: "./.github/actions/pre-steps"
- name: Linux build for benchmarks
run: make V=1 J=8 -j8 release
shell: bash
-33
View File
@@ -1,33 +0,0 @@
name: cache-folly
description: Cache folly build to speed up CI
outputs:
cache-hit:
description: Whether the cache was hit
value: ${{ steps.cache-folly-build.outputs.cache-hit }}
runs:
using: composite
steps:
- name: Extract FOLLY_MK_HASH
id: extract-folly-hash
shell: bash
run: |
FOLLY_MK_HASH=$(md5sum folly.mk | cut -d' ' -f1)
echo "hash=$FOLLY_MK_HASH" >> $GITHUB_OUTPUT
- name: Extract FOLLY_INSTALL_DIR
id: extract-folly-install-dir
shell: bash
run: |
FOLLY_INSTALL_DIR=$(cd third-party/folly && python3 build/fbcode_builder/getdeps.py show-inst-dir)
echo "dir=$(echo $FOLLY_INSTALL_DIR | sed 's|installed/folly|installed|')" >> $GITHUB_OUTPUT
- name: Cache folly build
id: cache-folly-build
uses: actions/cache@v4
with:
# Cache the folly build directory
path: ${{ steps.extract-folly-install-dir.outputs.dir }}
# Key is based on:
# - OS and architecture
# - The docker image, which may not always be specified/known
# - Hash of folly.mk, which includes the folly repository commit hash
# NOTE: this is still only intended for DEBUG folly builds
key: folly-build-${{ runner.os }}-${{ runner.arch }}-${{ github.job_container.image }}-${{ steps.extract-folly-hash.outputs.hash }}-${{ env.PORTABLE == '1' && 'portable' || 'native' }}
@@ -1,21 +0,0 @@
name: cache-getdeps-downloads
description: Cache getdeps downloads to avoid unreliable mirrors and speed up builds
outputs:
cache-hit:
description: Whether the cache was hit
value: ${{ steps.cache-downloads.outputs.cache-hit }}
runs:
using: composite
steps:
- name: Cache getdeps downloads
id: cache-downloads
uses: actions/cache@v4
with:
# Use a fixed path that we control - folly.mk will sync with getdeps downloads dir
path: /tmp/rocksdb-getdeps-cache
# Use a rolling cache key - the cache accumulates downloads over time
# The key includes a weekly timestamp to ensure periodic refresh
key: getdeps-downloads-${{ runner.os }}-${{ runner.arch }}-week-${{ github.run_id }}
restore-keys: |
getdeps-downloads-${{ runner.os }}-${{ runner.arch }}-week-
getdeps-downloads-${{ runner.os }}-${{ runner.arch }}-
@@ -1,10 +0,0 @@
name: increase-max-open-files-on-macos
runs:
using: composite
steps:
- name: Increase max open files
run: |-
sudo sysctl -w kern.maxfiles=1048576
sudo sysctl -w kern.maxfilesperproc=1048576
sudo launchctl limit maxfiles 1048576
shell: bash
@@ -1,7 +0,0 @@
name: install-gflags-on-macos
runs:
using: composite
steps:
- name: Install gflags on macos
run: HOMEBREW_NO_AUTO_UPDATE=1 brew install gflags
shell: bash
@@ -1,7 +0,0 @@
name: install-gflags
runs:
using: composite
steps:
- name: Install gflags
run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
shell: bash
-11
View File
@@ -1,11 +0,0 @@
name: install-maven
runs:
using: composite
steps:
- name: Install Maven
run: |
wget --no-check-certificate https://archive.apache.org/dist/maven/maven-3/3.9.11/binaries/apache-maven-3.9.11-bin.tar.gz
tar zxf apache-maven-3.9.11-bin.tar.gz
echo "export M2_HOME=$(pwd)/apache-maven-3.9.11" >> $GITHUB_ENV
echo "$(pwd)/apache-maven-3.9.11/bin" >> $GITHUB_PATH
shell: bash
@@ -1,22 +0,0 @@
name: perform-benchmarks
runs:
using: composite
steps:
- name: Test low-variance benchmarks
run: "./tools/benchmark_ci.py --db_dir ${{ runner.temp }}/rocksdb-benchmark-datadir --output_dir ${{ runner.temp }}/benchmark-results --num_keys 20000000"
env:
LD_LIBRARY_PATH: "/usr/local/lib"
DURATION_RO: 300
DURATION_RW: 500
NUM_THREADS: 1
MAX_BACKGROUND_JOBS: 4
CI_TESTS_ONLY: 'true'
WRITE_BUFFER_SIZE_MB: 16
TARGET_FILE_SIZE_BASE_MB: 16
MAX_BYTES_FOR_LEVEL_BASE_MB: 64
COMPRESSION_TYPE: none
CACHE_INDEX_AND_FILTER_BLOCKS: 1
MIN_LEVEL_TO_COMPRESS: 3
CACHE_SIZE_MB: 10240
MB_WRITE_PER_SEC: 2
shell: bash
@@ -1,17 +0,0 @@
name: post-benchmarks
runs:
using: composite
steps:
- name: Upload Benchmark Results artifact
uses: actions/upload-artifact@v4.0.0
with:
name: benchmark-results
path: "${{ runner.temp }}/benchmark-results/**"
if-no-files-found: error
- name: Send benchmark report to visualisation
run: |-
set +e
set +o pipefail
./build_tools/benchmark_log_tool.py --tsvfile ${{ runner.temp }}/benchmark-results/report.tsv --esdocument https://search-rocksdb-bench-k2izhptfeap2hjfxteolsgsynm.us-west-2.es.amazonaws.com/bench_test3_rix/_doc
true
shell: bash
-38
View File
@@ -1,38 +0,0 @@
name: post-steps
description: Steps that are taken after a RocksDB job
inputs:
artifact-prefix:
description: Prefix to append to the name of artifacts that are uploaded
required: true
default: "${{ github.job }}"
runs:
using: composite
steps:
- name: Upload Test Results artifact
uses: actions/upload-artifact@v4.0.0
with:
name: "${{ inputs.artifact-prefix }}-test-results"
path: "${{ runner.temp }}/test-results/**"
- name: Upload DB LOG file artifact
uses: actions/upload-artifact@v4.0.0
with:
name: "${{ inputs.artifact-prefix }}-db-log-file"
path: LOG
- name: Copy Test Logs (on Failure)
if: ${{ failure() }}
run: |
mkdir -p ${{ runner.temp }}/failure-test-logs
cp -r t/* ${{ runner.temp }}/failure-test-logs
shell: bash
- name: Upload Test Logs (on Failure) artifact
uses: actions/upload-artifact@v4.0.0
with:
name: "${{ inputs.artifact-prefix }}-failure-test-logs"
path: ${{ runner.temp }}/failure-test-logs/**
if-no-files-found: ignore
- name: Upload Core Dumps artifact
uses: actions/upload-artifact@v4.0.0
with:
name: "${{ inputs.artifact-prefix }}-core-dumps"
path: "core.*"
if-no-files-found: ignore
@@ -1,5 +0,0 @@
name: pre-steps-macos
runs:
using: composite
steps:
- uses: "./.github/actions/pre-steps"
-28
View File
@@ -1,28 +0,0 @@
name: pre-steps
runs:
using: composite
steps:
- name: Dump ulimits for diagnostics
run: |
echo "=== ulimits (soft) ==="
ulimit -a -S || true
echo "=== ulimits (hard) ==="
ulimit -a -H || true
shell: bash
- name: Install lld linker for faster builds
run: apt-get update -y && apt-get install -y lld 2>/dev/null || true
shell: bash
- name: Setup Environment Variables
run: |-
echo "GTEST_THROW_ON_FAILURE=0" >> "$GITHUB_ENV"
echo "GTEST_OUTPUT=\"xml:${{ runner.temp }}/test-results/\"" >> "$GITHUB_ENV"
echo "SKIP_FORMAT_BUCK_CHECKS=1" >> "$GITHUB_ENV"
echo "GTEST_COLOR=1" >> "$GITHUB_ENV"
echo "CTEST_OUTPUT_ON_FAILURE=1" >> "$GITHUB_ENV"
echo "CTEST_TEST_TIMEOUT=300" >> "$GITHUB_ENV"
echo "ZLIB_DOWNLOAD_BASE=https://rocksdb-deps.s3.us-west-2.amazonaws.com/pkgs/zlib" >> "$GITHUB_ENV"
echo "BZIP2_DOWNLOAD_BASE=https://rocksdb-deps.s3.us-west-2.amazonaws.com/pkgs/bzip2" >> "$GITHUB_ENV"
echo "SNAPPY_DOWNLOAD_BASE=https://rocksdb-deps.s3.us-west-2.amazonaws.com/pkgs/snappy" >> "$GITHUB_ENV"
echo "LZ4_DOWNLOAD_BASE=https://rocksdb-deps.s3.us-west-2.amazonaws.com/pkgs/lz4" >> "$GITHUB_ENV"
echo "ZSTD_DOWNLOAD_BASE=https://rocksdb-deps.s3.us-west-2.amazonaws.com/pkgs/zstd" >> "$GITHUB_ENV"
shell: bash
-54
View File
@@ -1,54 +0,0 @@
name: setup-ccache
description: Setup ccache for faster C++ compilation caching
inputs:
cache-key-prefix:
description: Unique prefix for the cache key (e.g., 'build-linux')
required: true
portable:
description: Set PORTABLE=1 to disable -march=native (set to "false" for jobs linking pre-built Folly)
required: false
default: "true"
runs:
using: composite
steps:
- name: Set ccache environment variables
run: |
echo "CCACHE_DIR=${{ github.workspace }}/.ccache" >> $GITHUB_ENV
echo "CCACHE_BASEDIR=${{ github.workspace }}" >> $GITHUB_ENV
echo "CCACHE_NOHASHDIR=true" >> $GITHUB_ENV
echo "CCACHE_COMPILERCHECK=content" >> $GITHUB_ENV
echo "CCACHE_SLOPPINESS=clang_index_store,file_stat_matches,include_file_ctime,include_file_mtime,ivfsoverlay,pch_defines,modules,system_headers,time_macros" >> $GITHUB_ENV
echo "CCACHE_MAXSIZE=4G" >> $GITHUB_ENV
if [ "${{ inputs.portable }}" = "true" ]; then
echo "PORTABLE=1" >> $GITHUB_ENV
fi
shell: bash
- name: Restore ccache
uses: actions/cache@v4
with:
path: ${{ github.workspace }}/.ccache
key: ccache-${{ inputs.cache-key-prefix }}-${{ github.head_ref || github.ref }}-${{ github.sha }}
restore-keys: |-
ccache-${{ inputs.cache-key-prefix }}-${{ github.head_ref || github.ref }}-
ccache-${{ inputs.cache-key-prefix }}-refs/heads/main-
- name: Install ccache
run: |
if [ "$RUNNER_OS" = "macOS" ]; then
which ccache || brew install ccache
else
which ccache || (apt-get update && apt-get install -y ccache)
fi
shell: bash
- name: Add ccache to PATH
run: |
if [ "$RUNNER_OS" = "macOS" ]; then
echo "$(brew --prefix ccache)/libexec" >> $GITHUB_PATH
else
echo "/usr/lib/ccache" >> $GITHUB_PATH
fi
shell: bash
- name: Zero ccache stats and set build marker
run: |
ccache -z
touch "$CCACHE_DIR/.build_marker"
shell: bash
-11
View File
@@ -1,11 +0,0 @@
name: setup-folly
runs:
using: composite
steps:
- name: Checkout folly sources
run: |
make checkout_folly
shell: bash
- name: Install patchelf and libaio
run: apt-get update -y && apt-get install -y patchelf libaio-dev
shell: bash
@@ -1,17 +0,0 @@
name: setup-jdk-on-macos
runs:
using: composite
steps:
- name: Set up JDK on macos
run: |-
set -euo pipefail
JAVA_HOME="$(/usr/libexec/java_home -v 21)"
echo "JAVA_HOME=${JAVA_HOME}" >> "$GITHUB_ENV"
echo "JAVAC_ARGS=--release 8" >> "$GITHUB_ENV"
echo "${JAVA_HOME}/bin" >> "$GITHUB_PATH"
echo "Using JDK at ${JAVA_HOME}"
"${JAVA_HOME}/bin/java" -version
"${JAVA_HOME}/bin/javac" -version
shell: bash
-20
View File
@@ -1,20 +0,0 @@
name: build-folly
runs:
using: composite
steps:
- name: Fix repo ownership
# Needed in some cases, as safe.directory setting doesn't take effect
# under env -i
run: chown `whoami` . || true
shell: bash
- name: Set upstream
run: git remote add upstream https://github.com/facebook/rocksdb.git
shell: bash
- name: Fetch upstream
run: git fetch upstream
shell: bash
- name: Git status
# NOTE: some old branch builds under check_format_compatible.sh invoke
# git under env -i
run: git status && git remote -v && env -i git branch
shell: bash
@@ -1,15 +0,0 @@
name: teardown-ccache
description: Trim stale ccache entries and print stats
runs:
using: composite
steps:
- name: Trim and print ccache stats
run: |
if [ -z "$CCACHE_DIR" ]; then
echo "teardown-ccache: CCACHE_DIR not set, skipping (setup-ccache may not have run)"
exit 0
fi
.github/scripts/ccache-trim.sh || true
ccache -s || echo "teardown-ccache: ccache not found, skipping stats"
if: always()
shell: bash
@@ -1,140 +0,0 @@
name: windows-build-steps
inputs:
suite-run:
description: Comma-separated list of test suites to run (empty to skip C++ tests)
required: false
default: arena_test,db_basic_test,db_test,db_etc2_test,db_merge_operand_test,bloom_test,c_test,coding_test,crc32c_test,dynamic_bloom_test,env_basic_test,env_test,hash_test,random_test
run-java:
description: Whether to run Java tests
required: false
default: "true"
runs:
using: composite
steps:
- name: Detect Visual Studio generator
id: detect_vs
shell: pwsh
run: |
$ErrorActionPreference = "Stop"
$vswhere = Join-Path ([Environment]::GetFolderPath("ProgramFilesX86")) "Microsoft Visual Studio\Installer\vswhere.exe"
if (!(Test-Path $vswhere)) {
throw "vswhere.exe was not found at $vswhere"
}
$vsInstallationsJson = & $vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -format json -utf8
$vsInstallations = @($vsInstallationsJson | ConvertFrom-Json)
if ($vsInstallations.Count -eq 0) {
throw "No Visual Studio instance with C++ tools was found."
}
$vsInstallation = $vsInstallations[0]
$vsMajor = [int]$vsInstallation.catalog.productLineVersion
if ($vsMajor -eq 0) {
$vsMajor = [int]($vsInstallation.installationVersion.Split(".")[0])
}
$generatorLine = cmake --help | Where-Object {
$_ -match "^\*?\s*Visual Studio $vsMajor\s+\d{4}\s*="
} | Select-Object -First 1
if (!$generatorLine) {
throw "CMake does not support a Visual Studio generator for VS major version $vsMajor."
}
$cmakeGenerator = ($generatorLine -replace "^\*?\s*", "" -replace "\s*=.*$", "").Trim()
$vsVersionRange = "[{0}.0,{1}.0)" -f $vsMajor, ($vsMajor + 1)
echo "Detected Visual Studio at $($vsInstallation.installationPath)"
echo "Using CMake generator: $cmakeGenerator"
echo "Using setup-msbuild vs-version: $vsVersionRange"
"CMAKE_GENERATOR=$cmakeGenerator" >> $env:GITHUB_ENV
"vs_version=$vsVersionRange" >> $env:GITHUB_OUTPUT
- name: Add msbuild to PATH
uses: microsoft/setup-msbuild@v2
with:
vs-version: ${{ steps.detect_vs.outputs.vs_version }}
- name: ccache
uses: hendrikmuhs/ccache-action@v1.2
with:
max-size: "10GB"
key: ccache-windows-${{ github.workflow }}
restore-keys: |
ccache-windows-
- name: Configure ccache
shell: pwsh
run: |
ccache --set-config=base_dir=C:\a\rocksdb\rocksdb
ccache --set-config=hash_dir=false
ccache --set-config=compiler_check=content
- name: Custom steps
env:
THIRDPARTY_HOME: ${{ github.workspace }}/thirdparty
CMAKE_HOME: C:/Program Files/CMake
CMAKE_BIN: C:/Program Files/CMake/bin/cmake.exe
CTEST_BIN: C:/Program Files/CMake/bin/ctest.exe
SNAPPY_HOME: ${{ github.workspace }}/thirdparty/snappy-1.2.2
SNAPPY_INCLUDE: ${{ github.workspace }}/thirdparty/snappy-1.2.2;${{ github.workspace }}/thirdparty/snappy-1.2.2/build
SNAPPY_LIB_DEBUG: ${{ github.workspace }}/thirdparty/snappy-1.2.2/build/Debug/snappy.lib
run: |-
# NOTE: if ... Exit $LASTEXITCODE lines needed to exit and report failure
echo ===================== Install Dependencies =====================
if (!$env:JAVA_HOME -or !(Test-Path (Join-Path $env:JAVA_HOME "bin\javac.exe"))) {
$javac = Get-Command javac -ErrorAction SilentlyContinue
if (!$javac) {
Write-Error "javac was not found on PATH and JAVA_HOME is not set."
Exit 1
}
$env:JAVA_HOME = Split-Path (Split-Path $javac.Source -Parent) -Parent
}
echo "JAVA_HOME=$env:JAVA_HOME"
& "$env:JAVA_HOME\bin\java.exe" -version
& "$env:JAVA_HOME\bin\javac.exe" -version
mkdir $Env:THIRDPARTY_HOME
cd $Env:THIRDPARTY_HOME
echo "Building Snappy dependency..."
curl -Lo snappy-1.2.2.zip https://github.com/google/snappy/archive/refs/tags/1.2.2.zip
if(!$?) { Exit $LASTEXITCODE }
unzip -q snappy-1.2.2.zip
if(!$?) { Exit $LASTEXITCODE }
cd snappy-1.2.2
mkdir build
cd build
& cmake -G "$Env:CMAKE_GENERATOR" .. -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF
if(!$?) { Exit $LASTEXITCODE }
cmake --build . --config Debug --parallel 32 -- /p:Platform=x64
if(!$?) { Exit $LASTEXITCODE }
echo ======================== Build RocksDB =========================
cd ${{ github.workspace }}
$env:Path = "$env:JAVA_HOME\bin;" + $env:Path
mkdir build
cd build
& cmake -G "$Env:CMAKE_GENERATOR" -DCMAKE_BUILD_TYPE=Debug -DWIN_CI=1 -DPORTABLE="$Env:CMAKE_PORTABLE" -DSNAPPY=1 -DXPRESS=1 -DJNI=1 ..
if(!$?) { Exit $LASTEXITCODE }
cd ..
echo "Building with VS version: $Env:CMAKE_GENERATOR"
# use more parallel processes than the number of processes available, as most of the compile command would be cache hit
cmake --build build --config Debug --parallel 32 -- /p:LinkIncremental=false /p:Platform=x64
if(!$?) { Exit $LASTEXITCODE }
echo ========================= Test RocksDB =========================
$suiteRun = "${{ inputs.suite-run }}"
if ($suiteRun -ne "") {
$suiteArray = $suiteRun -split ','
build_tools\run_ci_db_test.ps1 -SuiteRun $suiteArray -Concurrency 16
if(!$?) { Exit $LASTEXITCODE }
} else {
echo "Skipping C++ tests (suite-run is empty)"
}
if ("${{ inputs.run-java }}" -eq "true") {
echo ======================== Test RocksJava ========================
cd build\java
& ctest -C Debug -j 16
if(!$?) { Exit $LASTEXITCODE }
} else {
echo "Skipping Java tests"
}
shell: pwsh
- name: Show ccache stats
shell: pwsh
run: |
ccache --show-stats -v
@@ -1,27 +0,0 @@
// Shared markdown builder for AI review comment bodies.
//
// Usage:
// const build = require('./build-ai-review-comment.js');
// return build({ icon, headerTitle, triggerLine, responseBody, footerLines
// });
module.exports = function buildAiReviewComment(
{icon, headerTitle, triggerLine, responseBody, footerLines}) {
return [
`## ${icon} ${headerTitle}`,
'',
triggerLine,
'',
'---',
'',
responseBody,
'',
'---',
'',
'<details>',
'<summary>️ About this response</summary>',
'',
...footerLines,
'</details>',
].join('\n');
};
-39
View File
@@ -1,39 +0,0 @@
#!/bin/bash
# Trim ccache to keep only entries accessed during the current build.
#
# Usage:
# 1. Before build: touch "$CCACHE_DIR/.build_marker"
# 2. Run build (ccache updates mtime on hits, creates new files for misses)
# 3. After build: .github/scripts/ccache-trim.sh
#
# This removes cache files not accessed during the build (stale entries from
# previous commits). Only intended for CI where each run builds one commit.
# Do NOT use on local builds where multiple worktrees may share the cache.
set -e
CCACHE_DIR="${CCACHE_DIR:?CCACHE_DIR must be set}"
MARKER="$CCACHE_DIR/.build_marker"
if [ ! -f "$MARKER" ]; then
echo "ccache-trim: No .build_marker found, skipping (was the marker created before build?)"
exit 0
fi
# Count files before cleanup
before=$(find "$CCACHE_DIR" -type f \( -name '*R' -o -name '*M' \) | wc -l)
# Delete cache files (results and manifests) older than the marker
find "$CCACHE_DIR" -type f \( -name '*R' -o -name '*M' \) ! -newer "$MARKER" -delete
# Clean up empty directories
find "$CCACHE_DIR" -mindepth 2 -type d -empty -delete 2>/dev/null || true
# Recalculate size counters
ccache -c 2>/dev/null || true
after=$(find "$CCACHE_DIR" -type f \( -name '*R' -o -name '*M' \) | wc -l)
echo "ccache-trim: $before -> $after cache files (removed $((before - after)) stale entries)"
# Clean up marker
rm -f "$MARKER"
-32
View File
@@ -1,32 +0,0 @@
#!/bin/bash
# Compute test shard for parallel CI execution.
# Distributes tests round-robin across N shards for balanced load.
# Outputs ROCKSDBTESTS_SUBSET — the list of test binaries for this shard.
# The Makefile uses this to build and run only the assigned tests.
#
# Usage: compute-test-shard.sh <shard_index> <num_shards>
set -euo pipefail
shard=${1:?Usage: compute-test-shard.sh <shard_index> <num_shards>}
nshards=${2:?Usage: compute-test-shard.sh <shard_index> <num_shards>}
# Get sorted test list (db_test first since it's the heaviest, then alpha)
make -s list_all_tests 2>/dev/null | tr ' ' '\n' | grep '_test$' | sort -u > /tmp/sorted.txt
(echo db_test; grep -v '^db_test$' /tmp/sorted.txt) > /tmp/all_tests.txt
total=$(wc -l < /tmp/all_tests.txt)
# Round-robin: assign test i to shard (i % nshards).
# This spreads heavy tests (which are scattered alphabetically) evenly.
awk -v s="$shard" -v n="$nshards" 'NR > 0 && (NR - 1) % n == s' /tmp/all_tests.txt > /tmp/include.txt
included=$(wc -l < /tmp/include.txt)
first=$(head -1 /tmp/include.txt)
last=$(tail -1 /tmp/include.txt)
# Output space-separated list for ROCKSDBTESTS_SUBSET
subset=$(tr '\n' ' ' < /tmp/include.txt)
echo "subset=${subset}" >> "$GITHUB_OUTPUT"
echo "Shard $shard/$nshards: $included tests (round-robin), first=$first last=$last (total $total)"
-118
View File
@@ -1,118 +0,0 @@
// Parse Claude Code execution log and produce a markdown review comment.
//
// Usage from actions/github-script:
// const parse = require('./.github/scripts/parse-claude-review.js');
// const markdown = parse({ executionFile, conclusion, meta });
//
// Parameters:
// executionFile - path to the JSON execution log from claude-code-base-action
// conclusion - 'success' or 'failure' from the action output
// meta - { trigger, autoMode, headSha, reviewer, isQuery, isPartial
// }
const fs = require('fs');
const buildComment = require('./build-ai-review-comment.js');
module.exports = function parseClaude({executionFile, conclusion, meta}) {
function getTriggerLine() {
if (meta.trigger !== 'auto') {
return `*Requested by @${meta.reviewer}*`;
}
const shortSha = meta.headSha ? meta.headSha.substring(0, 7) : 'unknown';
if (meta.autoMode === 'early') {
return `*Auto-triggered after CI reached the early-review threshold — reviewing commit ${
shortSha}*`;
}
return `*Auto-triggered after CI passed — reviewing commit ${shortSha}*`;
}
let responseBody = '';
try {
const executionLog = JSON.parse(fs.readFileSync(executionFile, 'utf8'));
if (!Array.isArray(executionLog)) {
throw new Error('Expected array format from claude-code-base-action');
}
const resultMessage = executionLog.find(m => m.type === 'result');
// Helper: extract the last substantial assistant text from the log.
// Used as a fallback when Claude ran out of turns and the recovery
// session also failed to produce a result.
function getLastAssistantText(log, minLength = 200) {
for (let i = log.length - 1; i >= 0; i--) {
const m = log[i];
if (m.type !== 'assistant') continue;
const content = m.message && m.message.content;
if (!Array.isArray(content)) continue;
for (let j = content.length - 1; j >= 0; j--) {
if (content[j].type === 'text' && content[j].text &&
content[j].text.trim().length >= minLength) {
const text = content[j].text.trim();
// Truncate to avoid enormous PR comments
return text.length > 50000 ? text.substring(0, 50000) +
'\n\n*[Truncated — full output in execution log artifact]*' :
text;
}
}
}
return null;
}
if (!resultMessage) {
responseBody = '⚠️ No result message found in execution log.';
} else if (resultMessage.subtype === 'success' && resultMessage.result) {
responseBody = resultMessage.result;
} else if (resultMessage.is_error || resultMessage.subtype === 'error') {
const errorInfo =
resultMessage.result || resultMessage.error || 'Unknown error';
responseBody = `❌ **Claude encountered an error:**\n\n${errorInfo}`;
} else if (resultMessage.subtype === 'error_max_turns') {
// The workflow runs a recovery session when this happens, so this
// branch is typically only hit if recovery wasn't attempted (e.g.,
// no findings file was written). Extract what we can.
const partial = getLastAssistantText(executionLog);
if (partial) {
responseBody =
`⚠️ **Review incomplete — Claude hit the turn limit.**\n\nBelow is the last partial output. You can request a fresh review with \`/claude-review\`.\n\n---\n\n${
partial}`;
} else {
responseBody =
'⚠️ **Review incomplete — Claude hit the turn limit before producing output.** You can request a fresh review with `/claude-review`.';
}
} else if (resultMessage.result) {
responseBody = `⚠️ **Completed with status: ${
resultMessage.subtype}**\n\n${resultMessage.result}`;
} else {
responseBody = '⚠️ Claude completed but produced no output.';
}
} catch (error) {
responseBody = `❌ Error parsing Claude response: ${error.message}`;
}
const isPartial = !!meta.isPartial;
const icon = isPartial ? '🟡' : (conclusion === 'success' ? '✅' : '⚠️');
const headerTitle = meta.isQuery ? 'Claude Response' : 'Claude Code Review';
const triggerLine = getTriggerLine();
return buildComment({
icon,
headerTitle,
triggerLine,
responseBody,
footerLines: [
'Generated by [Claude Code](https://github.com/anthropics/claude-code-action).',
'Review methodology: `claude_md/code_review.md`',
'',
'**Limitations:**',
'- Claude may miss context from files not in the diff',
'- Large PRs may be truncated',
'- Always apply human judgment to AI suggestions',
'',
'**Commands:**',
'- `/claude-review [context]` — Request a code review',
'- `/claude-query <question>` — Ask about the PR or codebase',
],
});
};
-98
View File
@@ -1,98 +0,0 @@
// Parse Codex review artifacts and produce a markdown review comment.
//
// Usage from actions/github-script:
// const parse = require('./.github/scripts/parse-codex-review.js');
// const markdown = parse({ responseFile, recoveryFile, findingsFile, logFile,
// exitCode, meta });
//
// Parameters:
// responseFile - path to Codex final response output
// recoveryFile - path to recovery output formatted from review-findings.md
// findingsFile - path to incremental findings file written during review
// logFile - path to Codex stdout/stderr log
// exitCode - Codex process exit code
// meta - { trigger, autoMode, headSha, reviewer, isQuery, isPartial }
const fs = require('fs');
const buildComment = require('./build-ai-review-comment.js');
module.exports = function parseCodex(
{responseFile, recoveryFile, findingsFile, logFile, exitCode, meta}) {
function getTriggerLine() {
if (meta.trigger !== 'auto') {
return `*Requested by @${meta.reviewer}*`;
}
const shortSha = meta.headSha ? meta.headSha.substring(0, 7) : 'unknown';
if (meta.autoMode === 'early') {
return `*Auto-triggered after CI reached the early-review threshold — reviewing commit ${
shortSha}*`;
}
return `*Auto-triggered after CI passed — reviewing commit ${shortSha}*`;
}
function readIfPresent(path) {
if (!path || !fs.existsSync(path)) {
return '';
}
const text = fs.readFileSync(path, 'utf8').trim();
return text;
}
function tailFile(path, maxChars = 12000) {
const text = readIfPresent(path);
if (!text) {
return '';
}
if (text.length <= maxChars) {
return text;
}
return text.slice(text.length - maxChars);
}
let responseBody = '';
const recovered = readIfPresent(recoveryFile);
const direct = readIfPresent(responseFile);
const findings = readIfPresent(findingsFile);
if (recovered) {
responseBody = recovered;
} else if (direct) {
responseBody = direct;
} else if (findings) {
responseBody =
'⚠️ **Review incomplete — Codex did not produce a final response.**\n\n' +
'Below are the incremental findings recovered from `review-findings.md`.\n\n---\n\n' +
findings;
} else {
const logTail = tailFile(logFile);
responseBody = logTail ?
`❌ **Codex review failed before producing findings.**\n\n\`\`\`\n${
logTail}\n\`\`\`` :
'❌ Codex review failed before producing any output.';
}
const isPartial = !!meta.isPartial;
const code = Number.parseInt(exitCode || '1', 10);
const icon = isPartial ? '🟡' : (code === 0 ? '✅' : '⚠️');
const triggerLine = getTriggerLine();
return buildComment({
icon,
headerTitle: meta.isQuery ? 'Codex Response' : 'Codex Code Review',
triggerLine,
responseBody,
footerLines: [
'Generated by Codex CLI.',
'Review methodology: `claude_md/code_review.md`',
'',
'**Limitations:**',
'- Codex may miss context from files not in the diff',
'- Large PRs may be truncated',
'- Always apply human judgment to AI suggestions',
'',
'**Commands:**',
'- `/codex-review [context]` — Request a code review',
'- `/codex-query <question>` — Ask about the PR or codebase',
],
});
};
-204
View File
@@ -1,204 +0,0 @@
// Shared PR comment posting utility.
// Used by both clang-tidy-comment.yml and claude-review-comment.yml.
//
// Usage from actions/github-script:
// const post = require('./.github/scripts/post-pr-comment.js');
// await post({ github, context, core, prNumber, body, marker });
//
// Parameters:
// github - octokit instance from actions/github-script
// context - GitHub Actions context
// core - @actions/core for logging
// prNumber - PR number to comment on
// body - comment body (markdown string)
// marker - HTML comment marker for dedup (e.g. '<!-- claude-review -->')
// If an existing comment with this marker is found, it is updated.
// If not found, a new comment is created.
// legacyMarkers - optional list of legacy markers that should be migrated by
// being considered part of the same comment family.
// prunePrefix - optional marker prefix whose older comments should be
// superseded. If obsoleteTitle is set, older comments are
// collapsed instead of deleted.
// preserveLatest - optional count of active comments to keep when
// prunePrefix is set.
// obsoleteMarker - optional HTML marker used to detect already-obsolete
// comments.
// obsoleteTitle - optional heading to use when collapsing superseded
// comments into a details block.
module.exports = async function postPrComment({
github,
context,
core,
prNumber,
body,
marker,
legacyMarkers = [],
prunePrefix = '',
preserveLatest = 0,
obsoleteMarker = '',
obsoleteTitle = '',
}) {
if (!prNumber || !body) {
core.warning('Missing prNumber or body; skipping comment.');
return;
}
const owner = context.repo.owner;
const repo = context.repo.repo;
// Ensure marker is embedded in the body
const markedBody = body.includes(marker) ? body : `${marker}\n${body}`;
async function listComments() {
return await github.paginate(github.rest.issues.listComments, {
owner,
repo,
issue_number: prNumber,
per_page: 100,
});
}
function commentActivityTime(comment) {
const timestamp =
Date.parse(comment.updated_at || comment.created_at || '');
return Number.isNaN(timestamp) ? 0 : timestamp;
}
function commentSortDescending(left, right) {
const timeDelta = commentActivityTime(right) - commentActivityTime(left);
if (timeDelta !== 0) {
return timeDelta;
}
return Number(right.id || 0) - Number(left.id || 0);
}
function isObsoleteComment(comment) {
return !!obsoleteMarker && typeof comment.body === 'string' &&
comment.body.includes(obsoleteMarker);
}
function buildObsoleteBody(comment) {
const originalBody =
typeof comment.body === 'string' && comment.body.trim() ?
comment.body :
'*No original review body preserved.*';
const title = obsoleteTitle || 'AI Review - OBSOLETE';
return `${obsoleteMarker}\n## ${
title}\n\n<details>\n<summary>Superseded by a newer AI review. Expand to see the original review.</summary>\n\n${
originalBody}\n\n</details>`;
}
async function deleteCommentIfPresent(comment, reason) {
try {
await github.rest.issues.deleteComment({
owner,
repo,
comment_id: comment.id,
});
core.info(`${reason} ${comment.id}`);
} catch (error) {
if (error.status === 404) {
core.info(`Comment ${comment.id} was already deleted by another run.`);
return;
}
throw error;
}
}
async function supersedeCommentIfPresent(comment, reason) {
if (!obsoleteTitle) {
await deleteCommentIfPresent(comment, reason);
return;
}
if (isObsoleteComment(comment)) {
return;
}
try {
await github.rest.issues.updateComment({
owner,
repo,
comment_id: comment.id,
body: buildObsoleteBody(comment),
});
core.info(`${reason} ${comment.id}`);
} catch (error) {
if (error.status === 404) {
core.info(`Comment ${comment.id} was already deleted by another run.`);
return;
}
throw error;
}
}
let comments = await listComments();
const existing = comments.find(
comment =>
typeof comment.body === 'string' && comment.body.includes(marker));
let currentCommentId = null;
if (existing) {
try {
const response = await github.rest.issues.updateComment({
owner,
repo,
comment_id: existing.id,
body: markedBody,
});
currentCommentId = existing.id;
core.info(`Updated existing comment ${existing.id}`);
} catch (error) {
if (error.status !== 404) {
throw error;
}
core.info(
`Comment ${existing.id} disappeared before update; ` +
'creating a fresh comment instead.');
}
}
if (!currentCommentId) {
const response = await github.rest.issues.createComment({
owner,
repo,
issue_number: prNumber,
body: markedBody,
});
currentCommentId = response.data.id;
core.info('Created new PR comment');
}
if (prunePrefix || legacyMarkers.length > 0) {
comments = await listComments();
const relatedComments = comments.filter(
comment => comment.id !== currentCommentId &&
typeof comment.body === 'string' &&
((prunePrefix && comment.body.includes(prunePrefix)) ||
legacyMarkers.some(
legacyMarker => comment.body.includes(legacyMarker))));
const activeRelatedComments =
comments
.filter(
comment => typeof comment.body === 'string' &&
!isObsoleteComment(comment) &&
(comment.id === currentCommentId ||
relatedComments.some(
relatedComment => relatedComment.id === comment.id)))
.sort(commentSortDescending);
const keep =
new Set(activeRelatedComments.slice(0, Math.max(preserveLatest, 1))
.map(comment => comment.id));
const supersedeCandidates = obsoleteTitle ?
activeRelatedComments.filter(comment => !keep.has(comment.id)) :
relatedComments.filter(comment => !keep.has(comment.id));
for (const comment of supersedeCandidates) {
if (keep.has(comment.id)) {
continue;
}
await supersedeCommentIfPresent(comment, 'Superseded old AI review');
}
}
};
-347
View File
@@ -1,347 +0,0 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const postPrComment = require('./post-pr-comment.js');
const OBSOLETE_MARKER = '<!-- claude-review-obsolete -->';
const OBSOLETE_TITLE = 'Claude Code Review - OBSOLETE';
function makeComment(id, body, createdAt, updatedAt) {
return {
id,
body,
created_at: createdAt,
updated_at: updatedAt || createdAt,
};
}
function createHarness(initialComments, options = {}) {
let comments = initialComments.map(comment => ({...comment}));
let nextCommentId = options.nextCommentId || 1000;
let paginateCount = 0;
const calls = {
update: [],
create: [],
delete: [],
};
const github = {
paginate: async () => {
paginateCount++;
if (options.onPaginate) {
const updated = options.onPaginate({
paginateCount,
comments: comments.map(comment => ({...comment})),
});
if (updated) {
comments = updated.map(comment => ({...comment}));
}
}
return comments.map(comment => ({...comment}));
},
rest: {
issues: {
listComments: () => {
throw new Error('listComments should only be used through paginate');
},
updateComment: async ({comment_id, body}) => {
calls.update.push({comment_id, body});
const error =
options.updateErrors && options.updateErrors[comment_id];
if (error) {
throw error;
}
const index =
comments.findIndex(comment => comment.id === comment_id);
if (index === -1) {
const notFound = new Error(`Comment ${comment_id} not found`);
notFound.status = 404;
throw notFound;
}
const updated = {
...comments[index],
body,
updated_at: options.updateTimestamp || '2026-04-24T00:00:00Z',
};
comments[index] = updated;
return {data: {...updated}};
},
createComment: async ({issue_number, body}) => {
calls.create.push({issue_number, body});
const created = makeComment(
nextCommentId++, body,
options.createTimestamp || '2026-04-24T00:00:00Z');
comments.push(created);
return {data: {id: created.id}};
},
deleteComment: async ({comment_id}) => {
calls.delete.push({comment_id});
const error =
options.deleteErrors && options.deleteErrors[comment_id];
if (error) {
throw error;
}
comments = comments.filter(comment => comment.id !== comment_id);
},
},
},
};
return {
github,
calls,
getComments: () => comments.map(comment => ({...comment})),
};
}
function createCore() {
return {
info: () => {},
warning: () => {},
};
}
function escapeRegExp(value) {
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
function assertObsoleteComment(comment, originalBody) {
assert.ok(comment);
assert.match(comment.body, new RegExp(escapeRegExp(OBSOLETE_MARKER)));
assert.match(comment.body, new RegExp(`## ${escapeRegExp(OBSOLETE_TITLE)}`));
assert.match(comment.body, /<details>/);
assert.match(comment.body, /Superseded by a newer AI review/);
assert.match(comment.body, new RegExp(escapeRegExp(originalBody)));
}
const context = {
repo: {
owner: 'facebook',
repo: 'rocksdb',
},
};
test(
'creates a fresh review comment and supersedes legacy comments',
async () => {
const harness = createHarness(
[
makeComment(
1, '<!-- claude-review-auto -->\nold legacy', 'not-a-date'),
makeComment(
2, '<!-- claude-review-auto -->\nnew legacy',
'2026-04-21T00:00:00Z'),
],
{
updateTimestamp: '2026-04-24T00:00:00Z',
});
await postPrComment({
github: harness.github,
context,
core: createCore(),
prNumber: 14659,
body: 'review body',
marker: '<!-- claude-review-auto-run-500 -->',
legacyMarkers: ['<!-- claude-review-auto -->'],
prunePrefix: '<!-- claude-review-auto-',
preserveLatest: 1,
obsoleteMarker: OBSOLETE_MARKER,
obsoleteTitle: OBSOLETE_TITLE,
});
assert.equal(harness.calls.create.length, 1);
assert.deepEqual(
harness.calls.update.map(call => call.comment_id), [2, 1]);
assert.equal(harness.calls.delete.length, 0);
const comments = harness.getComments();
assert.equal(comments.length, 3);
assert.match(
comments.find(comment => comment.id === 1000).body,
/<!-- claude-review-auto-run-500 -->/);
assertObsoleteComment(
comments.find(comment => comment.id === 2),
'<!-- claude-review-auto -->\nnew legacy');
assertObsoleteComment(
comments.find(comment => comment.id === 1),
'<!-- claude-review-auto -->\nold legacy');
});
test(
'updates an exact-match comment and supersedes leftover legacy comments',
async () => {
const harness = createHarness(
[
makeComment(
3, '<!-- claude-review-auto-abcdef0 -->\ncurrent body',
'2026-04-24T00:00:00Z'),
makeComment(
4, '<!-- claude-review-auto -->\nlegacy body',
'2026-04-20T00:00:00Z'),
],
{
updateTimestamp: '2026-04-24T01:00:00Z',
});
await postPrComment({
github: harness.github,
context,
core: createCore(),
prNumber: 14659,
body: 'refreshed body',
marker: '<!-- claude-review-auto-abcdef0 -->',
legacyMarkers: ['<!-- claude-review-auto -->'],
prunePrefix: '<!-- claude-review-auto-',
preserveLatest: 1,
obsoleteMarker: OBSOLETE_MARKER,
obsoleteTitle: OBSOLETE_TITLE,
});
assert.deepEqual(
harness.calls.update.map(call => call.comment_id), [3, 4]);
assert.equal(harness.calls.create.length, 0);
assert.equal(harness.calls.delete.length, 0);
const comments = harness.getComments();
assert.match(
comments.find(comment => comment.id === 3).body,
/<!-- claude-review-auto-abcdef0 -->\nrefreshed body/);
assertObsoleteComment(
comments.find(comment => comment.id === 4),
'<!-- claude-review-auto -->\nlegacy body');
});
test(
'creates a new comment if the target disappears before update',
async () => {
const missing = new Error('gone');
missing.status = 404;
const harness = createHarness(
[
makeComment(
10, '<!-- claude-review-auto-abcdef0 -->\nold body',
'2026-04-20T00:00:00Z'),
],
{
updateErrors: {
10: missing,
},
});
await postPrComment({
github: harness.github,
context,
core: createCore(),
prNumber: 14659,
body: 'replacement body',
marker: '<!-- claude-review-auto-abcdef0 -->',
});
assert.equal(harness.calls.update.length, 1);
assert.equal(harness.calls.create.length, 1);
assert.match(
harness.calls.create[0].body, /<!-- claude-review-auto-abcdef0 -->/);
});
test(
'ignores 404 when superseding a comment already deleted by another run',
async () => {
const missing = new Error('gone');
missing.status = 404;
const harness = createHarness(
[
makeComment(
20, '<!-- claude-review-auto-oldest -->\noldest',
'2026-04-20T00:00:00Z'),
makeComment(
21, '<!-- claude-review-auto-newer -->\nnewer',
'2026-04-21T00:00:00Z'),
],
{
nextCommentId: 30,
createTimestamp: '2026-04-24T00:00:00Z',
deleteErrors: {
20: missing,
},
});
await postPrComment({
github: harness.github,
context,
core: createCore(),
prNumber: 14659,
body: 'fresh body',
marker: '<!-- claude-review-auto-latest -->',
prunePrefix: '<!-- claude-review-auto-',
preserveLatest: 1,
obsoleteMarker: OBSOLETE_MARKER,
obsoleteTitle: OBSOLETE_TITLE,
});
assert.equal(harness.calls.create.length, 1);
assert.equal(harness.calls.delete.length, 0);
assert.deepEqual(
harness.calls.update.map(call => call.comment_id), [21, 20]);
const comments = harness.getComments();
assertObsoleteComment(
comments.find(comment => comment.id === 21),
'<!-- claude-review-auto-newer -->\nnewer');
});
test(
'supersedes the current review when a newer concurrent one appears',
async () => {
const harness = createHarness(
[
makeComment(
40, '<!-- claude-review-auto-current -->\ncurrent',
'2026-04-20T00:00:00Z'),
makeComment(
41, '<!-- claude-review-auto-older -->\nolder',
'2026-04-19T00:00:00Z'),
],
{
updateTimestamp: '2026-04-24T00:00:00Z',
onPaginate: ({paginateCount, comments}) => {
if (paginateCount !== 2) {
return comments;
}
return comments.concat([
makeComment(
42, '<!-- claude-review-auto-newer -->\nnewer',
'2026-04-25T00:00:00Z'),
]);
},
});
await postPrComment({
github: harness.github,
context,
core: createCore(),
prNumber: 14659,
body: 'updated current body',
marker: '<!-- claude-review-auto-current -->',
prunePrefix: '<!-- claude-review-auto-',
preserveLatest: 1,
obsoleteMarker: OBSOLETE_MARKER,
obsoleteTitle: OBSOLETE_TITLE,
});
assert.deepEqual(
harness.calls.update.map(call => call.comment_id), [40, 40, 41]);
assert.equal(harness.calls.delete.length, 0);
const comments = harness.getComments();
assert.match(
comments.find(comment => comment.id === 42).body,
/<!-- claude-review-auto-newer -->\nnewer/);
assertObsoleteComment(
comments.find(comment => comment.id === 40),
'<!-- claude-review-auto-current -->\nupdated current body');
assertObsoleteComment(
comments.find(comment => comment.id === 41),
'<!-- claude-review-auto-older -->\nolder');
});
File diff suppressed because it is too large Load Diff
-185
View File
@@ -1,185 +0,0 @@
# Shared comment-posting workflow for AI reviews.
name: AI Review Comment
on:
workflow_call:
inputs:
provider:
description: Provider key, for example "claude" or "codex"
required: true
type: string
result_artifact_name:
description: Artifact name produced by the analysis workflow
required: true
type: string
comment_file:
description: Markdown comment file produced by the analysis workflow
required: true
type: string
permissions:
pull-requests: write
issues: write
jobs:
comment:
if: github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
steps:
- name: Checkout scripts
uses: actions/checkout@v4
with:
sparse-checkout: .github/scripts
sparse-checkout-cone-mode: true
- name: Download review artifact
id: download
uses: actions/download-artifact@v4
with:
name: ${{ inputs.result_artifact_name }}
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
continue-on-error: true
- name: Post or update PR comment
if: steps.download.outcome == 'success'
env:
PROVIDER: ${{ inputs.provider }}
COMMENT_FILE: ${{ inputs.comment_file }}
RUN_ID: ${{ github.event.workflow_run.id }}
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
if (!fs.existsSync(process.env.COMMENT_FILE) ||
!fs.existsSync('pr_number.txt')) {
core.info('No review results found; skipping.');
return;
}
const post = require('./.github/scripts/post-pr-comment.js');
const provider = process.env.PROVIDER;
const providerTitle = provider === 'codex' ? 'Codex' : 'Claude';
const body = fs.readFileSync(process.env.COMMENT_FILE, 'utf8');
const prNumber = parseInt(
fs.readFileSync('pr_number.txt', 'utf8').trim(),
10
);
const trigger = fs.existsSync('trigger_type.txt') ?
fs.readFileSync('trigger_type.txt', 'utf8').trim() :
'auto';
const headSha = fs.existsSync('head_sha.txt') ?
fs.readFileSync('head_sha.txt', 'utf8').trim() :
'';
const shortSha = headSha ? headSha.substring(0, 7) : '';
const runId = process.env.RUN_ID;
let marker = '';
let legacyMarkers = [];
let prunePrefix = '';
let preserveLatest = 0;
let obsoleteMarker = '';
let obsoleteTitle = '';
if (trigger === 'auto') {
if (!shortSha) {
core.warning(
'Missing head_sha.txt for auto review; skipping comment.'
);
return;
}
marker = `<!-- ${provider}-review-auto-run-${runId} -->`;
legacyMarkers = [`<!-- ${provider}-review-auto -->`];
prunePrefix = `<!-- ${provider}-review-auto-`;
preserveLatest = 1;
obsoleteMarker = `<!-- ${provider}-review-obsolete -->`;
obsoleteTitle = `${providerTitle} Code Review - OBSOLETE`;
} else {
marker = `<!-- ${provider}-review-manual-run-${runId} -->`;
}
await post({
github,
context,
core,
prNumber,
body,
marker,
legacyMarkers,
prunePrefix,
preserveLatest,
obsoleteMarker,
obsoleteTitle,
});
- name: Add reaction to trigger comment
if: steps.download.outcome == 'success'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
if (!fs.existsSync('trigger_type.txt')) return;
const trigger = fs.readFileSync('trigger_type.txt', 'utf8').trim();
if (trigger !== 'manual') return;
if (!fs.existsSync('comment_id.txt')) return;
const commentId = parseInt(
fs.readFileSync('comment_id.txt', 'utf8').trim(),
10
);
if (!commentId) return;
await github.rest.reactions.createForIssueComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: commentId,
content: 'rocket'
});
failure-notice:
if: github.event.workflow_run.conclusion == 'failure'
runs-on: ubuntu-latest
steps:
- name: Download artifact for PR number
id: download
uses: actions/download-artifact@v4
with:
name: ${{ inputs.result_artifact_name }}
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
continue-on-error: true
- name: React with failure
if: steps.download.outcome == 'success'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
if (!fs.existsSync('comment_id.txt')) return;
const commentId = parseInt(
fs.readFileSync('comment_id.txt', 'utf8').trim(),
10
);
if (!commentId) return;
await github.rest.reactions.createForIssueComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: commentId,
content: 'confused'
});
unauthorized-notice:
runs-on: ubuntu-latest
if: >-
github.event.workflow_run.event == 'issue_comment' &&
github.event.workflow_run.conclusion == 'skipped'
steps:
- name: Log skipped unauthorized request
uses: actions/github-script@v7
with:
script: |
core.info(
'Analysis workflow was skipped, which usually means the ' +
'issue_comment requester was not in the authorized list.'
);
-15
View File
@@ -1,15 +0,0 @@
name: facebook/rocksdb/benchmark-linux
on: workflow_dispatch
permissions: {}
# FIXME: Disabled temporarily
# schedule:
# - cron: 7 */2 * * * # At minute 7 past every 2nd hour
jobs:
benchmark-linux:
if: ${{ github.repository_owner == 'facebook' }}
runs-on: ubuntu-latest # FIXME: change this back to self-hosted when ready
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/build-for-benchmarks"
- uses: "./.github/actions/perform-benchmarks"
- uses: "./.github/actions/post-benchmarks"
-51
View File
@@ -1,51 +0,0 @@
name: Post clang-tidy PR comment
on:
workflow_run:
workflows: ["clang-tidy"]
types: [completed]
permissions:
pull-requests: write
jobs:
comment:
if: github.event.workflow_run.event == 'pull_request'
runs-on: ubuntu-latest
steps:
- name: Checkout scripts
uses: actions/checkout@v4
with:
sparse-checkout: .github/scripts
sparse-checkout-cone-mode: true
- name: Download clang-tidy results
id: download
uses: actions/download-artifact@v4.1.3
with:
name: clang-tidy-result
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
continue-on-error: true
- name: Post or update PR comment
if: steps.download.outcome == 'success'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
if (!fs.existsSync('clang-tidy-comment.md') || !fs.existsSync('pr_number.txt')) {
core.info('No clang-tidy results found; skipping.');
return;
}
const body = fs.readFileSync('clang-tidy-comment.md', 'utf8');
const prNumber = parseInt(fs.readFileSync('pr_number.txt', 'utf8').trim());
const post = require('./.github/scripts/post-pr-comment.js');
await post({
github,
context,
core,
prNumber,
body,
marker: '<!-- clang-tidy-bot -->',
});
-74
View File
@@ -1,74 +0,0 @@
name: clang-tidy
on:
push:
pull_request:
permissions: {}
jobs:
clang-tidy:
if: github.repository_owner == 'facebook'
runs-on:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
steps:
- uses: actions/checkout@v4.1.0
with:
fetch-depth: 2
- name: Mark workspace as safe for git
run: git config --global --add safe.directory $GITHUB_WORKSPACE
- name: Determine diff base
id: diff-base
run: |
if [ "${{ github.event_name }}" = "pull_request" ]; then
BASE="${{ github.event.pull_request.base.sha }}"
git fetch --depth=1 origin "$BASE"
else
BASE="${{ github.event.before }}"
if echo "$BASE" | grep -q '^0\{40\}$'; then
echo "skip=true" >> "$GITHUB_OUTPUT"
echo "New branch push; skipping clang-tidy."
exit 0
fi
fi
echo "ref=$BASE" >> "$GITHUB_OUTPUT"
echo "skip=false" >> "$GITHUB_OUTPUT"
- name: Install clang-tidy
if: steps.diff-base.outputs.skip != 'true'
run: apt-get update && apt-get install -y clang-tidy-21 && ln -sf /usr/bin/clang-tidy-21 /usr/local/bin/clang-tidy
- name: Generate compile_commands.json
if: steps.diff-base.outputs.skip != 'true'
run: |
mkdir build && cd build
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DCMAKE_C_COMPILER=clang-21 \
-DCMAKE_CXX_COMPILER=clang++-21 ..
cd ..
ln -sf build/compile_commands.json compile_commands.json
- name: Run clang-tidy on changed files
id: clang-tidy
if: steps.diff-base.outputs.skip != 'true'
run: |
python3 tools/run_clang_tidy.py \
-j 4 \
--diff-base ${{ steps.diff-base.outputs.ref }} \
--github-annotations \
--github-step-summary \
--comment-output clang-tidy-comment.md
continue-on-error: true
- name: Save PR number
if: github.event_name == 'pull_request' && always()
run: echo "${{ github.event.pull_request.number }}" > pr_number.txt
- name: Upload clang-tidy results
if: always()
uses: actions/upload-artifact@v4.0.0
with:
name: clang-tidy-result
path: |
clang-tidy-comment.md
pr_number.txt
if-no-files-found: ignore
- name: Fail if clang-tidy found issues
if: steps.clang-tidy.outcome == 'failure'
run: exit 1
@@ -1,25 +0,0 @@
# Claude Code Review — Comment Posting Workflow
#
# Thin wrapper around .github/workflows/ai-review-comment.yml so the provider
# specific workflow name stays stable while the shared implementation lives in
# one reusable workflow.
name: Post Claude Review Comment
on:
workflow_run:
workflows: ["Claude Code Review"]
types: [completed]
permissions:
pull-requests: write
issues: write
jobs:
review-comment:
uses: ./.github/workflows/ai-review-comment.yml
with:
provider: claude
result_artifact_name: claude-review-result
comment_file: claude-review-comment.md
secrets: inherit
-69
View File
@@ -1,69 +0,0 @@
# Claude Code Review — Analysis Workflow
#
# Thin wrapper around .github/workflows/ai-review-analysis.yml so provider
# specific triggers and workflow_dispatch inputs stay stable while the shared
# implementation lives in one reusable workflow.
name: Claude Code Review
on:
workflow_run:
workflows: ["facebook/rocksdb/pr-jobs"]
types: [completed]
# The early pull_request_target path is limited to same-repo PRs by the job
# condition below. Fork PRs skip this path and rely on workflow_run instead.
pull_request_target:
types: [opened, reopened, synchronize]
issue_comment:
types: [created]
workflow_dispatch:
inputs:
pr_number:
description: PR number to review
required: true
type: number
model:
description: Claude model to use (defaults to latest Opus)
required: false
type: choice
options:
- claude-opus-4-6
- claude-sonnet-4-6
default: claude-opus-4-6
thinking_budget:
description: Override MAX_THINKING_TOKENS (blank = auto-classify, capped at 24000)
required: false
type: string
default: ""
permissions:
contents: read
# The shared analysis workflow polls check state, inspects prior runs, and
# reads PR metadata. Keep those permissions read-only in the caller.
actions: read
checks: read
pull-requests: read
jobs:
review:
if: >-
github.event_name != 'pull_request_target' ||
github.event.pull_request.head.repo.full_name == github.repository
uses: ./.github/workflows/ai-review-analysis.yml
with:
provider: claude
display_name: Claude
review_command: /claude-review
query_command: /claude-query
result_artifact_name: claude-review-result
comment_file: claude-review-comment.md
default_model: claude-opus-4-6
selected_model: ${{ github.event_name == 'workflow_dispatch' && inputs.model || '' }}
classifier_model: claude-sonnet-4-6
recovery_model: claude-sonnet-4-6
thinking_budget: ${{ github.event_name == 'workflow_dispatch' && inputs.thinking_budget || '' }}
dispatch_pr_number: ${{ github.event_name == 'workflow_dispatch' && inputs.pr_number || '' }}
secrets: inherit
@@ -1,25 +0,0 @@
# Codex Code Review — Comment Posting Workflow
#
# Thin wrapper around .github/workflows/ai-review-comment.yml so the provider
# specific workflow name stays stable while the shared implementation lives in
# one reusable workflow.
name: Post Codex Review Comment
on:
workflow_run:
workflows: ["Codex Code Review"]
types: [completed]
permissions:
pull-requests: write
issues: write
jobs:
review-comment:
uses: ./.github/workflows/ai-review-comment.yml
with:
provider: codex
result_artifact_name: codex-review-result
comment_file: codex-review-comment.md
secrets: inherit
-68
View File
@@ -1,68 +0,0 @@
# Codex Code Review — Analysis Workflow
#
# Thin wrapper around .github/workflows/ai-review-analysis.yml so provider
# specific triggers and workflow_dispatch inputs stay stable while the shared
# implementation lives in one reusable workflow.
name: Codex Code Review
on:
workflow_run:
workflows: ["facebook/rocksdb/pr-jobs"]
types: [completed]
# The early pull_request_target path is limited to same-repo PRs by the job
# condition below. Fork PRs skip this path and rely on workflow_run instead.
pull_request_target:
types: [opened, reopened, synchronize]
issue_comment:
types: [created]
workflow_dispatch:
inputs:
pr_number:
description: PR number to review
required: true
type: number
model:
description: Codex model to use
required: false
type: choice
options:
- gpt-5.5
- gpt-5.3-codex
- gpt-5.2-codex
default: gpt-5.5
thinking_budget:
description: Override MAX_THINKING_TOKENS (blank = auto-classify)
required: false
type: string
default: ""
permissions:
contents: read
# The shared analysis workflow polls check state, inspects prior runs, and
# reads PR metadata. Keep those permissions read-only in the caller.
actions: read
checks: read
pull-requests: read
jobs:
review:
if: >-
github.event_name != 'pull_request_target' ||
github.event.pull_request.head.repo.full_name == github.repository
uses: ./.github/workflows/ai-review-analysis.yml
with:
provider: codex
display_name: Codex
review_command: /codex-review
query_command: /codex-query
result_artifact_name: codex-review-result
comment_file: codex-review-comment.md
default_model: gpt-5.5
selected_model: ${{ github.event_name == 'workflow_dispatch' && inputs.model || '' }}
thinking_budget: ${{ github.event_name == 'workflow_dispatch' && inputs.thinking_budget || '' }}
dispatch_pr_number: ${{ github.event_name == 'workflow_dispatch' && inputs.pr_number || '' }}
secrets: inherit
-19
View File
@@ -1,19 +0,0 @@
name: facebook/rocksdb/nightly
on: workflow_dispatch
permissions: {}
jobs:
# These jobs would be in nightly but are failing or otherwise broken for
# some reason.
build-linux-arm-test-full:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: arm64large
container:
image: ubuntu-2004:202111-02
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/install-gflags"
- run: make V=1 J=4 -j4 check
- uses: "./.github/actions/post-steps"
-189
View File
@@ -1,189 +0,0 @@
name: facebook/rocksdb/nightly
on:
schedule:
- cron: 0 9 * * *
workflow_dispatch:
permissions: {}
jobs:
build-format-compatible:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
with:
fetch-depth: 0 # Need full repo history
fetch-tags: true
- uses: "./.github/actions/setup-upstream"
- uses: "./.github/actions/pre-steps"
- name: test
run: |-
export TEST_TMPDIR=/dev/shm/rocksdb
rm -rf /dev/shm/rocksdb
mkdir /dev/shm/rocksdb
git config --global --add safe.directory /__w/rocksdb/rocksdb
tools/check_format_compatible.sh
- uses: "./.github/actions/post-steps"
build-linux-non-shm:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
env:
TEST_TMPDIR: "/tmp/rocksdb_test_tmp"
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- run: make V=1 -j32 check
- uses: "./.github/actions/post-steps"
build-linux-clang-21-asan-ubsan-with-folly:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
env:
CC: clang-21
CXX: clang++-21
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/cache-getdeps-downloads"
- uses: "./.github/actions/setup-folly"
- name: Build folly and dependencies
run: |
clean_path=()
IFS=: read -ra path_entries <<< "$PATH"
for entry in "${path_entries[@]}"; do
if [[ "$entry" != "/usr/lib/ccache" ]]; then
clean_path+=("$entry")
fi
done
export PATH="$(IFS=:; echo "${clean_path[*]}")"
ccache_bin="$(command -v ccache || true)"
if [[ -n "$ccache_bin" && -x "$ccache_bin" ]]; then
mv "$ccache_bin" "${ccache_bin}.disabled"
fi
export CC=gcc
export CXX=g++
export USE_CCACHE=0
export CCACHE_DISABLE=1
make build_folly
shell: bash
- run: LIB_MODE=static USE_CLANG=1 USE_FOLLY=1 COMPILE_WITH_UBSAN=1 COMPILE_WITH_ASAN=1 make -j32 check
- uses: "./.github/actions/post-steps"
build-linux-cmake-with-folly:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/cache-getdeps-downloads"
- uses: "./.github/actions/setup-folly"
- uses: "./.github/actions/cache-folly"
id: cache-folly
- uses: "./.github/actions/build-folly"
with:
cache-hit: ${{ steps.cache-folly.outputs.cache-hit }}
- run: "(mkdir build && cd build && cmake -DUSE_FOLLY=1 -DWITH_GFLAGS=1 -DROCKSDB_BUILD_SHARED=0 .. && make VERBOSE=1 -j20 && ctest -j20)"
- uses: "./.github/actions/post-steps"
build-linux-release-with-folly:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/cache-getdeps-downloads"
- uses: "./.github/actions/setup-folly"
- run: "DEBUG_LEVEL=0 make -j20 build_folly"
- run: "USE_FOLLY=1 LIB_MODE=static DEBUG_LEVEL=0 V=1 make -j20 release"
- run: "(mkdir build && cd build && cmake -DUSE_FOLLY=1 -DWITH_GFLAGS=1 -DROCKSDB_BUILD_SHARED=0 -DCMAKE_BUILD_TYPE=Release .. && make VERBOSE=1 -j20 && ctest -j20)"
- uses: "./.github/actions/post-steps"
build-windows-msvc-avx2:
if: ${{ github.repository_owner == 'facebook' }}
runs-on: windows-8-core
env:
CMAKE_PORTABLE: AVX2
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/windows-build-steps"
build-linux-arm-test-full:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 4-core-ubuntu-arm
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- run: sudo apt-get update && sudo apt-get install -y build-essential libgflags-dev
- run: make V=1 J=4 -j4 check
- uses: "./.github/actions/post-steps"
build-linux-arm-crashtest:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 4-core-ubuntu-arm
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- run: sudo apt-get update && sudo apt-get install -y build-essential libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev
- run: sudo mount -o remount,size=16G /dev/shm
- run: sudo dd bs=1048576 count=4096 if=/dev/zero of=/swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile
- run: ulimit -S -n `ulimit -H -n` && make V=1 -j8 CRASH_TEST_EXT_ARGS='--duration=1800 --max_key=2500000' blackbox_crash_test_with_atomic_flush
- run: rm -rf /dev/shm/rocksdb.*
- run: ulimit -S -n `ulimit -H -n` && make V=1 -j8 CRASH_TEST_EXT_ARGS='--duration=1800 --max_key=2500000' blackbox_crash_test_with_multiops_wc_txn
- uses: "./.github/actions/post-steps"
build-examples:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- name: Build examples
run: make V=1 -j4 static_lib && cd examples && make V=1 -j4
- uses: "./.github/actions/post-steps"
build-fuzzers:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- name: Build rocksdb lib
run: CC=clang-21 CXX=clang++-21 USE_CLANG=1 make -j4 static_lib
- name: Build fuzzers
run: cd fuzz && make sst_file_writer_fuzzer db_fuzzer db_map_fuzzer
- uses: "./.github/actions/post-steps"
build-linux-cmake-with-folly-lite:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/cache-getdeps-downloads"
- uses: "./.github/actions/setup-folly"
- run: "(mkdir build && cd build && cmake -DUSE_FOLLY_LITE=1 -DWITH_GFLAGS=1 -DCMAKE_CXX_FLAGS=-DGLOG_USE_GLOG_EXPORT .. && make VERBOSE=1 -j20 && ctest -j20)"
- uses: "./.github/actions/post-steps"
-47
View File
@@ -1,47 +0,0 @@
name: facebook/rocksdb/pr-jobs-candidate
on: workflow_dispatch
permissions: {}
jobs:
# These jobs would be in pr-jobs but are failing or otherwise broken for
# some reason.
# =========================== ARM Jobs ============================ #
build-linux-arm:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: arm64large # GitHub hosted ARM runners do not yet exist
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/install-gflags"
- run: ROCKSDBTESTS_PLATFORM_DEPENDENT=only make V=1 J=4 -j4 all_but_some_tests check_some
- uses: "./.github/actions/post-steps"
build-linux-arm-cmake-no_test_run:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: arm64large # GitHub hosted ARM runners do not yet exist
env:
JAVA_HOME: "/usr/lib/jvm/java-8-openjdk-arm64"
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/install-gflags"
- name: Set Java Environment
run: |-
echo "JAVA_HOME=${JAVA_HOME}"
echo 'export PATH=$JAVA_HOME/bin:$PATH' >> $BASH_ENV
which java && java -version
which javac && javac -version
- name: Build with cmake
run: |-
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DWITH_TESTS=0 -DWITH_GFLAGS=1 -DWITH_BENCHMARK_TOOLS=0 -DWITH_TOOLS=0 -DWITH_CORE_TOOLS=1 ..
make -j4
- name: Build Java with cmake
run: |-
rm -rf build
mkdir build
cd build
cmake -DJNI=1 -DCMAKE_BUILD_TYPE=Release -DWITH_GFLAGS=1 ..
make -j4 rocksdb rocksdbjni
- uses: "./.github/actions/post-steps"
-741
View File
@@ -1,741 +0,0 @@
name: facebook/rocksdb/pr-jobs
on: [push, pull_request]
permissions: {}
env:
# Set to a job name to run only that job (on any repo), or leave empty for
# normal behavior (all jobs on facebook repo only).
ONLY_JOB: ''
jobs:
config:
runs-on: ubuntu-latest
outputs:
only_job: ${{ steps.set.outputs.only_job }}
steps:
- id: set
run: echo "only_job=$ONLY_JOB" >> "$GITHUB_OUTPUT"
# NOTE: multiple workflows would be recommended, but the current GHA UI in
# PRs doesn't make it clear when there's an overall error with a workflow,
# making it easy to overlook something broken. Grouping everything into one
# workflow minimizes the problem because it will be suspicious if there are
# no GHA results.
#
# The if: ${{ github.repository_owner == 'facebook' }} lines prevent the
# jobs from attempting to run on repo forks, because of a few problems:
# * runs-on labels are repository (owner) specific, so the job might wait
# for days waiting for a runner that simply isn't available.
# * Pushes to branches on forks for pull requests (the normal process) would
# run the workflow jobs twice: once in the pull-from fork and once for the PR
# destination repo. This is wasteful and dumb.
# * It is not known how to avoid copy-pasting the line to each job,
# increasing the risk of misconfiguration, especially on forks that might
# want to run with this GHA setup.
#
# SELECTIVE JOB EXECUTION: Set the ONLY_JOB env var at the top of this file
# to a job name (e.g. "build-linux-clang-tidy") to run only that job,
# bypassing the repository owner check. Leave it empty for normal behavior.
#
# DEBUGGING WITH SSH: Temporarily add this as a job step, either before the
# step of interest without the "if:" line or after the failing step with the
# "if:" line. Then use ssh command printed in CI output.
# - name: Setup tmate session # TEMPORARY!
# if: ${{ failure() }}
# uses: mxschmitt/action-tmate@v3
# with:
# limit-access-to-actor: true
# ======================== Fast Initial Checks ====================== #
check-format-and-targets:
if: needs.config.outputs.only_job == 'check-format-and-targets' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4.1.0
with:
fetch-depth: 0 # Need full checkout to determine merge base
fetch-tags: true
- uses: "./.github/actions/setup-upstream"
- name: Setup Python
uses: actions/setup-python@v5
- name: Install Dependencies
run: python -m pip install --upgrade pip
- name: Install argparse
run: pip install argparse
- name: Install clang-format
run: |
pip install https://files.pythonhosted.org/packages/fb/ac/3c04772acc0257f5730e83adb542b2603c1a62d1315010ab593a980af404/clang_format-21.1.2-py2.py3-none-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
clang-format --version
- name: Download clang-format-diff.py
run: wget https://rocksdb-deps.s3.us-west-2.amazonaws.com/llvm/llvm-project/release/12.x/clang/tools/clang-format/clang-format-diff.py
- name: Check format
run: VERBOSE_CHECK=1 make check-format
- name: Compare buckify output
run: make check-buck-targets
- name: Simple source code checks
run: make check-sources
- name: Validate GitHub Actions YAML
run: make check-workflow-yaml
- name: Sanity check check_format_compatible.sh
run: |-
export TEST_TMPDIR=/dev/shm/rocksdb
rm -rf /dev/shm/rocksdb
mkdir /dev/shm/rocksdb
git reset --hard
git config --global --add safe.directory /__w/rocksdb/rocksdb
SANITY_CHECK=1 LONG_TEST=1 tools/check_format_compatible.sh
# ========================= Linux With Tests ======================== #
build-linux:
if: needs.config.outputs.only_job == 'build-linux' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: build-linux
- run: make V=1 J=32 -j32 check
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-cmake-mingw:
if: needs.config.outputs.only_job == 'build-linux-cmake-mingw' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: cmake-mingw
- run: update-alternatives --set x86_64-w64-mingw32-g++ /usr/bin/x86_64-w64-mingw32-g++-posix
- name: Build cmake-mingw
run: |-
export PATH=$JAVA_HOME/bin:$PATH
echo "JAVA_HOME=${JAVA_HOME}"
which java && java -version
which javac && javac -version
mkdir build && cd build && cmake -DJNI=1 -DWITH_GFLAGS=OFF .. -DCMAKE_C_COMPILER=x86_64-w64-mingw32-gcc -DCMAKE_CXX_COMPILER=x86_64-w64-mingw32-g++ -DCMAKE_SYSTEM_NAME=Windows && make -j4 rocksdb rocksdbjni
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-make-with-folly:
if: needs.config.outputs.only_job == 'build-linux-make-with-folly' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 32-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/cache-getdeps-downloads"
- uses: "./.github/actions/setup-folly"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: make-with-folly
- uses: "./.github/actions/cache-folly"
id: cache-folly
- uses: "./.github/actions/build-folly"
with:
cache-hit: ${{ steps.cache-folly.outputs.cache-hit }}
- run: USE_FOLLY=1 LIB_MODE=static V=1 make -j64 check
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-make-with-folly-lite-no-test:
if: needs.config.outputs.only_job == 'build-linux-make-with-folly-lite-no-test' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/cache-getdeps-downloads"
- uses: "./.github/actions/setup-folly"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: make-folly-lite
- run: USE_FOLLY_LITE=1 EXTRA_CXXFLAGS=-DGLOG_USE_GLOG_EXPORT V=1 make -j32 all
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-cmake-with-folly-coroutines:
if: needs.config.outputs.only_job == 'build-linux-cmake-with-folly-coroutines' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 32-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/cache-getdeps-downloads"
- uses: "./.github/actions/setup-folly"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: cmake-folly-coroutines
- uses: "./.github/actions/cache-folly"
id: cache-folly
- uses: "./.github/actions/build-folly"
with:
cache-hit: ${{ steps.cache-folly.outputs.cache-hit }}
- run: "(mkdir build && cd build && cmake -DUSE_COROUTINES=1 -DWITH_GFLAGS=1 -DROCKSDB_BUILD_SHARED=0 -DPORTABLE=ON .. && make VERBOSE=1 -j64 && ctest -j64)"
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-cmake-with-benchmark-no-thread-status:
if: needs.config.outputs.only_job == 'build-linux-cmake-with-benchmark-no-thread-status' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 16-core-ubuntu
strategy:
fail-fast: false
matrix:
shard: [0, 1, 2, 3]
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: cmake-benchmark
- name: Build
run: mkdir build && cd build && cmake -DWITH_GFLAGS=1 -DWITH_BENCHMARK=1 -DPORTABLE=ON -DCMAKE_CXX_FLAGS=-DNROCKSDB_THREAD_STATUS .. && make VERBOSE=1 -j20
- name: Test shard ${{ matrix.shard }} of 4
run: cd build && ctest -j20 -I ${{ matrix.shard }},,4
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
with:
artifact-prefix: "${{ github.job }}-shard${{ matrix.shard }}"
build-linux-encrypted_env-no_compression:
if: needs.config.outputs.only_job == 'build-linux-encrypted_env-no_compression' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: encrypted-env-no-compression
- run: ENCRYPTED_ENV=1 ROCKSDB_DISABLE_SNAPPY=1 ROCKSDB_DISABLE_ZLIB=1 ROCKSDB_DISABLE_BZIP=1 ROCKSDB_DISABLE_LZ4=1 ROCKSDB_DISABLE_ZSTD=1 make V=1 J=32 -j32 check
- run: "./sst_dump --help | grep -E -q 'Supported built-in compression types: kNoCompression$' # Verify no compiled in compression\n"
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
# ======================== Linux No Test Runs ======================= #
build-linux-release:
if: needs.config.outputs.only_job == 'build-linux-release' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: release
- run: make V=1 -j32 LIB_MODE=shared release
- run: ls librocksdb.so
- run: "./trace_analyzer --version" # A tool dependent on gflags that can run in release build
- run: make clean
- run: USE_RTTI=1 make V=1 -j32 release
- run: ls librocksdb.a
- run: "./trace_analyzer --version"
- run: make clean
- run: apt-get remove -y libgflags-dev
- run: make V=1 -j32 LIB_MODE=shared release
- run: ls librocksdb.so
- run: if ./trace_analyzer --version; then false; else true; fi
- run: make clean
- run: USE_RTTI=1 make V=1 -j32 release
- run: ls librocksdb.a
- run: if ./trace_analyzer --version; then false; else true; fi
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-clang-13-no_test_run:
if: needs.config.outputs.only_job == 'build-linux-clang-13-no_test_run' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 8-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: clang-13
# FIXME: get back to "all microbench" targets
- run: CC=clang-13 CXX=clang++-13 USE_CLANG=1 EXTRA_CXXFLAGS=-stdlib=libc++ EXTRA_LDFLAGS=-stdlib=libc++ make -j32 shared_lib
- run: make clean
# FIXME: get back to "release" target
- run: CC=clang-13 CXX=clang++-13 USE_CLANG=1 EXTRA_CXXFLAGS=-stdlib=libc++ EXTRA_LDFLAGS=-stdlib=libc++ DEBUG_LEVEL=0 make -j32 shared_lib
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-clang-21-no_test_run:
if: needs.config.outputs.only_job == 'build-linux-clang-21-no_test_run' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: clang-21
- run: CC=clang-21 CXX=clang++-21 USE_CLANG=1 make -j32 all microbench
- run: make clean
- run: CC=clang-21 CXX=clang++-21 USE_CLANG=1 DEBUG_LEVEL=0 make -j32 release
- name: Install clang-format-21 for C API gen checks
run: apt-get update && apt-get install -y clang-format-21
- name: Check C API codegen (link-complete, backward-compatible, up to date)
# Run the make target in STRICT mode so this core CI job HARD-FAILS rather
# than silently skipping if a prerequisite (clang++, clang-format, or the
# compat baseline ref) is ever missing. See `make check-c-api-gen`.
run: |
# The container checkout is owned by a different user, so git refuses to
# operate on it ("dubious ownership") -- mark it safe for the `git fetch`
# and the `git show` the compat checker runs.
git config --global --add safe.directory "$GITHUB_WORKSPACE"
git config --global --add safe.directory '*'
git fetch --no-tags --depth=1 origin main
CXX=clang++-21 make check-c-api-gen \
CHECK_C_API_GEN_STRICT=1 \
API_COMPAT_REF=FETCH_HEAD \
CLANG_FORMAT_BINARY=clang-format-21
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-gcc-14-no_test_run:
if: needs.config.outputs.only_job == 'build-linux-gcc-14-no_test_run' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: gcc-14
- run: CC=gcc-14 CXX=g++-14 V=1 make -j32 all microbench
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
# ======================== Linux Other Checks ======================= #
build-linux-unity-and-headers:
if: needs.config.outputs.only_job == 'build-linux-unity-and-headers' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 4-core-ubuntu
container:
image: gcc:latest
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- run: apt-get update -y && apt-get install -y libgflags-dev
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: unity-headers
- name: Unity build
run: make V=1 -j8 unity_test
- run: make V=1 -j8 -k check-headers
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-mini-crashtest:
if: needs.config.outputs.only_job == 'build-linux-mini-crashtest' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 4-core-ubuntu
strategy:
fail-fast: false
matrix:
include:
- crash_test_target: blackbox_crash_test_with_atomic_flush
crash_duration: 480
- crash_test_target: blackbox_crash_test
crash_duration: 240
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb --ulimit nofile=1048576:1048576
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: mini-crashtest
- run: |
ulimit -S -n $(ulimit -Hn) # Raise open files soft limit to hard limit
echo "Updated ulimit -Hn: $(ulimit -Hn) -Sn: $(ulimit -Sn)"
make V=1 -j8 CRASH_TEST_EXT_ARGS='--duration=${{ matrix.crash_duration }} --max_key=2500000' ${{ matrix.crash_test_target }}
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
with:
artifact-prefix: "${{ github.job }}-${{ matrix.crash_test_target }}"
# ======================= Linux with Sanitizers ===================== #
build-linux-clang21-asan-ubsan:
if: needs.config.outputs.only_job == 'build-linux-clang21-asan-ubsan' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 32-core-ubuntu
strategy:
fail-fast: false
matrix:
shard: [0, 1, 2]
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: clang21-asan-ubsan
- run: COMPILE_WITH_ASAN=1 COMPILE_WITH_UBSAN=1 CC=clang-21 CXX=clang++-21 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j40 check
env:
CI_SHARD_INDEX: ${{ matrix.shard }}
CI_TOTAL_SHARDS: 3
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
with:
artifact-prefix: "${{ github.job }}-shard${{ matrix.shard }}"
build-linux-clang21-mini-tsan:
if: needs.config.outputs.only_job == 'build-linux-clang21-mini-tsan' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 32-core-ubuntu
strategy:
fail-fast: false
matrix:
shard: [0, 1, 2]
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: clang21-tsan
- run: COMPILE_WITH_TSAN=1 CC=clang-21 CXX=clang++-21 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j32 check
env:
CI_SHARD_INDEX: ${{ matrix.shard }}
CI_TOTAL_SHARDS: 3
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
with:
artifact-prefix: "${{ github.job }}-shard${{ matrix.shard }}"
build-linux-static_lib-alt_namespace-status_checked:
if: needs.config.outputs.only_job == 'build-linux-static_lib-alt_namespace-status_checked' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: static-alt-namespace
- run: ASSERT_STATUS_CHECKED=1 TEST_UINT128_COMPAT=1 ROCKSDB_MODIFY_NPHASH=1 LIB_MODE=static OPT="-DROCKSDB_USE_STD_SEMAPHORES -DROCKSDB_NAMESPACE=alternative_rocksdb_ns" make V=1 -j24 check
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
# ========================= MacOS build only ======================== #
build-macos:
if: needs.config.outputs.only_job == 'build-macos' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on: macos-15-xlarge
env:
ROCKSDB_DISABLE_JEMALLOC: 1
steps:
- uses: actions/checkout@v4.1.0
- uses: maxim-lobanov/setup-xcode@v1.6.0
with:
xcode-version: 16.4.0
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: macos
- uses: "./.github/actions/increase-max-open-files-on-macos"
- uses: "./.github/actions/install-gflags-on-macos"
- uses: "./.github/actions/pre-steps-macos"
- name: Build
run: ulimit -S -n `ulimit -H -n` && make V=1 J=16 -j8 all
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
# ========================= MacOS with Tests ======================== #
build-macos-cmake:
if: needs.config.outputs.only_job == 'build-macos-cmake' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on: macos-15-xlarge
strategy:
matrix:
run_sharded_tests: [0, 1, 2, 3]
steps:
- uses: actions/checkout@v4.1.0
- uses: maxim-lobanov/setup-xcode@v1.6.0
with:
xcode-version: 16.4.0
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: macos-cmake
- uses: "./.github/actions/increase-max-open-files-on-macos"
- uses: "./.github/actions/install-gflags-on-macos"
- uses: "./.github/actions/pre-steps-macos"
- name: cmake generate project file
run: ulimit -S -n `ulimit -H -n` && mkdir build && cd build && cmake -DWITH_GFLAGS=1 ..
- name: Build tests
run: cd build && make VERBOSE=1 -j8
- name: Run shard 0 out of 4 test shards
run: ulimit -S -n `ulimit -H -n` && cd build && ctest -j8 -I 0,,4
if: ${{ matrix.run_sharded_tests == 0 }}
- name: Run shard 1 out of 4 test shards
run: ulimit -S -n `ulimit -H -n` && cd build && ctest -j8 -I 1,,4
if: ${{ matrix.run_sharded_tests == 1 }}
- name: Run shard 2 out of 4 test shards
run: ulimit -S -n `ulimit -H -n` && cd build && ctest -j8 -I 2,,4
if: ${{ matrix.run_sharded_tests == 2 }}
- name: Run shard 3 out of 4 test shards
run: ulimit -S -n `ulimit -H -n` && cd build && ctest -j8 -I 3,,4
if: ${{ matrix.run_sharded_tests == 3 }}
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
# ======================== Windows with Tests ======================= #
# NOTE: some windows jobs are in "nightly" to save resources
build-windows-msvc:
if: needs.config.outputs.only_job == 'build-windows-msvc' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on: windows-8-core
strategy:
fail-fast: false
matrix:
include:
- test_shard: db_test
suite_run: db_test
run_java: "false"
- test_shard: other
suite_run: arena_test,db_basic_test,db_etc2_test,db_merge_operand_test,bloom_test,c_test,coding_test,crc32c_test,dynamic_bloom_test,env_basic_test,env_test,hash_test,random_test
run_java: "false"
- test_shard: java
suite_run: ""
run_java: "true"
env:
CMAKE_PORTABLE: 1
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/windows-build-steps"
with:
suite-run: ${{ matrix.suite_run }}
run-java: ${{ matrix.run_java }}
# ============================ Java Jobs ============================ #
build-linux-java:
if: needs.config.outputs.only_job == 'build-linux-java' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: java
- name: Set Java Environment
run: |-
echo "JAVA_HOME=${JAVA_HOME}"
which java && java -version
which javac && javac -version
- name: Test RocksDBJava
run: make V=1 J=8 -j8 jtest
- uses: "./.github/actions/teardown-ccache"
if: always()
build-linux-java-static:
if: needs.config.outputs.only_job == 'build-linux-java-static' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: java-static
- name: Set Java Environment
run: |-
echo "JAVA_HOME=${JAVA_HOME}"
which java && java -version
which javac && javac -version
- name: Build RocksDBJava Static Library
run: make V=1 J=8 -j8 rocksdbjavastatic
- uses: "./.github/actions/teardown-ccache"
if: always()
build-macos-java:
if: needs.config.outputs.only_job == 'build-macos-java' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on: macos-15-xlarge
env:
ROCKSDB_DISABLE_JEMALLOC: 1
steps:
- uses: actions/checkout@v4.1.0
- uses: maxim-lobanov/setup-xcode@v1.6.0
with:
xcode-version: 16.4.0
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: macos-java
- uses: "./.github/actions/increase-max-open-files-on-macos"
- uses: "./.github/actions/install-gflags-on-macos"
- uses: "./.github/actions/setup-jdk-on-macos"
- uses: "./.github/actions/pre-steps-macos"
- name: Set Java Environment
run: |-
echo "JAVA_HOME=${JAVA_HOME}"
which java && java -version
which javac && javac -version
- name: Test RocksDBJava
run: make V=1 J=16 -j16 jtest
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-macos-java-static:
if: needs.config.outputs.only_job == 'build-macos-java-static' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on: macos-15-xlarge
steps:
- uses: actions/checkout@v4.1.0
- uses: maxim-lobanov/setup-xcode@v1.6.0
with:
xcode-version: 16.4.0
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: macos-java-static
- uses: "./.github/actions/increase-max-open-files-on-macos"
- uses: "./.github/actions/install-gflags-on-macos"
- uses: "./.github/actions/setup-jdk-on-macos"
- uses: "./.github/actions/pre-steps-macos"
- name: Set Java Environment
run: |-
echo "JAVA_HOME=${JAVA_HOME}"
which java && java -version
which javac && javac -version
- name: Build RocksDBJava x86 and ARM Static Libraries
run: make V=1 J=16 -j16 rocksdbjavastaticosx
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-macos-java-static-universal:
if: needs.config.outputs.only_job == 'build-macos-java-static-universal' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on: macos-15-xlarge
steps:
- uses: actions/checkout@v4.1.0
- uses: maxim-lobanov/setup-xcode@v1.6.0
with:
xcode-version: 16.4.0
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: macos-java-static-universal
- uses: "./.github/actions/increase-max-open-files-on-macos"
- uses: "./.github/actions/install-gflags-on-macos"
- uses: "./.github/actions/setup-jdk-on-macos"
- uses: "./.github/actions/pre-steps-macos"
- name: Set Java Environment
run: |-
echo "JAVA_HOME=${JAVA_HOME}"
which java && java -version
which javac && javac -version
- name: Build RocksDBJava Universal Binary Static Library
run: make V=1 J=16 -j16 rocksdbjavastaticosx_ub
- uses: "./.github/actions/teardown-ccache"
if: always()
- uses: "./.github/actions/post-steps"
build-linux-java-pmd:
if: needs.config.outputs.only_job == 'build-linux-java-pmd' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 4-core-ubuntu
container:
image: evolvedbinary/rocksjava:alpine3_x64-be
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/install-maven"
- uses: "./.github/actions/pre-steps"
- name: Set Java Environment
run: |-
echo "JAVA_HOME=${JAVA_HOME}"
which java && java -version
which javac && javac -version
- name: PMD RocksDBJava
run: make V=1 J=8 -j8 jpmd
- uses: actions/upload-artifact@v4.0.0
with:
name: pmd-report
path: "${{ github.workspace }}/java/target/pmd.xml"
- uses: actions/upload-artifact@v4.0.0
with:
name: maven-site
path: "${{ github.workspace }}/java/target/site"
build-linux-arm:
if: needs.config.outputs.only_job == 'build-linux-arm' || (needs.config.outputs.only_job == '' && github.repository_owner == 'facebook')
needs: config
runs-on:
labels: 4-core-ubuntu-arm
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- run: sudo apt-get update && sudo apt-get install -y build-essential ccache
- uses: "./.github/actions/setup-ccache"
with:
cache-key-prefix: arm
- run: ROCKSDBTESTS_PLATFORM_DEPENDENT=only make V=1 J=4 -j4 all_but_some_tests check_some
- name: Print ccache stats
run: ccache -s
if: always()
shell: bash
- uses: "./.github/actions/post-steps"
-20
View File
@@ -1,20 +0,0 @@
name: facebook/rocksdb/weekly
on:
schedule:
- cron: 0 9 * * 0
workflow_dispatch:
permissions: {}
jobs:
build-linux-valgrind:
if: ${{ github.repository_owner == 'facebook' }}
runs-on:
labels: 16-core-ubuntu
timeout-minutes: 840
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
- run: make V=1 -j20 valgrind_test
- uses: "./.github/actions/post-steps"
+3 -54
View File
@@ -1,7 +1,5 @@
TARGETS
make_config.mk
rocksdb.pc
.build_signature
build_tools/__pycache__/
*.a
*.arc
@@ -10,7 +8,6 @@ build_tools/__pycache__/
*.gcda
*.gcno
*.o
*.o.tmp
*.so
*.so.*
*_test
@@ -28,7 +25,6 @@ build_tools/__pycache__/
*.vcxproj.filters
*.sln
*.cmake
.watchmanconfig
CMakeCache.txt
CMakeFiles/
build/
@@ -36,32 +32,16 @@ build/
ldb
manifest_dump
sst_dump
blob_dump
block_cache_trace_analyzer
tools/block_cache_analyzer/*.pyc
column_aware_encoding_exp
util/build_version.cc
build_tools/VALGRIND_LOGS/
coverage/COVERAGE_REPORT
.gdbhistory
.gdb_history
package/
.phutil_module_cache
unity.a
tags
etags
GPATH
GRTAGS
GTAGS
rocksdb_dump
rocksdb_undump
db_etc2_test
trace_analyzer
block_cache_trace_analyzer
io_tracer_parser
.DS_Store
.vs
.vscode
.clangd
java/out
java/target
@@ -77,40 +57,9 @@ rocksdb.h
unity.cc
java/crossbuild/.vagrant
.vagrant/
java/**/*.asc
java/**.asc
java/javadoc
scan_build_report/
t
LOG
db_logs/
tp2/
fbcode/
fbcode
buckifier/*.pyc
buckifier/__pycache__
tools/__pycache__/
tools/c_api_gen/__pycache__/
.arcconfig
compile_commands.json
meta_gen_cpp_compile_commands.json
clang-format-diff.py
.py3/
fuzz/proto/gen/
fuzz/crash-*
cmake-build-*
third-party/folly/
.cache
*.sublime-*
# Claude Code local settings
.claude/settings.local.json
tools/__pycache__/
# Keep documentation trackable even if broader ignores match names like "*_test".
!docs/
!docs/**
-4
View File
@@ -1,4 +0,0 @@
extraction:
cpp:
index:
build_command: make static_lib
+43
View File
@@ -0,0 +1,43 @@
sudo: false
language: cpp
matrix:
include:
- os: linux
compiler: clang
env: COMPILER=clang++-3.6
addons:
apt:
sources: ['ubuntu-toolchain-r-test', 'llvm-toolchain-precise-3.6']
packages: ['clang-3.6', 'clang-format-3.6', 'zlib1g-dev', 'libbz2-dev', 'libsnappy-dev', 'curl']
- os: osx
compiler: clang
install:
# Build gflags
# TODO(noetzli): Remove when gflags available through Travis
- pushd /tmp/ && curl -L https://github.com/gflags/gflags/archive/v2.1.2.tar.gz -o gflags.tar.gz && tar xfz gflags.tar.gz && cd gflags-2.1.2 && cmake . && make && popd
# Download clang-format-diff.py to check source code formatting
- pushd /tmp/ && curl -L http://llvm.org/svn/llvm-project/cfe/trunk/tools/clang-format/clang-format-diff.py -o clang-format-diff.py && chmod +x clang-format-diff.py && popd
before_script:
# Add gflags to include/library paths
# TODO(noetzli): Remove when gflags available through Travis
- export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/gflags-2.1.2/lib"
- export LIBRARY_PATH="$LIBRARY_PATH:/tmp/gflags-2.1.2/lib"
- export CPLUS_INCLUDE_PATH="$CPLUS_INCLUDE_PATH:/tmp/gflags-2.1.2/include"
- if [ -n "${COMPILER}" ]; then CXX=${COMPILER}; fi
- if [[ "${TRAVIS_OS_NAME}" == 'osx' ]]; then brew install gflags snappy; fi
- ulimit -n 2000 || true
# Lousy hack to disable use and testing of fallocate, which doesn't behave quite
# as EnvPosixTest::AllocateTest expects within the Travis OpenVZ environment.
script:
- if [[ "${TRAVIS_OS_NAME}" == 'linux' ]]; then OPT=-DTRAVIS CLANG_FORMAT_DIFF=/tmp/clang-format-diff.py make format || true; fi
- OPT=-DTRAVIS V=1 make -j4 check && OPT=-DTRAVIS V=1 make clean jclean rocksdbjava jtest && make clean && OPT="-DTRAVIS -DROCKSDB_LITE" V=1 make -j4 check
notifications:
email:
- leveldb@fb.com
webhooks:
- https://buildtimetrend.herokuapp.com/travis
-6
View File
@@ -1,6 +0,0 @@
{
"content_hash_warming": true,
"content_hash_max_items": 333333,
"hint_num_files_per_dir": 8,
"fsevents_latency": 0.05
}
-9
View File
@@ -1,9 +0,0 @@
# Agent Instructions
This repository's authoritative agent instructions live in `CLAUDE.md`.
Read and follow [`CLAUDE.md`](./CLAUDE.md) in full before making changes or
reviewing code in this checkout.
If there is any ambiguity between this file and `CLAUDE.md`, `CLAUDE.md` takes
precedence.
-1
View File
@@ -9,4 +9,3 @@ Sanjay Ghemawat <sanjay@google.com>
# Partial list of contributors:
Kevin Regan <kevin.d.regan@gmail.com>
Johan Bilien <jobi@litl.com>
Matthew Von-Maszewski <https://github.com/matthewvon> (Basho Technologies)
-5800
View File
File diff suppressed because it is too large Load Diff
-435
View File
@@ -1,435 +0,0 @@
# RocksDB Code Generation and Review Guidance
This document provides guidance for generating and reviewing code in the RocksDB project, derived from analysis of code review feedback across hundreds of complex merged Pull Requests. Use this as a reference when writing code with AI assistants or conducting code reviews.
---
## General Best Practices
### Code Quality and Maintainability
**Clarity and Readability:** Write clear, self-documenting code. Use meaningful variable names, add comments for complex logic, and structure code to minimize cognitive load. Avoid clever tricks that sacrifice readability for marginal performance gains unless absolutely necessary.
**Consistent Style:** Follow existing code style conventions. RocksDB uses `.clang-format` for formatting, specific naming conventions, and structural patterns. Deviations from these patterns are frequently flagged in reviews.
**Error Handling:** Ensure robust error handling throughout the codebase. Use RocksDB's `Status` type consistently, propagate errors appropriately, and avoid silently ignoring failures. Reviewers pay close attention to edge cases and failure modes.
### Testing Philosophy
**Comprehensive Coverage:** Every change should include appropriate test coverage. This includes unit tests for isolated functionality, integration tests for component interactions, and stress tests for concurrency and performance validation. Reviewers will ask for additional tests if coverage is insufficient.
**Edge Cases and Failure Modes:** Tests should explicitly cover edge cases, boundary conditions, and potential failure scenarios. This is especially important for changes affecting core database operations, compaction, or recovery logic.
**Platform-Specific Testing:** RocksDB supports multiple platforms (Linux, Windows, macOS) and compilers (GCC, Clang, MSVC). Changes should be tested across relevant platforms, particularly when touching platform-specific code or using compiler-specific features.
### Performance Considerations
**⚠️ PERFORMANCE IS CRITICAL:** RocksDB is a high-performance storage engine where every CPU cycle and memory access matters. When writing code, always evaluate from a performance perspective. This is not optional—performance-aware coding is a fundamental requirement for all contributions.
**Benchmarking and Profiling:** Performance claims should be backed by empirical evidence. Use RocksDB's benchmarking tools (e.g., `db_bench`) to validate improvements. Reviewers will request benchmark results for changes that could impact performance.
**Memory Allocation:** Minimize dynamic memory allocations, especially in hot paths. Prefer stack allocation over heap allocation. Reuse buffers when possible. Consider using arena allocators or memory pools for frequent small allocations. Every `new`, `malloc`, or container resize has a cost.
**Memory Copy:** Avoid unnecessary memory copies. Use move semantics, `std::string_view`, `Slice`, and pass-by-reference where appropriate. Be aware of implicit copies in STL containers and function returns. Prefer in-place operations over copy-and-modify patterns.
**CPU Cache Efficiency:** Design data structures and access patterns to be cache-friendly. Keep frequently accessed data together (data locality). Prefer sequential memory access over random access. Be mindful of cache line sizes (typically 64 bytes) and avoid false sharing in concurrent code. Consider struct packing and field ordering to improve cache utilization.
**Loop Optimization:** Look for opportunities to collapse nested loops, reduce loop overhead, and minimize branch mispredictions. Hoist invariant computations out of loops. Consider loop unrolling for tight inner loops. Batch operations when possible to amortize per-operation overhead.
**SIMD and Vectorization:** Leverage SIMD instructions (SSE, AVX) for data-parallel operations when appropriate. Structure data to enable auto-vectorization by the compiler. Consider explicit SIMD intrinsics for critical hot paths like checksum computation, encoding/decoding, and bulk data processing.
**Branch Prediction:** Minimize unpredictable branches in hot paths. Use `LIKELY`/`UNLIKELY` macros to hint branch prediction, e.g. for error cases and other rare or otherwise costly cases, but NOT for predicting popular configurations. Consider branchless alternatives for simple conditionals. Order switch cases and if-else chains by frequency.
**Memory and Resource Management:** Be mindful of memory allocations, especially in hot paths. Use RAII patterns, smart pointers, and RocksDB's memory management utilities appropriately.
**Hot Path Analysis:** When deciding how aggressively to optimize code, consider whether it's on a hot path:
- **Hot path** (executed thousands+ times, e.g., data access, iteration, compaction loops): Performance is paramount. Apply all optimization techniques—loop collapsing, SIMD, cache optimization, pre-allocation, etc. The cost of each operation is multiplied by execution frequency.
- **Cold path** (executed rarely, e.g., DB open, configuration parsing, error handling): Maintainability and clarity are more important. Prefer readable code over micro-optimizations. Complex optimizations here add maintenance burden with negligible performance benefit.
- **Warm path** (moderate frequency): Balance both concerns. Use profiling data to guide optimization decisions.
**Avoid Premature Optimization:** While performance is critical, focus on correctness first, then optimize based on profiling data. However, be performance-aware from the start—choosing the right algorithm and data structure upfront is not premature optimization. Use the hot path analysis above to decide how much optimization effort is warranted.
### API Design and Compatibility
**Backwards Compatibility:** RocksDB maintains strong backwards compatibility guarantees. Breaking changes are rare and require extensive justification. When deprecating features, follow the project's deprecation policy (typically spanning multiple releases).
**API Consistency:** New APIs should be consistent with existing patterns. Use similar naming conventions, parameter ordering, and return types. Reviewers will suggest changes to improve consistency with the broader codebase.
**Documentation:** Public APIs must be thoroughly documented. Include usage examples, parameter descriptions, and notes on thread safety, performance characteristics, and compatibility considerations.
---
## Component-Specific Guidance
### Database Core (`db`)
The database core handles write-ahead logging (WAL), memtables, compaction, and recovery. This component receives the most scrutiny in code reviews.
**Concurrency and Thread Safety:** Database operations are highly concurrent. Reviewers carefully examine locking strategies, atomic operations, and memory ordering. Document synchronization assumptions clearly. Use appropriate memory ordering semantics (`acquire`/`release` vs. `seq_cst`).
**Compaction Logic:** Changes to compaction are complex and high-risk. Ensure that compaction logic respects configured parameters, handles edge cases (empty databases, single-file compactions), and maintains correctness under concurrent operations.
**Error Propagation:** Database operations can fail in many ways (I/O errors, corruption, resource exhaustion). Ensure that errors are properly propagated, logged, and handled. Avoid assertions in production code paths.
**Testing:** Database core changes require extensive testing, including unit tests, integration tests, and stress tests. Test with various configurations, compaction styles, and concurrent workloads.
### Public Headers (`include`)
Public headers define RocksDB's API surface. Changes here have the highest compatibility impact.
**API Design:** New APIs should be intuitive, consistent with existing patterns, and well-documented. Consider how the API will be used in practice and avoid adding unnecessary complexity.
**Backwards Compatibility:** Breaking changes to public APIs require extensive justification and a deprecation plan. Maintain ABI compatibility for bug fixes and patch releases.
**Documentation:** Every public API must be thoroughly documented with usage examples, parameter descriptions, and notes on thread safety and performance characteristics.
**Deprecation:** When deprecating APIs, follow the project's policy. Mark deprecated APIs clearly, provide migration guidance, and maintain support for at least one major release.
### Internal Utilities (`util`)
Internal utilities provide common functionality used throughout the codebase.
**Code Reuse:** Utilities should be general-purpose and reusable. Avoid duplicating functionality that already exists elsewhere in the codebase.
**Error Handling:** Utility functions should handle errors robustly and propagate them appropriately. Consider edge cases like overflow, underflow, and invalid inputs.
**Testing:** Utility functions should have comprehensive test coverage, including edge cases and failure modes. Consider adding death tests for assertions.
**Performance:** Utilities are often used in hot paths. Ensure that implementations are efficient and avoid unnecessary allocations or copies.
### Table Management (`table`)
Table management handles SST file format, block-based tables, and table readers/writers.
**Block Format and Checksums:** Changes to block format require extreme care. Ensure that checksums are computed and verified correctly. Test with various compression algorithms and block sizes.
**Iterator Correctness:** Table iterators are used throughout the codebase. Ensure that iterator semantics (Seek, Next, Prev) are correct, especially at boundaries and with deletions.
**Caching and Prefetching:** Table readers interact with the block cache and prefetching logic. Ensure that cache keys are unique and that prefetching respects configured limits.
**Performance:** Table operations are performance-critical. Benchmark changes that could impact read or write performance.
### Utilities (`utilities`)
Utilities include optional features like transactions, backup engine, and checkpoint.
**Feature Isolation:** Utilities should be self-contained and not introduce unnecessary dependencies on core database internals.
**Deprecation and Cleanup:** Legacy features are being phased out. When removing deprecated code, ensure that migration paths are documented and that users have sufficient warning.
**Cross-Platform Compatibility:** Utilities often interact with OS-specific APIs. Ensure that code works on all supported platforms.
### Options and Configuration (`options`)
Options define RocksDB's configuration system.
**Type Safety:** Use appropriate types for options (e.g., `uint32_t` for flags, scoped enums for enumerated values).
**Deprecation Policy:** When deprecating options, follow the project's policy. Document the deprecation, provide migration guidance, and maintain support for at least one major release.
**Dynamic Configuration:** Some options can be changed dynamically. Ensure that dynamic changes are thread-safe and take effect correctly.
**Validation:** Validate option values and provide clear error messages for invalid configurations.
### Cache (`cache`)
Cache management is critical for RocksDB's performance.
**Concurrency:** Cache operations are highly concurrent. Ensure that implementations are thread-safe and use appropriate synchronization primitives.
**Performance:** Cache operations are in the hot path. Optimize for low latency and high throughput. Benchmark changes carefully.
**Memory Management:** Cache implementations must manage memory carefully to avoid leaks and excessive allocations.
**Eviction Policies:** Changes to eviction policies should be well-tested and benchmarked to ensure they improve overall performance.
---
## Code Review Checklist
When reviewing RocksDB code (or preparing code for review), use this checklist:
### Contract Boundaries
- [ ] Is each behavior owned by the right layer? High-level policy (for example,
"compaction wants this I/O mode") should live at the caller/policy layer, while
lower layers should expose generic mechanisms (for example, "open a fresh
reader", "skip shared cache insertion", or "use these FileOptions").
- [ ] Do comments and names describe local contracts rather than leaking a
specific caller's rationale into reusable APIs? Generic code should not need to
know about one current use case unless the API itself is intentionally
use-case-specific.
- [ ] Does each flag or parameter control one coherent behavior? If one boolean
starts implying ownership, cache policy, I/O mode, prefetching, and caller
identity, split it into explicit flags or an options struct.
- [ ] Could a future caller use this lower-level API without accidentally
inheriting assumptions from compaction, backup, user reads, or a particular
table format? If not, tighten the contract with assertions, clearer names, or
a narrower API.
- [ ] Are implementation details not being used as policy signals? Prefer an
explicit contract over inferring behavior from incidental fields such as file
options, cache handles, or current table-reader state.
### Correctness
- [ ] Does the change preserve database semantics (e.g., snapshot isolation, key ordering)?
- [ ] Are all error cases handled appropriately?
- [ ] Is the change thread-safe? Are synchronization primitives used correctly?
- [ ] Are there any potential data races or deadlocks?
### Testing
- [ ] Does the change include appropriate test coverage?
- [ ] Are edge cases and failure modes tested?
- [ ] Have the tests been run on all supported platforms?
- [ ] Are stress tests passing?
### Performance
- [ ] Are there benchmark results for performance-sensitive changes?
- [ ] Does the change avoid unnecessary allocations or copies?
- [ ] Are hot paths optimized appropriately?
### API and Compatibility
- [ ] Is the change backwards compatible?
- [ ] Are new APIs consistent with existing patterns?
- [ ] Is the public API documented?
- [ ] Are deprecated features handled according to policy?
### Code Quality
- [ ] Does the code follow RocksDB's style conventions?
- [ ] Is the code clear and maintainable?
- [ ] Are comments and documentation sufficient?
- [ ] Are there any code smells or anti-patterns?
---
## Common Review Feedback Patterns
The following patterns emerged as frequent sources of review feedback:
1. **Test Coverage:** Reviewers frequently request additional tests for edge cases, platform-specific behavior, and failure modes. Complex changes require comprehensive test coverage including unit tests, integration tests, and stress tests.
2. **Error Handling:** Ensure proper error propagation using RocksDB's `Status` type. Avoid silent failures and provide clear error messages that include context about what failed and why.
3. **API Design:** New APIs should be consistent with existing patterns. Use descriptive names that follow established conventions. Avoid breaking changes without strong justification and a clear deprecation plan.
4. **Documentation:** Public APIs must be documented with usage examples and notes on thread safety, performance characteristics, and compatibility considerations. Complex internal logic should also be well-commented.
5. **Performance:** Performance-sensitive changes require benchmark results to validate improvements. Use `db_bench` and other profiling tools to measure impact. Avoid premature optimization that adds complexity without measurable benefit.
6. **Concurrency:** Thread safety is critical in RocksDB. Document synchronization assumptions clearly. Use appropriate memory ordering semantics. Consider potential race conditions and deadlocks.
7. **Code Style:** Follow existing conventions for naming, formatting, and structure. Use `.clang-format` for consistent formatting. Prefer scoped enums (`enum class`) over unscoped enums.
8. **Backwards Compatibility:** RocksDB maintains strong compatibility guarantees. Breaking changes require extensive justification. When deprecating features, provide migration guidance and maintain support across multiple releases.
9. **Refactoring:** Reviewers appreciate refactoring that improves code readability and maintainability. Look for opportunities to deduplicate code and simplify complex logic.
10. **Platform Compatibility:** Ensure changes work correctly on all supported platforms (Linux, Windows, macOS) and with all supported compilers (GCC, Clang, MSVC).
11. **Contract Boundary Leaks:** When a change plumbs a new option or use-case
specific behavior through multiple subsystems, review the call chain for
contract leaks. Caller-specific rationale belongs at the call site or public API
documentation; reusable layers should expose precise, layer-local capabilities.
Watch especially for comments mentioning one caller in generic code, booleans
that silently bundle several behaviors, and downstream code inferring policy
from an implementation detail instead of an explicit option.
---
## Important tips
### Build system
* There are 3 build system. Make for git clones, BUCK (meta internal) for hg
clones, and CMake for some special cases.
* When a new .cc file is added, update Makefile, CMakeLists.txt, src.mk, BUCK.
* Don't manually edit BUCK file, after updating src.mk, run
/usr/local/bin/python3 buckifier/buckify_rocksdb.py to update it
* For -j in make command, use the number of CPU cores to decide it.
### Avoiding mixed build modes with Make (use `AUTO_CLEAN=1`)
Object files are written to the same paths regardless of build flags, so
reusing objects from a prior build with different flags causes confusing
linker errors, etc. This problem is essentially avoidable by ALWAYS using
`AUTO_CLEAN=1 make -j<n> <something>` for manual make invocations. This
will automatically clean object files if the build parameters/flavor have
changed. The `build_tools/rockstest.sh` / `rocksptest.sh` helpers described
below set `AUTO_CLEAN=1` for you.
### Source checks
* Run `make check-sources` before committing. This catches non-ASCII
characters in source files and other source-level issues that CI will
reject. In particular, **do not use Unicode characters** (em dashes,
smart quotes, etc.) in comments or strings -- use ASCII equivalents
(`--` instead of em dash, `'` instead of smart quote, etc.).
### License headers
* Every new source file needs a license header. For a file that does **not**
carry an outside/third-party copyright, use the standard Meta dual-licensed
header (the dual-license designation is required -- a bare
"All Rights Reserved" copyright is not an acceptable open-source header):
```
// Copyright (c) Meta Platforms, Inc. and affiliates.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
```
Use a `#` comment prefix instead of `//` for shell, Python, and Makefile
fragments.
* Files derived from an external source (e.g. LevelDB) keep their original
upstream copyright line in addition to the header above.
### RTTI and dynamic_cast
* Production code and `db_stress` must build in **release mode
(`-fno-rtti`)**. Do not use `dynamic_cast` anywhere except unit tests.
Use `static_cast_with_check` from `util/cast_util.h` (validates with
`dynamic_cast` in debug builds, plain `static_cast` in release).
* Unit tests (`*_test.cc`) are built in debug mode with RTTI enabled.
### Cross-platform / portability
Local `make` only exercises Linux with GCC/Clang, but CI
(`.github/workflows/pr-jobs.yml` and `nightly.yml`) gates on a much wider
matrix, so portability breaks are invisible locally until CI fails. Code must
build (and where noted, run tests) across:
| Axis | Must support |
|------|--------------|
| OS | Linux (x86_64 + ARM), macOS, Windows |
| Compiler | GCC, Clang (libstdc++ **and** libc++), AppleClang, **MSVC (VS2022)**, MinGW (Linux cross-compile, build-only, no gflags) |
| Build system | Make, CMake, and BUCK (internal) -- keep all in sync (see "Build system" above) |
| Config | release (`-fno-rtti`), `ASSERT_STATUS_CHECKED`, ASAN/UBSAN/TSAN, folly, unity build, JNI/Java |
Treat these as constraints to satisfy and infer the specifics from them before
adding any system header, libc call, or compiler-specific construct. The most
common trap: anything that compiles under GCC/Clang on Linux but not under
**MSVC/MinGW** -- e.g. unguarded POSIX-only headers/functions (`<unistd.h>`,
`<sys/*.h>`, `getpid`, `_exit`, ...) or GCC/Clang extensions
(`__attribute__`, `__builtin_*`, VLAs, `alloca`). Prefer the `port::`/`Env`
abstractions; otherwise guard with `#ifdef OS_WIN` (POSIX `<unistd.h>` ->
Windows `<process.h>`). Because libc++ is also tested, include what you use
rather than relying on libstdc++ transitive includes.
### Unit Test
* After all of the unit tests are added, review them and try to extract common
reusable utility functions to reduce code duplication due to copy past between
unit tests. This should be done every time unit test is updated.
* Don't use sleep to wait for certain events to happen. This will cause test to
be flaky. Instead, use sync point to synchronize thread progress.
* Cap unit test execution with 60 seconds timeout.
* To build and run unit tests locally, prefer these helper scripts:
* `build_tools/rocksptest.sh <test_binary> [more_binaries...] [args...]`
builds the binary(ies) with parallel make and `AUTO_CLEAN=1` and runs
them under gtest-parallel, sharding the test cases across CPUs. Prefer
this whenever running more than a couple of test cases, e.g.
`build_tools/rocksptest.sh table_test` or
`build_tools/rocksptest.sh db_test env_test --gtest_filter=*Foo*`.
* `build_tools/rockstest.sh <test_binary> [args...]` builds with parallel
make and `AUTO_CLEAN=1` and runs the binary directly (serially).
Use it only for a very small number of test cases, e.g.
`build_tools/rockstest.sh db_test --gtest_filter=*MixedSlowdown*`.
* After writing a test, stress-test for flakiness (AUTO_CLEAN handles the
rebuild needed by the `COERCE_CONTEXT_SWITCH=1` flag change):
```bash
COERCE_CONTEXT_SWITCH=1 build_tools/rockstest.sh {test_binary} -r100 \
--gtest_filter="*YourTestName*"
```
* For CI-style flaky tests that do not reproduce with `gtest_parallel.py`,
`--gtest_repeat`, or normal coerce-mode runs, inspect
`tools/gtest_parallel_repro.py --help`.
### Unit test dedup guidelines
* Extract helper functions for repeated patterns such as object
construction, round-trip (encode → decode → verify), and common
assertion sequences.
* Use table-driven tests (struct array + loop) when multiple test cases
share the same logic but differ only in input/expected data.
* Prefer randomized tests over exhaustive parameter permutations. Use
`Random` from `util/random.h` (not `std::mt19937`). Use a time-based
seed with `SCOPED_TRACE("seed=" + std::to_string(seed))` so failures
are reproducible.
* Keep deterministic edge-case tests separate from randomized tests
(error paths, boundary conditions, format verification).
* Methods only used in tests should be private with `friend class` +
`TEST_F` fixture wrappers. In wrappers, always fully qualify the
target method to avoid infinite recursion.
### Adding new public API
Refer to claude_md/add_public_api.md
### Adding new option
Refer to claude_md/add_option.md
### Removing deprecated option
Refer to claude_md/remove_option.md
### Metrics
* When adding a new feature, evaluate whether there is opportunity to add
metrics. Try to avoid causing performance regression on hot path when adding
metrics.
### Stress test
* When adding a new feature, make sure stress test covers the new option.
### Component docs
* For component-level design notes and implementation walkthroughs, start with
`docs/components/index.md`.
* Documentation under `docs/components/` is organized by subsystem in
`docs/components/<area>/`.
* Each subsystem directory should have an `index.md` entry point plus focused
chapter files for deeper topics.
### DB bench update
* When adding a performance related feature, support it in db_bench
### Adding release note
* Release note should be kept short at high level for external user consumption.
### Blog posts (docs/_posts)
* Blog post authors must be defined in `docs/_data/authors.yml` to be displayed
### Final verification of the change
* Execute `AUTO_CLEAN=1 make check` to build all of the changes and execute all
of the tests. `AUTO_CLEAN=1` ensures a clean rebuild if your previous build
used different parameters. Note that executing all of the tests could take
multiple minutes.
* Run `AUTO_CLEAN=1 ASSERT_STATUS_CHECKED=1 make check` to verify all Status
objects are properly checked. This catches missing error handling that can
lead to silent data corruption.
### Monitoring make check progress
* Use `make check-progress` to get machine-parseable JSON progress while
`make check` is running. This is useful for Claude Code to monitor long
builds without timeout issues.
* Run `make check` in background, then poll progress:
```bash
AUTO_CLEAN=1 make check &
# Poll periodically:
make check-progress
```
* The output shows current phase and progress:
```json
{"status":"running","phase":"compiling","completed":300,"total":919,...}
{"status":"running","phase":"testing","completed":1500,"total":29962,"failed":0,"percent":5,...}
{"status":"completed","phase":"testing","completed":29962,"total":29962,"failed":0,"percent":100,...}
```
* Phases: `compiling` -> `linking` -> `generating` -> `testing` -> `completed`
* Key fields: `status`, `phase`, `completed`, `total`, `failed`, `percent`
* When tests fail, `failed_tests` array shows details (up to 10 failures):
```json
{"status":"running",...,"failed":3,"failed_tests":[
{"test":"cache_test-CacheTest.Usage","exit_code":1,"signal":0,"output":"...test log..."},
{"test":"env_test-EnvTest.Open","exit_code":0,"signal":11,"output":"...Segmentation fault..."}
]}
```
* `exit_code`: non-zero means test assertion failed
* `signal`: non-zero means test was killed (e.g., 9=SIGKILL, 6=SIGABRT, 11=SIGSEGV)
* `output`: last 50 lines of test log including error messages and stack traces
### Executing benchmark using db_bench
* Since the goal is to measure performance, we need to build a release binary
using `AUTO_CLEAN=1 DEBUG_LEVEL=0 make db_bench`. If there is an engine
crash due to a bug, switch back to a debug build with
`AUTO_CLEAN=1 make dbg`; `AUTO_CLEAN=1` handles the release<->debug rebuild
automatically.
### Formatting code
* After making change, use `make format-auto` to auto-apply formatting without
interactive prompts (Claude Code friendly).
+259 -1616
View File
File diff suppressed because it is too large Load Diff
-77
View File
@@ -1,77 +0,0 @@
# Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to make participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies within all project spaces, and it also applies when
an individual is representing the project or its community in public spaces.
Examples of representing a project or community include using an official
project e-mail address, posting via an official social media account, or acting
as an appointed representative at an online or offline event. Representation of
a project may be further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at <opensource-conduct@fb.com>. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq
+5 -3
View File
@@ -1,8 +1,5 @@
# Contributing to RocksDB
## Code of Conduct
The code of conduct is described in [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)
## Contributor License Agreement ("CLA")
In order to accept your pull request, we need you to submit a CLA. You
@@ -15,3 +12,8 @@ Complete your CLA here: <https://code.facebook.com/cla>
If you prefer to sign a paper copy, we can send you a PDF. Send us an
e-mail or create a new github issue to request the CLA in PDF format.
## License
By contributing to RocksDB, you agree that your contributions will be
licensed under the [BSD License](LICENSE).
-339
View File
@@ -1,339 +0,0 @@
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Ty Coon>, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License.
-24
View File
@@ -1,24 +0,0 @@
# RocksDB default options change log (NO LONGER MAINTAINED)
## Unreleased
* delayed_write_rate takes the rate given by rate_limiter if not specified.
## 5.2
* Change the default of delayed slowdown value to 16MB/s and further increase the L0 stop condition to 36 files.
## 5.0 (11/17/2016)
* Options::allow_concurrent_memtable_write and Options::enable_write_thread_adaptive_yield are now true by default
* Options.level0_stop_writes_trigger default value changes from 24 to 32.
## 4.8.0 (5/2/2016)
* options.max_open_files changes from 5000 to -1. It improves performance, but users need to set file descriptor limit to be large enough and watch memory usage for index and bloom filters.
* options.base_background_compactions changes from max_background_compactions to 1. When users set higher max_background_compactions but the write throughput is not high, the writes are less spiky to disks.
* options.wal_recovery_mode changes from kTolerateCorruptedTailRecords to kPointInTimeRecovery. Avoid some false positive when file system or hardware reorder the writes for file data and metadata.
## 4.7.0 (4/8/2016)
* options.write_buffer_size changes from 4MB to 64MB.
* options.target_file_size_base changes from 2MB to 64MB.
* options.max_bytes_for_level_base changes from 10MB to 256MB.
* options.soft_pending_compaction_bytes_limit changes from 0 (disabled) to 64GB.
* options.hard_pending_compaction_bytes_limit changes from 0 (disabled) to 256GB.
* table_cache_numshardbits changes from 4 to 6.
* max_file_opening_threads changes from 1 to 16.
-9
View File
@@ -1,9 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<Project>
<PropertyGroup>
<CLToolExe>ccache_msvc_compiler.bat</CLToolExe>
<CLToolPath>$(MSBuildThisFileDirectory)</CLToolPath>
<UseMultiToolTask>true</UseMultiToolTask>
<EnforceProcessCountAcrossBuilds>true</EnforceProcessCountAcrossBuilds>
</PropertyGroup>
</Project>
+30 -2991
View File
File diff suppressed because it is too large Load Diff
+27 -160
View File
@@ -1,12 +1,12 @@
## Compilation
**Important**: If you plan to run RocksDB in production, don't compile using default
**Important**: If you plan to run RocksDB in production, don't compile using default
`make` or `make all`. That will compile RocksDB in debug mode, which is much slower
than release mode.
RocksDB's library should be able to compile without any dependency installed,
although we recommend installing some compression libraries (see below).
We do depend on newer gcc/clang with C++20 support (GCC >= 11, Clang >= 10).
We do depend on newer gcc/clang with C++11 support.
There are few options when compiling RocksDB:
@@ -17,50 +17,30 @@ There are few options when compiling RocksDB:
* `make check` will compile and run all the unit tests. `make check` will compile RocksDB in debug mode.
* `make all` will compile our static library, and all our tools and unit tests. Our tools
depend on gflags 2.2.0 or newer. You will need to have gflags installed to run `make all`. This will compile RocksDB in debug mode. Don't
depend on gflags. You will need to have gflags installed to run `make all`. This will compile RocksDB in debug mode. Don't
use binaries compiled by `make all` in production.
* By default the binary we produce is optimized for the CPU you're compiling on
(`-march=native` or the equivalent). To build a binary compatible with the most
general architecture supported by your CPU and compiler, set `PORTABLE=1` for
the build, but performance will suffer as many operations benefit from newer
and wider instructions. In addition to `PORTABLE=0` (default) and `PORTABLE=1`,
it can be set to an architecture name recognized by your compiler. For example,
on 64-bit x86, a reasonable compromise is `PORTABLE=haswell` which supports
many or most of the available optimizations while still being compatible with
most processors made since roughly 2013.
* By default the binary we produce is optimized for the platform you're compiling on
(-march=native). If you want to build a portable binary, add 'PORTABLE=1' before
your make commands, like this: `PORTABLE=1 make static_lib`
## Dependencies
* You can link RocksDB with following compression libraries:
- [zlib](http://www.zlib.net/) - a library for data compression.
- [bzip2](http://www.bzip.org/) - a library for data compression.
- [lz4](https://github.com/lz4/lz4) - a library for extremely fast data compression.
- [snappy](http://google.github.io/snappy/) - a library for fast
- [snappy](https://code.google.com/p/snappy/) - a library for fast
data compression.
- [zstandard](http://www.zstd.net) - Fast real-time compression
algorithm.
* All our tools depend on:
- [gflags](https://gflags.github.io/gflags/) - a library that handles
command line flags processing. You can compile rocksdb library even
if you don't have gflags installed.
* `make check` will also check code formatting, which requires [clang-format](https://clang.llvm.org/docs/ClangFormat.html)
* If you wish to build the RocksJava static target, then cmake is required for building Snappy.
* If you wish to run microbench (e.g, `make microbench`, `make ribbon_bench` or `cmake -DWITH_BENCHMARK=1`), Google benchmark >= 1.6.0 is needed.
* You can do the following to install Google benchmark. These commands are copied from `./build_tools/ubuntu20_image/Dockerfile`:
`$ git clone --depth 1 --branch v1.7.0 https://github.com/google/benchmark.git ~/benchmark`
`$ cd ~/benchmark && mkdir build && cd build && cmake .. -GNinja -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_ENABLE_GTEST_TESTS=0 && ninja && ninja install`
## Supported platforms
* **Linux - Ubuntu**
* Upgrade your gcc to version at least 11 to get C++20 support.
* Upgrade your gcc to version at least 4.7 to get C++11 support.
* Install gflags. First, try: `sudo apt-get install libgflags-dev`
If this doesn't work and you're using Ubuntu, here's a nice tutorial:
(http://askubuntu.com/questions/312173/installing-gflags-12-04)
@@ -68,157 +48,44 @@ most processors made since roughly 2013.
`sudo apt-get install libsnappy-dev`.
* Install zlib. Try: `sudo apt-get install zlib1g-dev`.
* Install bzip2: `sudo apt-get install libbz2-dev`.
* Install lz4: `sudo apt-get install liblz4-dev`.
* Install zstandard: `sudo apt-get install libzstd-dev`.
* **Linux - CentOS / RHEL**
* Upgrade your gcc to version at least 11 to get C++20 support
* **Linux - CentOS**
* Upgrade your gcc to version at least 4.7 to get C++11 support:
`yum install gcc47-c++`
* Install gflags:
git clone https://github.com/gflags/gflags.git
cd gflags
git checkout v2.2.0
wget https://gflags.googlecode.com/files/gflags-2.0-no-svn-files.tar.gz
tar -xzvf gflags-2.0-no-svn-files.tar.gz
cd gflags-2.0
./configure && make && sudo make install
**Notice**: Once installed, please add the include path for gflags to your `CPATH` environment variable and the
lib path to `LIBRARY_PATH`. If installed with default settings, the include path will be `/usr/local/include`
and the lib path will be `/usr/local/lib`.
* Install snappy:
sudo yum install snappy snappy-devel
wget https://snappy.googlecode.com/files/snappy-1.1.1.tar.gz
tar -xzvf snappy-1.1.1.tar.gz
cd snappy-1.1.1
./configure && make && sudo make install
* Install zlib:
sudo yum install zlib zlib-devel
sudo yum install zlib
sudo yum install zlib-devel
* Install bzip2:
sudo yum install bzip2 bzip2-devel
* Install lz4:
sudo yum install lz4-devel
* Install ASAN (optional for debugging):
sudo yum install libasan
* Install zstandard:
* With [EPEL](https://fedoraproject.org/wiki/EPEL):
sudo yum install libzstd-devel
* With CentOS 8:
sudo dnf install libzstd-devel
* From source:
wget https://github.com/facebook/zstd/archive/v1.1.3.tar.gz
mv v1.1.3.tar.gz zstd-1.1.3.tar.gz
tar zxvf zstd-1.1.3.tar.gz
cd zstd-1.1.3
make && sudo make install
sudo yum install bzip2
sudo yum install bzip2-devel
* **OS X**:
* Install latest C++ compiler that supports C++20:
* Install latest C++ compiler that supports C++ 11:
* Update XCode: run `xcode-select --install` (or install it from XCode App's settting).
* Install via [homebrew](http://brew.sh/).
* If you're first time developer in MacOS, you still need to run: `xcode-select --install` in your command line.
* run `brew tap homebrew/versions; brew install gcc47 --use-llvm` to install gcc 4.7 (or higher).
* run `brew install rocksdb`
* **FreeBSD** (11.01):
* You can either install RocksDB from the Ports system using `cd /usr/ports/databases/rocksdb && make install`, or you can follow the details below to install dependencies and compile from source code:
* Install the dependencies for RocksDB:
export BATCH=YES
cd /usr/ports/devel/gmake && make install
cd /usr/ports/devel/gflags && make install
cd /usr/ports/archivers/snappy && make install
cd /usr/ports/archivers/bzip2 && make install
cd /usr/ports/archivers/liblz4 && make install
cd /usr/ports/archivesrs/zstd && make install
cd /usr/ports/devel/git && make install
* Install the dependencies for RocksJava (optional):
export BATCH=yes
cd /usr/ports/java/openjdk7 && make install
* Build RocksDB from source:
cd ~
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
gmake static_lib
* Build RocksJava from source (optional):
cd rocksdb
export JAVA_HOME=/usr/local/openjdk7
gmake rocksdbjava
* **OpenBSD** (6.3/-current):
* As RocksDB is not available in the ports yet you have to build it on your own:
* Install the dependencies for RocksDB:
`pkg_add gmake gflags snappy bzip2 lz4 zstd git bash findutils gnuwatch`
* Build RocksDB from source:
```bash
cd ~
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
gmake static_lib
```
* Build RocksJava from source (optional):
* In OpenBSD, JDK depends on XWindows system, so please check that you installed OpenBSD with `xbase` package.
* Install dependencies : `pkg_add -v jdk%1.8`
```bash
cd rocksdb
export JAVA_HOME=/usr/local/jdk-1.8.0
export PATH=$PATH:/usr/local/jdk-1.8.0/bin
gmake rocksdbjava SHA256_CMD='sha256 -q'
```
* **iOS**:
* Run: `TARGET_OS=IOS make static_lib`. When building the project which uses rocksdb iOS library, make sure to define an important pre-processing macros: `IOS_CROSS_COMPILE`.
* Run: `TARGET_OS=IOS make static_lib`. When building the project which uses rocksdb iOS library, make sure to define two important pre-processing macros: `ROCKSDB_LITE` and `IOS_CROSS_COMPILE`.
* **Windows** (Visual Studio 2017 to up):
* **Windows**:
* For building with MS Visual Studio 13 you will need Update 4 installed.
* Read and follow the instructions at CMakeLists.txt
* Or install via [vcpkg](https://github.com/microsoft/vcpkg)
* run `vcpkg install rocksdb:x64-windows`
* **AIX 6.1**
* Install AIX Toolbox rpms with gcc
* Use these environment variables:
export PORTABLE=1
export CC=gcc
export AR="ar -X64"
export EXTRA_ARFLAGS=-X64
export EXTRA_CFLAGS=-maix64
export EXTRA_CXXFLAGS=-maix64
export PLATFORM_LDFLAGS="-static-libstdc++ -static-libgcc"
export LIBPATH=/opt/freeware/lib
export JAVA_HOME=/usr/java8_64
export PATH=/opt/freeware/bin:$PATH
* **Solaris Sparc**
* Install GCC 11 and higher.
* Use these environment variables:
export CC=gcc
export EXTRA_CFLAGS=-m64
export EXTRA_CXXFLAGS=-m64
export EXTRA_LDFLAGS=-m64
export PORTABLE=1
export PLATFORM_LDFLAGS="-static-libstdc++ -static-libgcc"
-27
View File
@@ -1,27 +0,0 @@
This is the list of all known third-party language bindings for RocksDB. If something is missing, please open a pull request to add it.
* Java - https://github.com/facebook/rocksdb/tree/main/java
* Python
* https://github.com/rocksdict/RocksDict
* http://python-rocksdb.readthedocs.io/en/latest/ (unmaintained)
* http://pyrocksdb.readthedocs.org/en/latest/ (unmaintained)
* Perl - https://metacpan.org/pod/RocksDB
* Node.js - https://npmjs.org/package/rocksdb
* Go
* https://github.com/linxGnu/grocksdb
* https://github.com/tecbot/gorocksdb (unmaintained)
* Ruby - http://rubygems.org/gems/rocksdb-ruby
* Haskell - https://hackage.haskell.org/package/rocksdb-haskell
* PHP - https://github.com/Photonios/rocksdb-php
* C#
* https://github.com/warrenfalk/rocksdb-sharp
* https://github.com/curiosity-ai/rocksdb-sharp
* Rust
* https://github.com/pingcap/rust-rocksdb (used in production fork of https://github.com/spacejam/rust-rocksdb)
* https://github.com/spacejam/rust-rocksdb
* https://github.com/bh1xuw/rust-rocks
* D programming language - https://github.com/b1naryth1ef/rocksdb
* Erlang - https://gitlab.com/barrel-db/erlang-rocksdb
* Elixir - https://github.com/urbint/rox
* Nim - https://github.com/status-im/nim-rocksdb
* Swift and Objective-C (iOS/OSX) - https://github.com/iabudiab/ObjectiveRocks
+7 -1
View File
@@ -1,4 +1,10 @@
This contains code that is from LevelDB, and that code is under the following license:
BSD License
For rocksdb software
Copyright (c) 2014, Facebook, Inc.
All rights reserved.
---------------------------------------------------------------------
Copyright (c) 2011 The LevelDB Authors. All rights reserved.
-202
View File
@@ -1,202 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+646 -2340
View File
File diff suppressed because it is too large Load Diff
+33
View File
@@ -0,0 +1,33 @@
Additional Grant of Patent Rights Version 2
"Software" means the RocksDB software distributed by Facebook, Inc.
Facebook, Inc. ("Facebook") hereby grants to each recipient of the Software
("you") a perpetual, worldwide, royalty-free, non-exclusive, irrevocable
(subject to the termination provision below) license under any Necessary
Claims, to make, have made, use, sell, offer to sell, import, and otherwise
transfer the Software. For avoidance of doubt, no license is granted under
Facebooks rights in any patent claims that are infringed by (i) modifications
to the Software made by you or any third party or (ii) the Software in
combination with any software or other technology.
The license granted hereunder will terminate, automatically and without notice,
if you (or any of your subsidiaries, corporate affiliates or agents) initiate
directly or indirectly, or take a direct financial interest in, any Patent
Assertion: (i) against Facebook or any of its subsidiaries or corporate
affiliates, (ii) against any party if such Patent Assertion arises in whole or
in part from any software, technology, product or service of Facebook or any of
its subsidiaries or corporate affiliates, or (iii) against any party relating
to the Software. Notwithstanding the foregoing, if Facebook or any of its
subsidiaries or corporate affiliates files a lawsuit alleging patent
infringement against you in the first instance, and you respond by filing a
patent infringement counterclaim in that lawsuit against that party that is
unrelated to the Software, the license granted hereunder will not terminate
under section (i) of this paragraph due to such counterclaim.
A "Necessary Claim" is a claim of a patent owned by Facebook that is
necessarily infringed by the Software standing alone.
A "Patent Assertion" is any lawsuit or other action alleging direct, indirect,
or contributory infringement or inducement to infringe any patent, including a
cross-claim or counterclaim.
-9
View File
@@ -1,9 +0,0 @@
This is the list of all known third-party plugins for RocksDB. If something is missing, please open a pull request to add it.
* [Dedupfs](https://github.com/ajkr/dedupfs): an example for plugin developers to reference
* [HDFS](https://github.com/riversand963/rocksdb-hdfs-env): an Env used for interacting with HDFS. Migrated from main RocksDB repo
* [ZenFS](https://github.com/westerndigitalcorporation/zenfs): a file system for zoned block devices
* [RADOS](https://github.com/riversand963/rocksdb-rados-env): an Env used for interacting with RADOS. Migrated from RocksDB main repo.
* [PMEM](https://github.com/pmem/pmem-rocksdb-plugin): a collection of plugins to enable Persistent Memory on RocksDB.
* [IPPCP](https://github.com/intel/ippcp-plugin-rocksdb): a plugin to enable encryption on RocksDB based on Intel optimized open source IPP-Crypto library.
* [encfs](https://github.com/pegasus-kv/encfs): a plugin to enable encryption on RocksDB based on OpenSSL library.
+6 -10
View File
@@ -1,20 +1,20 @@
## RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
[![CircleCI Status](https://circleci.com/gh/facebook/rocksdb.svg?style=svg)](https://circleci.com/gh/facebook/rocksdb)
[![Build Status](https://travis-ci.org/facebook/rocksdb.svg?branch=master)](https://travis-ci.org/facebook/rocksdb)
RocksDB is developed and maintained by Facebook Database Engineering Team.
It is built on earlier work on [LevelDB](https://github.com/google/leveldb) by Sanjay Ghemawat (sanjay@google.com)
It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com)
and Jeff Dean (jeff@google.com)
This code is a library that forms the core building block for a fast
key-value server, especially suited for storing data on flash drives.
key value server, especially suited for storing data on flash drives.
It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs
between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF)
and Space-Amplification-Factor (SAF). It has multi-threaded compactions,
making it especially suitable for storing multiple terabytes of data in a
making it specially suitable for storing multiple terabytes of data in a
single database.
Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples
Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples
See the [github wiki](https://github.com/facebook/rocksdb/wiki) for more explanation.
@@ -22,8 +22,4 @@ The public interface is in `include/`. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Questions and discussions are welcome on the [RocksDB Developers Public](https://www.facebook.com/groups/rocksdb.dev/) Facebook group and [email list](https://groups.google.com/g/rocksdb) on Google Groups.
## License
RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.
Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/
+21
View File
@@ -0,0 +1,21 @@
# RocksDBLite
RocksDBLite is a project focused on mobile use cases, which don't need a lot of fancy things we've built for server workloads and they are very sensitive to binary size. For that reason, we added a compile flag ROCKSDB_LITE that comments out a lot of the nonessential code and keeps the binary lean.
Some examples of the features disabled by ROCKSDB_LITE:
* compiled-in support for LDB tool
* No backupable DB
* No support for replication (which we provide in form of TrasactionalIterator)
* No advanced monitoring tools
* No special-purpose memtables that are highly optimized for specific use cases
* No Transactions
When adding a new big feature to RocksDB, please add ROCKSDB_LITE compile guard if:
* Nobody from mobile really needs your feature,
* Your feature is adding a lot of weight to the binary.
Don't add ROCKSDB_LITE compile guard if:
* It would introduce a lot of code complexity. Compile guards make code harder to read. It's a trade-off.
* Your feature is not adding a lot of weight.
If unsure, ask. :)
+9 -132
View File
@@ -1,60 +1,24 @@
This document lists users of RocksDB and their use cases. If you are using RocksDB, please open a pull request and add yourself to the list.
## Facebook
At Facebook, we use RocksDB as storage engines in multiple data management services and a backend for many different stateful services, including:
At Facebook, we use RocksDB as a backend for many different stateful services. We're also experimenting with running RocksDB as a storage engine for two databases:
1. MyRocks -- https://github.com/MySQLOnRocksDB/mysql-5.6
2. MongoRocks -- https://github.com/mongodb-partners/mongo-rocks
3. ZippyDB -- Facebook's distributed key-value store with Paxos-style replication, built on top of RocksDB.[1] https://www.youtube.com/watch?v=DfiN7pG0D0khtt
4. Laser -- Laser is a high query throughput, low (millisecond) latency, key-value storage service built on top of RocksDB.[1]
4. Dragon -- a distributed graph query engine. https://code.facebook.com/posts/1737605303120405/dragon-a-distributed-graph-query-engine/
5. Stylus -- a low-level stream processing framework writtenin C++.[1]
6. LogDevice -- a distributed data store for logs [2]
[1] https://research.facebook.com/publications/realtime-data-processing-at-facebook/
[2] https://code.facebook.com/posts/357056558062811/logdevice-a-distributed-data-store-for-logs/
## Bilibili
[Bilibili](bilibili.com) [uses](https://www.alluxio.io/blog/when-ai-meets-alluxio-at-bilibili-building-an-efficient-ai-platform-for-data-preprocessing-and-model-training/) Alluxio to speed up its ML training workloads, and Alluxio uses RocksDB to store its filesystem metadata, so Bilibili uses RocksDB.
Bilibili's [real-time platform](https://www.alibabacloud.com/blog/architecture-and-practices-of-bilibilis-real-time-platform_596676) uses Flink, and uses RocksDB as Flink's state store.
## TikTok
TikTok, or its parent company ByteDance, uses RocksDB as the storage engine for some storage systems, such as its distributed graph database [ByteGraph](https://vldb.org/pvldb/vol15/p3306-li.pdf).
Also, TikTok uses [Alluxio](alluxio.io) to [speed up Presto queries](https://www.alluxio.io/resources/videos/improving-presto-performance-with-alluxio-at-tiktok/), and Alluxio stores the files' metadata in RocksDB.
## FoundationDB
[FoundationDB](https://www.foundationdb.org/) [uses](https://github.com/apple/foundationdb/blob/377f1f692da6ab2fe5bdac57035651db3e5fb66d/fdbserver/KeyValueStoreRocksDB.actor.cpp) RocksDB to implement a [key-value store interface](https://github.com/apple/foundationdb/blob/377f1f692da6ab2fe5bdac57035651db3e5fb66d/fdbserver/KeyValueStoreRocksDB.actor.cpp#L1127) in its server backend.
## Apple
Apple [uses](https://opensource.apple.com/projects/foundationdb/) FoundationDB, so it also uses RocksDB.
## Snowflake
Snowflake [uses](https://www.snowflake.com/blog/how-foundationdb-powers-snowflake-metadata-forward/) FoundationDB, so it also uses RocksDB.
## Microsoft
The Bing search engine from Microsoft uses RocksDB as the storage engine for its web data platform: https://blogs.bing.com/Engineering-Blog/october-2021/RocksDB-in-Microsoft-Bing
## LinkedIn
1. [Venice](https://venicedb.org/) is a derived data platform using RocksDB as its storage engine. It is LinkedIn's ML feature store, powering thousands of recommender use cases, including the Feed, Video recommendations, and People You May Know.
2. LinkedIn's follow feed for storing user's activities. Check out the blog post: https://engineering.linkedin.com/blog/2016/03/followfeed--linkedin-s-feed-made-faster-and-smarter
3. Apache Samza, open source framework for stream processing.
Two different use cases at Linkedin are using RocksDB as a storage engine:
Learn more about LinkedIn's follow feed and Apache Samza in a Tech Talk by Ankit Gupta and Naveen Somasundaram: http://www.youtube.com/watch?v=plqVp_OnSzg
1. LinkedIn's follow feed for storing user's activities
2. Apache Samza, open source framework for stream processing
Learn more about those use cases in a Tech Talk by Ankit Gupta and Naveen Somasundaram: http://www.youtube.com/watch?v=plqVp_OnSzg
## Yahoo
Yahoo is using RocksDB as a storage engine for their biggest distributed data store Sherpa. Learn more about it here: http://yahooeng.tumblr.com/post/120730204806/sherpa-scales-new-heights
## Tencent
[PaxosStore](https://github.com/Tencent/paxosstore) is a distributed database supporting WeChat. It uses RocksDB as its storage engine.
## Baidu
[Apache Doris](http://doris.apache.org/master/en/) is a MPP analytical database engine released by Baidu. It [uses RocksDB](http://doris.apache.org/master/en/administrator-guide/operation/tablet-meta-tool.html) to manage its tablet's metadata.
## CockroachDB
CockroachDB is an open-source geo-replicated transactional database. They are using RocksDB as their storage engine. Check out their github: https://github.com/cockroachdb/cockroach
CockroachDB is an open-source geo-replicated transactional database (still in development). They are using RocksDB as their storage engine. Check out their github: https://github.com/cockroachdb/cockroach
## DNANexus
DNANexus is using RocksDB to speed up processing of genomics data.
@@ -71,101 +35,14 @@ Tango is using RocksDB as a graph storage to store all users' connection data an
Turn is using RocksDB as a storage layer for their key/value store, serving at peak 2.4MM QPS out of different datacenters.
Check out our RocksDB Protobuf merge operator at: https://github.com/vladb38/rocksdb_protobuf
## Santander UK/Cloudera Profession Services
## Santanader UK/Cloudera Profession Services
Check out their blog post: http://blog.cloudera.com/blog/2015/08/inside-santanders-near-real-time-data-ingest-architecture/
## Airbnb
Airbnb is using RocksDB as a storage engine for their personalized search service. You can learn more about it here: https://www.youtube.com/watch?v=ASQ6XMtogMs
## Alluxio
[Alluxio](https://www.alluxio.io) uses RocksDB to serve and scale file system metadata to beyond 1 Billion files. The detailed design and implementation is described in this engineering blog:
https://www.alluxio.io/blog/scalable-metadata-service-in-alluxio-storing-billions-of-files/
## Pinterest
Pinterest's Object Retrieval System uses RocksDB for storage: https://www.youtube.com/watch?v=MtFEVEs_2Vo
## Smyte
[Smyte](https://www.smyte.com/) uses RocksDB as the storage layer for their core key-value storage, high-performance counters and time-windowed HyperLogLog services.
## Rakuten Marketing
[Rakuten Marketing](https://marketing.rakuten.com/) uses RocksDB as the disk cache layer for the real-time bidding service in their Performance DSP.
## VWO, Wingify
[VWO's](https://vwo.com/) Smart Code checker and URL helper uses RocksDB to store all the URLs where VWO's Smart Code is installed.
## quasardb
[quasardb](https://www.quasardb.net) is a high-performance, distributed, transactional key-value database that integrates well with in-memory analytics engines such as Apache Spark.
quasardb uses a heavily tuned RocksDB as its persistence layer.
## Netflix
[Netflix](http://techblog.netflix.com/2016/05/application-data-caching-using-ssds.html) Netflix uses RocksDB on AWS EC2 instances with local SSD drives to cache application data.
## TiKV
[TiKV](https://github.com/pingcap/tikv) is a GEO-replicated, high-performance, distributed, transactional key-value database. TiKV is powered by Rust and Raft. TiKV uses RocksDB as its persistence layer.
## TiDB
[TiDB](https://github.com/pingcap/tidb) uses the TiKV distributed key-value database, so it uses RocksDB.
## PingCAP
[PingCAP](https://www.pingcap.com/) is the company behind TiDB, its cloud database service uses RocksDB.
## Apache Spark
[Spark Structured Streaming](https://docs.databricks.com/structured-streaming/rocksdb-state-store.html) uses RocksDB as the local state store.
## Databricks
[Databricks](https://www.databricks.com/) [replaces AWS RDS with TiDB](https://www.pingcap.com/case-study/how-databricks-tackles-the-scalability-limit-with-a-mysql-alternative/) for scalability, so it uses RocksDB.
## Apache Flink
[Apache Flink](https://flink.apache.org/news/2016/03/08/release-1.0.0.html) uses RocksDB to store state locally on a machine.
## Dgraph
[Dgraph](https://github.com/dgraph-io/dgraph) is an open-source, scalable, distributed, low latency, high throughput Graph database .They use RocksDB to store state locally on a machine.
## Uber
[Uber](http://eng.uber.com/cherami/) uses RocksDB as a durable and scalable task queue.
## 360 Pika
[360](http://www.360.cn/) [Pika](https://github.com/Qihoo360/pika) is a nosql compatible with redis. With the huge amount of data stored, redis may suffer for a capacity bottleneck, and pika was born for solving it. It has widely been used in many companies.
## LzLabs
LzLabs is using RocksDB as a storage engine in their multi-database distributed framework to store application configuration and user data.
## ProfaneDB
[ProfaneDB](https://profanedb.gitlab.io/) is a database for Protocol Buffers, and uses RocksDB for storage. It is accessible via gRPC, and the schema is defined using directly `.proto` files.
## IOTA Foundation
[IOTA Foundation](https://www.iota.org/) is using RocksDB in the [IOTA Reference Implementation (IRI)](https://github.com/iotaledger/iri) to store the local state of the Tangle. The Tangle is the first open-source distributed ledger powering the future of the Internet of Things.
## Avrio Project
[Avrio Project](http://avrio-project.github.io/avrio.network/) is using RocksDB in [Avrio ](https://github.com/avrio-project/avrio) to store blocks, account balances and data and other blockchain-releated data. Avrio is a multiblockchain decentralized cryptocurrency empowering monetary transactions.
## Crux
[Crux](https://github.com/juxt/crux) is a document database that uses RocksDB for local [EAV](https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model) index storage to enable point-in-time bitemporal Datalog queries. The "unbundled" architecture uses Kafka to provide horizontal scalability.
## Nebula Graph
[Nebula Graph](https://github.com/vesoft-inc/nebula) is a distributed, scalable, lightning-fast, open source graph database capable of hosting super large scale graphs with dozens of billions of vertices (nodes) and trillions of edges, with milliseconds of latency.
## YugabyteDB
[YugabyteDB](https://www.yugabyte.com/) is an open source, high performance, distributed SQL database that uses RocksDB as its storage layer. For more information, please see https://github.com/yugabyte/yugabyte-db/.
## ArangoDB
[ArangoDB](https://www.arangodb.com/) is a native multi-model database with flexible data models for documents, graphs, and key-values, for building high performance applications using a convenient SQL-like query language or JavaScript extensions. It uses RocksDB as its storage engine.
## Qdrant
[Qdrant](https://qdrant.tech/) is an open source vector database, it [uses](https://qdrant.tech/documentation/concepts/storage/) RocksDB as its persistent storage.
## Milvus
[Milvus](https://milvus.io/) is an open source vector database for unstructured data. It uses RocksDB not only as one of the supported kv storage engines, but also as a message queue.
## Kafka
[Kafka](https://kafka.apache.org/) is an open-source distributed event streaming platform, it uses RocksDB to store state in Kafka Streams: https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance/.
## Solana Labs
[Solana](https://github.com/solana-labs/solana) is a fast, secure, scalable, and decentralized blockchain. It uses RocksDB as the underlying storage for its ledger store.
## Apache Kvrocks
[Apache Kvrocks](https://github.com/apache/kvrocks) is an open-source distributed key-value NoSQL database built on top of RocksDB. It serves as a cost-saving and capacity-increasing alternative drop-in replacement for Redis.
## Others
More databases using RocksDB can be found at [dbdb.io](https://dbdb.io/browse?embeds=rocksdb).
[Smyte](https://www.smyte.com/) uses RocksDB as the storage layer for their core key-value storage, high-performance counters and time-windowed HyperLogLog services.
Vendored
-6
View File
@@ -1,4 +1,3 @@
# Vagrant file
Vagrant.configure("2") do |config|
config.vm.provider "virtualbox" do |v|
@@ -14,11 +13,6 @@ Vagrant.configure("2") do |config|
box.vm.box = "chef/centos-6.5"
end
config.vm.define "centos7" do |box|
box.vm.box = "centos/7"
box.vm.provision "shell", path: "build_tools/setup_centos7.sh"
end
config.vm.define "FreeBSD10" do |box|
box.vm.guest = :freebsd
box.vm.box = "robin/freebsd-10"
+4 -4
View File
@@ -24,7 +24,7 @@ We strive to achieve the following goals:
* make all unit test pass both in debug and release builds.
* Note: latest introduction of SyncPoint seems to disable running db_test in Release.
* make performance on par with published benchmarks accounting for HW differences
* we would like to keep the port code inline with the main branch with no forking
* we would like to keep the port code inline with the master branch with no forking
## Build system
We have chosen CMake as a widely accepted build system to build the Windows port. It is very fast and convenient.
@@ -43,9 +43,9 @@ We plan to use this port for our business purposes here at Bing and this provide
* Certain headers that are not present and not necessary on Windows were simply `#ifndef OS_WIN` in a few places (`unistd.h`)
* All posix specific headers were replaced to port/port.h which worked well
* Replaced `dirent.h` for `port/port_dirent.h` (very few places) with the implementation of the relevant interfaces within `rocksdb::port` namespace
* Replaced `dirent.h` for `port/dirent.h` (very few places) with the implementation of the relevant interfaces within `rocksdb::port` namespace
* Replaced `sys/time.h` to `port/sys_time.h` (few places) implemented equivalents within `rocksdb::port`
* `printf %z` specification is not supported on Windows. To imitate existing standards we came up with a string macro `ROCKSDB_PRIszt` which expands to `zu` on posix systems and to `Iu` on windows.
* `printf %z` specification is not supported on Windows. To imitate existing standards we came up with a string macro `ROCKSDB_PRIszt` which expands to `%z` on posix systems and to Iu on windows.
* in class member initialization were moved to a __ctors in some cases
* `constexpr` is not supported. We had to replace `std::numeric_limits<>::max/min()` to its C macros for constants. Sometimes we had to make class members `static const` and place a definition within a .cc file.
* `constexpr` for functions was replaced to a template specialization (1 place)
@@ -66,7 +66,7 @@ We endeavored to make it functionally on par with posix_env. This means we repli
Even though Windows provides its own efficient thread-pool implementation we chose to replicate posix logic using `std::thread` primitives. This allows anyone to quickly detect any changes within the posix source code and replicate them within windows env. This has proven to work very well. At the same time for anyone who wishes to replace the built-in thread-pool can do so using RocksDB stackable environments.
For disk access we implemented all of the functionality present within the posix_env which includes memory mapped files, random access, rate-limiter support etc.
The `use_os_buffer` flag on Posix platforms currently denotes disabling read-ahead log via `fadvise` mechanism. Windows does not have `fadvise` system call. What is more, it implements disk cache in a way that differs from Linux greatly. It's not an uncommon practice on Windows to perform un-buffered disk access to gain control of the memory consumption. We think that in our use case this may also be a good configuration option at the expense of disk throughput. To compensate one may increase the configured in-memory cache size instead. Thus we have chosen `use_os_buffer=false` to disable OS disk buffering for `WinWritableFile` and `WinRandomAccessFile`. The OS imposes restrictions on the alignment of the disk offsets, buffers used and the amount of data that is read/written when accessing files in un-buffered mode. When the option is true, the classes behave in a standard way. This allows to perform writes and reads in cases when un-buffered access does not make sense such as WAL and MANIFEST.
The `use_os_buffer` flag on Posix platforms currently denotes disabling read-ahead log via `fadvise` mechanism. Windows does not have `fadvise` system call. What is more, it implements disk cache in a way that differs from Linux greatly. Its not an uncommon practice on Windows to perform un-buffered disk access to gain control of the memory consumption. We think that in our use case this may also be a good configuration option at the expense of disk throughput. To compensate one may increase the configured in-memory cache size instead. Thus we have chosen `use_os_buffer=false` to disable OS disk buffering for `WinWritableFile` and `WinRandomAccessFile`. The OS imposes restrictions on the alignment of the disk offsets, buffers used and the amount of data that is read/written when accessing files in un-buffered mode. When the option is true, the classes behave in a standard way. This allows to perform writes and reads in cases when un-buffered access does not make sense such as WAL and MANIFEST.
We have replaced `pread/pwrite` with `WriteFile/ReadFile` with `OVERLAPPED` structure so we can atomically seek to the position of the disk operation but still perform the operation synchronously. Thus we able to emulate that functionality of `pread/pwrite` reasonably well. The only difference is that the file pointer is not returned to its original position but that hardly matters given the random nature of access.
+15
View File
@@ -0,0 +1,15 @@
version: 1.0.{build}
before_build:
- md %APPVEYOR_BUILD_FOLDER%\build
- cd %APPVEYOR_BUILD_FOLDER%\build
- cmake -G "Visual Studio 12 Win64" -DOPTDBG=1 ..
- cd ..
build:
project: build\rocksdb.sln
parallel: true
verbosity: minimal
test:
test_script:
- ps: build_tools\run_ci_db_test.ps1 -EnableRerun -Run db_test -Exclude DBTest.Randomized,DBTest.FileCreationRandomFailure -Concurrency 18
- ps: build_tools\run_ci_db_test.ps1 -Run env_test -Concurrency 1
@@ -0,0 +1,3 @@
<?php
phutil_register_library('arcanist_util', __FILE__);
+38
View File
@@ -0,0 +1,38 @@
<?php
/**
* This file is automatically generated. Use 'arc liberate' to rebuild it.
* @generated
* @phutil-library-version 2
*/
phutil_register_library_map(array(
'__library_version__' => 2,
'class' =>
array(
'ArcanistCpplintLinter' => 'cpp_linter/ArcanistCpplintLinter.php',
'BaseDirectoryScopedFormatLinter' => 'cpp_linter/BaseDirectoryScopedFormatLinter.php',
'FacebookArcanistConfiguration' => 'config/FacebookArcanistConfiguration.php',
'FacebookFbcodeLintEngine' => 'lint_engine/FacebookFbcodeLintEngine.php',
'FacebookFbcodeUnitTestEngine' => 'unit_engine/FacebookFbcodeUnitTestEngine.php',
'FacebookHowtoevenLintEngine' => 'lint_engine/FacebookHowtoevenLintEngine.php',
'FacebookHowtoevenLinter' => 'cpp_linter/FacebookHowtoevenLinter.php',
'FbcodeClangFormatLinter' => 'cpp_linter/FbcodeClangFormatLinter.php',
'FbcodeCppLinter' => 'cpp_linter/FbcodeCppLinter.php',
),
'function' =>
array(
),
'xmap' =>
array(
'ArcanistCpplintLinter' => 'ArcanistLinter',
'BaseDirectoryScopedFormatLinter' => 'ArcanistLinter',
'FacebookArcanistConfiguration' => 'ArcanistConfiguration',
'FacebookFbcodeLintEngine' => 'ArcanistLintEngine',
'FacebookFbcodeUnitTestEngine' => 'ArcanistBaseUnitTestEngine',
'FacebookHowtoevenLintEngine' => 'ArcanistLintEngine',
'FacebookHowtoevenLinter' => 'ArcanistLinter',
'FbcodeClangFormatLinter' => 'BaseDirectoryScopedFormatLinter',
'FbcodeCppLinter' => 'ArcanistLinter',
),
));
@@ -0,0 +1,35 @@
<?php
// Copyright 2004-present Facebook. All Rights Reserved.
// This source code is licensed under the BSD-style license found in the
// LICENSE file in the root directory of this source tree. An additional grant
// of patent rights can be found in the PATENTS file in the same directory.
class FacebookArcanistConfiguration extends ArcanistConfiguration {
public function didRunWorkflow($command,
ArcanistBaseWorkflow $workflow,
$error_code) {
if ($command == 'diff' && !$workflow->isRawDiffSource()) {
$this->maybePushToJenkins($workflow);
}
}
//////////////////////////////////////////////////////////////////////
/* Send off builds to jenkins */
function maybePushToJenkins($workflow) {
$diffID = $workflow->getDiffID();
if ($diffID === null) {
return;
}
$results = $workflow->getTestResults();
if (!$results) {
return;
}
$url = "https://ci-builds.fb.com/view/rocksdb/job/rocksdb_diff_check/"
."buildWithParameters?token=AUTH&DIFF_ID=$diffID";
system("curl --noproxy '*' \"$url\" > /dev/null 2>&1");
}
}
@@ -0,0 +1,88 @@
<?php
/**
* Uses google's cpplint.py to check code. RocksDB team forked this file from
* phabricator's /src/lint/linter/ArcanistCpplintLinter.php, and customized it
* for its own use.
*
* You can get it here:
* http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py
* @group linter
*/
final class ArcanistCpplintLinter extends ArcanistLinter {
public function willLintPaths(array $paths) {
return;
}
public function getLinterName() {
return 'cpplint.py';
}
public function getLintPath() {
$bin = 'cpplint.py';
// Search under current dir
list($err) = exec_manual('which %s/%s', $this->linterDir(), $bin);
if (!$err) {
return $this->linterDir().'/'.$bin;
}
// Look for globally installed cpplint.py
list($err) = exec_manual('which %s', $bin);
if ($err) {
throw new ArcanistUsageException(
"cpplint.py does not appear to be installed on this system. Install ".
"it (e.g., with 'wget \"http://google-styleguide.googlecode.com/".
"svn/trunk/cpplint/cpplint.py\"') ".
"in your .arcconfig to point to the directory where it resides. ".
"Also don't forget to chmod a+x cpplint.py!");
}
return $bin;
}
public function lintPath($path) {
$bin = $this->getLintPath();
$path = $this->rocksdbDir().'/'.$path;
$f = new ExecFuture("%C $path", $bin);
list($err, $stdout, $stderr) = $f->resolve();
if ($err === 2) {
throw new Exception("cpplint failed to run correctly:\n".$stderr);
}
$lines = explode("\n", $stderr);
$messages = array();
foreach ($lines as $line) {
$line = trim($line);
$matches = null;
$regex = '/^[^:]+:(\d+):\s*(.*)\s*\[(.*)\] \[(\d+)\]$/';
if (!preg_match($regex, $line, $matches)) {
continue;
}
foreach ($matches as $key => $match) {
$matches[$key] = trim($match);
}
$message = new ArcanistLintMessage();
$message->setPath($path);
$message->setLine($matches[1]);
$message->setCode($matches[3]);
$message->setName($matches[3]);
$message->setDescription($matches[2]);
$message->setSeverity(ArcanistLintSeverity::SEVERITY_WARNING);
$this->addLintMessage($message);
}
}
// The path of this linter
private function linterDir() {
return dirname(__FILE__);
}
// TODO(kaili) a quick and dirty way to figure out rocksdb's root dir.
private function rocksdbDir() {
return $this->linterDir()."/../..";
}
}
@@ -0,0 +1,74 @@
<?php
// Copyright 2004-present Facebook. All Rights Reserved.
// This source code is licensed under the BSD-style license found in the
// LICENSE file in the root directory of this source tree. An additional grant
// of patent rights can be found in the PATENTS file in the same directory.
abstract class BaseDirectoryScopedFormatLinter extends ArcanistLinter {
const LINT_FORMATTING = 1;
private $changedLines = array();
private $rawLintOutput = array();
abstract protected function getPathsToLint();
protected function shouldLintPath($path) {
foreach ($this->getPathsToLint() as $p) {
// check if $path starts with $p
if (strncmp($path, $p, strlen($p)) === 0) {
return true;
}
}
return false;
}
// API to tell this linter which lines were changed
final public function setPathChangedLines($path, $changed) {
$this->changedLines[$path] = $changed;
}
final public function willLintPaths(array $paths) {
$futures = array();
foreach ($paths as $path) {
if (!$this->shouldLintPath($path)) {
continue;
}
$changed = $this->changedLines[$path];
if (!isset($changed)) {
// do not run linter if there are no changes
continue;
}
$futures[$path] = $this->getFormatFuture($path, $changed);
}
foreach (Futures($futures)->limit(8) as $p => $f) {
$this->rawLintOutput[$p] = $f->resolvex();
}
}
abstract protected function getFormatFuture($path, array $changed);
abstract protected function getLintMessage($diff);
final public function lintPath($path) {
if (!isset($this->rawLintOutput[$path])) {
return;
}
list($new_content) = $this->rawLintOutput[$path];
$old_content = $this->getData($path);
if ($new_content != $old_content) {
$diff = ArcanistDiffUtils::renderDifferences($old_content, $new_content);
$this->raiseLintAtOffset(
0,
self::LINT_FORMATTING,
$this->getLintMessage($diff),
$old_content,
$new_content);
}
}
}
@@ -0,0 +1,223 @@
<?php
// Copyright 2015-present Facebook. All Rights Reserved.
// This source code is licensed under the BSD-style license found in the
// LICENSE file in the root directory of this source tree. An additional grant
// of patent rights can be found in the PATENTS file in the same directory.
final class FacebookHowtoevenLinter extends ArcanistLinter {
const VERSION = 'fd9192f324c36d28136d14380f0b552a1385b59b';
private $parsedTargets = array();
public function getLinterName() {
return 'Howtoeven';
}
protected function getSeverity($code) {
$severities = array(
ArcanistLintSeverity::SEVERITY_DISABLED,
ArcanistLintSeverity::SEVERITY_ADVICE,
ArcanistLintSeverity::SEVERITY_WARNING,
ArcanistLintSeverity::SEVERITY_ERROR,
);
return idx($severities, $code, ArcanistLintSeverity::SEVERITY_WARNING);
}
public function willLintPaths(array $paths) {
// Cleanup previous runs.
$this->localExecx("rm -rf _build/_lint");
// Build compilation database.
$lintable_paths = $this->getLintablePaths($paths);
$interesting_paths = $this->getInterestingPaths($lintable_paths);
if (!$lintable_paths) {
return;
}
// Run lint.
try {
$this->localExecx(
"%C %C -p _build/dev/ %Ls",
$this->getBinaryPath(),
$this->getFilteredIssues(),
$lintable_paths);
} catch (CommandException $exception) {
PhutilConsole::getConsole()->writeErr($exception->getMessage());
}
// Load results.
$result = id(
new SQLite3(
$this->getProjectRoot().'/_build/_lint/lint.db',
SQLITE3_OPEN_READONLY))
->query("SELECT * FROM raised_issues");
while ($issue = $result->fetchArray(SQLITE3_ASSOC)) {
// Skip issues not part of the linted file.
if (in_array($issue['file'], $interesting_paths)) {
$this->addLintMessage(id(new ArcanistLintMessage())
->setPath($issue['file'])
->setLine($issue['line'])
->setChar($issue['column'])
->setCode('Howtoeven')
->setSeverity($this->getSeverity($issue['severity']))
->setName('Hte-'.$issue['name'])
->setDescription(
sprintf(
"%s\n\n%s",
($issue['message']) ? $issue['message'] : $issue['description'],
$issue['explanation']))
->setOriginalText(idx($issue, 'original', ''))
->setReplacementText(idx($issue, 'replacement', '')));
}
}
}
public function lintPath($path) {
}
/**
* Get the paths that we know how to lint.
*
* The strategy is to first look whether there's an existing compilation
* database and use that if it's exhaustive. We generate our own only if
* necessary.
*/
private function getLintablePaths($paths) {
// Replace headers with existing sources.
for ($i = 0; $i < count($paths); $i++) {
if (preg_match("/\.h$/", $paths[$i])) {
$header = preg_replace("/\.h$/", ".cpp", $paths[$i]);
if (file_exists($header)) {
$paths[$i] = $header;
}
}
}
// Check if database exists and is exhaustive.
$available_paths = $this->getAvailablePaths();
$lintable_paths = array_intersect($paths, $available_paths);
if ($paths === $lintable_paths) {
return $lintable_paths;
}
// Generate our own database.
$targets = $this->getTargetsFor($paths);
if (!$targets) {
PhutilConsole::getConsole()->writeErr(
"No build targets found for %s\n",
implode(', ', $paths));
return array();
}
$this->localExecx("./tools/build/bin/fbconfig.par -r %Ls", $targets);
$this->localExecx("./tools/build/bin/fbmake.par gen_cdb");
$available_paths = $this->getAvailablePaths();
$lintable_paths = array_intersect($paths, $available_paths);
if ($paths != $lintable_paths) {
PhutilConsole::getConsole()->writeErr(
"Can't lint %s\n",
implode(', ', array_diff($paths, $available_paths)));
}
// Return what we know how to lint.
return $lintable_paths;
}
/**
* Get the available paths in the current compilation database.
*/
private function getAvailablePaths() {
$database_path = $this->getProjectRoot()
.'/_build/dev/compile_commands.json';
if (!file_exists($database_path)) {
return array();
}
$entries = json_decode(file_get_contents($database_path), true);
$paths = array();
foreach ($entries as $entry) {
$paths[] = $entry['file'];
}
return $paths;
}
/**
* Search for the targets directories for the given files.
*/
private static function getTargetsFor($paths) {
$targets = array();
foreach ($paths as $path) {
while (($path = dirname($path)) !== '.') {
if (in_array('TARGETS', scandir($path))) {
$contents = file_get_contents($path.'/TARGETS');
if (strpos($contents, 'cpp_binary') !== false) {
$targets[] = $path;
break;
}
}
}
}
return array_unique($targets);
}
/**
* The paths that we actually want to report on.
*/
private function getInterestingPaths($paths) {
$headers = array();
foreach ($paths as $path) {
$headers[] = preg_replace("/\.cpp$/", ".h", $path);
}
return array_merge($paths, $headers);
}
/**
* The path where the binary is located. Will return the current dewey binary
* unless the `HOWTOEVEN_BUILD` environment variable is set.
*/
private function getBinaryPath() {
$path = sprintf(
"/mnt/dewey/fbcode/.commits/%s/builds/howtoeven/client",
self::VERSION);
$build = getenv('HOWTOEVEN_BUILD');
if ($build) {
$path = sprintf(
"./_build/%s/tools/howtoeven/client",
$build);
if (!file_exists($path)) {
PhutilConsole::getConsole()->writeErr(">> %s does not exist\n", $path);
exit(1);
}
}
return $path;
}
/**
* Execute the command in the root directory.
*/
private function localExecx($command /* , ... */) {
$arguments = func_get_args();
return newv('ExecFuture', $arguments)
->setCWD($this->getProjectRoot())
->resolvex();
}
/**
* The root of the project.
*/
private function getProjectRoot() {
return $this->getEngine()->getWorkingCopy()->getProjectRoot();
}
private function getFilteredIssues() {
$issues = getenv('HOWTOEVEN_ISSUES');
return ($issues) ? csprintf('-issues %s', $issues) : '';
}
}
@@ -0,0 +1,58 @@
<?php
// Copyright 2004-present Facebook. All Rights Reserved.
// This source code is licensed under the BSD-style license found in the
// LICENSE file in the root directory of this source tree. An additional grant
// of patent rights can be found in the PATENTS file in the same directory.
final class FbcodeClangFormatLinter extends BaseDirectoryScopedFormatLinter {
const LINT_FORMATTING = 1;
const CLANG_FORMAT_BINARY = '/mnt/vol/engshare/admin/scripts/clang-format';
protected function getPathsToLint() {
return array('');
}
public function getLinterName() {
return 'CLANG_FORMAT';
}
public function getLintSeverityMap() {
return array(
self::LINT_FORMATTING => ArcanistLintSeverity::SEVERITY_ADVICE,
);
}
public function getLintNameMap() {
return array(
self::LINT_FORMATTING => pht('Changes are not clang-formatted'),
);
}
protected function getFormatFuture($path, array $changed) {
$args = "";
foreach ($changed as $key => $value) {
$args .= " --lines=$key:$key";
}
$binary = self::CLANG_FORMAT_BINARY;
if (!file_exists($binary)) {
// trust the $PATH
$binary = "clang-format";
}
return new ExecFuture(
"%s %s $args",
$binary,
$this->getEngine()->getFilePathOnDisk($path));
}
protected function getLintMessage($diff) {
$link_to_clang_format =
"[[ http://fburl.com/clang-format | clang-format ]]";
return <<<LINT_MSG
Changes in this file were not formatted using $link_to_clang_format.
Please run build_tools/format-diff.sh or `make format`
LINT_MSG;
}
}
@@ -0,0 +1,126 @@
<?php
// Copyright 2004-present Facebook. All rights reserved.
class FbcodeCppLinter extends ArcanistLinter {
const FLINT = "/home/engshare/tools/flint";
const LINT_ERROR = 1;
const LINT_WARNING = 2;
const LINT_ADVICE = 3;
const C_FLAG = "--c_mode=true";
private $rawLintOutput = array();
public function willLintPaths(array $paths) {
if (!file_exists(self::FLINT)) {
return;
}
$futures = array();
foreach ($paths as $p) {
$lpath = $this->getEngine()->getFilePathOnDisk($p);
$lpath_file = file($lpath);
if (preg_match('/\.(c)$/', $lpath) ||
preg_match('/-\*-.*Mode: C[; ].*-\*-/', $lpath_file[0]) ||
preg_match('/vim(:.*)*:\s*(set\s+)?filetype=c\s*:/', $lpath_file[0])
) {
$futures[$p] = new ExecFuture("%s %s %s 2>&1",
self::FLINT, self::C_FLAG,
$this->getEngine()->getFilePathOnDisk($p));
} else {
$futures[$p] = new ExecFuture("%s %s 2>&1",
self::FLINT, $this->getEngine()->getFilePathOnDisk($p));
}
}
foreach (Futures($futures)->limit(8) as $p => $f) {
$this->rawLintOutput[$p] = $f->resolvex();
}
return;
}
public function getLinterName() {
return "FBCPP";
}
public function lintPath($path) {
$this->runCppLint($path);
}
private function runCppLint($path) {
$msgs = $this->getCppLintOutput($path);
foreach ($msgs as $m) {
$this->raiseLintAtLine($m['line'], 0, $m['severity'], $m['msg']);
}
}
private function adviseOnEachPattern(
$path,
$regex,
$message,
$lint_type = self::LINT_ADVICE,
$match_idx = 0) {
$file_data = $this->getData($path);
$matches = array();
if (!preg_match_all($regex, $file_data, $matches, PREG_OFFSET_CAPTURE)) {
return;
}
foreach ($matches[$match_idx] as $match) {
list($match_str, $offset) = $match;
$this->raiseLintAtOffset($offset, $lint_type, $message, $match_str);
}
}
public function getLintSeverityMap() {
return array(
self::LINT_WARNING => ArcanistLintSeverity::SEVERITY_WARNING,
self::LINT_ADVICE => ArcanistLintSeverity::SEVERITY_ADVICE,
self::LINT_ERROR => ArcanistLintSeverity::SEVERITY_ERROR
);
}
public function getLintNameMap() {
return array(
self::LINT_ADVICE => "CppLint Advice",
self::LINT_WARNING => "CppLint Warning",
self::LINT_ERROR => "CppLint Error"
);
}
private function getCppLintOutput($path) {
if (!array_key_exists($path, $this->rawLintOutput)) {
return array();
}
list($output) = $this->rawLintOutput[$path];
$msgs = array();
$current = null;
$matches = array();
foreach (explode("\n", $output) as $line) {
if (preg_match('/.*?:(\d+):(.*)/', $line, $matches)) {
if ($current) {
$msgs[] = $current;
}
$line = $matches[1];
$text = $matches[2];
if (preg_match('/.*Warning.*/', $text)) {
$sev = self::LINT_WARNING;
} else if (preg_match('/.*Advice.*/', $text)) {
$sev = self::LINT_ADVICE;
} else {
$sev = self::LINT_ERROR;
}
$current = array('line' => $line,
'msg' => $text,
'severity' => $sev);
} else if ($current) {
$current['msg'] .= ' ' . $line;
}
}
if ($current) {
$msgs[] = $current;
}
return $msgs;
}
}
+4767
View File
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,140 @@
<?php
// Copyright 2004-present Facebook. All rights reserved.
// This source code is licensed under the BSD-style license found in the
// LICENSE file in the root directory of this source tree. An additional grant
// of patent rights can be found in the PATENTS file in the same directory.
class FacebookFbcodeLintEngine extends ArcanistLintEngine {
public function buildLinters() {
$linters = array();
$paths = $this->getPaths();
// Remove all deleted files, which are not checked by the
// following linters.
foreach ($paths as $key => $path) {
if (!Filesystem::pathExists($this->getFilePathOnDisk($path))) {
unset($paths[$key]);
}
}
$generated_linter = new ArcanistGeneratedLinter();
$linters[] = $generated_linter;
$nolint_linter = new ArcanistNoLintLinter();
$linters[] = $nolint_linter;
$text_linter = new ArcanistTextLinter();
$text_linter->setCustomSeverityMap(array(
ArcanistTextLinter::LINT_LINE_WRAP
=> ArcanistLintSeverity::SEVERITY_ADVICE,
));
$linters[] = $text_linter;
$java_text_linter = new ArcanistTextLinter();
$java_text_linter->setMaxLineLength(100);
$java_text_linter->setCustomSeverityMap(array(
ArcanistTextLinter::LINT_LINE_WRAP
=> ArcanistLintSeverity::SEVERITY_ADVICE,
));
$linters[] = $java_text_linter;
$python_linter = new ArcanistPEP8Linter();
$linters[] = $python_linter;
if (!$this->getCommitHookMode()) {
$cpp_linters = array();
$cpp_linters[] = $linters[] = new ArcanistCpplintLinter();
$cpp_linters[] = $linters[] = new FbcodeCppLinter();
$clang_format_linter = new FbcodeClangFormatLinter();
$linters[] = $clang_format_linter;
}
$spelling_linter = new ArcanistSpellingLinter();
$linters[] = $spelling_linter;
foreach ($paths as $path) {
$is_text = false;
$text_extensions = (
'/\.('.
'cpp|cxx|c|cc|h|hpp|hxx|tcc|'.
'py|rb|hs|pl|pm|tw|'.
'php|phpt|css|js|'.
'java|'.
'thrift|'.
'lua|'.
'siv|'.
'txt'.
')$/'
);
if (preg_match($text_extensions, $path)) {
$is_text = true;
}
if ($is_text) {
$nolint_linter->addPath($path);
$generated_linter->addPath($path);
$generated_linter->addData($path, $this->loadData($path));
if (preg_match('/\.java$/', $path)) {
$java_text_linter->addPath($path);
$java_text_linter->addData($path, $this->loadData($path));
} else {
$text_linter->addPath($path);
$text_linter->addData($path, $this->loadData($path));
}
$spelling_linter->addPath($path);
$spelling_linter->addData($path, $this->loadData($path));
}
if (preg_match('/\.(cpp|c|cc|cxx|h|hh|hpp|hxx|tcc)$/', $path)
&& !preg_match('/third-party/', $path)) {
foreach ($cpp_linters as &$linter) {
$linter->addPath($path);
$linter->addData($path, $this->loadData($path));
}
$clang_format_linter->addPath($path);
$clang_format_linter->addData($path, $this->loadData($path));
$clang_format_linter->setPathChangedLines(
$path, $this->getPathChangedLines($path));
}
// Match *.py and contbuild config files
if (preg_match('/(\.(py|tw|smcprops)|^contbuild\/configs\/[^\/]*)$/',
$path)) {
$space_count = 4;
$real_path = $this->getFilePathOnDisk($path);
$dir = dirname($real_path);
do {
if (file_exists($dir.'/.python2space')) {
$space_count = 2;
break;
}
$dir = dirname($dir);
} while ($dir != '/' && $dir != '.');
$cur_path_linter = $python_linter;
$cur_path_linter->addPath($path);
$cur_path_linter->addData($path, $this->loadData($path));
if (preg_match('/\.tw$/', $path)) {
$cur_path_linter->setCustomSeverityMap(array(
'E251' => ArcanistLintSeverity::SEVERITY_DISABLED,
));
}
}
}
$name_linter = new ArcanistFilenameLinter();
$linters[] = $name_linter;
foreach ($paths as $path) {
$name_linter->addPath($path);
}
return $linters;
}
}
@@ -0,0 +1,27 @@
<?php
// Copyright 2015-present Facebook. All Rights Reserved.
// This source code is licensed under the BSD-style license found in the
// LICENSE file in the root directory of this source tree. An additional grant
// of patent rights can be found in the PATENTS file in the same directory.
final class FacebookHowtoevenLintEngine extends ArcanistLintEngine {
public function buildLinters() {
$paths = array();
foreach ($this->getPaths() as $path) {
// Don't try to lint deleted files or changed directories.
if (!Filesystem::pathExists($path) || is_dir($path)) {
continue;
}
if (preg_match('/\.(cpp|c|cc|cxx|h|hh|hpp|hxx|tcc)$/', $path)) {
$paths[] = $path;
}
}
$howtoeven = new FacebookHowtoevenLinter();
$howtoeven->setPaths($paths);
return array($howtoeven);
}
}
@@ -0,0 +1,21 @@
<?php
// Copyright 2004-present Facebook. All Rights Reserved.
// This source code is licensed under the BSD-style license found in the
// LICENSE file in the root directory of this source tree. An additional grant
// of patent rights can be found in the PATENTS file in the same directory.
class FacebookFbcodeUnitTestEngine extends ArcanistBaseUnitTestEngine {
public function run() {
// Here we create a new unit test "jenkins_async_test" and promise we'll
// update the results later.
// Jenkins updates the results using `arc call-conduit
// differential.updateunitresults` call. If you change the name here, also
// make sure to change the name in Jenkins script that updates the test
// result -- they have to be the same.
$result = new ArcanistUnitTestResult();
$result->setName("jenkins_async_test");
$result->setResult(ArcanistUnitTestResult::RESULT_POSTPONED);
return array($result);
}
}
File diff suppressed because it is too large Load Diff
-1594
View File
File diff suppressed because it is too large Load Diff
-395
View File
@@ -1,395 +0,0 @@
#!/usr/bin/env python3
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
try:
from builtins import str
except ImportError:
from __builtin__ import str
import fnmatch
import json
import os
import sys
from targets_builder import TARGETSBuilder, LiteralValue
from util import ColorString
# This script generates BUCK file for Buck.
# Buck is a build tool specifying dependencies among different build targets.
# User can pass extra dependencies as a JSON object via command line, and this
# script can include these dependencies in the generate BUCK file.
# Usage:
# $python3 buckifier/buckify_rocksdb.py
# (This generates a TARGET file without user-specified dependency for unit
# tests.)
# $python3 buckifier/buckify_rocksdb.py \
# '{"fake": {
# "extra_deps": [":test_dep", "//fakes/module:mock1"],
# "extra_compiler_flags": ["-DFOO_BAR", "-Os"]
# }
# }'
# (Generated BUCK file has test_dep and mock1 as dependencies for RocksDB
# unit tests, and will use the extra_compiler_flags to compile the unit test
# source.)
# tests to export as libraries for inclusion in other projects
_EXPORTED_TEST_LIBS = ["env_basic_test"]
# Parse src.mk files as a Dictionary of
# VAR_NAME => list of files
def parse_src_mk(repo_path):
src_mk = repo_path + "/src.mk"
src_files = {}
for line in open(src_mk):
line = line.strip()
if len(line) == 0 or line[0] == "#":
continue
if "=" in line:
current_src = line.split("=")[0].strip()
src_files[current_src] = []
elif ".c" in line:
src_path = line.split("\\")[0].strip()
src_files[current_src].append(src_path)
return src_files
# get all .cc / .c files
def get_cc_files(repo_path):
cc_files = []
for root, _dirnames, filenames in os.walk(
repo_path
): # noqa: B007 T25377293 Grandfathered in
root = root[(len(repo_path) + 1) :]
if "java" in root:
# Skip java
continue
for filename in fnmatch.filter(filenames, "*.cc"):
cc_files.append(os.path.join(root, filename))
for filename in fnmatch.filter(filenames, "*.c"):
cc_files.append(os.path.join(root, filename))
return cc_files
# Get non_parallel tests from Makefile
def get_non_parallel_tests(repo_path):
Makefile = repo_path + "/Makefile"
s = set({})
found_non_parallel_tests = False
for line in open(Makefile):
line = line.strip()
if line.startswith("NON_PARALLEL_TEST ="):
found_non_parallel_tests = True
elif found_non_parallel_tests:
if line.endswith("\\"):
# remove the trailing \
line = line[:-1]
line = line.strip()
s.add(line)
else:
# we consumed all the non_parallel tests
break
return s
# Parse extra dependencies passed by user from command line
def get_dependencies():
deps_map = {"": {"extra_deps": [], "extra_compiler_flags": []}}
if len(sys.argv) < 2:
return deps_map
def encode_dict(data):
rv = {}
for k, v in data.items():
if isinstance(v, dict):
v = encode_dict(v)
rv[k] = v
return rv
extra_deps = json.loads(sys.argv[1], object_hook=encode_dict)
for target_alias, deps in extra_deps.items():
deps_map[target_alias] = deps
return deps_map
# Prepare BUCK file for buck
def generate_buck(repo_path, deps_map):
print(ColorString.info("Generating BUCK"))
# parsed src.mk file
src_mk = parse_src_mk(repo_path)
# get all .cc files
cc_files = get_cc_files(repo_path)
# get non_parallel tests from Makefile
non_parallel_tests = get_non_parallel_tests(repo_path)
if src_mk is None or cc_files is None or non_parallel_tests is None:
return False
extra_argv = ""
if len(sys.argv) >= 2:
# Heuristically quote and canonicalize whitespace for inclusion
# in how the file was generated.
extra_argv = " '{}'".format(" ".join(sys.argv[1].split()))
BUCK = TARGETSBuilder("%s/BUCK" % repo_path, extra_argv)
# Add oncall("rocksdb_point_of_contact") at the top
BUCK.add_oncall("rocksdb_point_of_contact")
# rocksdb_lib
BUCK.add_library(
"rocksdb_lib",
src_mk["LIB_SOURCES"] +
# always add range_tree, it's only excluded on ppc64, which we don't use internally
src_mk["RANGE_TREE_SOURCES"] + src_mk["TOOL_LIB_SOURCES"],
deps=[
"//folly/container:f14_hash",
"//folly/coro:blocking_wait",
"//folly/coro:collect",
"//folly/coro:coroutine",
"//folly/coro:task",
"//folly/synchronization:distributed_mutex",
],
headers=LiteralValue("glob([\"**/*.h\"])")
)
# rocksdb_whole_archive_lib
BUCK.add_library(
"rocksdb_whole_archive_lib",
[],
deps=[
":rocksdb_lib",
],
extra_external_deps="",
link_whole=True,
)
# rocksdb_with_faiss_lib
BUCK.add_library(
"rocksdb_with_faiss_lib",
src_mk.get("WITH_FAISS_LIB_SOURCES", []),
deps=[
"//faiss:faiss",
":rocksdb_lib",
],
)
# rocksdb_test_lib
BUCK.add_library(
"rocksdb_test_lib",
src_mk.get("MOCK_LIB_SOURCES", [])
+ src_mk.get("TEST_LIB_SOURCES", [])
+ src_mk.get("EXP_LIB_SOURCES", [])
+ src_mk.get("ANALYZER_LIB_SOURCES", []),
[":rocksdb_lib"],
extra_test_libs=True,
)
# rocksdb_with_faiss_test_lib
BUCK.add_library(
"rocksdb_with_faiss_test_lib",
src_mk.get("MOCK_LIB_SOURCES", [])
+ src_mk.get("TEST_LIB_SOURCES", [])
+ src_mk.get("EXP_LIB_SOURCES", [])
+ src_mk.get("ANALYZER_LIB_SOURCES", []),
deps=[
":rocksdb_with_faiss_lib",
],
extra_test_libs=True,
)
# rocksdb_tools_lib
BUCK.add_library(
"rocksdb_tools_lib",
src_mk.get("BENCH_LIB_SOURCES", [])
+ src_mk.get("ANALYZER_LIB_SOURCES", [])
+ ["test_util/testutil.cc"],
[":rocksdb_lib"],
)
# rocksdb_cache_bench_tools_lib
BUCK.add_library(
"rocksdb_cache_bench_tools_lib",
src_mk.get("CACHE_BENCH_LIB_SOURCES", []),
[":rocksdb_lib"],
)
# rocksdb_point_lock_bench_tools_lib
BUCK.add_library(
"rocksdb_point_lock_bench_tools_lib",
src_mk.get("POINT_LOCK_BENCH_LIB_SOURCES", []),
[":rocksdb_lib"],
)
# rocksdb_stress_lib
BUCK.add_rocksdb_library(
"rocksdb_stress_lib",
src_mk.get("ANALYZER_LIB_SOURCES", [])
+ src_mk.get("STRESS_LIB_SOURCES", [])
+ ["test_util/testutil.cc"],
)
# ldb binary
BUCK.add_binary(
"ldb", ["tools/ldb.cc"], [":rocksdb_tools_lib"]
)
# db_stress binary
BUCK.add_binary(
"db_stress", ["db_stress_tool/db_stress.cc"], [":rocksdb_stress_lib"]
)
# db_bench binary
BUCK.add_binary(
"db_bench", ["tools/db_bench.cc"], [":rocksdb_tools_lib"]
)
# cache_bench binary
BUCK.add_binary(
"cache_bench", ["cache/cache_bench.cc"], [":rocksdb_cache_bench_tools_lib"]
)
# point_lock_bench binary
BUCK.add_binary(
"point_lock_bench",
["utilities/transactions/lock/point/point_lock_bench.cc"],
[":rocksdb_point_lock_bench_tools_lib"]
)
# bench binaries
for src in src_mk.get("MICROBENCH_SOURCES", []):
name = src.rsplit("/", 1)[1].split(".")[0] if "/" in src else src.split(".")[0]
BUCK.add_binary(name, [src], [], extra_bench_libs=True)
print(f"Extra dependencies:\n{json.dumps(deps_map)}")
# Dictionary test executable name -> relative source file path
test_source_map = {}
# c_test.c is added through BUCK.add_c_test(). If there
# are more than one .c test file, we need to extend
# BUCK.add_c_test() to include other C tests too.
for test_src in src_mk.get("TEST_MAIN_SOURCES_C", []):
if test_src != "db/c_test.c":
print("Don't know how to deal with " + test_src)
return False
BUCK.add_c_test()
try:
with open(f"{repo_path}/buckifier/bench.json") as json_file:
fast_fancy_bench_config_list = json.load(json_file)
for config_dict in fast_fancy_bench_config_list:
clean_benchmarks = {}
benchmarks = config_dict["benchmarks"]
for binary, benchmark_dict in benchmarks.items():
clean_benchmarks[binary] = {}
for benchmark, overloaded_metric_list in benchmark_dict.items():
clean_benchmarks[binary][benchmark] = []
for metric in overloaded_metric_list:
if not isinstance(metric, dict):
clean_benchmarks[binary][benchmark].append(metric)
BUCK.add_fancy_bench_config(
config_dict["name"],
clean_benchmarks,
False,
config_dict["expected_runtime_one_iter"],
config_dict["sl_iterations"],
config_dict["regression_threshold"],
)
with open(f"{repo_path}/buckifier/bench-slow.json") as json_file:
slow_fancy_bench_config_list = json.load(json_file)
for config_dict in slow_fancy_bench_config_list:
clean_benchmarks = {}
benchmarks = config_dict["benchmarks"]
for binary, benchmark_dict in benchmarks.items():
clean_benchmarks[binary] = {}
for benchmark, overloaded_metric_list in benchmark_dict.items():
clean_benchmarks[binary][benchmark] = []
for metric in overloaded_metric_list:
if not isinstance(metric, dict):
clean_benchmarks[binary][benchmark].append(metric)
for config_dict in slow_fancy_bench_config_list:
BUCK.add_fancy_bench_config(
config_dict["name"] + "_slow",
clean_benchmarks,
True,
config_dict["expected_runtime_one_iter"],
config_dict["sl_iterations"],
config_dict["regression_threshold"],
)
# it is better servicelab experiments break
# than rocksdb github ci
except Exception:
pass
BUCK.add_test_header()
for test_src in src_mk.get("TEST_MAIN_SOURCES", []):
test = test_src.split(".c")[0].strip().split("/")[-1].strip()
test_source_map[test] = (test_src, False)
print("" + test + " " + test_src)
for test_src in src_mk.get("WITH_FAISS_TEST_MAIN_SOURCES", []):
test = test_src.split(".c")[0].strip().split("/")[-1].strip()
test_source_map[test] = (test_src, True)
print("" + test + " " + test_src + " [FAISS]")
for target_alias, deps in deps_map.items():
for test, (test_src, with_faiss) in sorted(test_source_map.items()):
if len(test) == 0:
print(ColorString.warning("Failed to get test name for %s" % test_src))
continue
test_target_name = test if not target_alias else test + "_" + target_alias
if test in _EXPORTED_TEST_LIBS:
test_library = "%s_lib" % test_target_name
BUCK.add_library(
test_library,
[test_src],
deps=[":rocksdb_test_lib"],
extra_test_libs=True,
)
BUCK.register_test(
test_target_name,
test_src,
deps=json.dumps(deps["extra_deps"] + [":" + test_library]),
extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
)
else:
if with_faiss:
BUCK.register_test(
test_target_name,
test_src,
deps=json.dumps(deps["extra_deps"] + [":rocksdb_with_faiss_test_lib"]),
extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
)
else:
BUCK.register_test(
test_target_name,
test_src,
deps=json.dumps(deps["extra_deps"] + [":rocksdb_test_lib"]),
extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
)
BUCK.export_file("tools/db_crashtest.py")
BUCK.export_file("tools/fault_injection_log_parser.py")
print(ColorString.info("Generated BUCK Summary:"))
print(ColorString.info("- %d libs" % BUCK.total_lib))
print(ColorString.info("- %d binarys" % BUCK.total_bin))
print(ColorString.info("- %d tests" % BUCK.total_test))
return True
def get_rocksdb_path():
# rocksdb = {script_dir}/..
script_dir = os.path.dirname(sys.argv[0])
script_dir = os.path.abspath(script_dir)
rocksdb_path = os.path.abspath(os.path.join(script_dir, "../"))
return rocksdb_path
def exit_with_error(msg):
print(ColorString.error(msg))
sys.exit(1)
def main():
deps_map = get_dependencies()
# Generate BUCK file for buck
ok = generate_buck(get_rocksdb_path(), deps_map)
if not ok:
exit_with_error("Failed to generate BUCK files")
if __name__ == "__main__":
main()
-44
View File
@@ -1,44 +0,0 @@
#!/usr/bin/env bash
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
if [[ ! -f "BUCK" ]]
then
echo "BUCK file is missing!"
echo "Please do not remove / rename BUCK file in your commit(s)."
exit 1
fi
TGT_DIFF=`git diff BUCK | head -n 1`
if [ ! -z "$TGT_DIFF" ]
then
echo "BUCK file has uncommitted changes. Skip this check."
exit 0
fi
echo Backup original BUCK file.
cp BUCK BUCK.bkp
${PYTHON:-python3} buckifier/buckify_rocksdb.py
if [[ ! -f "BUCK" ]]
then
echo "BUCK file went missing after running buckifier/buckify_rocksdb.py!"
echo "Please do not remove the BUCK file."
exit 1
fi
TGT_DIFF=`git diff BUCK | head -n 1`
if [ -z "$TGT_DIFF" ]
then
mv BUCK.bkp BUCK
exit 0
else
echo "Please run '${PYTHON:-python3} buckifier/buckify_rocksdb.py' to update BUCK file."
echo "Do not manually update BUCK file."
${PYTHON:-python3} --version
mv BUCK.bkp BUCK
exit 1
fi
-6
View File
@@ -1,6 +0,0 @@
#!/usr/bin/env bash
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
# Create a tmp directory for the test to use
TEST_DIR=$(mktemp -d /dev/shm/fbcode_rocksdb_XXXXXXX)
# shellcheck disable=SC2068
TEST_TMPDIR="$TEST_DIR" $@ && rm -rf "$TEST_DIR"
-181
View File
@@ -1,181 +0,0 @@
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
try:
from builtins import object, str
except ImportError:
from __builtin__ import object, str
import pprint
import targets_cfg
class LiteralValue:
def __init__(self, value):
self.value = value
def __str__(self):
return str(self.value)
def smart_quote_value(val):
if isinstance(val, LiteralValue):
return str(val)
return '"%s"' % val
def pretty_list(lst, indent=8):
if lst is None or len(lst) == 0:
return ""
if len(lst) == 1:
return smart_quote_value(lst[0])
separator = ',\n%s' % (" " * indent)
res = separator.join(sorted(map(smart_quote_value, lst)))
res = "\n" + (" " * indent) + res + ',\n' + (" " * (indent - 4))
return res
class TARGETSBuilder:
def __init__(self, path, extra_argv):
self.path = path
header = targets_cfg.rocksdb_target_header_template.format(
extra_argv=extra_argv
)
with open(path, "wb") as targets_file:
targets_file.write(header.encode("utf-8"))
self.total_lib = 0
self.total_bin = 0
self.total_test = 0
self.tests_cfg = ""
def add_oncall(self, oncall):
with open(self.path, "ab") as targets_file:
targets_file.write(targets_cfg.oncall_template.format(name=oncall).encode("utf-8"))
def add_library(
self,
name,
srcs,
deps=None,
headers=None,
extra_external_deps="",
link_whole=False,
external_dependencies=None,
extra_test_libs=False,
):
if headers is not None:
if isinstance(headers, LiteralValue):
headers = str(headers)
else:
headers = "[" + pretty_list(headers) + "]"
else:
headers = "[]"
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.library_template.format(
name=name,
srcs=pretty_list(srcs),
headers=headers,
deps=pretty_list(deps),
extra_external_deps=extra_external_deps,
link_whole=link_whole,
external_dependencies=pretty_list(external_dependencies),
extra_test_libs=extra_test_libs,
).encode("utf-8")
)
self.total_lib = self.total_lib + 1
def add_rocksdb_library(self, name, srcs, headers=None, external_dependencies=None):
headers = "[" + pretty_list(headers) + "]"
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.rocksdb_library_template.format(
name=name,
srcs=pretty_list(srcs),
headers=headers,
external_dependencies=pretty_list(external_dependencies),
).encode("utf-8")
)
self.total_lib = self.total_lib + 1
def add_binary(
self,
name,
srcs,
deps=None,
extra_preprocessor_flags=None,
extra_bench_libs=False,
):
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.binary_template.format(
name=name,
srcs=pretty_list(srcs),
deps=pretty_list(deps),
extra_preprocessor_flags=pretty_list(extra_preprocessor_flags),
extra_bench_libs=extra_bench_libs,
).encode("utf-8")
)
self.total_bin = self.total_bin + 1
def add_c_test(self):
# The actual c_test_bin target is defined by add_c_test_wrapper in the
# internal //rocks/buckifier:defs.bzl (not in this OSS repo). db/c_test.c
# #includes the generated c_api_gen/*.inc fragments, so under Buck's
# hermetic sandbox that wrapper must expose them as headers, e.g.
# headers = native.glob(["c_api_gen/**/*.inc"])
# (Make/CMake resolve the include via -I. / PROJECT_SOURCE_DIR.)
with open(self.path, "ab") as targets_file:
targets_file.write(
b"""
add_c_test_wrapper()
"""
)
def add_test_header(self):
with open(self.path, "ab") as targets_file:
targets_file.write(
b"""
# Generate a test rule for each entry in ROCKS_TESTS
# Do not build the tests in opt mode, since SyncPoint and other test code
# will not be included.
"""
)
def add_fancy_bench_config(
self,
name,
bench_config,
slow,
expected_runtime,
sl_iterations,
regression_threshold,
):
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.fancy_bench_template.format(
name=name,
bench_config=pprint.pformat(bench_config),
slow=slow,
expected_runtime=expected_runtime,
sl_iterations=sl_iterations,
regression_threshold=regression_threshold,
).encode("utf-8")
)
def register_test(self, test_name, src, deps, extra_compiler_flags):
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.unittests_template.format(
test_name=test_name,
test_cc=str(src),
deps=deps,
extra_compiler_flags=extra_compiler_flags,
).encode("utf-8")
)
self.total_test = self.total_test + 1
def export_file(self, name):
with open(self.path, "a") as targets_file:
targets_file.write(
targets_cfg.export_file_template.format(name=name)
)
-50
View File
@@ -1,50 +0,0 @@
# Copyright (c) Meta Platforms, Inc. and affiliates.
# This source code is licensed under both the GPLv2 (found in the COPYING file in the root directory)
# and the Apache 2.0 License (found in the LICENSE.Apache file in the root directory).
rocksdb_target_header_template = """# This file \100generated by:
#$ python3 buckifier/buckify_rocksdb.py{extra_argv}
# --> DO NOT EDIT MANUALLY <--
# This file is a Meta-specific integration for buck builds, so can
# only be validated by Meta employees.
load("//rocks/buckifier:defs.bzl", "cpp_library_wrapper","rocks_cpp_library_wrapper","cpp_binary_wrapper","cpp_unittest_wrapper","fancy_bench_wrapper","add_c_test_wrapper")
load("@fbcode_macros//build_defs:export_files.bzl", "export_file")
"""
library_template = """
cpp_library_wrapper(name="{name}", srcs=[{srcs}], deps=[{deps}], headers={headers}, link_whole={link_whole}, extra_test_libs={extra_test_libs})
"""
rocksdb_library_template = """
rocks_cpp_library_wrapper(name="{name}", srcs=[{srcs}], headers={headers})
"""
binary_template = """
cpp_binary_wrapper(name="{name}", srcs=[{srcs}], deps=[{deps}], extra_preprocessor_flags=[{extra_preprocessor_flags}], extra_bench_libs={extra_bench_libs})
"""
unittests_template = """
cpp_unittest_wrapper(name="{test_name}",
srcs=["{test_cc}"],
deps={deps},
extra_compiler_flags={extra_compiler_flags})
"""
fancy_bench_template = """
fancy_bench_wrapper(suite_name="{name}", binary_to_bench_to_metric_list_map={bench_config}, slow={slow}, expected_runtime={expected_runtime}, sl_iterations={sl_iterations}, regression_threshold={regression_threshold})
"""
export_file_template = """
export_file(name = "{name}")
"""
oncall_template = """
oncall("{name}")
"""
-117
View File
@@ -1,117 +0,0 @@
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
"""
This module keeps commonly used components.
"""
try:
from builtins import object
except ImportError:
from __builtin__ import object
import os
import subprocess
import sys
import time
class ColorString:
"""Generate colorful strings on terminal"""
HEADER = "\033[95m"
BLUE = "\033[94m"
GREEN = "\033[92m"
WARNING = "\033[93m"
FAIL = "\033[91m"
ENDC = "\033[0m"
@staticmethod
def _make_color_str(text, color):
# In Python2, default encoding for unicode string is ASCII
if sys.version_info.major <= 2:
return "".join([color, text.encode("utf-8"), ColorString.ENDC])
# From Python3, default encoding for unicode string is UTF-8
return "".join([color, text, ColorString.ENDC])
@staticmethod
def ok(text):
if ColorString.is_disabled:
return text
return ColorString._make_color_str(text, ColorString.GREEN)
@staticmethod
def info(text):
if ColorString.is_disabled:
return text
return ColorString._make_color_str(text, ColorString.BLUE)
@staticmethod
def header(text):
if ColorString.is_disabled:
return text
return ColorString._make_color_str(text, ColorString.HEADER)
@staticmethod
def error(text):
if ColorString.is_disabled:
return text
return ColorString._make_color_str(text, ColorString.FAIL)
@staticmethod
def warning(text):
if ColorString.is_disabled:
return text
return ColorString._make_color_str(text, ColorString.WARNING)
is_disabled = False
def run_shell_command(shell_cmd, cmd_dir=None):
"""Run a single shell command.
@returns a tuple of shell command return code, stdout, stderr"""
if cmd_dir is not None and not os.path.exists(cmd_dir):
run_shell_command("mkdir -p %s" % cmd_dir)
start = time.time()
print("\t>>> Running: " + shell_cmd)
p = subprocess.Popen( # noqa
shell_cmd,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
cwd=cmd_dir,
)
stdout, stderr = p.communicate()
end = time.time()
# Report time if we spent more than 5 minutes executing a command
execution_time = end - start
if execution_time > (60 * 5):
mins = execution_time / 60
secs = execution_time % 60
print("\t>time spent: %d minutes %d seconds" % (mins, secs))
return p.returncode, stdout, stderr
def run_shell_commands(shell_cmds, cmd_dir=None, verbose=False):
"""Execute a sequence of shell commands, which is equivalent to
running `cmd1 && cmd2 && cmd3`
@returns boolean indication if all commands succeeds.
"""
if cmd_dir:
print("\t=== Set current working directory => %s" % cmd_dir)
for shell_cmd in shell_cmds:
ret_code, stdout, stderr = run_shell_command(shell_cmd, cmd_dir)
if stdout:
if verbose or ret_code != 0:
print(ColorString.info("stdout: \n"), stdout)
if stderr:
# contents in stderr is not necessarily to be error messages.
if verbose or ret_code != 0:
print(ColorString.error("stderr: \n"), stderr)
if ret_code != 0:
return False
return True
+19 -76
View File
@@ -1,5 +1,4 @@
#!/usr/bin/python
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
# amalgamate.py creates an amalgamation from a unity build.
# It can be run with either Python 2 or 3.
@@ -25,17 +24,17 @@
#
# The solution is to move the include out of the #ifdef.
from __future__ import print_function
import argparse
from os import path
import re
import sys
from os import path
include_re = re.compile('^[ \t]*#include[ \t]+"(.*)"[ \t]*$')
included = set()
excluded = set()
def find_header(name, abs_path, include_paths):
samedir = path.join(path.dirname(abs_path), name)
if path.exists(samedir):
@@ -46,31 +45,17 @@ def find_header(name, abs_path, include_paths):
return include_path
return None
def expand_include(
include_path,
f,
abs_path,
source_out,
header_out,
include_paths,
public_include_paths,
):
def expand_include(include_path, f, abs_path, source_out, header_out, include_paths, public_include_paths):
if include_path in included:
return False
included.add(include_path)
with open(include_path) as f:
print(f'#line 1 "{include_path}"', file=source_out)
process_file(
f, include_path, source_out, header_out, include_paths, public_include_paths
)
print('#line 1 "{}"'.format(include_path), file=source_out)
process_file(f, include_path, source_out, header_out, include_paths, public_include_paths)
return True
def process_file(
f, abs_path, source_out, header_out, include_paths, public_include_paths
):
def process_file(f, abs_path, source_out, header_out, include_paths, public_include_paths):
for (line, text) in enumerate(f):
m = include_re.match(text)
if m:
@@ -82,15 +67,7 @@ def process_file(
source_out.write(text)
expanded = False
else:
expanded = expand_include(
include_path,
f,
abs_path,
source_out,
header_out,
include_paths,
public_include_paths,
)
expanded = expand_include(include_path, f, abs_path, source_out, header_out, include_paths, public_include_paths)
else:
# now try public headers
include_path = find_header(filename, abs_path, public_include_paths)
@@ -100,52 +77,23 @@ def process_file(
if include_path in excluded:
source_out.write(text)
else:
expand_include(
include_path,
f,
abs_path,
header_out,
None,
public_include_paths,
[],
)
expand_include(include_path, f, abs_path, header_out, None, public_include_paths, [])
else:
sys.exit(
"unable to find {}, included in {} on line {}".format(
filename, abs_path, line
)
)
sys.exit("unable to find {}, included in {} on line {}".format(filename, abs_path, line))
if expanded:
print(f'#line {line + 1} "{abs_path}"', file=source_out)
print('#line {} "{}"'.format(line+1, abs_path), file=source_out)
elif text != "#pragma once\n":
source_out.write(text)
def main():
parser = argparse.ArgumentParser(
description="Transform a unity build into an amalgamation"
)
parser = argparse.ArgumentParser(description="Transform a unity build into an amalgamation")
parser.add_argument("source", help="source file")
parser.add_argument(
"-I",
action="append",
dest="include_paths",
help="include paths for private headers",
)
parser.add_argument(
"-i",
action="append",
dest="public_include_paths",
help="include paths for public headers",
)
parser.add_argument(
"-x", action="append", dest="excluded", help="excluded header files"
)
parser.add_argument("-I", action="append", dest="include_paths", help="include paths for private headers")
parser.add_argument("-i", action="append", dest="public_include_paths", help="include paths for public headers")
parser.add_argument("-x", action="append", dest="excluded", help="excluded header files")
parser.add_argument("-o", dest="source_out", help="output C++ file", required=True)
parser.add_argument(
"-H", dest="header_out", help="output C++ header file", required=True
)
parser.add_argument("-H", dest="header_out", help="output C++ header file", required=True)
args = parser.parse_args()
include_paths = list(map(path.abspath, args.include_paths or []))
@@ -153,15 +101,10 @@ def main():
excluded.update(map(path.abspath, args.excluded or []))
filename = args.source
abs_path = path.abspath(filename)
with open(filename) as f, open(args.source_out, "w") as source_out, open(
args.header_out, "w"
) as header_out:
print(f'#line 1 "{filename}"', file=source_out)
print(f'#include "{header_out.name}"', file=source_out)
process_file(
f, abs_path, source_out, header_out, include_paths, public_include_paths
)
with open(filename) as f, open(args.source_out, 'w') as source_out, open(args.header_out, 'w') as header_out:
print('#line 1 "{}"'.format(filename), file=source_out)
print('#include "{}"'.format(header_out.name), file=source_out)
process_file(f, abs_path, source_out, header_out, include_paths, public_include_paths)
if __name__ == "__main__":
main()
-238
View File
@@ -1,238 +0,0 @@
#!/usr/bin/env python3
# Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
# This source code is licensed under both the GPLv2 (found in the
# COPYING file in the root directory) and Apache 2.0 License
# (found in the LICENSE.Apache file in the root directory).
"""Access the results of benchmark runs
Send these results on to OpenSearch graphing service
"""
import argparse
import itertools
import logging
import os
import re
import sys
import requests
from dateutil import parser
logging.basicConfig(level=logging.DEBUG)
class Configuration:
opensearch_user = os.environ["ES_USER"]
opensearch_pass = os.environ["ES_PASS"]
class BenchmarkResultException(Exception):
def __init__(self, message, content):
super().__init__(self, message)
self.content = content
class BenchmarkUtils:
expected_keys = [
"ops_sec",
"mb_sec",
"lsm_sz",
"blob_sz",
"c_wgb",
"w_amp",
"c_mbps",
"c_wsecs",
"c_csecs",
"b_rgb",
"b_wgb",
"usec_op",
"p50",
"p99",
"p99.9",
"p99.99",
"pmax",
"uptime",
"stall%",
"Nstall",
"u_cpu",
"s_cpu",
"rss",
"test",
"date",
"version",
"job_id",
]
def sanity_check(row):
if "test" not in row:
logging.debug(f"not 'test' in row: {row}")
return False
if row["test"] == "":
logging.debug(f"row['test'] == '': {row}")
return False
if "date" not in row:
logging.debug(f"not 'date' in row: {row}")
return False
if "ops_sec" not in row:
logging.debug(f"not 'ops_sec' in row: {row}")
return False
try:
_ = int(row["ops_sec"])
except (ValueError, TypeError):
logging.debug(f"int(row['ops_sec']): {row}")
return False
try:
(_, _) = parser.parse(row["date"], fuzzy_with_tokens=True)
except (parser.ParserError):
logging.error(
f"parser.parse((row['date']): not a valid format for date in row: {row}"
)
return False
return True
def conform_opensearch(row):
(dt, _) = parser.parse(row["date"], fuzzy_with_tokens=True)
# create a test_date field, which was previously what was expected
# repair the date field, which has what can be a WRONG ISO FORMAT, (no leading 0 on single-digit day-of-month)
# e.g. 2022-07-1T00:14:55 should be 2022-07-01T00:14:55
row["test_date"] = dt.isoformat()
row["date"] = dt.isoformat()
return {key.replace(".", "_"): value for key, value in row.items()}
class ResultParser:
def __init__(self, field=r"(\w|[+-:.%])+", intrafield=r"(\s)+", separator="\t"):
self.field = re.compile(field)
self.intra = re.compile(intrafield)
self.sep = re.compile(separator)
def ignore(self, l_in: str):
if len(l_in) == 0:
return True
if l_in[0:1] == "#":
return True
return False
def line(self, line_in: str):
"""Parse a line into items
Being clever about separators
"""
line = line_in
row = []
while line != "":
match_item = self.field.match(line)
if match_item:
item = match_item.group(0)
row.append(item)
line = line[len(item) :]
else:
match_intra = self.intra.match(line)
if match_intra:
intra = match_intra.group(0)
# Count the separators
# If there are >1 then generate extra blank fields
# White space with no true separators fakes up a single separator
tabbed = self.sep.split(intra)
sep_count = len(tabbed) - 1
if sep_count == 0:
sep_count = 1
for _ in range(sep_count - 1):
row.append("")
line = line[len(intra) :]
else:
raise BenchmarkResultException(
"Invalid TSV line", f"{line_in} at {line}"
)
return row
def parse(self, lines):
"""Parse something that iterates lines"""
rows = [self.line(line) for line in lines if not self.ignore(line)]
header = rows[0]
width = len(header)
records = [
{k: v for (k, v) in itertools.zip_longest(header, row[:width])}
for row in rows[1:]
]
return records
def load_report_from_tsv(filename: str):
file = open(filename)
contents = file.readlines()
file.close()
parser = ResultParser()
report = parser.parse(contents)
logging.debug(f"Loaded TSV Report: {report}")
return report
def push_report_to_opensearch(report, esdocument):
sanitized = [
BenchmarkUtils.conform_opensearch(row)
for row in report
if BenchmarkUtils.sanity_check(row)
]
logging.debug(
f"upload {len(sanitized)} sane of {len(report)} benchmarks to opensearch"
)
for single_benchmark in sanitized:
logging.debug(f"upload benchmark: {single_benchmark}")
response = requests.post(
esdocument,
json=single_benchmark,
auth=(os.environ["ES_USER"], os.environ["ES_PASS"]),
)
logging.debug(
f"Sent to OpenSearch, status: {response.status_code}, result: {response.text}"
)
response.raise_for_status()
def push_report_to_null(report):
for row in report:
if BenchmarkUtils.sanity_check(row):
logging.debug(f"row {row}")
conformed = BenchmarkUtils.conform_opensearch(row)
logging.debug(f"conformed row {conformed}")
def main():
"""Tool for fetching, parsing and uploading benchmark results to OpenSearch / ElasticSearch
This tool will
(1) Open a local tsv benchmark report file
(2) Upload to OpenSearch document, via https/JSON
"""
parser = argparse.ArgumentParser(description="CircleCI benchmark scraper.")
# --tsvfile is the name of the file to read results from
# --esdocument is the ElasticSearch document to push these results into
#
parser.add_argument(
"--tsvfile",
default="build_tools/circle_api_scraper_input.txt",
help="File from which to read tsv report",
)
parser.add_argument(
"--esdocument",
help="ElasticSearch/OpenSearch document URL to upload report into",
)
parser.add_argument(
"--upload", choices=["opensearch", "none"], default="opensearch"
)
args = parser.parse_args()
logging.debug(f"Arguments: {args}")
reports = load_report_from_tsv(args.tsvfile)
if args.upload == "opensearch":
push_report_to_opensearch(reports, args.esdocument)
else:
push_report_to_null(reports)
if __name__ == "__main__":
sys.exit(main())
+142 -596
View File
@@ -1,4 +1,4 @@
#!/usr/bin/env bash
#!/bin/sh
#
# Detects OS we're compiling on and outputs a file specified by the first
# argument, which in turn gets read while processing Makefile.
@@ -9,7 +9,6 @@
# PLATFORM_LDFLAGS Linker flags
# JAVA_LDFLAGS Linker flags for RocksDBJava
# JAVA_STATIC_LDFLAGS Linker flags for RocksDBJava static build
# JAVAC_ARGS Arguments for javac
# PLATFORM_SHARED_EXT Extension for shared libraries
# PLATFORM_SHARED_LDFLAGS Flags for building shared library
# PLATFORM_SHARED_CFLAGS Flags for compiling objects for shared library
@@ -17,26 +16,25 @@
# PLATFORM_CXXFLAGS C++ compiler flags. Will contain:
# PLATFORM_SHARED_VERSIONED Set to 'true' if platform supports versioned
# shared libraries, empty otherwise.
# FIND Command for the find utility
# WATCH Command for the watch utility
#
# The PLATFORM_CCFLAGS and PLATFORM_CXXFLAGS might include the following:
#
# -DROCKSDB_PLATFORM_POSIX if posix-platform based
# -DLEVELDB_PLATFORM_POSIX if cstdatomic is present
# -DLEVELDB_PLATFORM_NOATOMIC if it is not
# -DSNAPPY if the Snappy library is present
# -DLZ4 if the LZ4 library is present
# -DZSTD if the ZSTD library is present
# -DNUMA if the NUMA library is present
# -DMEMKIND if the memkind library is present
#
# Using gflags in rocksdb:
# Our project depends on gflags, which requires users to take some extra steps
# before they can compile the whole repository:
# 1. Install gflags. You may download it from here:
# https://gflags.github.io/gflags/ (Mac users can `brew install gflags`)
# 2. Once installed, add the include path for gflags to your CPATH env var and
# the lib path to LIBRARY_PATH. If installed with default settings, the lib
# will be /usr/local/lib and the include path will be /usr/local/include
# https://code.google.com/p/gflags/
# 2. Once install, add the include path/lib path for gflags to CPATH and
# LIBRARY_PATH respectively. If installed with default mode, the
# lib and include path will be /usr/local/lib and /usr/local/include
# Mac user can do this by having brew installed and running brew install gflags
OUTPUT=$1
if test -z "$OUTPUT"; then
@@ -44,33 +42,17 @@ if test -z "$OUTPUT"; then
exit 1
fi
# we depend on C++20, but should be compatible with newer standards
if [ "$ROCKSDB_CXX_STANDARD" ]; then
PLATFORM_CXXFLAGS="-std=$ROCKSDB_CXX_STANDARD"
else
PLATFORM_CXXFLAGS="-std=c++20"
fi
# we depend on C++11
PLATFORM_CXXFLAGS="-std=c++11"
# we currently depend on POSIX platform
COMMON_FLAGS="-DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX"
# Default to fbcode gcc on Meta internal machines
IS_META_HOST="$(hostname | grep -E '(facebook|meta).com|fbinfra.net')"
if [ -z "$ROCKSDB_NO_FBCODE" -a "$IS_META_HOST" ]; then
if [ -d /mnt/gvfs/third-party ]; then
echo "NOTE: Using fbcode build" >&2
# Default to fbcode gcc on internal fb machines
if [ -z "$ROCKSDB_NO_FBCODE" -a -d /mnt/gvfs/third-party ]; then
FBCODE_BUILD="true"
# If we're compiling with TSAN or shared lib, we need pic build
# If we're compiling with TSAN we need pic build
PIC_BUILD=$COMPILE_WITH_TSAN
if [ "$LIB_MODE" == "shared" ]; then
PIC_BUILD=1
fi
source "$PWD/build_tools/fbcode_config_platform010.sh"
else
echo "************************************************************************" >&2
echo "WARNING: -d /mnt/gvfs/third-party failed; no fbcode build" >&2
echo "************************************************************************" >&2
fi
source "$PWD/build_tools/fbcode_config.sh"
fi
# Delete existing output, if it exists
@@ -78,39 +60,11 @@ rm -f "$OUTPUT"
touch "$OUTPUT"
if test -z "$CC"; then
if [ "$USE_CLANG" -a -x "$(command -v clang)" ]; then
CC=clang
elif [ -x "$(command -v cc)" ]; then
CC=cc
elif [ -x "$(command -v clang)" ]; then
CC=clang
else
CC=cc
fi
CC=cc
fi
if test -z "$CXX"; then
if [ "$USE_CLANG" -a -x "$(command -v clang++)" ]; then
CXX=clang++
elif [ -x "$(command -v g++)" ]; then
CXX=g++
elif [ -x "$(command -v clang++)" ]; then
CXX=clang++
else
CXX=g++
fi
fi
if test -z "$AR"; then
if [ "$USE_CLANG" -a -x "$(command -v llvm-ar)" ]; then
AR=llvm-ar
elif [ -x "$(command -v gcc-ar)" ]; then
AR=gcc-ar
elif [ -x "$(command -v llvm-ar)" ]; then
AR=llvm-ar
else
AR=ar
fi
CXX=g++
fi
# Detect OS
@@ -127,15 +81,7 @@ if test -z "$CLANG_SCAN_BUILD"; then
fi
if test -z "$CLANG_ANALYZER"; then
CLANG_ANALYZER=$(command -v clang++ 2> /dev/null)
fi
if test -z "$FIND"; then
FIND=find
fi
if test -z "$WATCH"; then
WATCH=watch
CLANG_ANALYZER=$(which clang++ 2> /dev/null)
fi
COMMON_FLAGS="$COMMON_FLAGS ${CFLAGS}"
@@ -147,24 +93,6 @@ PLATFORM_SHARED_LDFLAGS="-Wl,--no-as-needed -shared -Wl,-soname -Wl,"
PLATFORM_SHARED_CFLAGS="-fPIC"
PLATFORM_SHARED_VERSIONED=true
# Prefer lld linker when available on Linux. lld is typically 5-10x faster
# than the default ld.bfd for large C++ projects. macOS uses ld64 (or
# ld-prime) which is already fast, so we skip lld detection there.
# Set ROCKSDB_NO_FAST_LINKER=1 to disable this auto-detection.
if [ -z "$ROCKSDB_NO_FAST_LINKER" ] && [ "$TARGET_OS" = "Linux" ]; then
if $CXX -fuse-ld=lld -L/usr/local/lib -x c++ - -o /dev/null 2>/dev/null <<EOF
int main() { return 0; }
EOF
then
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -fuse-ld=lld"
# Ensure lld can find libraries in /usr/local/lib (lld does not
# search there by default, unlike ld.bfd)
if [ -d /usr/local/lib ]; then
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -L/usr/local/lib"
fi
fi
fi
# generic port files (working on all platform by #ifdef) go directly in /port
GENERIC_PORT_FILES=`cd "$ROCKSDB_ROOT"; find port -name '*.cc' | tr "\n" " "`
@@ -179,7 +107,7 @@ case "$TARGET_OS" in
;;
IOS)
PLATFORM=IOS
COMMON_FLAGS="$COMMON_FLAGS -DOS_MACOSX -DIOS_CROSS_COMPILE "
COMMON_FLAGS="$COMMON_FLAGS -DOS_MACOSX -DIOS_CROSS_COMPILE -DROCKSDB_LITE"
PLATFORM_SHARED_EXT=dylib
PLATFORM_SHARED_LDFLAGS="-dynamiclib -install_name "
CROSS_COMPILE=true
@@ -190,43 +118,22 @@ case "$TARGET_OS" in
COMMON_FLAGS="$COMMON_FLAGS -DOS_LINUX"
if [ -z "$USE_CLANG" ]; then
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp"
else
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -latomic"
fi
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt -ldl"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt"
# PORT_FILES=port/linux/linux_specific.cc
;;
SunOS)
PLATFORM=OS_SOLARIS
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_SOLARIS -m64"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt -static-libstdc++ -static-libgcc -m64"
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_SOLARIS"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt"
# PORT_FILES=port/sunos/sunos_specific.cc
;;
AIX)
PLATFORM=OS_AIX
CC=gcc
COMMON_FLAGS="$COMMON_FLAGS -maix64 -pthread -fno-builtin-memcmp -D_REENTRANT -DOS_AIX -D__STDC_FORMAT_MACROS"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -pthread -lpthread -lrt -maix64 -static-libstdc++ -static-libgcc"
# PORT_FILES=port/aix/aix_specific.cc
;;
FreeBSD)
PLATFORM=OS_FREEBSD
CXX=clang++
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_FREEBSD"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread"
# PORT_FILES=port/freebsd/freebsd_specific.cc
;;
GNU/kFreeBSD)
PLATFORM=OS_GNU_KFREEBSD
COMMON_FLAGS="$COMMON_FLAGS -DOS_GNU_KFREEBSD"
if [ -z "$USE_CLANG" ]; then
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp"
else
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -latomic"
fi
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt"
# PORT_FILES=port/gnu_kfreebsd/gnu_kfreebsd_specific.cc
;;
NetBSD)
PLATFORM=OS_NETBSD
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_NETBSD"
@@ -235,12 +142,9 @@ case "$TARGET_OS" in
;;
OpenBSD)
PLATFORM=OS_OPENBSD
CXX=clang++
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_OPENBSD"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -pthread"
# PORT_FILES=port/openbsd/openbsd_specific.cc
FIND=gfind
WATCH=gnuwatch
;;
DragonFly)
PLATFORM=OS_DRAGONFLYBSD
@@ -255,15 +159,13 @@ case "$TARGET_OS" in
COMMON_FLAGS="$COMMON_FLAGS -DCYGWIN"
if [ -z "$USE_CLANG" ]; then
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp"
else
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -latomic"
fi
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt"
# PORT_FILES=port/linux/linux_specific.cc
;;
OS_ANDROID_CROSSCOMPILE)
PLATFORM=OS_ANDROID
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_ANDROID -DROCKSDB_PLATFORM_POSIX"
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_ANDROID -DLEVELDB_PLATFORM_POSIX"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS " # All pthread features are in the Android C library
# PORT_FILES=port/android/android.cc
CROSS_COMPILE=true
@@ -276,375 +178,150 @@ esac
PLATFORM_CXXFLAGS="$PLATFORM_CXXFLAGS ${CXXFLAGS}"
JAVA_LDFLAGS="$PLATFORM_LDFLAGS"
JAVA_STATIC_LDFLAGS="$PLATFORM_LDFLAGS"
JAVAC_ARGS="-source 8"
if [ "$CROSS_COMPILE" = "true" -o "$FBCODE_BUILD" = "true" ]; then
# Cross-compiling; do not try any compilation tests.
# Also don't need any compilation tests if compiling on fbcode
if [ "$FBCODE_BUILD" = "true" ]; then
# Enable backtrace on fbcode since the necessary libraries are present
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_BACKTRACE"
FOLLY_DIR="third-party/folly"
fi
true
else
if ! test $ROCKSDB_DISABLE_FALLOCATE; then
# Test whether fallocate is available
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <fcntl.h>
#include <linux/falloc.h>
int main() {
int fd = open("/dev/null", 0);
fallocate(fd, FALLOC_FL_KEEP_SIZE, 0, 1024);
}
# Test whether fallocate is available
$CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <fcntl.h>
#include <linux/falloc.h>
int main() {
int fd = open("/dev/null", 0);
fallocate(fd, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE, 0, 1024);
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_FALLOCATE_PRESENT"
fi
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_FALLOCATE_PRESENT"
fi
if ! test $ROCKSDB_DISABLE_SNAPPY; then
# Test whether Snappy library is installed
# http://code.google.com/p/snappy/
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <snappy.h>
int main() {}
# Test whether Snappy library is installed
# http://code.google.com/p/snappy/
$CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <snappy.h>
int main() {}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DSNAPPY"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lsnappy"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lsnappy"
fi
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DSNAPPY"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lsnappy"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lsnappy"
fi
if ! test $ROCKSDB_DISABLE_GFLAGS; then
# Test whether gflags library is installed
# http://gflags.github.io/gflags/
# check if the namespace is gflags
if $CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null << EOF
#include <gflags/gflags.h>
using namespace GFLAGS_NAMESPACE;
int main() {}
# Test whether gflags library is installed
# http://gflags.github.io/gflags/
# check if the namespace is gflags
$CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
#include <gflags/gflags.h>
using namespace gflags;
int main() {}
EOF
then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=1"
# Hack: don't link extra gflags assuming it comes with folly
[ "$USE_FOLLY" ] || PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
# check if namespace is gflags
elif $CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null << EOF
#include <gflags/gflags.h>
using namespace gflags;
int main() {}
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=gflags"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
else
# check if namespace is google
$CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
#include <gflags/gflags.h>
using namespace google;
int main() {}
EOF
then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=1 -DGFLAGS_NAMESPACE=gflags"
# Hack: don't link extra gflags assuming it comes with folly
[ "$USE_FOLLY" ] || PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
# check if namespace is google
elif $CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null << EOF
#include <gflags/gflags.h>
using namespace google;
int main() {}
EOF
then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=1 -DGFLAGS_NAMESPACE=google"
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=google"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
fi
fi
fi
if ! test $ROCKSDB_DISABLE_ZLIB; then
# Test whether zlib library is installed
$CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <zlib.h>
int main() {}
# Test whether zlib library is installed
$CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <zlib.h>
int main() {}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DZLIB"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lz"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lz"
fi
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DZLIB"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lz"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lz"
fi
if ! test $ROCKSDB_DISABLE_BZIP; then
# Test whether bzip library is installed
$CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <bzlib.h>
int main() {}
# Test whether bzip library is installed
$CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <bzlib.h>
int main() {}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DBZIP2"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lbz2"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lbz2"
fi
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DBZIP2"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lbz2"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lbz2"
fi
if ! test $ROCKSDB_DISABLE_LZ4; then
# Test whether lz4 library is installed
$CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <lz4.h>
#include <lz4hc.h>
int main() {}
# Test whether lz4 library is installed
$CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <lz4.h>
#include <lz4hc.h>
int main() {}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DLZ4"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -llz4"
JAVA_LDFLAGS="$JAVA_LDFLAGS -llz4"
fi
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DLZ4"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -llz4"
JAVA_LDFLAGS="$JAVA_LDFLAGS -llz4"
fi
if ! test $ROCKSDB_DISABLE_ZSTD; then
# Test whether zstd library is installed with minimum version
# (Keep in sync with compression.h)
$CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <zstd.h>
#if ZSTD_VERSION_NUMBER < 10400
#error "ZSTD support requires version >= 1.4.0 (libzstd-devel)"
#endif // ZSTD_VERSION_NUMBER
int main() {}
# Test whether zstd library is installed
$CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <zstd.h>
int main() {}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DZSTD"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lzstd"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lzstd"
fi
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DZSTD"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lzstd"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lzstd"
fi
if ! test $ROCKSDB_DISABLE_NUMA; then
# Test whether numa is available
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o -lnuma 2>/dev/null <<EOF
#include <numa.h>
#include <numaif.h>
int main() {}
# Test whether numa is available
$CXX $CFLAGS -x c++ - -o /dev/null -lnuma 2>/dev/null <<EOF
#include <numa.h>
#inlcude <numaif.h>
int main() {}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DNUMA"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lnuma"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lnuma"
fi
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DNUMA"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lnuma"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lnuma"
fi
if ! test $ROCKSDB_DISABLE_JEMALLOC; then
# Test whether jemalloc is available
if echo 'int main() {}' | $CXX $PLATFORM_CXXFLAGS $LDFLAGS -x c++ - -o test.o -ljemalloc \
2>/dev/null; then
# This will enable some preprocessor identifiers in the Makefile
JEMALLOC=1
# JEMALLOC can be enabled either using the flag (like here) or by
# providing direct link to the jemalloc library
WITH_JEMALLOC_FLAG=1
# check for JEMALLOC installed with HomeBrew
if [ "$PLATFORM" == "OS_MACOSX" ]; then
if [ "$TARGET_ARCHITECTURE" = "arm64" ]; then
# on M1 Macs, homebrew installs here instead of /usr/local
JEMALLOC_PREFIX="/opt/homebrew"
else
JEMALLOC_PREFIX="/usr/local"
fi
if hash brew 2>/dev/null && brew ls --versions jemalloc > /dev/null; then
JEMALLOC_VER=$(brew ls --versions jemalloc | tail -n 1 | cut -f 2 -d ' ')
JEMALLOC_INCLUDE="-I${JEMALLOC_PREFIX}/Cellar/jemalloc/${JEMALLOC_VER}/include"
JEMALLOC_LIB="${JEMALLOC_PREFIX}/Cellar/jemalloc/${JEMALLOC_VER}/lib/libjemalloc_pic.a"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -L${JEMALLOC_PREFIX}/lib $JEMALLOC_LIB"
JAVA_LDFLAGS="$JAVA_LDFLAGS -L${JEMALLOC_PREFIX}/lib $JEMALLOC_LIB"
JAVA_STATIC_LDFLAGS="$JAVA_STATIC_LDFLAGS -L${JEMALLOC_PREFIX}/lib $JEMALLOC_LIB"
fi
fi
fi
fi
if ! test $JEMALLOC && ! test $ROCKSDB_DISABLE_TCMALLOC; then
# Test whether jemalloc is available
if echo 'int main() {}' | $CXX $CFLAGS -x c++ - -o /dev/null -ljemalloc \
2>/dev/null; then
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -ljemalloc"
JAVA_LDFLAGS="$JAVA_LDFLAGS -ljemalloc"
else
# jemalloc is not available. Let's try tcmalloc
if echo 'int main() {}' | $CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o \
-ltcmalloc 2>/dev/null; then
if echo 'int main() {}' | $CXX $CFLAGS -x c++ - -o /dev/null \
-ltcmalloc 2>/dev/null; then
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -ltcmalloc"
JAVA_LDFLAGS="$JAVA_LDFLAGS -ltcmalloc"
fi
fi
if ! test $ROCKSDB_DISABLE_MALLOC_USABLE_SIZE; then
# Test whether malloc_usable_size is available
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <malloc.h>
int main() {
size_t res = malloc_usable_size(0);
(void)res;
return 0;
}
# Test whether malloc_usable_size is available
$CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <malloc.h>
int main() {
size_t res = malloc_usable_size(0);
return 0;
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_MALLOC_USABLE_SIZE"
fi
fi
if ! test $ROCKSDB_DISABLE_MEMKIND; then
# Test whether memkind library is installed
$CXX $PLATFORM_CXXFLAGS $LDFLAGS -x c++ - -o test.o -lmemkind 2>/dev/null <<EOF
#include <memkind.h>
int main() {
memkind_malloc(MEMKIND_DAX_KMEM, 1024);
return 0;
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DMEMKIND"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lmemkind"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lmemkind"
fi
fi
if ! test $ROCKSDB_DISABLE_PTHREAD_MUTEX_ADAPTIVE_NP; then
# Test whether PTHREAD_MUTEX_ADAPTIVE_NP mutex type is available
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <pthread.h>
int main() {
int x = PTHREAD_MUTEX_ADAPTIVE_NP;
(void)x;
return 0;
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_PTHREAD_ADAPTIVE_MUTEX"
fi
fi
if ! test $ROCKSDB_DISABLE_BACKTRACE; then
# Test whether backtrace is available
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <execinfo.h>
int main() {
void* frames[1];
backtrace_symbols(frames, backtrace(frames, 1));
return 0;
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_BACKTRACE"
else
# Test whether execinfo library is installed
$CXX $PLATFORM_CXXFLAGS -lexecinfo -x c++ - -o test.o 2>/dev/null <<EOF
#include <execinfo.h>
int main() {
void* frames[1];
backtrace_symbols(frames, backtrace(frames, 1));
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_BACKTRACE"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lexecinfo"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lexecinfo"
fi
fi
fi
if ! test $ROCKSDB_DISABLE_PG; then
# Test if -pg is supported
$CXX $PLATFORM_CXXFLAGS -pg -x c++ - -o test.o 2>/dev/null <<EOF
int main() {
return 0;
}
EOF
if [ "$?" = 0 ]; then
PROFILING_FLAGS=-pg
fi
fi
if ! test $ROCKSDB_DISABLE_SYNC_FILE_RANGE; then
# Test whether sync_file_range is supported for compatibility with an old glibc
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <fcntl.h>
int main() {
int fd = open("/dev/null", 0);
sync_file_range(fd, 0, 1024, SYNC_FILE_RANGE_WRITE);
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_RANGESYNC_PRESENT"
fi
fi
if ! test $ROCKSDB_DISABLE_SCHED_GETCPU; then
# Test whether sched_getcpu is supported
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <sched.h>
int main() {
int cpuid = sched_getcpu();
(void)cpuid;
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_SCHED_GETCPU_PRESENT"
fi
fi
if ! test $ROCKSDB_DISABLE_AUXV_GETAUXVAL; then
# Test whether getauxval is supported
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <sys/auxv.h>
int main() {
uint64_t auxv = getauxval(AT_HWCAP);
(void)auxv;
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_AUXV_GETAUXVAL_PRESENT"
fi
fi
if ! test $ROCKSDB_DISABLE_ALIGNED_NEW; then
# Test whether c++17 aligned-new is supported
$CXX $PLATFORM_CXXFLAGS -faligned-new -x c++ - -o test.o 2>/dev/null <<EOF
struct alignas(1024) t {int a;};
int main() {}
EOF
if [ "$?" = 0 ]; then
PLATFORM_CXXFLAGS="$PLATFORM_CXXFLAGS -faligned-new -DHAVE_ALIGNED_NEW"
fi
fi
if ! test $ROCKSDB_DISABLE_BENCHMARK; then
# Test whether google benchmark is available
$CXX $PLATFORM_CXXFLAGS -x c++ - -o /dev/null -lbenchmark -lpthread 2>/dev/null <<EOF
#include <benchmark/benchmark.h>
int main() {}
EOF
if [ "$?" = 0 ]; then
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lbenchmark"
fi
fi
if test $USE_FOLLY || test $USE_FOLLY_LITE; then
# Test whether libfolly library is installed
$CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <folly/synchronization/DistributedMutex.h>
int main() {}
EOF
if [ "$?" != 0 ]; then
FOLLY_DIR="./third-party/folly"
fi
fi
if test -z "$ROCKSDB_USE_IO_URING"; then
ROCKSDB_USE_IO_URING=1
fi
if [ "$ROCKSDB_USE_IO_URING" -ne 0 -a "$PLATFORM" = OS_LINUX ]; then
# check for liburing
$CXX $PLATFORM_CXXFLAGS -x c++ - -luring -o test.o 2>/dev/null <<EOF
#include <liburing.h>
int main() {
struct io_uring ring;
io_uring_queue_init(1, &ring, 0);
return 0;
}
EOF
if [ "$?" = 0 ]; then
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -luring"
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_IOURING_PRESENT"
fi
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_MALLOC_USABLE_SIZE"
fi
fi
# TODO(tec): Fix -Wshorten-64-to-32 errors on FreeBSD and enable the warning.
# -Wshorten-64-to-32 breaks compilation on FreeBSD aarch64 and i386
if ! { [ "$TARGET_OS" = FreeBSD -o "$TARGET_OS" = OpenBSD ] && [ "$TARGET_ARCHITECTURE" = arm64 -o "$TARGET_ARCHITECTURE" = i386 ]; }; then
# -Wshorten-64-to-32 breaks compilation on FreeBSD i386
if ! [ "$TARGET_OS" = FreeBSD -a "$TARGET_ARCHITECTURE" = i386 ]; then
# Test whether -Wshorten-64-to-32 is available
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o -Wshorten-64-to-32 2>/dev/null <<EOF
$CXX $CFLAGS -x c++ - -o /dev/null -Wshorten-64-to-32 2>/dev/null <<EOF
int main() {}
EOF
if [ "$?" = 0 ]; then
@@ -652,134 +329,32 @@ EOF
fi
fi
if [ "$PORTABLE" == "" ] || [ "$PORTABLE" == 0 ]; then
if test -n "`echo $TARGET_ARCHITECTURE | grep ^ppc64`"; then
# Tune for this POWER processor, treating '+' models as base models
POWER=`LD_SHOW_AUXV=1 /bin/true | grep AT_PLATFORM | grep -E -o power[0-9]+`
COMMON_FLAGS="$COMMON_FLAGS -mcpu=$POWER -mtune=$POWER "
elif test -n "`echo $TARGET_ARCHITECTURE | grep -e^arm -e^aarch64`"; then
# TODO: Handle this with approprite options.
COMMON_FLAGS="$COMMON_FLAGS"
elif test -n "`echo $TARGET_ARCHITECTURE | grep ^aarch64`"; then
COMMON_FLAGS="$COMMON_FLAGS"
elif test -n "`echo $TARGET_ARCHITECTURE | grep ^s390x`"; then
if echo 'int main() {}' | $CXX $PLATFORM_CXXFLAGS -x c++ \
-march=native - -o /dev/null 2>/dev/null; then
COMMON_FLAGS="$COMMON_FLAGS -march=native "
else
COMMON_FLAGS="$COMMON_FLAGS -march=z196 "
fi
COMMON_FLAGS="$COMMON_FLAGS"
elif test -n "`echo $TARGET_ARCHITECTURE | grep ^riscv64`"; then
RISC_ISA=$(cat /proc/cpuinfo | grep -E '^isa\s*:' | head -1 | cut --delimiter=: -f 2 | cut -b 2-)
if [ -n "${RISCV_ISA}" ]; then
COMMON_FLAGS="$COMMON_FLAGS -march=${RISC_ISA}"
fi
elif [ "$TARGET_OS" == "IOS" ]; then
COMMON_FLAGS="$COMMON_FLAGS"
else
COMMON_FLAGS="$COMMON_FLAGS -march=native "
fi
else
# PORTABLE specified
if [ "$PORTABLE" == 1 ]; then
if test -n "`echo $TARGET_ARCHITECTURE | grep ^s390x`"; then
COMMON_FLAGS="$COMMON_FLAGS -march=z196 "
elif test -n "`echo $TARGET_ARCHITECTURE | grep ^riscv64`"; then
COMMON_FLAGS="$COMMON_FLAGS -march=rv64gc"
elif test "$USE_SSE"; then
# USE_SSE is DEPRECATED
# This is a rough approximation of the old USE_SSE behavior
COMMON_FLAGS="$COMMON_FLAGS -march=haswell"
fi
# Other than those cases, not setting -march= here.
else
# Assume PORTABLE is a minimum assumed cpu type, e.g. PORTABLE=haswell
COMMON_FLAGS="$COMMON_FLAGS -march=${PORTABLE}"
fi
# shall we use HDFS?
if [[ "${PLATFORM}" == "OS_MACOSX" ]]; then
# For portability compile for macOS 10.14 (2018) or newer
COMMON_FLAGS="$COMMON_FLAGS -mmacosx-version-min=10.14"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -mmacosx-version-min=10.14"
# -mmacosx-version-min must come first here.
PLATFORM_SHARED_LDFLAGS="-mmacosx-version-min=10.14 $PLATFORM_SHARED_LDFLAGS"
PLATFORM_CMAKE_FLAGS="-DCMAKE_OSX_DEPLOYMENT_TARGET=10.14"
JAVA_STATIC_DEPS_COMMON_FLAGS="-mmacosx-version-min=10.14"
JAVA_STATIC_DEPS_LDFLAGS="$JAVA_STATIC_DEPS_COMMON_FLAGS"
JAVA_STATIC_DEPS_CCFLAGS="$JAVA_STATIC_DEPS_COMMON_FLAGS"
JAVA_STATIC_DEPS_CXXFLAGS="$JAVA_STATIC_DEPS_COMMON_FLAGS"
if test "$USE_HDFS"; then
if test -z "$JAVA_HOME"; then
echo "JAVA_HOME has to be set for HDFS usage."
exit 1
fi
HDFS_CCFLAGS="$HDFS_CCFLAGS -I$JAVA_HOME/include -I$JAVA_HOME/include/linux -DUSE_HDFS"
HDFS_LDFLAGS="$HDFS_LDFLAGS -lhdfs -L$JAVA_HOME/jre/lib/amd64"
HDFS_LDFLAGS="$HDFS_LDFLAGS -L$JAVA_HOME/jre/lib/amd64/server -L$GLIBC_RUNTIME_PATH/lib"
HDFS_LDFLAGS="$HDFS_LDFLAGS -ldl -lverify -ljava -ljvm"
COMMON_FLAGS="$COMMON_FLAGS $HDFS_CCFLAGS"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS $HDFS_LDFLAGS"
JAVA_LDFLAGS="$JAVA_LDFLAGS $HDFS_LDFLAGS"
fi
if test -n "`echo $TARGET_ARCHITECTURE | grep ^ppc64`"; then
# check for GNU libc on ppc64
$CXX -x c++ - -o /dev/null 2>/dev/null <<EOF
#include <stdio.h>
#include <stdlib.h>
#include <gnu/libc-version.h>
int main(int argc, char *argv[]) {
printf("GNU libc version: %s\n", gnu_get_libc_version());
return 0;
}
EOF
if [ "$?" != 0 ]; then
PPC_LIBC_IS_GNU=0
fi
if [ "$TARGET_OS" = FreeBSD -a "$TARGET_ARCHITECTURE" = i386 ]; then
# Intel SSE instructions breaks compilation on FreeBSD i386
unset USE_SSE
fi
$CXX $PLATFORM_CXXFLAGS $COMMON_FLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <cstdint>
int main() {
uint64_t a = 0xffffFFFFffffFFFF;
__uint128_t b = __uint128_t(a) * a;
a = static_cast<uint64_t>(b >> 64);
(void)a;
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DHAVE_UINT128_EXTENSION"
fi
if [ "$FBCODE_BUILD" != "true" -a "$PLATFORM" = OS_LINUX ]; then
$CXX $COMMON_FLAGS $PLATFORM_SHARED_CFLAGS -x c++ -c - -o test_dl.o 2>/dev/null <<EOF
void dummy_func() {}
EOF
if [ "$?" = 0 ]; then
$CXX $COMMON_FLAGS $PLATFORM_SHARED_LDFLAGS test_dl.o -o test.o 2>/dev/null
if [ "$?" = 0 ]; then
EXEC_LDFLAGS+="-ldl"
rm -f test_dl.o
fi
fi
fi
# check for F_FULLFSYNC
$CXX $PLATFORM_CXXFALGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <fcntl.h>
int main() {
fcntl(0, F_FULLFSYNC);
return 0;
}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DHAVE_FULLFSYNC"
fi
rm -f test.o test_dl.o
# Get the path for the folly installation dir
if [ "$USE_FOLLY" ]; then
if [ "$FOLLY_DIR" ]; then
FOLLY_PATH=`cd $FOLLY_DIR && $PYTHON build/fbcode_builder/getdeps.py show-inst-dir folly`
fi
fi
if [ "$USE_FOLLY_LITE" ]; then
if [ "$FOLLY_DIR" ]; then
BOOST_SOURCE_PATH=`cd $FOLLY_DIR && $PYTHON build/fbcode_builder/getdeps.py show-source-dir boost`
FMT_SOURCE_PATH=`cd $FOLLY_DIR && $PYTHON build/fbcode_builder/getdeps.py show-source-dir fmt`
fi
if test "$USE_SSE"; then
# if Intel SSE instruction set is supported, set USE_SSE=1
COMMON_FLAGS="$COMMON_FLAGS -msse -msse4.2 "
elif test -z "$PORTABLE"; then
COMMON_FLAGS="$COMMON_FLAGS -march=native "
fi
PLATFORM_CCFLAGS="$PLATFORM_CCFLAGS $COMMON_FLAGS"
@@ -793,16 +368,10 @@ ROCKSDB_PATCH=`build_tools/version.sh patch`
echo "CC=$CC" >> "$OUTPUT"
echo "CXX=$CXX" >> "$OUTPUT"
echo "AR=$AR" >> "$OUTPUT"
echo "PLATFORM=$PLATFORM" >> "$OUTPUT"
echo "PLATFORM_LDFLAGS=$PLATFORM_LDFLAGS" >> "$OUTPUT"
echo "PLATFORM_CMAKE_FLAGS=$PLATFORM_CMAKE_FLAGS" >> "$OUTPUT"
echo "JAVA_LDFLAGS=$JAVA_LDFLAGS" >> "$OUTPUT"
echo "JAVA_STATIC_LDFLAGS=$JAVA_STATIC_LDFLAGS" >> "$OUTPUT"
echo "JAVA_STATIC_DEPS_CCFLAGS=$JAVA_STATIC_DEPS_CCFLAGS" >> "$OUTPUT"
echo "JAVA_STATIC_DEPS_CXXFLAGS=$JAVA_STATIC_DEPS_CXXFLAGS" >> "$OUTPUT"
echo "JAVA_STATIC_DEPS_LDFLAGS=$JAVA_STATIC_DEPS_LDFLAGS" >> "$OUTPUT"
echo "JAVAC_ARGS=$JAVAC_ARGS" >> "$OUTPUT"
echo "VALGRIND_VER=$VALGRIND_VER" >> "$OUTPUT"
echo "PLATFORM_CCFLAGS=$PLATFORM_CCFLAGS" >> "$OUTPUT"
echo "PLATFORM_CXXFLAGS=$PLATFORM_CXXFLAGS" >> "$OUTPUT"
@@ -818,26 +387,3 @@ echo "ROCKSDB_MINOR=$ROCKSDB_MINOR" >> "$OUTPUT"
echo "ROCKSDB_PATCH=$ROCKSDB_PATCH" >> "$OUTPUT"
echo "CLANG_SCAN_BUILD=$CLANG_SCAN_BUILD" >> "$OUTPUT"
echo "CLANG_ANALYZER=$CLANG_ANALYZER" >> "$OUTPUT"
echo "PROFILING_FLAGS=$PROFILING_FLAGS" >> "$OUTPUT"
echo "FIND=$FIND" >> "$OUTPUT"
echo "WATCH=$WATCH" >> "$OUTPUT"
echo "FOLLY_PATH=$FOLLY_PATH" >> "$OUTPUT"
echo "BOOST_SOURCE_PATH=$BOOST_SOURCE_PATH" >> "$OUTPUT"
echo "FMT_SOURCE_PATH=$FMT_SOURCE_PATH" >> "$OUTPUT"
# This will enable some related identifiers for the preprocessor
if test -n "$JEMALLOC"; then
echo "JEMALLOC=1" >> "$OUTPUT"
fi
# Indicates that jemalloc should be enabled using -ljemalloc flag
# The alternative is to porvide a direct link to the library via JEMALLOC_LIB
# and JEMALLOC_INCLUDE
if test -n "$WITH_JEMALLOC_FLAG"; then
echo "WITH_JEMALLOC_FLAG=$WITH_JEMALLOC_FLAG" >> "$OUTPUT"
fi
if test -n "$USE_FOLLY"; then
echo "USE_FOLLY=$USE_FOLLY" >> "$OUTPUT"
fi
if test -n "$PPC_LIBC_IS_GNU"; then
echo "PPC_LIBC_IS_GNU=$PPC_LIBC_IS_GNU" >> "$OUTPUT"
fi
-34
View File
@@ -1,34 +0,0 @@
#!/usr/bin/env bash
# Copyright (c) Meta Platforms, Inc. and affiliates.
# This source code is licensed under both the GPLv2 (found in the
# COPYING file in the root directory) and Apache 2.0 License
# (found in the LICENSE.Apache file in the root directory).
#
# Check for some simple mistakes in public headers (on the command line)
# that should prevent commit or push
BAD=""
# Look for potential for ODR violations caused by public headers depending on
# build parameters that could vary between RocksDB build and application build.
# * Cases like ROCKSDB_NAMESPACE, and ROCKSDB_ASSERT_STATUS_CHECKED are
# intentional, hard to avoid. (We expect definitions to change and the user
# should also.)
# * Cases like _WIN32, OS_WIN, and __cplusplus are essentially ODR-safe.
# * Cases like
# #ifdef BLAH // ODR-SAFE
# #undef BLAH
# #endif
# that should not cause ODR violations can be exempted with the ODR-SAFE
# marker recognized here.
grep -nHE '^#if' -- "$@" | grep -vE 'ROCKSDB_NAMESPACE|ROCKSDB_ASSERT_STATUS_CHECKED|_WIN32|OS_WIN|ODR-SAFE|__cplusplus|ROCKSDB_DLL|ROCKSDB_LIBRARY_EXPORTS'
if [ "$?" != "1" ]; then
echo "^^^^^ #if in public API could cause an ODR violation."
echo " Add // ODR-SAFE if verified safe."
BAD=1
fi
if [ "$BAD" ]; then
exit 1
fi
-48
View File
@@ -1,48 +0,0 @@
#!/usr/bin/env bash
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
#
# Check for some simple mistakes that should prevent commit or push
BAD=""
git grep -n 'namespace rocksdb' -- '*.[ch]*'
if [ "$?" != "1" ]; then
echo "^^^^^ Do not hardcode namespace rocksdb. Use ROCKSDB_NAMESPACE"
BAD=1
fi
git grep -n -i 'nocommit' -- ':!build_tools/check-sources.sh'
if [ "$?" != "1" ]; then
echo "^^^^^ Code was not intended to be committed"
BAD=1
fi
git grep -n 'include <rocksdb/' -- ':!build_tools/check-sources.sh'
if [ "$?" != "1" ]; then
echo '^^^^^ Use double-quotes as in #include "rocksdb/something.h"'
BAD=1
fi
git grep -n 'include "include/rocksdb/' -- ':!build_tools/check-sources.sh'
if [ "$?" != "1" ]; then
echo '^^^^^ Use #include "rocksdb/something.h" instead of #include "include/rocksdb/something.h"'
BAD=1
fi
git grep -n 'using namespace' -- ':!build_tools' ':!docs' \
':!third-party/folly/folly/lang/Align.h' \
':!third-party/gtest-1.8.1/fused-src/gtest/gtest.h'
if [ "$?" != "1" ]; then
echo '^^^^ Do not use "using namespace"'
BAD=1
fi
LC_ALL=C git grep -n $'[\x80-\xff]' -- ':!docs' ':!*.md' ':!.github'
if [ "$?" != "1" ]; then
echo '^^^^ Use only ASCII characters in source files'
BAD=1
fi
if [ "$BAD" ]; then
exit 1
fi
-42
View File
@@ -1,42 +0,0 @@
#!/usr/bin/env bash
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
#
# Validate GitHub Actions workflow YAML before it reaches CI runtime.
set -euo pipefail
if ! command -v ruby >/dev/null 2>&1; then
echo "ruby is required to validate GitHub Actions workflow YAML"
echo "On CentOS Stream: sudo dnf install ruby rubygems rubygem-psych"
exit 1
fi
if ! ruby -e 'require "psych"' 2>/dev/null; then
echo "ruby is installed but cannot load required library 'psych'"
echo "On CentOS Stream: sudo dnf install rubygems rubygem-psych"
exit 1
fi
ruby <<'RUBY'
require "psych"
bad = false
workflow_files = Dir[".github/workflows/*.{yml,yaml}"].sort
if workflow_files.empty?
warn "No workflow YAML files found under .github/workflows"
exit 1
end
workflow_files.each do |path|
begin
Psych.parse_file(path)
puts "OK #{path}"
rescue Psych::Exception => e
warn "Invalid YAML in #{path}: #{e.message}"
bad = true
end
end
exit(bad ? 1 : 0)
RUBY
-231
View File
@@ -1,231 +0,0 @@
#!/bin/bash
# Output test progress in JSON format for machine parsing
# Usage: build_tools/check_progress.sh
LOG_FILE="LOG"
T_DIR="t"
SRC_MK="src.mk"
# Maximum lines of test output to include per failed test
MAX_OUTPUT_LINES=50
# Helper to escape string for JSON (handles newlines, quotes, backslashes, tabs)
json_escape() {
local str="$1"
# Use python for reliable JSON escaping if available, otherwise use sed
if command -v python3 &>/dev/null; then
printf '%s' "$str" | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read())[1:-1], end="")'
else
printf '%s' "$str" | sed 's/\\/\\\\/g; s/"/\\"/g; s/\t/\\t/g; s/\r/\\r/g' | awk '{printf "%s\\n", $0}' | sed 's/\\n$//'
fi
}
# Helper to output JSON and exit
output_json() {
local status="$1"
local completed="${2:-0}"
local total="${3:-0}"
local failed="${4:-0}"
local percent="${5:-0}"
local eta="${6:-0}"
local avg_time="${7:-0}"
local last_item="${8:-}"
local phase="${9:-}"
local failed_tests="${10:-}"
# Build JSON output
local json="{\"status\":\"$status\""
if [[ -n "$phase" ]]; then
json="$json,\"phase\":\"$phase\""
fi
json="$json,\"completed\":$completed,\"total\":$total,\"failed\":$failed,\"percent\":$percent"
json="$json,\"eta_seconds\":$eta,\"avg_time\":\"$avg_time\",\"last_item\":\"$(json_escape "$last_item")\""
if [[ -n "$failed_tests" ]]; then
json="$json,\"failed_tests\":[$failed_tests]"
fi
json="$json}"
echo "$json"
}
# Get failed test info with log output
get_failed_tests_json() {
local log_file="$1"
local t_dir="$2"
local max_failures=10
local count=0
local first=true
# Get failed tests from LOG file
while IFS=$'\t' read -r seq host starttime runtime send recv exitval signal cmd; do
# Skip header line
[[ "$seq" == "Seq" ]] && continue
# Check if failed (exitval != 0 or signal != 0)
if [[ "$exitval" != "0" || "$signal" != "0" ]]; then
# Extract test name from command
test_name=$(echo "$cmd" | sed 's,.*/run-,,;s, .*,,')
# Get log file path
log_path="$t_dir/log-run-$test_name"
# Read test output (last N lines)
if [[ -f "$log_path" ]]; then
output=$(tail -n "$MAX_OUTPUT_LINES" "$log_path" 2>/dev/null)
else
output="(log file not found: $log_path)"
fi
# Escape output for JSON
escaped_output=$(json_escape "$output")
# Build JSON object for this failure
if [[ "$first" == "true" ]]; then
first=false
else
printf ","
fi
printf '{"test":"%s","exit_code":%d,"signal":%d,"output":"%s"}' \
"$test_name" "$exitval" "$signal" "$escaped_output"
((count++))
if [[ $count -ge $max_failures ]]; then
break
fi
fi
done < "$log_file"
}
# Check if tests are running (LOG file exists)
if [[ -f "$LOG_FILE" ]]; then
# Count total tests from t/run-* files
if [[ -d "$T_DIR" ]]; then
total=$(find "$T_DIR" -name 'run-*' -type f 2>/dev/null | wc -l)
else
total=0
fi
# If no parallel tests generated yet
if [[ "$total" -eq 0 ]]; then
output_json "running" 0 0 0 0 0 "0" "" "generating"
exit 0
fi
# Parse LOG file (skip header line)
# LOG format: Seq Host Starttime JobRuntime Send Receive Exitval Signal Command
completed=$(tail -n +2 "$LOG_FILE" 2>/dev/null | wc -l)
# Count failures
failed=$(awk -F'\t' 'NR>1 && ($7 != 0 || $8 != 0) {count++} END {print count+0}' "$LOG_FILE" 2>/dev/null)
# Get failed tests JSON with output (only if there are failures)
if [[ "$failed" -gt 0 ]]; then
failed_tests=$(get_failed_tests_json "$LOG_FILE" "$T_DIR")
else
failed_tests=""
fi
# Calculate percentage
if [[ "$total" -gt 0 ]]; then
percent=$((completed * 100 / total))
else
percent=0
fi
# Get last completed test name (extract from command column)
last_test=$(tail -1 "$LOG_FILE" 2>/dev/null | awk -F'\t' '{print $9}' | sed 's,.*/run-,,;s, .*,,;s,^./,,')
# Calculate ETA based on average time
if [[ "$completed" -gt 0 ]]; then
avg_time=$(awk -F'\t' 'NR>1 {sum+=$4; count++} END {if(count>0) printf "%.1f", sum/count; else print "0"}' "$LOG_FILE")
remaining=$((total - completed))
eta=$(awk "BEGIN {printf \"%.0f\", $avg_time * $remaining}")
else
avg_time="0"
eta="0"
fi
# Determine status
if [[ "$completed" -ge "$total" ]]; then
status="completed"
elif [[ "$completed" -gt 0 ]]; then
status="running"
else
status="starting"
fi
output_json "$status" "$completed" "$total" "$failed" "$percent" "$eta" "$avg_time" "$last_test" "testing" "$failed_tests"
exit 0
fi
# No LOG file - check if we're in compilation/linking phase
# Count expected source files from src.mk
if [[ -f "$SRC_MK" ]]; then
# Count LIB_SOURCES (library object files to compile)
expected_lib_objects=$(grep -E '\.cc\s*\\?$' "$SRC_MK" | grep -v '^#' | wc -l)
# Count TEST_MAIN_SOURCES (test binaries to link)
expected_test_binaries=$(sed -n '/^TEST_MAIN_SOURCES =/,/^[^ ]/p' "$SRC_MK" | grep -cE '\.cc\s*\\?$' 2>/dev/null || echo 0)
else
expected_lib_objects=0
expected_test_binaries=0
fi
# Check for test generation phase (t/ directory being created)
if [[ -d "$T_DIR" ]]; then
total=$(find "$T_DIR" -name 'run-*' -type f 2>/dev/null | wc -l)
if [[ "$total" -gt 0 ]]; then
output_json "running" 0 "$total" 0 0 0 "0" "" "generating"
exit 0
fi
fi
# Count compiled object files (in subdirectories matching source structure)
# Object files are created as dir/file.o (e.g., cache/cache.o, db/db_impl.o)
compiled_objects=0
if [[ "$expected_lib_objects" -gt 0 ]]; then
# Count .o files in source directories
compiled_objects=$(find cache db env file logging memory memtable monitoring options port table test_util trace_replay util utilities -name '*.o' -type f 2>/dev/null | wc -l)
fi
# Count linked test binaries (test binaries are in current directory with _test suffix)
linked_tests=0
if [[ "$expected_test_binaries" -gt 0 ]]; then
linked_tests=$(find . -maxdepth 1 -name '*_test' -type f -executable 2>/dev/null | wc -l)
fi
# Determine phase based on what exists
if [[ "$compiled_objects" -eq 0 && "$linked_tests" -eq 0 ]]; then
# Nothing compiled yet - not started or just beginning
output_json "not_started" 0 0 0 0 0 "0" ""
exit 0
fi
# Calculate total work units: compiling + linking
total_work=$((expected_lib_objects + expected_test_binaries))
completed_work=$((compiled_objects + linked_tests))
if [[ "$total_work" -gt 0 ]]; then
percent=$((completed_work * 100 / total_work))
else
percent=0
fi
# Determine phase
if [[ "$compiled_objects" -lt "$expected_lib_objects" ]]; then
phase="compiling"
# Get most recently modified .o file as last_item
last_item=$(find cache db env file logging memory memtable monitoring options port table test_util trace_replay util utilities -name '*.o' -type f -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2- | sed 's,^\./,,;s,\.o$,,')
elif [[ "$linked_tests" -lt "$expected_test_binaries" ]]; then
phase="linking"
# Get most recently modified test binary as last_item
last_item=$(find . -maxdepth 1 -name '*_test' -type f -executable -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2- | sed 's,^\./,,')
else
phase="generating"
last_item=""
fi
output_json "running" "$completed_work" "$total_work" 0 "$percent" 0 "0" "$last_item" "$phase"
+19
View File
@@ -0,0 +1,19 @@
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
GCC_BASE=/mnt/gvfs/third-party2/gcc/7331085db891a2ef4a88a48a751d834e8d68f4cb/7.x/centos7-native/b2ef2b6
CLANG_BASE=/mnt/gvfs/third-party2/llvm-fb/963d9aeda70cc4779885b1277484fe7544a04e3e/9.0.0/platform007/9e92d53/
LIBGCC_BASE=/mnt/gvfs/third-party2/libgcc/6ace84e956873d53638c738b6f65f3f469cca74c/7.x/platform007/5620abc
GLIBC_BASE=/mnt/gvfs/third-party2/glibc/192b0f42d63dcf6210d6ceae387b49af049e6e0c/2.26/platform007/f259413
SNAPPY_BASE=/mnt/gvfs/third-party2/snappy/7f9bdaada18f59bc27ec2b0871eb8a6144343aef/1.1.3/platform007/ca4da3d
ZLIB_BASE=/mnt/gvfs/third-party2/zlib/2d9f0b9a4274cc21f61272a9e89bdb859bce8f1f/1.2.8/platform007/ca4da3d
BZIP2_BASE=/mnt/gvfs/third-party2/bzip2/dc49a21c5fceec6456a7a28a94dcd16690af1337/1.0.6/platform007/ca4da3d
LZ4_BASE=/mnt/gvfs/third-party2/lz4/0f607f8fc442ea7d6b876931b1898bb573d5e5da/1.9.1/platform007/ca4da3d
ZSTD_BASE=/mnt/gvfs/third-party2/zstd/ca22bc441a4eb709e9e0b1f9fec9750fed7b31c5/1.4.x/platform007/15a3614
GFLAGS_BASE=/mnt/gvfs/third-party2/gflags/0b9929d2588991c65a57168bf88aff2db87c5d48/2.2.0/platform007/ca4da3d
JEMALLOC_BASE=/mnt/gvfs/third-party2/jemalloc/c26f08f47ac35fc31da2633b7da92d6b863246eb/master/platform007/c26c002
NUMA_BASE=/mnt/gvfs/third-party2/numa/3f3fb57a5ccc5fd21c66416c0b83e0aa76a05376/2.0.11/platform007/ca4da3d
LIBUNWIND_BASE=/mnt/gvfs/third-party2/libunwind/40c73d874898b386a71847f1b99115d93822d11f/1.4/platform007/6f3e0a9
TBB_BASE=/mnt/gvfs/third-party2/tbb/4ce8e8dba77cdbd81b75d6f0c32fd7a1b76a11ec/2018_U5/platform007/ca4da3d
KERNEL_HEADERS_BASE=/mnt/gvfs/third-party2/kernel-headers/fb251ecd2f5ae16f8671f7014c246e52a748fe0b/fb/platform007/da39a3e
BINUTILS_BASE=/mnt/gvfs/third-party2/binutils/ab9f09bba370e7066cafd4eb59752db93f2e8312/2.29.1/platform007/15a3614
VALGRIND_BASE=/mnt/gvfs/third-party2/valgrind/d42d152a15636529b0861ec493927200ebebca8e/3.15.0/platform007/ca4da3d
LUA_BASE=/mnt/gvfs/third-party2/lua/f0cd714433206d5139df61659eb7b28b1dea6683/5.3.4/platform007/5007832
+16
View File
@@ -0,0 +1,16 @@
GCC_BASE=/mnt/gvfs/third-party2/gcc/ebc96bc2fb751b5a0300b8d91a95bdf24ac1d88b/4.8.1/centos6-native/cc6c9dc
CLANG_BASE=/mnt/vol/engshare/fbcode/third-party2/clang/3.7.1/centos6-native/9d9ecb9/
LIBGCC_BASE=/mnt/gvfs/third-party2/libgcc/ea2fd1278810d3af2ea52218d2767e09d786dbd0/4.8.1/gcc-4.8.1-glibc-2.17/8aac7fc
GLIBC_BASE=/mnt/gvfs/third-party2/glibc/f5484f168c0e4d19823d41df052c5870c6e575a4/2.17/gcc-4.8.1-glibc-2.17/99df8fc
SNAPPY_BASE=/mnt/gvfs/third-party2/snappy/cbf6f1f209e5bd160bdc5d971744e039f36b1566/1.1.3/gcc-4.8.1-glibc-2.17/c3f970a
ZLIB_BASE=/mnt/gvfs/third-party2/zlib/6d39cb54708049f527e713ad19f2aadb9d3667e8/1.2.8/gcc-4.8.1-glibc-2.17/c3f970a
BZIP2_BASE=/mnt/gvfs/third-party2/bzip2/2ddd45f0853bfc8bb1c27f0f447236a1a26c338a/1.0.6/gcc-4.8.1-glibc-2.17/c3f970a
LZ4_BASE=/mnt/gvfs/third-party2/lz4/6858fac689e0f92e584224d91bdb0e39f6c8320d/r131/gcc-4.8.1-glibc-2.17/c3f970a
ZSTD_BASE=/mnt/gvfs/third-party2/zstd/9c833012dea9ae6de1df2cbdcb7aebc57ba2cb39/0.5.0/gcc-4.8.1-glibc-2.17/c3f970a
GFLAGS_BASE=/mnt/gvfs/third-party2/gflags/c7275a4ceae0aca0929e56964a31dafc53c1ee96/2.1.1/gcc-4.8.1-glibc-2.17/c3f970a
JEMALLOC_BASE=/mnt/gvfs/third-party2/jemalloc/378c8f95cf28c833e121ca73432101e451c3e62a/4.0.3/gcc-4.8.1-glibc-2.17/8d31e51
NUMA_BASE=/mnt/gvfs/third-party2/numa/ae54a5ed22cdabb1c6446dce4e8ffae5b4446d73/2.0.8/gcc-4.8.1-glibc-2.17/c3f970a
LIBUNWIND_BASE=/mnt/gvfs/third-party2/libunwind/303048f72efc92ae079e62dfc84823401aecfd94/trunk/gcc-4.8.1-glibc-2.17/675d945
KERNEL_HEADERS_BASE=/mnt/gvfs/third-party2/kernel-headers/1a48835975c66d30e47770ec419758ed3b9ba010/3.10.62-62_fbk17_03959_ge29cc63/gcc-4.8.1-glibc-2.17/da39a3e
BINUTILS_BASE=/mnt/gvfs/third-party2/binutils/a5b8152b2a15ce8a98808cf954fbccec825a97bc/2.25/centos6-native/da39a3e
VALGRIND_BASE=/mnt/gvfs/third-party2/valgrind/af85c56f424cd5edfc2c97588299b44ecdec96bb/3.8.1/gcc-4.8.1-glibc-2.17/c3f970a
-20
View File
@@ -1,20 +0,0 @@
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
# The file is generated using update_dependencies.sh.
GCC_BASE=/mnt/gvfs/third-party2/gcc/e9cb2d4550b5a95024125084420342d256b349cd/11.x/centos9-native/3bed279
CLANG_BASE=/mnt/gvfs/third-party2/llvm-fb/4e46e6967e15412702e51960c69790a78933856f/21/platform010/72a2ff8
LIBGCC_BASE=/mnt/gvfs/third-party2/libgcc/63c5179edc90fec473e53c1b2f92269c65c39e07/11.x/platform010/5684a5a
GLIBC_BASE=/mnt/gvfs/third-party2/glibc/b4a6248a76a163bcccb1e4751e247712c535846e/2.34/platform010/f259413
SNAPPY_BASE=/mnt/gvfs/third-party2/snappy/ef895164499fdfedb9d9b9c85520de10aa8ff9ca/1.1.8/platform010/76ebdda
ZLIB_BASE=/mnt/gvfs/third-party2/zlib/643dbf0136f869562485cfcb2953f5cab60447ef/1.2.8/platform010/76ebdda
BZIP2_BASE=/mnt/gvfs/third-party2/bzip2/09703139cfc376bd8a82642385a0e97726b28287/1.0.6/platform010/76ebdda
LZ4_BASE=/mnt/gvfs/third-party2/lz4/823bf8e000654abef784311296ffc8166d9d3986/1.9.4/platform010/76ebdda
ZSTD_BASE=/mnt/gvfs/third-party2/zstd/0cda78647b5712bec7ec6d9ba2f0dcdabfb6f44f/1.4.x/platform010/64091f4
GFLAGS_BASE=/mnt/gvfs/third-party2/gflags/a02e0de00945729ad553c187c64e86b57f45a33b/2.2.0/platform010/76ebdda
JEMALLOC_BASE=/mnt/gvfs/third-party2/jemalloc/a1a50742bec7195bfafb594e174ea45dc197a420/master/platform010/779a252
NUMA_BASE=/mnt/gvfs/third-party2/numa/5b602edd46fda54cdd7ea45f77dbe4061206e174/2.0.11/platform010/76ebdda
LIBUNWIND_BASE=/mnt/gvfs/third-party2/libunwind/f3cce2e1a79ac8d64e102886603a6997d6e88640/1.8/platform010/76ebdda
LIBURING_BASE=/mnt/gvfs/third-party2/liburing/a98e2d137007e3ebf7f33bd6f99c2c56bdaf8488/20210212/platform010/76ebdda
BENCHMARK_BASE=/mnt/gvfs/third-party2/benchmark/780c7a0f9cf0967961e69ad08e61cddd85d61821/trunk/platform010/76ebdda
KERNEL_HEADERS_BASE=/mnt/gvfs/third-party2/kernel-headers/0883cccda758e68b47e4c04e4fa01142e1f60f32/fb/platform010/da39a3e
BINUTILS_BASE=/mnt/gvfs/third-party2/binutils/1d5d19135a406a941d3a441eb2bf506bb9d19210/2.43/centos9-native/be7139e
VALGRIND_BASE=/mnt/gvfs/third-party2/valgrind/3e2b7ccc315c9a0f2b678ce6fd774f4970f9bf3b/3.22.0/platform010/76ebdda

Some files were not shown because too many files have changed in this diff Show More