An odd fix for GCC 7

Summary: Currently, RocksDB returns error if a db written with prefix hash index, is later opened without providing a prefix extractor. This is uncessarily harsh. Without a prefix extractor, we could always fallback to the normal binary index.

Test Plan: unit test, also manually veried LOG that fallback did occur.

Reviewers: sdong, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D19191

.gitignore

@@ -19,6 +19,7 @@ build_config.mk
 *.*jnilib*
 *.d-e
 *.o-*
+*.swp
 
 ldb
 manifest_dump

.travis.yml

@@ -0,0 +1,20 @@
+language: cpp
+compiler: gcc
+before_install:
+# As of this writing (10 May 2014) the Travis build environment is Ubuntu 12.04,
+# which needs the following ugly dependency incantations to build RocksDB:
+ - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
+ - sudo apt-get update -qq
+ - sudo apt-get install -y -qq gcc-4.8 g++-4.8 zlib1g-dev libbz2-dev libsnappy-dev libjemalloc-dev
+ - sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50
+ - sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 50
+ - wget https://gflags.googlecode.com/files/libgflags0_2.0-1_amd64.deb
+ - sudo dpkg -i libgflags0_2.0-1_amd64.deb
+ - wget https://gflags.googlecode.com/files/libgflags-dev_2.0-1_amd64.deb
+ - sudo dpkg -i libgflags-dev_2.0-1_amd64.deb
+# Lousy hack to disable use and testing of fallocate, which doesn't behave quite
+# as EnvPosixTest::AllocateTest expects within the Travis OpenVZ environment.
+ - sed -i "s/fallocate(/HACK_NO_fallocate(/" build_tools/build_detect_platform
+script: make check -j8
+notifications:
+    email: false

HISTORY.md

@@ -1,10 +1,22 @@
 # Rocksdb Change Log
 
+## 3.1.0 (05/21/2014)
+
+### Public API changes
+* Replaced ColumnFamilyOptions::table_properties_collectors with ColumnFamilyOptions::table_properties_collector_factories
+* Add two paramters to NewHashLinkListRepFactory() for logging on too many entries in a hash bucket when flushing.
+
+### New Features
+* Hash index for block-based table will be materialized and reconstructed more efficiently. Previously hash index is constructed by scanning the whole table during every table open.
+* FIFO compaction style
+* Add AdaptiveTableFactory, which is used to convert from a DB of PlainTable to BlockBasedTabe, or vise versa. It can be created using NewAdaptiveTableFactory()
+
 ## 3.0.0 (05/05/2014)
 
 ### Public API changes
 * Added _LEVEL to all InfoLogLevel enums
 * Deprecated ReadOptions.prefix and ReadOptions.prefix_seek. Seek() defaults to prefix-based seek when Options.prefix_extractor is supplied. More detail is documented in https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes
+* MemTableRepFactory::CreateMemTableRep() takes info logger as an extra parameter.
 
 ### New Features
 * Column family support

INSTALL.md

@@ -1,19 +1,32 @@
+## Compilation
+
+RocksDB's library should be able to compile without any dependency installed,
+although we recommend installing some compression libraries (see below).
+We do depend on newer gcc with C++11 support.
+
+There are few options when compiling RocksDB:
+
+* [recommended] `make static_lib` will compile librocksdb.a, RocksDB static library.
+
+* `make shared_lib` will compile librocksdb.so, RocksDB shared library.
+
+* `make check` will compile and run all the unit tests
+
+* `make all` will compile our static library, and all our tools and unit tests. Our tools
+depend on gflags. You will need to have gflags installed to run `make all`.
+
 ## Dependencies
 
-RocksDB is developed on Linux (CentOS release 5.2), with gcc 4.8.1.
-It depends on gcc with C++11 support.
-
-* RocksDB depends on the following libraries:
+* You can link RocksDB with following compression libraries:
   - [zlib](http://www.zlib.net/) - a library for data compression.
   - [bzip2](http://www.bzip.org/) - a library for data compression.
   - [snappy](https://code.google.com/p/snappy/) - a library for fast
       data compression.
-  - [gflags](https://code.google.com/p/gflags/) - a library that handles
-      command line flags processing.
 
-RocksDB will successfully compile without the compression libraries included,
-but some things may fail. We do not support releases without the compression
-libraries. You are on your own.
+* All our tools depend on:
+  - [gflags](https://code.google.com/p/gflags/) - a library that handles
+      command line flags processing. You can compile rocksdb library even
+      if you don't have gflags installed.
 
 ## Supported platforms
 
@@ -61,7 +74,7 @@ libraries. You are on your own.
             * run `brew tap homebrew/dupes; brew install gcc47 --use-llvm` to install gcc 4.7 (or higher).
     * Install zlib, bzip2 and snappy libraries for compression.
     * Install gflags. We have included a script
-    `build_tools/mac-install-gflags.sh`, which should automatically install it.
+    `build_tools/mac-install-gflags.sh`, which should automatically install it (execute this file instead of runing using "source" command).
     If you installed gflags by other means (for example, `brew install gflags`),
     please set `LIBRARY_PATH` and `CPATH` accordingly.
     * Please note that some of the optimizations/features are disabled in OSX.
@@ -69,12 +82,3 @@ libraries. You are on your own.
 
 * **iOS**:
   * Run: `TARGET_OS=IOS make static_lib`
-
-## Compilation
-`make clean; make` will compile librocksdb.a (RocksDB static library) and all
-the unit tests. You can run all unit tests with `make check`.
-
-For shared library builds, exec `make shared_lib` instead.
-
-If you followed the above steps and your compile or unit tests fail,
-please submit an issue: (https://github.com/facebook/rocksdb/issues)

Makefile

@@ -14,19 +14,17 @@ else
 endif
 
 ifeq ($(MAKECMDGOALS),shared_lib)
-PLATFORM_SHARED_LDFLAGS=-fPIC
 OPT += -DNDEBUG
 endif
 
 ifeq ($(MAKECMDGOALS),static_lib)
-PLATFORM_SHARED_LDFLAGS=-fPIC
 OPT += -DNDEBUG
 endif
 
 #-----------------------------------------------
 
 # detect what platform we're building on
-$(shell (export ROCKSDB_ROOT=$(CURDIR); $(CURDIR)/build_tools/build_detect_platform $(CURDIR)/build_config.mk))
+$(shell (export ROCKSDB_ROOT="$(CURDIR)"; "$(CURDIR)/build_tools/build_detect_platform" "$(CURDIR)/build_config.mk"))
 # this file is generated by the previous line to set build flags and sources
 include build_config.mk
 
@@ -51,7 +49,6 @@ else
 	PLATFORM_CCFLAGS += $(JEMALLOC_INCLUDE) -DHAVE_JEMALLOC
 endif
 
-WARNING_FLAGS = -Wall -Werror
 CFLAGS += $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CCFLAGS) $(OPT)
 CXXFLAGS += $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CXXFLAGS) $(OPT) -Woverloaded-virtual
 
@@ -148,7 +145,7 @@ SHARED = $(SHARED1)
 else
 # Update db.h if you change these.
 SHARED_MAJOR = 3
-SHARED_MINOR = 0
+SHARED_MINOR = 2
 SHARED1 = ${LIBNAME}.$(PLATFORM_SHARED_EXT)
 SHARED2 = $(SHARED1).$(SHARED_MAJOR)
 SHARED3 = $(SHARED1).$(SHARED_MAJOR).$(SHARED_MINOR)
@@ -181,10 +178,6 @@ release:
 	$(MAKE) clean
 	OPT="-DNDEBUG -O2" $(MAKE) static_lib $(PROGRAMS) -j32
 
-release_shared_lib:
-	$(MAKE) clean
-	OPT="-DNDEBUG -O2" $(MAKE) shared_lib -j32
-
 coverage:
 	$(MAKE) clean
 	COVERAGEFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS+="-lgcov" $(MAKE) all check -j32
@@ -192,11 +185,11 @@ coverage:
 	# Delete intermediate files
 	find . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" -exec rm {} \;
 
-check: $(PROGRAMS) $(TESTS) $(TOOLS)
+check: $(TESTS) ldb
 	for t in $(TESTS); do echo "***** Running $$t"; ./$$t || exit 1; done
 	python tools/ldb_test.py
 
-ldb_tests: all $(PROGRAMS) $(TESTS) $(TOOLS)
+ldb_tests: ldb
 	python tools/ldb_test.py
 
 crash_test: whitebox_crash_test blackbox_crash_test

README

@@ -1,82 +0,0 @@
-rocksdb: A persistent key-value store for flash storage
-Authors: * The Facebook Database Engineering Team
-         * Build on earlier work on leveldb by Sanjay Ghemawat
-           (sanjay@google.com) and Jeff Dean (jeff@google.com)
-
-This code is a library that forms the core building block for a fast
-key value server, especially suited for storing data on flash drives.
-It has an Log-Structured-Merge-Database (LSM) design with flexible tradeoffs
-between Write-Amplification-Factor(WAF), Read-Amplification-Factor (RAF)
-and Space-Amplification-Factor(SAF). It has multi-threaded compactions,
-making it specially suitable for storing multiple terabytes of data in a
-single database.
-
-The core of this code has been derived from open-source leveldb.
-
-The code under this directory implements a system for maintaining a
-persistent key/value store.
-
-See doc/index.html and github wiki (https://github.com/facebook/rocksdb/wiki)
-for more explanation.
-
-The public interface is in include/*.  Callers should not include or
-rely on the details of any other header files in this package.  Those
-internal APIs may be changed without warning.
-
-Guide to header files:
-
-include/rocksdb/db.h
-    Main interface to the DB: Start here
-
-include/rocksdb/options.h
-    Control over the behavior of an entire database, and also
-    control over the behavior of individual reads and writes.
-
-include/rocksdb/comparator.h
-    Abstraction for user-specified comparison function.  If you want
-    just bytewise comparison of keys, you can use the default comparator,
-    but clients can write their own comparator implementations if they
-    want custom ordering (e.g. to handle different character
-    encodings, etc.)
-
-include/rocksdb/iterator.h
-    Interface for iterating over data. You can get an iterator
-    from a DB object.
-
-include/rocksdb/write_batch.h
-    Interface for atomically applying multiple updates to a database.
-
-include/rocksdb/slice.h
-    A simple module for maintaining a pointer and a length into some
-    other byte array.
-
-include/rocksdb/status.h
-    Status is returned from many of the public interfaces and is used
-    to report success and various kinds of errors.
-
-include/rocksdb/env.h
-    Abstraction of the OS environment.  A posix implementation of
-    this interface is in util/env_posix.cc
-
-include/rocksdb/table_builder.h
-    Lower-level modules that most clients probably won't use directly
-
-include/rocksdb/cache.h
-    An API for the block cache.
-
-include/rocksdb/compaction_filter.h
-    An API for a application filter invoked on every compaction.
-
-include/rocksdb/filter_policy.h
-    An API for configuring a bloom filter.
-
-include/rocksdb/memtablerep.h
-    An API for implementing a memtable.
-
-include/rocksdb/statistics.h
-    An API to retrieve various database statistics.
-
-include/rocksdb/transaction_log.h
-    An API to retrieve transaction logs from a database.
-
-Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/

README.md

@@ -0,0 +1,25 @@
+## RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
+
+[![Build Status](https://travis-ci.org/facebook/rocksdb.svg?branch=master)](https://travis-ci.org/facebook/rocksdb)
+
+RocksDB is developed and maintained by Facebook Database Engineering Team.
+It is built on on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com)
+and Jeff Dean (jeff@google.com)
+
+This code is a library that forms the core building block for a fast
+key value server, especially suited for storing data on flash drives.
+It has an Log-Structured-Merge-Database (LSM) design with flexible tradeoffs
+between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF)
+and Space-Amplification-Factor (SAF). It has multi-threaded compactions,
+making it specially suitable for storing multiple terabytes of data in a
+single database.
+
+Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples
+
+See the [github wiki](https://github.com/facebook/rocksdb/wiki) for more explanation.
+
+The public interface is in `include/`.  Callers should not include or
+rely on the details of any other header files in this package.  Those
+internal APIs may be changed without warning.
+
+Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/

build_tools/build_detect_platform

@@ -47,21 +47,15 @@ COMMON_FLAGS="-DROCKSDB_PLATFORM_POSIX"
 if [ -d /mnt/gvfs/third-party -a -z "$CXX" ]; then
     FBCODE_BUILD="true"
     if [ -z "$USE_CLANG" ]; then
-        CENTOS_VERSION=`rpm -q --qf "%{VERSION}" \
-          $(rpm -q --whatprovides redhat-release)`
-        if [ "$CENTOS_VERSION" = "6" ]; then
-          source $PWD/build_tools/fbcode.gcc481.sh
-        else
-          source $PWD/build_tools/fbcode.gcc471.sh
-        fi
+        source $PWD/build_tools/fbcode_config.sh
     else
-        source $PWD/build_tools/fbcode.clang31.sh
+        source "$PWD/build_tools/fbcode.clang31.sh"
     fi
 fi
 
 # Delete existing output, if it exists
-rm -f $OUTPUT
-touch $OUTPUT
+rm -f "$OUTPUT"
+touch "$OUTPUT"
 
 if test -z "$CC"; then
    CC=cc
@@ -87,7 +81,7 @@ PLATFORM_SHARED_CFLAGS="-fPIC"
 PLATFORM_SHARED_VERSIONED=false
 
 # generic port files (working on all platform by #ifdef) go directly in /port
-GENERIC_PORT_FILES=`cd $ROCKSDB_ROOT; find port -name '*.cc' | tr "\n" " "`
+GENERIC_PORT_FILES=`cd "$ROCKSDB_ROOT"; find port -name '*.cc' | tr "\n" " "`
 
 # On GCC, we pick libc's memcmp over GCC's memcmp via -fno-builtin-memcmp
 case "$TARGET_OS" in
@@ -157,7 +151,7 @@ case "$TARGET_OS" in
 esac
 
 if test -z "$DO_NOT_RUN_BUILD_DETECT_VERSION"; then
-  $PWD/build_tools/build_detect_version
+  "$PWD/build_tools/build_detect_version"
 fi
 
 # We want to make a list of all cc files within util, db, table, and helpers
@@ -169,27 +163,21 @@ DIRS="util db table utilities"
 set -f # temporarily disable globbing so that our patterns arent expanded
 PRUNE_TEST="-name *test*.cc -prune"
 PRUNE_BENCH="-name *bench*.cc -prune"
-PORTABLE_FILES=`cd $ROCKSDB_ROOT; find $DIRS $PRUNE_TEST -o $PRUNE_BENCH -o -name '*.cc' -print | sort | tr "\n" " "`
-PORTABLE_CPP=`cd $ROCKSDB_ROOT; find $DIRS $PRUNE_TEST -o $PRUNE_BENCH -o -name '*.cpp' -print | sort | tr "\n" " "`
+PORTABLE_FILES=`cd "$ROCKSDB_ROOT"; find $DIRS $PRUNE_TEST -o $PRUNE_BENCH -o -name '*.cc' -print | sort | tr "\n" " "`
+PORTABLE_CPP=`cd "$ROCKSDB_ROOT"; find $DIRS $PRUNE_TEST -o $PRUNE_BENCH -o -name '*.cpp' -print | sort | tr "\n" " "`
 set +f # re-enable globbing
 
 # The sources consist of the portable files, plus the platform-specific port
 # file.
-echo "SOURCES=$PORTABLE_FILES $GENERIC_PORT_FILES $PORT_FILES" >> $OUTPUT
-echo "SOURCESCPP=$PORTABLE_CPP" >> $OUTPUT
-echo "MEMENV_SOURCES=helpers/memenv/memenv.cc" >> $OUTPUT
+echo "SOURCES=$PORTABLE_FILES $GENERIC_PORT_FILES $PORT_FILES" >> "$OUTPUT"
+echo "SOURCESCPP=$PORTABLE_CPP" >> "$OUTPUT"
+echo "MEMENV_SOURCES=helpers/memenv/memenv.cc" >> "$OUTPUT"
 
 if [ "$CROSS_COMPILE" = "true" -o "$FBCODE_BUILD" = "true" ]; then
     # Cross-compiling; do not try any compilation tests.
     # Also don't need any compilation tests if compiling on fbcode
     true
 else
-    # do fPIC on 64 bit in non-fbcode environment
-    case "$TARGET_OS" in
-        x86_64)
-            PLATFORM_CXXFLAGS="$PLATFORM_CXXFLAGS -fPIC"
-    esac
-
     # If -std=c++0x works, use <atomic>.  Otherwise use port_posix.h.
     $CXX $CFLAGS -std=c++0x -x c++ - -o /dev/null 2>/dev/null  <<EOF
       #include <atomic>
@@ -225,12 +213,25 @@ EOF
 
     # Test whether gflags library is installed
     # http://code.google.com/p/gflags/
+    # check if the namespace is gflags
     $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
       #include <gflags/gflags.h>
+      using namespace gflags;
       int main() {}
 EOF
     if [ "$?" = 0 ]; then
-        COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS"
+        COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=gflags"
+        PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
+    fi
+
+    # check if namespace is google
+    $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
+      #include <gflags/gflags.h>
+      using namespace google;
+      int main() {}
+EOF
+    if [ "$?" = 0 ]; then
+        COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=gflags"
         PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
     fi
 
@@ -282,7 +283,7 @@ if test "$USE_HDFS"; then
     exit 1
   fi
   HDFS_CCFLAGS="$HDFS_CCFLAGS -I$JAVA_HOME/include -I$JAVA_HOME/include/linux -DUSE_HDFS"
-  HDFS_LDFLAGS="$HDFS_LDFLAGS -Wl,--no-whole-archive hdfs/libhdfs.a -L$JAVA_HOME/jre/lib/amd64"
+  HDFS_LDFLAGS="$HDFS_LDFLAGS -Wl,--no-whole-archive -lhdfs -L$JAVA_HOME/jre/lib/amd64"
   HDFS_LDFLAGS="$HDFS_LDFLAGS -L$JAVA_HOME/jre/lib/amd64/server -L$GLIBC_RUNTIME_PATH/lib"
   HDFS_LDFLAGS="$HDFS_LDFLAGS -ldl -lverify -ljava -ljvm"
   COMMON_FLAGS="$COMMON_FLAGS $HDFS_CCFLAGS"
@@ -297,17 +298,17 @@ PLATFORM_CXXFLAGS="$PLATFORM_CXXFLAGS $COMMON_FLAGS"
 
 VALGRIND_VER="$VALGRIND_VER"
 
-echo "CC=$CC" >> $OUTPUT
-echo "CXX=$CXX" >> $OUTPUT
-echo "PLATFORM=$PLATFORM" >> $OUTPUT
-echo "PLATFORM_LDFLAGS=$PLATFORM_LDFLAGS" >> $OUTPUT
-echo "VALGRIND_VER=$VALGRIND_VER" >> $OUTPUT
-echo "PLATFORM_CCFLAGS=$PLATFORM_CCFLAGS" >> $OUTPUT
-echo "PLATFORM_CXXFLAGS=$PLATFORM_CXXFLAGS" >> $OUTPUT
-echo "PLATFORM_SHARED_CFLAGS=$PLATFORM_SHARED_CFLAGS" >> $OUTPUT
-echo "PLATFORM_SHARED_EXT=$PLATFORM_SHARED_EXT" >> $OUTPUT
-echo "PLATFORM_SHARED_LDFLAGS=$PLATFORM_SHARED_LDFLAGS" >> $OUTPUT
-echo "PLATFORM_SHARED_VERSIONED=$PLATFORM_SHARED_VERSIONED" >> $OUTPUT
-echo "EXEC_LDFLAGS=$EXEC_LDFLAGS" >> $OUTPUT
-echo "JEMALLOC_INCLUDE=$JEMALLOC_INCLUDE" >> $OUTPUT
-echo "JEMALLOC_LIB=$JEMALLOC_LIB" >> $OUTPUT
+echo "CC=$CC" >> "$OUTPUT"
+echo "CXX=$CXX" >> "$OUTPUT"
+echo "PLATFORM=$PLATFORM" >> "$OUTPUT"
+echo "PLATFORM_LDFLAGS=$PLATFORM_LDFLAGS" >> "$OUTPUT"
+echo "VALGRIND_VER=$VALGRIND_VER" >> "$OUTPUT"
+echo "PLATFORM_CCFLAGS=$PLATFORM_CCFLAGS" >> "$OUTPUT"
+echo "PLATFORM_CXXFLAGS=$PLATFORM_CXXFLAGS" >> "$OUTPUT"
+echo "PLATFORM_SHARED_CFLAGS=$PLATFORM_SHARED_CFLAGS" >> "$OUTPUT"
+echo "PLATFORM_SHARED_EXT=$PLATFORM_SHARED_EXT" >> "$OUTPUT"
+echo "PLATFORM_SHARED_LDFLAGS=$PLATFORM_SHARED_LDFLAGS" >> "$OUTPUT"
+echo "PLATFORM_SHARED_VERSIONED=$PLATFORM_SHARED_VERSIONED" >> "$OUTPUT"
+echo "EXEC_LDFLAGS=$EXEC_LDFLAGS" >> "$OUTPUT"
+echo "JEMALLOC_INCLUDE=$JEMALLOC_INCLUDE" >> "$OUTPUT"
+echo "JEMALLOC_LIB=$JEMALLOC_LIB" >> "$OUTPUT"

build_tools/dependencies.sh

@@ -0,0 +1,19 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
+GCC_BASE=/mnt/gvfs/third-party2/gcc/7331085db891a2ef4a88a48a751d834e8d68f4cb/7.x/centos7-native/b2ef2b6
+CLANG_BASE=/mnt/gvfs/third-party2/llvm-fb/963d9aeda70cc4779885b1277484fe7544a04e3e/9.0.0/platform007/9e92d53/
+LIBGCC_BASE=/mnt/gvfs/third-party2/libgcc/6ace84e956873d53638c738b6f65f3f469cca74c/7.x/platform007/5620abc
+GLIBC_BASE=/mnt/gvfs/third-party2/glibc/192b0f42d63dcf6210d6ceae387b49af049e6e0c/2.26/platform007/f259413
+SNAPPY_BASE=/mnt/gvfs/third-party2/snappy/7f9bdaada18f59bc27ec2b0871eb8a6144343aef/1.1.3/platform007/ca4da3d
+ZLIB_BASE=/mnt/gvfs/third-party2/zlib/2d9f0b9a4274cc21f61272a9e89bdb859bce8f1f/1.2.8/platform007/ca4da3d
+BZIP2_BASE=/mnt/gvfs/third-party2/bzip2/dc49a21c5fceec6456a7a28a94dcd16690af1337/1.0.6/platform007/ca4da3d
+LZ4_BASE=/mnt/gvfs/third-party2/lz4/0f607f8fc442ea7d6b876931b1898bb573d5e5da/1.9.1/platform007/ca4da3d
+ZSTD_BASE=/mnt/gvfs/third-party2/zstd/ca22bc441a4eb709e9e0b1f9fec9750fed7b31c5/1.4.x/platform007/15a3614
+GFLAGS_BASE=/mnt/gvfs/third-party2/gflags/0b9929d2588991c65a57168bf88aff2db87c5d48/2.2.0/platform007/ca4da3d
+JEMALLOC_BASE=/mnt/gvfs/third-party2/jemalloc/c26f08f47ac35fc31da2633b7da92d6b863246eb/master/platform007/c26c002
+NUMA_BASE=/mnt/gvfs/third-party2/numa/3f3fb57a5ccc5fd21c66416c0b83e0aa76a05376/2.0.11/platform007/ca4da3d
+LIBUNWIND_BASE=/mnt/gvfs/third-party2/libunwind/40c73d874898b386a71847f1b99115d93822d11f/1.4/platform007/6f3e0a9
+TBB_BASE=/mnt/gvfs/third-party2/tbb/4ce8e8dba77cdbd81b75d6f0c32fd7a1b76a11ec/2018_U5/platform007/ca4da3d
+KERNEL_HEADERS_BASE=/mnt/gvfs/third-party2/kernel-headers/fb251ecd2f5ae16f8671f7014c246e52a748fe0b/fb/platform007/da39a3e
+BINUTILS_BASE=/mnt/gvfs/third-party2/binutils/ab9f09bba370e7066cafd4eb59752db93f2e8312/2.29.1/platform007/15a3614
+VALGRIND_BASE=/mnt/gvfs/third-party2/valgrind/d42d152a15636529b0861ec493927200ebebca8e/3.15.0/platform007/ca4da3d
+LUA_BASE=/mnt/gvfs/third-party2/lua/f0cd714433206d5139df61659eb7b28b1dea6683/5.3.4/platform007/5007832

build_tools/fbcode.gcc471.sh

@@ -55,7 +55,7 @@ CFLAGS="-B$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/gold -m64
 CFLAGS+=" -I $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/include -DHAVE_JEMALLOC"
 CFLAGS+=" $LIBGCC_INCLUDE $GLIBC_INCLUDE"
 CFLAGS+=" -DROCKSDB_PLATFORM_POSIX -DROCKSDB_ATOMIC_PRESENT -DROCKSDB_FALLOCATE_PRESENT"
-CFLAGS+=" -DSNAPPY -DGFLAGS -DZLIB -DBZIP2"
+CFLAGS+=" -DSNAPPY -DGFLAGS=google -DZLIB -DBZIP2"
 
 EXEC_LDFLAGS=" -Wl,--whole-archive $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/lib/libjemalloc.a"
 EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/350336c/lib/libunwind.a"

build_tools/fbcode.gcc481.sh

@@ -66,7 +66,7 @@ RANLIB=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ranlib
 CFLAGS="-B$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/gold -m64 -mtune=generic"
 CFLAGS+=" $LIBGCC_INCLUDE $GLIBC_INCLUDE"
 CFLAGS+=" -DROCKSDB_PLATFORM_POSIX -DROCKSDB_ATOMIC_PRESENT -DROCKSDB_FALLOCATE_PRESENT"
-CFLAGS+=" -DSNAPPY -DGFLAGS -DZLIB -DBZIP2 -DLZ4"
+CFLAGS+=" -DSNAPPY -DGFLAGS=google -DZLIB -DBZIP2 -DLZ4"
 
 EXEC_LDFLAGS="-Wl,--dynamic-linker,/usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/ld.so"
 EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/675d945/lib/libunwind.a"

build_tools/fbcode_config.sh

@@ -0,0 +1,139 @@
+#!/bin/sh
+#
+# Set environment variables so that we can compile rocksdb using
+# fbcode settings.  It uses the latest g++ and clang compilers and also
+# uses jemalloc
+# Environment variables that change the behavior of this script:
+# PIC_BUILD -- if true, it will only take pic versions of libraries from fbcode. libraries that don't have pic variant will not be included
+
+
+BASEDIR=`dirname $BASH_SOURCE`
+source "$BASEDIR/dependencies.sh"
+
+CFLAGS=""
+
+# libgcc
+LIBGCC_INCLUDE="$LIBGCC_BASE/include/c++/7.3.0"
+LIBGCC_LIBS=" -L $LIBGCC_BASE/lib"
+
+# glibc
+GLIBC_INCLUDE="$GLIBC_BASE/include"
+GLIBC_LIBS=" -L $GLIBC_BASE/lib"
+
+# snappy
+SNAPPY_INCLUDE=" -I $SNAPPY_BASE/include/"
+if test -z $PIC_BUILD; then
+  SNAPPY_LIBS=" $SNAPPY_BASE/lib/libsnappy.a"
+else
+  SNAPPY_LIBS=" $SNAPPY_BASE/lib/libsnappy_pic.a"
+fi
+CFLAGS+=" -DSNAPPY"
+
+if test -z $PIC_BUILD; then
+  # location of zlib headers and libraries
+  ZLIB_INCLUDE=" -I $ZLIB_BASE/include/"
+  ZLIB_LIBS=" $ZLIB_BASE/lib/libz.a"
+  CFLAGS+=" -DZLIB"
+
+  LZ4_INCLUDE=" -I $LZ4_BASE/include/"
+  LZ4_LIBS=" $LZ4_BASE/lib/liblz4.a"
+  CFLAGS+=" -DLZ4"
+fi
+
+# location of gflags headers and libraries
+GFLAGS_INCLUDE=" -I $GFLAGS_BASE/include/"
+if test -z $PIC_BUILD; then
+  GFLAGS_LIBS=" $GFLAGS_BASE/lib/libgflags.a"
+else
+  GFLAGS_LIBS=" $GFLAGS_BASE/lib/libgflags_pic.a"
+fi
+CFLAGS+=" -DGFLAGS=gflags"
+
+# location of jemalloc
+JEMALLOC_INCLUDE=" -I $JEMALLOC_BASE/include/"
+JEMALLOC_LIB=" $JEMALLOC_BASE/lib/libjemalloc.a"
+
+if test -z $PIC_BUILD; then
+  # location of numa
+  NUMA_INCLUDE=" -I $NUMA_BASE/include/"
+  NUMA_LIB=" $NUMA_BASE/lib/libnuma.a"
+  CFLAGS+=" -DNUMA"
+
+  # location of libunwind
+  LIBUNWIND="$LIBUNWIND_BASE/lib/libunwind.a"
+fi
+
+# location of TBB
+TBB_INCLUDE=" -isystem $TBB_BASE/include/"
+if test -z $PIC_BUILD; then
+  TBB_LIBS="$TBB_BASE/lib/libtbb.a"
+else
+  TBB_LIBS="$TBB_BASE/lib/libtbb_pic.a"
+fi
+CFLAGS+=" -DTBB"
+
+# use Intel SSE support for checksum calculations
+export USE_SSE=" -msse -msse4.2 "
+
+BINUTILS="$BINUTILS_BASE/bin"
+AR="$BINUTILS/ar"
+
+DEPS_INCLUDE="$SNAPPY_INCLUDE $ZLIB_INCLUDE $LZ4_INCLUDE $GFLAGS_INCLUDE"
+
+STDLIBS="-L $GCC_BASE/lib64"
+
+CLANG_BIN="$CLANG_BASE/bin"
+CLANG_LIB="$CLANG_BASE/lib"
+CLANG_SRC="$CLANG_BASE/../../src"
+
+CLANG_ANALYZER="$CLANG_BIN/clang++"
+CLANG_SCAN_BUILD="$CLANG_SRC/llvm/tools/clang/tools/scan-build/bin/scan-build"
+
+if [ -z "$USE_CLANG" ]; then
+  # gcc
+  CC="$GCC_BASE/bin/gcc"
+  CXX="$GCC_BASE/bin/g++"
+
+  CFLAGS+=" -B$BINUTILS/gold"
+  CFLAGS+=" -isystem $LIBGCC_INCLUDE"
+  CFLAGS+=" -isystem $GLIBC_INCLUDE"
+  JEMALLOC=1
+else
+  # clang
+  CLANG_INCLUDE="$CLANG_LIB/clang/stable/include"
+  CC="$CLANG_BIN/clang"
+  CXX="$CLANG_BIN/clang++"
+
+  KERNEL_HEADERS_INCLUDE="$KERNEL_HEADERS_BASE/include"
+
+  CFLAGS+=" -B$BINUTILS/gold -nostdinc -nostdlib"
+  CFLAGS+=" -isystem $LIBGCC_BASE/include/c++/7.x "
+  CFLAGS+=" -isystem $LIBGCC_BASE/include/c++/7.x/x86_64-facebook-linux "
+  CFLAGS+=" -isystem $GLIBC_INCLUDE"
+  CFLAGS+=" -isystem $LIBGCC_INCLUDE"
+  CFLAGS+=" -isystem $CLANG_INCLUDE"
+  CFLAGS+=" -isystem $KERNEL_HEADERS_INCLUDE/linux "
+  CFLAGS+=" -isystem $KERNEL_HEADERS_INCLUDE "
+  CFLAGS+=" -Wno-expansion-to-defined "
+  CXXFLAGS="-nostdinc++"
+fi
+
+CFLAGS+=" $DEPS_INCLUDE"
+CFLAGS+=" -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX -DROCKSDB_FALLOCATE_PRESENT -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DROCKSDB_SUPPORT_THREAD_LOCAL -DHAVE_SSE42"
+CXXFLAGS+=" $CFLAGS"
+
+EXEC_LDFLAGS=" $SNAPPY_LIBS $ZLIB_LIBS $LZ4_LIBS $ZSTD_LIBS $GFLAGS_LIBS $NUMA_LIB $TBB_LIBS"
+EXEC_LDFLAGS+=" -B$BINUTILS/gold"
+EXEC_LDFLAGS+=" -Wl,--dynamic-linker,/usr/local/fbcode/platform007/lib/ld.so"
+EXEC_LDFLAGS+=" $LIBUNWIND"
+EXEC_LDFLAGS+=" -Wl,-rpath=/usr/local/fbcode/platform007/lib"
+# required by libtbb
+EXEC_LDFLAGS+=" -ldl"
+
+PLATFORM_LDFLAGS="$LIBGCC_LIBS $GLIBC_LIBS $STDLIBS -lgcc -lstdc++"
+
+EXEC_LDFLAGS_SHARED="$SNAPPY_LIBS $ZLIB_LIBS $LZ4_LIBS $ZSTD_LIBS $GFLAGS_LIBS $TBB_LIBS"
+
+VALGRIND_VER="$VALGRIND_BASE/bin/"
+
+export CC CXX AR CFLAGS CXXFLAGS EXEC_LDFLAGS EXEC_LDFLAGS_SHARED VALGRIND_VER JEMALLOC_LIB JEMALLOC_INCLUDE CLANG_ANALYZER CLANG_SCAN_BUILD 

build_tools/regression_build_test.sh

@@ -283,6 +283,76 @@ make release
     --value_size=10 \
     --threads=16 > ${STAT_FILE}.memtablefillreadrandom
 
+common_in_mem_args="--db=/dev/shm/rocksdb \
+    --num_levels=6 \
+    --key_size=20 \
+    --prefix_size=12 \
+    --keys_per_prefix=10 \
+    --value_size=100 \
+    --compression_type=none \
+    --compression_ratio=1 \
+    --disable_seek_compaction=1 \
+    --hard_rate_limit=2 \
+    --write_buffer_size=134217728 \
+    --max_write_buffer_number=4 \
+    --level0_file_num_compaction_trigger=8 \
+    --level0_slowdown_writes_trigger=16 \
+    --level0_stop_writes_trigger=24 \
+    --target_file_size_base=134217728 \
+    --max_bytes_for_level_base=1073741824 \
+    --disable_wal=0 \
+    --wal_dir=/dev/shm/rocksdb \
+    --sync=0 \
+    --disable_data_sync=1 \
+    --verify_checksum=1 \
+    --delete_obsolete_files_period_micros=314572800 \
+    --max_grandparent_overlap_factor=10 \
+    --use_plain_table=1 \
+    --open_files=-1 \
+    --mmap_read=1 \
+    --mmap_write=0 \
+    --memtablerep=prefix_hash \
+    --bloom_bits=10 \
+    --bloom_locality=1 \
+    --perf_level=0"
+
+# prepare a in-memory DB with 50M keys, total DB size is ~6G
+./db_bench \
+    $common_in_mem_args \
+    --statistics=0 \
+    --max_background_compactions=16 \
+    --max_background_flushes=16 \
+    --benchmarks=filluniquerandom \
+    --use_existing_db=0 \
+    --num=52428800 \
+    --threads=1 > /dev/null
+
+# Readwhilewriting
+./db_bench \
+    $common_in_mem_args \
+    --statistics=1 \
+    --max_background_compactions=4 \
+    --max_background_flushes=0 \
+    --benchmarks=readwhilewriting\
+    --use_existing_db=1 \
+    --duration=600 \
+    --threads=32 \
+    --writes_per_second=81920 > ${STAT_FILE}.readwhilewriting_in_ram
+
+# Seekrandomwhilewriting
+./db_bench \
+    $common_in_mem_args \
+    --statistics=1 \
+    --max_background_compactions=4 \
+    --max_background_flushes=0 \
+    --benchmarks=seekrandomwhilewriting \
+    --use_existing_db=1 \
+    --use_tailing_iterator=1 \
+    --duration=600 \
+    --threads=32 \
+    --writes_per_second=81920 > ${STAT_FILE}.seekwhilewriting_in_ram
+
+
 # send data to ods
 function send_to_ods {
   key="$1"
@@ -308,9 +378,9 @@ function send_benchmark_to_ods {
   file="$3"
 
   QPS=$(grep $bench $file | awk '{print $5}')
-  P50_MICROS=$(grep $bench $file -A 4 | tail -n1 | awk '{print $3}' )
-  P75_MICROS=$(grep $bench $file -A 4 | tail -n1 | awk '{print $5}' )
-  P99_MICROS=$(grep $bench $file -A 4 | tail -n1 | awk '{print $7}' )
+  P50_MICROS=$(grep $bench $file -A 6 | grep "Percentiles" | awk '{print $3}' )
+  P75_MICROS=$(grep $bench $file -A 6 | grep "Percentiles" | awk '{print $5}' )
+  P99_MICROS=$(grep $bench $file -A 6 | grep "Percentiles" | awk '{print $7}' )
 
   send_to_ods rocksdb.build.$bench_key.qps $QPS
   send_to_ods rocksdb.build.$bench_key.p50_micros $P50_MICROS
@@ -328,3 +398,5 @@ send_benchmark_to_ods readrandom readrandom_fillunique_random $STAT_FILE.readran
 send_benchmark_to_ods fillrandom memtablefillrandom $STAT_FILE.memtablefillreadrandom
 send_benchmark_to_ods readrandom memtablereadrandom $STAT_FILE.memtablefillreadrandom
 send_benchmark_to_ods readwhilewriting readwhilewriting $STAT_FILE.readwhilewriting
+send_benchmark_to_ods readwhilewriting readwhilewriting_in_ram ${STAT_FILE}.readwhilewriting_in_ram
+send_benchmark_to_ods seekrandomwhilewriting seekwhilewriting_in_ram ${STAT_FILE}.seekwhilewriting_in_ram

db/builder.cc

@@ -42,7 +42,7 @@ Status BuildTable(const std::string& dbname, Env* env, const Options& options,
                   const SequenceNumber earliest_seqno_in_memtable,
                   const CompressionType compression) {
   Status s;
-  meta->file_size = 0;
+  meta->fd.file_size = 0;
   meta->smallest_seqno = meta->largest_seqno = 0;
   iter->SeekToFirst();
 
@@ -54,7 +54,7 @@ Status BuildTable(const std::string& dbname, Env* env, const Options& options,
     purge = false;
   }
 
-  std::string fname = TableFileName(dbname, meta->number);
+  std::string fname = TableFileName(dbname, meta->fd.GetNumber());
   if (iter->Valid()) {
     unique_ptr<WritableFile> file;
     s = env->NewWritableFile(fname, &file, soptions);
@@ -177,8 +177,8 @@ Status BuildTable(const std::string& dbname, Env* env, const Options& options,
     if (s.ok()) {
       s = builder->Finish();
       if (s.ok()) {
-        meta->file_size = builder->FileSize();
-        assert(meta->file_size > 0);
+        meta->fd.file_size = builder->FileSize();
+        assert(meta->fd.GetFileSize() > 0);
       }
     } else {
       builder->Abandon();
@@ -202,7 +202,7 @@ Status BuildTable(const std::string& dbname, Env* env, const Options& options,
     if (s.ok()) {
       // Verify that the table is usable
       Iterator* it = table_cache->NewIterator(ReadOptions(), soptions,
-                                              internal_comparator, *meta);
+                                              internal_comparator, meta->fd);
       s = it->status();
       delete it;
     }
@@ -213,7 +213,7 @@ Status BuildTable(const std::string& dbname, Env* env, const Options& options,
     s = iter->status();
   }
 
-  if (s.ok() && meta->file_size > 0) {
+  if (s.ok() && meta->fd.GetFileSize() > 0) {
     // Keep it
   } else {
     env->DeleteFile(fname);

db/builder.h

@@ -29,7 +29,7 @@ extern TableBuilder* NewTableBuilder(
     WritableFile* file, CompressionType compression_type);
 
 // Build a Table file from the contents of *iter.  The generated file
-// will be named according to meta->number.  On success, the rest of
+// will be named according to number specified in meta. On success, the rest of
 // *meta will be filled with metadata about the generated table.
 // If no data is present in *iter, meta->file_size will be set to
 // zero, and no Table file will be produced.

db/c.cc

@@ -14,6 +14,7 @@
 #include <stdlib.h>
 #include <unistd.h>
 #include "rocksdb/cache.h"
+#include "rocksdb/compaction_filter.h"
 #include "rocksdb/comparator.h"
 #include "rocksdb/db.h"
 #include "rocksdb/env.h"
@@ -30,6 +31,7 @@
 #include "rocksdb/table.h"
 
 using rocksdb::Cache;
+using rocksdb::CompactionFilter;
 using rocksdb::Comparator;
 using rocksdb::CompressionType;
 using rocksdb::DB;
@@ -77,6 +79,49 @@ struct rocksdb_logger_t          { shared_ptr<Logger>  rep; };
 struct rocksdb_cache_t           { shared_ptr<Cache>   rep; };
 struct rocksdb_livefiles_t       { std::vector<LiveFileMetaData> rep; };
 
+struct rocksdb_compactionfilter_t : public CompactionFilter {
+  void* state_;
+  void (*destructor_)(void*);
+  unsigned char (*filter_)(
+      void*,
+      int level,
+      const char* key, size_t key_length,
+      const char* existing_value, size_t value_length,
+      char** new_value, size_t *new_value_length,
+      unsigned char* value_changed);
+  const char* (*name_)(void*);
+
+  virtual ~rocksdb_compactionfilter_t() {
+    (*destructor_)(state_);
+  }
+
+  virtual bool Filter(
+      int level,
+      const Slice& key,
+      const Slice& existing_value,
+      std::string* new_value,
+      bool* value_changed) const {
+    char* c_new_value = NULL;
+    size_t new_value_length = 0;
+    unsigned char c_value_changed = 0;
+    unsigned char result = (*filter_)(
+        state_,
+        level,
+        key.data(), key.size(),
+        existing_value.data(), existing_value.size(),
+        &c_new_value, &new_value_length, &c_value_changed);
+    if (c_value_changed) {
+      new_value->assign(c_new_value, new_value_length);
+      *value_changed = true;
+    }
+    return result;
+  }
+
+  virtual const char* Name() const {
+    return (*name_)(state_);
+  }
+};
+
 struct rocksdb_comparator_t : public Comparator {
   void* state_;
   void (*destructor_)(void*);
@@ -631,6 +676,12 @@ void rocksdb_options_destroy(rocksdb_options_t* options) {
   delete options;
 }
 
+void rocksdb_options_set_compaction_filter(
+    rocksdb_options_t* opt,
+    rocksdb_compactionfilter_t* filter) {
+  opt->rep.compaction_filter = filter;
+}
+
 void rocksdb_options_set_comparator(
     rocksdb_options_t* opt,
     rocksdb_comparator_t* cmp) {
@@ -1119,10 +1170,32 @@ DB::GetUpdatesSince
 DB::GetDbIdentity
 DB::RunManualCompaction
 custom cache
-compaction_filter
 table_properties_collectors
 */
 
+rocksdb_compactionfilter_t* rocksdb_compactionfilter_create(
+    void* state,
+    void (*destructor)(void*),
+    unsigned char (*filter)(
+        void*,
+        int level,
+        const char* key, size_t key_length,
+        const char* existing_value, size_t value_length,
+        char** new_value, size_t *new_value_length,
+        unsigned char* value_changed),
+    const char* (*name)(void*)) {
+  rocksdb_compactionfilter_t* result = new rocksdb_compactionfilter_t;
+  result->state_ = state;
+  result->destructor_ = destructor;
+  result->filter_ = filter;
+  result->name_ = name;
+  return result;
+}
+
+void rocksdb_compactionfilter_destroy(rocksdb_compactionfilter_t* filter) {
+  delete filter;
+}
+
 rocksdb_comparator_t* rocksdb_comparator_create(
     void* state,
     void (*destructor)(void*),

db/c_test.c

@@ -154,6 +154,28 @@ static unsigned char FilterKeyMatch(
   return fake_filter_result;
 }
 
+// Custom compaction filter
+static void CFilterDestroy(void* arg) {}
+static const char* CFilterName(void* arg) { return "foo"; }
+static unsigned char CFilterFilter(void* arg, int level, const char* key,
+                                   size_t key_length,
+                                   const char* existing_value,
+                                   size_t value_length, char** new_value,
+                                   size_t* new_value_length,
+                                   unsigned char* value_changed) {
+  if (key_length == 3) {
+    if (memcmp(key, "bar", key_length) == 0) {
+      return 1;
+    } else if (memcmp(key, "baz", key_length) == 0) {
+      *value_changed = 1;
+      *new_value = "newbazvalue";
+      *new_value_length = 11;
+      return 0;
+    }
+  }
+  return 0;
+}
+
 // Custom merge operator
 static void MergeOperatorDestroy(void* arg) { }
 static const char* MergeOperatorName(void* arg) {
@@ -407,6 +429,37 @@ int main(int argc, char** argv) {
     rocksdb_filterpolicy_destroy(policy);
   }
 
+  StartPhase("compaction_filter");
+  {
+    rocksdb_compactionfilter_t* cfilter;
+    cfilter = rocksdb_compactionfilter_create(NULL, CFilterDestroy,
+                                              CFilterFilter, CFilterName);
+    // Create new database
+    rocksdb_close(db);
+    rocksdb_destroy_db(options, dbname, &err);
+    rocksdb_options_set_compaction_filter(options, cfilter);
+    db = rocksdb_open(options, dbname, &err);
+    CheckNoError(err);
+    rocksdb_put(db, woptions, "foo", 3, "foovalue", 8, &err);
+    CheckNoError(err);
+    CheckGet(db, roptions, "foo", "foovalue");
+    rocksdb_put(db, woptions, "bar", 3, "barvalue", 8, &err);
+    CheckNoError(err);
+    CheckGet(db, roptions, "bar", "barvalue");
+    rocksdb_put(db, woptions, "baz", 3, "bazvalue", 8, &err);
+    CheckNoError(err);
+    CheckGet(db, roptions, "baz", "bazvalue");
+
+    // Force compaction
+    rocksdb_compact_range(db, NULL, 0, NULL, 0);
+    // should have filtered bar, but not foo
+    CheckGet(db, roptions, "foo", "foovalue");
+    CheckGet(db, roptions, "bar", NULL);
+    CheckGet(db, roptions, "baz", "newbazvalue");
+
+    rocksdb_compactionfilter_destroy(cfilter);
+  }
+
   StartPhase("merge_operator");
   {
     rocksdb_mergeoperator_t* merge_operator;

db/column_family.cc

@@ -12,6 +12,7 @@
 #include <vector>
 #include <string>
 #include <algorithm>
+#include <limits>
 
 #include "db/db_impl.h"
 #include "db/version_set.h"
@@ -91,6 +92,9 @@ ColumnFamilyOptions SanitizeOptions(const InternalKeyComparator* icmp,
   if (result.soft_rate_limit > result.hard_rate_limit) {
     result.soft_rate_limit = result.hard_rate_limit;
   }
+  if (result.max_write_buffer_number < 2) {
+    result.max_write_buffer_number = 2;
+  }
   if (!result.prefix_extractor) {
     assert(result.memtable_factory);
     Slice name = result.memtable_factory->Name();
@@ -104,14 +108,26 @@ ColumnFamilyOptions SanitizeOptions(const InternalKeyComparator* icmp,
   // All user defined properties collectors will be wrapped by
   // UserKeyTablePropertiesCollector since for them they only have the
   // knowledge of the user keys; internal keys are invisible to them.
-  auto& collectors = result.table_properties_collectors;
-  for (size_t i = 0; i < result.table_properties_collectors.size(); ++i) {
-    assert(collectors[i]);
-    collectors[i] =
-        std::make_shared<UserKeyTablePropertiesCollector>(collectors[i]);
+  auto& collector_factories = result.table_properties_collector_factories;
+  for (size_t i = 0; i < result.table_properties_collector_factories.size();
+       ++i) {
+    assert(collector_factories[i]);
+    collector_factories[i] =
+        std::make_shared<UserKeyTablePropertiesCollectorFactory>(
+            collector_factories[i]);
   }
   // Add collector to collect internal key statistics
-  collectors.push_back(std::make_shared<InternalKeyPropertiesCollector>());
+  collector_factories.push_back(
+      std::make_shared<InternalKeyPropertiesCollectorFactory>());
+
+  if (result.compaction_style == kCompactionStyleFIFO) {
+    result.num_levels = 1;
+    // since we delete level0 files in FIFO compaction when there are too many
+    // of them, these options don't really mean anything
+    result.level0_file_num_compaction_trigger = std::numeric_limits<int>::max();
+    result.level0_slowdown_writes_trigger = std::numeric_limits<int>::max();
+    result.level0_stop_writes_trigger = std::numeric_limits<int>::max();
+  }
 
   return result;
 }
@@ -193,7 +209,7 @@ ColumnFamilyData::ColumnFamilyData(const std::string& dbname, uint32_t id,
       options_(*db_options, SanitizeOptions(&internal_comparator_,
                                             &internal_filter_policy_, options)),
       mem_(nullptr),
-      imm_(options.min_write_buffer_number_to_merge),
+      imm_(options_.min_write_buffer_number_to_merge),
       super_version_(nullptr),
       super_version_number_(0),
       local_sv_(new ThreadLocalPtr(&SuperVersionUnrefHandle)),
@@ -206,16 +222,20 @@ ColumnFamilyData::ColumnFamilyData(const std::string& dbname, uint32_t id,
 
   // if dummy_versions is nullptr, then this is a dummy column family.
   if (dummy_versions != nullptr) {
-    internal_stats_.reset(new InternalStats(options.num_levels, db_options->env,
-                                            db_options->statistics.get()));
+    internal_stats_.reset(new InternalStats(
+        options_.num_levels, db_options->env, db_options->statistics.get()));
     table_cache_.reset(
         new TableCache(dbname, &options_, storage_options, table_cache));
     if (options_.compaction_style == kCompactionStyleUniversal) {
       compaction_picker_.reset(
           new UniversalCompactionPicker(&options_, &internal_comparator_));
-    } else {
+    } else if (options_.compaction_style == kCompactionStyleLevel) {
       compaction_picker_.reset(
           new LevelCompactionPicker(&options_, &internal_comparator_));
+    } else {
+      assert(options_.compaction_style == kCompactionStyleFIFO);
+      compaction_picker_.reset(
+          new FIFOCompactionPicker(&options_, &internal_comparator_));
     }
 
     Log(options_.info_log, "Options for column family \"%s\":\n",
@@ -489,6 +509,10 @@ void ColumnFamilySet::UpdateMaxColumnFamily(uint32_t new_max_column_family) {
   max_column_family_ = std::max(new_max_column_family, max_column_family_);
 }
 
+size_t ColumnFamilySet::NumberOfColumnFamilies() const {
+  return column_families_.size();
+}
+
 // under a DB mutex
 ColumnFamilyData* ColumnFamilySet::CreateColumnFamily(
     const std::string& name, uint32_t id, Version* dummy_versions,

db/column_family.h

@@ -180,7 +180,7 @@ class ColumnFamilyData {
   void SetCurrent(Version* current);
   void CreateNewMemtable();
 
-  TableCache* table_cache() { return table_cache_.get(); }
+  TableCache* table_cache() const { return table_cache_.get(); }
 
   // See documentation in compaction_picker.h
   Compaction* PickCompaction(LogBuffer* log_buffer);
@@ -302,8 +302,8 @@ class ColumnFamilyData {
 // family might get dropped when the DB mutex is released
 // * GetDefault() -- thread safe
 // * GetColumnFamily() -- either inside of DB mutex or call Lock() <-> Unlock()
-// * GetNextColumnFamilyID(), GetMaxColumnFamily(), UpdateMaxColumnFamily() --
-// inside of DB mutex
+// * GetNextColumnFamilyID(), GetMaxColumnFamily(), UpdateMaxColumnFamily(),
+// NumberOfColumnFamilies -- inside of DB mutex
 class ColumnFamilySet {
  public:
   // ColumnFamilySet supports iteration
@@ -342,6 +342,7 @@ class ColumnFamilySet {
   uint32_t GetNextColumnFamilyID();
   uint32_t GetMaxColumnFamily();
   void UpdateMaxColumnFamily(uint32_t new_max_column_family);
+  size_t NumberOfColumnFamilies() const;
 
   ColumnFamilyData* CreateColumnFamily(const std::string& name, uint32_t id,
                                        Version* dummy_version,

db/column_family_test.cc

@@ -970,6 +970,15 @@ TEST(ColumnFamilyTest, FlushStaleColumnFamilies) {
   Close();
 }
 
+TEST(ColumnFamilyTest, CreateMissingColumnFamilies) {
+  Status s = TryOpen({"one", "two"});
+  ASSERT_TRUE(!s.ok());
+  db_options_.create_missing_column_families = true;
+  s = TryOpen({"default", "one", "two"});
+  ASSERT_TRUE(s.ok());
+  Close();
+}
+
 }  // namespace rocksdb
 
 int main(int argc, char** argv) {

db/compaction.cc

@@ -8,14 +8,20 @@
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
 #include "db/compaction.h"
+
+#define __STDC_FORMAT_MACROS
+#include <inttypes.h>
+#include <vector>
+
 #include "db/column_family.h"
+#include "util/logging.h"
 
 namespace rocksdb {
 
 static uint64_t TotalFileSize(const std::vector<FileMetaData*>& files) {
   uint64_t sum = 0;
   for (size_t i = 0; i < files.size() && files[i]; i++) {
-    sum += files[i]->file_size;
+    sum += files[i]->fd.GetFileSize();
   }
   return sum;
 }
@@ -23,7 +29,8 @@ static uint64_t TotalFileSize(const std::vector<FileMetaData*>& files) {
 Compaction::Compaction(Version* input_version, int level, int out_level,
                        uint64_t target_file_size,
                        uint64_t max_grandparent_overlap_bytes,
-                       bool seek_compaction, bool enable_compression)
+                       bool seek_compaction, bool enable_compression,
+                       bool deletion_compaction)
     : level_(level),
       out_level_(out_level),
       max_output_file_size_(target_file_size),
@@ -33,6 +40,7 @@ Compaction::Compaction(Version* input_version, int level, int out_level,
       cfd_(input_version_->cfd_),
       seek_compaction_(seek_compaction),
       enable_compression_(enable_compression),
+      deletion_compaction_(deletion_compaction),
       grandparent_index_(0),
       seen_key_(false),
       overlapped_bytes_(0),
@@ -77,15 +85,18 @@ bool Compaction::IsTrivialMove() const {
           TotalFileSize(grandparents_) <= max_grandparent_overlap_bytes_);
 }
 
+bool Compaction::IsDeletionCompaction() const { return deletion_compaction_; }
+
 void Compaction::AddInputDeletions(VersionEdit* edit) {
   for (int which = 0; which < 2; which++) {
     for (size_t i = 0; i < inputs_[which].size(); i++) {
-      edit->DeleteFile(level_ + which, inputs_[which][i]->number);
+      edit->DeleteFile(level_ + which, inputs_[which][i]->fd.GetNumber());
     }
   }
 }
 
 bool Compaction::IsBaseLevelForKey(const Slice& user_key) {
+  assert(cfd_->options()->compaction_style != kCompactionStyleFIFO);
   if (cfd_->options()->compaction_style == kCompactionStyleUniversal) {
     return bottommost_level_;
   }
@@ -116,7 +127,7 @@ bool Compaction::ShouldStopBefore(const Slice& internal_key) {
       icmp->Compare(internal_key,
                     grandparents_[grandparent_index_]->largest.Encode()) > 0) {
     if (seen_key_) {
-      overlapped_bytes_ += grandparents_[grandparent_index_]->file_size;
+      overlapped_bytes_ += grandparents_[grandparent_index_]->fd.GetFileSize();
     }
     assert(grandparent_index_ + 1 >= grandparents_.size() ||
            icmp->Compare(grandparents_[grandparent_index_]->largest.Encode(),
@@ -149,6 +160,7 @@ void Compaction::MarkFilesBeingCompacted(bool value) {
 
 // Is this compaction producing files at the bottommost level?
 void Compaction::SetupBottomMostLevel(bool isManual) {
+  assert(cfd_->options()->compaction_style != kCompactionStyleFIFO);
   if (cfd_->options()->compaction_style == kCompactionStyleUniversal) {
     // If universal compaction style is used and manual
     // compaction is occuring, then we are guaranteed that
@@ -191,71 +203,67 @@ void Compaction::ResetNextCompactionIndex() {
   input_version_->ResetNextCompactionIndex(level_);
 }
 
-/*
-for sizes >=10TB, print "XXTB"
-for sizes >=10GB, print "XXGB"
-etc.
-*/
-static void FileSizeSummary(unsigned long long sz, char* output, int len) {
-  const unsigned long long ull10 = 10;
-  if (sz >= ull10<<40) {
-    snprintf(output, len, "%lluTB", sz>>40);
-  } else if (sz >= ull10<<30) {
-    snprintf(output, len, "%lluGB", sz>>30);
-  } else if (sz >= ull10<<20) {
-    snprintf(output, len, "%lluMB", sz>>20);
-  } else if (sz >= ull10<<10) {
-    snprintf(output, len, "%lluKB", sz>>10);
-  } else {
-    snprintf(output, len, "%lluB", sz);
-  }
-}
-
-static int InputSummary(std::vector<FileMetaData*>& files, char* output,
-                         int len) {
+namespace {
+int InputSummary(const std::vector<FileMetaData*>& files, char* output,
+                 int len) {
   *output = '\0';
   int write = 0;
   for (unsigned int i = 0; i < files.size(); i++) {
     int sz = len - write;
     int ret;
     char sztxt[16];
-    FileSizeSummary((unsigned long long)files.at(i)->file_size, sztxt, 16);
-    ret = snprintf(output + write, sz, "%lu(%s) ",
-                   (unsigned long)files.at(i)->number,
-                   sztxt);
-    if (ret < 0 || ret >= sz)
-      break;
+    AppendHumanBytes(files.at(i)->fd.GetFileSize(), sztxt, 16);
+    ret = snprintf(output + write, sz, "%" PRIu64 "(%s) ",
+                   files.at(i)->fd.GetNumber(), sztxt);
+    if (ret < 0 || ret >= sz) break;
     write += ret;
   }
-  return write;
+  // if files.size() is non-zero, overwrite the last space
+  return write - !!files.size();
 }
+}  // namespace
 
 void Compaction::Summary(char* output, int len) {
-  int write = snprintf(output, len,
-      "Base version %lu Base level %d, seek compaction:%d, inputs: [",
-      (unsigned long)input_version_->GetVersionNumber(),
-      level_,
-      seek_compaction_);
+  int write =
+      snprintf(output, len, "Base version %" PRIu64
+                            " Base level %d, seek compaction:%d, inputs: [",
+               input_version_->GetVersionNumber(), level_, seek_compaction_);
   if (write < 0 || write >= len) {
     return;
   }
 
-  write += InputSummary(inputs_[0], output+write, len-write);
+  write += InputSummary(inputs_[0], output + write, len - write);
   if (write < 0 || write >= len) {
     return;
   }
 
-  write += snprintf(output+write, len-write, "],[");
+  write += snprintf(output + write, len - write, "], [");
   if (write < 0 || write >= len) {
     return;
   }
 
-  write += InputSummary(inputs_[1], output+write, len-write);
+  write += InputSummary(inputs_[1], output + write, len - write);
   if (write < 0 || write >= len) {
     return;
   }
 
-  snprintf(output+write, len-write, "]");
+  snprintf(output + write, len - write, "]");
+}
+
+uint64_t Compaction::OutputFilePreallocationSize() {
+  uint64_t preallocation_size = 0;
+
+  if (cfd_->options()->compaction_style == kCompactionStyleLevel) {
+    preallocation_size =
+        cfd_->compaction_picker()->MaxFileSizeForLevel(output_level());
+  } else {
+    for (const auto& f : inputs_[0]) {
+      preallocation_size += f->fd.GetFileSize();
+    }
+  }
+  // Over-estimate slightly so we don't end up just barely crossing
+  // the threshold
+  return preallocation_size * 1.1;
 }
 
 }  // namespace rocksdb

db/compaction.h

@@ -54,6 +54,9 @@ class Compaction {
   // moving a single input file to the next level (no merging or splitting)
   bool IsTrivialMove() const;
 
+  // If true, just delete all files in inputs_[0]
+  bool IsDeletionCompaction() const;
+
   // Add all inputs to this compaction as delete operations to *edit.
   void AddInputDeletions(VersionEdit* edit);
 
@@ -88,14 +91,21 @@ class Compaction {
   // Was this compaction triggered manually by the client?
   bool IsManualCompaction() { return is_manual_compaction_; }
 
+  // Returns a number of byte that the output file should be preallocated to
+  // In level compaction, that is max_file_size_. In universal compaction, that
+  // is the sum of all input file sizes
+  uint64_t OutputFilePreallocationSize();
+
  private:
   friend class CompactionPicker;
   friend class UniversalCompactionPicker;
+  friend class FIFOCompactionPicker;
   friend class LevelCompactionPicker;
 
   Compaction(Version* input_version, int level, int out_level,
              uint64_t target_file_size, uint64_t max_grandparent_overlap_bytes,
-             bool seek_compaction = false, bool enable_compression = true);
+             bool seek_compaction = false, bool enable_compression = true,
+             bool deletion_compaction = false);
 
   int level_;
   int out_level_; // levels to which output files are stored
@@ -108,6 +118,8 @@ class Compaction {
 
   bool seek_compaction_;
   bool enable_compression_;
+  // if true, just delete files in inputs_[0]
+  bool deletion_compaction_;
 
   // Each compaction reads inputs from "level_" and "level_+1"
   std::vector<FileMetaData*> inputs_[2];      // The two sets of inputs

db/compaction_picker.cc

@@ -9,6 +9,8 @@
 
 #include "db/compaction_picker.h"
 
+#define __STDC_FORMAT_MACROS
+#include <inttypes.h>
 #include <limits>
 #include "util/log_buffer.h"
 #include "util/statistics.h"
@@ -20,7 +22,7 @@ namespace {
 uint64_t TotalFileSize(const std::vector<FileMetaData*>& files) {
   uint64_t sum = 0;
   for (size_t i = 0; i < files.size() && files[i]; i++) {
-    sum += files[i]->file_size;
+    sum += files[i]->fd.GetFileSize();
   }
   return sum;
 }
@@ -78,7 +80,7 @@ void CompactionPicker::SizeBeingCompacted(std::vector<uint64_t>& sizes) {
     for (auto c : compactions_in_progress_[level]) {
       assert(c->level() == level);
       for (int i = 0; i < c->num_input_files(0); i++) {
-        total += c->input(0,i)->file_size;
+        total += c->input(0, i)->fd.GetFileSize();
       }
     }
     sizes[level] = total;
@@ -307,6 +309,9 @@ Compaction* CompactionPicker::CompactRange(Version* version, int input_level,
                                            const InternalKey* begin,
                                            const InternalKey* end,
                                            InternalKey** compaction_end) {
+  // CompactionPickerFIFO has its own implementation of compact range
+  assert(options_->compaction_style != kCompactionStyleFIFO);
+
   std::vector<FileMetaData*> inputs;
   bool covering_the_whole_range = true;
 
@@ -330,7 +335,7 @@ Compaction* CompactionPicker::CompactRange(Version* version, int input_level,
         MaxFileSizeForLevel(input_level) * options_->source_compaction_factor;
     uint64_t total = 0;
     for (size_t i = 0; i + 1 < inputs.size(); ++i) {
-      uint64_t s = inputs[i]->file_size;
+      uint64_t s = inputs[i]->fd.GetFileSize();
       total += s;
       if (total >= limit) {
         **compaction_end = inputs[i + 1]->smallest;
@@ -503,10 +508,11 @@ Compaction* LevelCompactionPicker::PickCompactionBySize(Version* version,
     FileMetaData* f = c->input_version_->files_[level][index];
 
     // check to verify files are arranged in descending size
-    assert((i == file_size.size() - 1) ||
-           (i >= Version::number_of_files_to_sort_ - 1) ||
-           (f->file_size >=
-            c->input_version_->files_[level][file_size[i + 1]]->file_size));
+    assert(
+        (i == file_size.size() - 1) ||
+        (i >= Version::number_of_files_to_sort_ - 1) ||
+        (f->fd.GetFileSize() >=
+         c->input_version_->files_[level][file_size[i + 1]]->fd.GetFileSize()));
 
     // do not pick a file to compact if it is being compacted
     // from n-1 level.
@@ -675,19 +681,21 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalReadAmp(
         candidate_count = 1;
         break;
       }
-      LogToBuffer(
-          log_buffer, "[%s] Universal: file %lu[%d] being compacted, skipping",
-          version->cfd_->GetName().c_str(), (unsigned long)f->number, loop);
+      LogToBuffer(log_buffer,
+                  "[%s] Universal: file %lu[%d] being compacted, skipping",
+                  version->cfd_->GetName().c_str(),
+                  (unsigned long)f->fd.GetNumber(), loop);
       f = nullptr;
     }
 
     // This file is not being compacted. Consider it as the
     // first candidate to be compacted.
-    uint64_t candidate_size =  f != nullptr? f->file_size : 0;
+    uint64_t candidate_size = f != nullptr ? f->fd.GetFileSize() : 0;
     if (f != nullptr) {
-      LogToBuffer(
-          log_buffer, "[%s] Universal: Possible candidate file %lu[%d].",
-          version->cfd_->GetName().c_str(), (unsigned long)f->number, loop);
+      LogToBuffer(log_buffer,
+                  "[%s] Universal: Possible candidate file %lu[%d].",
+                  version->cfd_->GetName().c_str(),
+                  (unsigned long)f->fd.GetNumber(), loop);
     }
 
     // Check if the suceeding files need compaction.
@@ -706,13 +714,13 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalReadAmp(
       // kCompactionStopStyleSimilarSize, it's simply the size of the last
       // picked file.
       uint64_t sz = (candidate_size * (100L + ratio)) /100;
-      if (sz < f->file_size) {
+      if (sz < f->fd.GetFileSize()) {
         break;
       }
       if (options_->compaction_options_universal.stop_style == kCompactionStopStyleSimilarSize) {
         // Similar-size stopping rule: also check the last picked file isn't
         // far larger than the next candidate file.
-        sz = (f->file_size * (100L + ratio)) / 100;
+        sz = (f->fd.GetFileSize() * (100L + ratio)) / 100;
         if (sz < candidate_size) {
           // If the small file we've encountered begins a run of similar-size
           // files, we'll pick them up on a future iteration of the outer
@@ -720,9 +728,9 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalReadAmp(
           // by the last-resort read amp strategy which disregards size ratios.
           break;
         }
-        candidate_size = f->file_size;
+        candidate_size = f->fd.GetFileSize();
       } else { // default kCompactionStopStyleTotalSize
-        candidate_size += f->file_size;
+        candidate_size += f->fd.GetFileSize();
       }
       candidate_count++;
     }
@@ -739,8 +747,9 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalReadAmp(
        FileMetaData* f = version->files_[level][index];
        LogToBuffer(log_buffer,
                    "[%s] Universal: Skipping file %lu[%d] with size %lu %d\n",
-                   version->cfd_->GetName().c_str(), (unsigned long)f->number,
-                   i, (unsigned long)f->file_size, f->being_compacted);
+                   version->cfd_->GetName().c_str(),
+                   (unsigned long)f->fd.GetNumber(), i,
+                   (unsigned long)f->fd.GetFileSize(), f->being_compacted);
       }
     }
   }
@@ -758,7 +767,8 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalReadAmp(
     uint64_t older_file_size = 0;
     for (unsigned int i = file_by_time.size() - 1; i >= first_index_after;
         i--) {
-      older_file_size += version->files_[level][file_by_time[i]]->file_size;
+      older_file_size +=
+          version->files_[level][file_by_time[i]]->fd.GetFileSize();
       if (older_file_size * 100L >= total_size * (long) ratio_to_compress) {
         enable_compression = false;
         break;
@@ -774,10 +784,10 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalReadAmp(
     int index = file_by_time[i];
     FileMetaData* f = c->input_version_->files_[level][index];
     c->inputs_[0].push_back(f);
-    LogToBuffer(log_buffer,
-                "[%s] Universal: Picking file %lu[%d] with size %lu\n",
-                version->cfd_->GetName().c_str(), (unsigned long)f->number, i,
-                (unsigned long)f->file_size);
+    LogToBuffer(
+        log_buffer, "[%s] Universal: Picking file %lu[%d] with size %lu\n",
+        version->cfd_->GetName().c_str(), (unsigned long)f->fd.GetNumber(), i,
+        (unsigned long)f->fd.GetFileSize());
   }
   return c;
 }
@@ -813,10 +823,10 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalSizeAmp(
       start_index = loop;         // Consider this as the first candidate.
       break;
     }
-    LogToBuffer(log_buffer,
-                "[%s] Universal: skipping file %lu[%d] compacted %s",
-                version->cfd_->GetName().c_str(), (unsigned long)f->number,
-                loop, " cannot be a candidate to reduce size amp.\n");
+    LogToBuffer(
+        log_buffer, "[%s] Universal: skipping file %lu[%d] compacted %s",
+        version->cfd_->GetName().c_str(), (unsigned long)f->fd.GetNumber(),
+        loop, " cannot be a candidate to reduce size amp.\n");
     f = nullptr;
   }
   if (f == nullptr) {
@@ -824,8 +834,9 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalSizeAmp(
   }
 
   LogToBuffer(log_buffer, "[%s] Universal: First candidate file %lu[%d] %s",
-              version->cfd_->GetName().c_str(), (unsigned long)f->number,
-              start_index, " to reduce size amp.\n");
+              version->cfd_->GetName().c_str(),
+              (unsigned long)f->fd.GetNumber(), start_index,
+              " to reduce size amp.\n");
 
   // keep adding up all the remaining files
   for (unsigned int loop = start_index; loop < file_by_time.size() - 1;
@@ -835,11 +846,12 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalSizeAmp(
     if (f->being_compacted) {
       LogToBuffer(
           log_buffer, "[%s] Universal: Possible candidate file %lu[%d] %s.",
-          version->cfd_->GetName().c_str(), (unsigned long)f->number, loop,
+          version->cfd_->GetName().c_str(), (unsigned long)f->fd.GetNumber(),
+          loop,
           " is already being compacted. No size amp reduction possible.\n");
       return nullptr;
     }
-    candidate_size += f->file_size;
+    candidate_size += f->fd.GetFileSize();
     candidate_count++;
   }
   if (candidate_count == 0) {
@@ -848,7 +860,7 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalSizeAmp(
 
   // size of earliest file
   int index = file_by_time[file_by_time.size() - 1];
-  uint64_t earliest_file_size = version->files_[level][index]->file_size;
+  uint64_t earliest_file_size = version->files_[level][index]->fd.GetFileSize();
 
   // size amplification = percentage of additional size
   if (candidate_size * 100 < ratio * earliest_file_size) {
@@ -880,10 +892,77 @@ Compaction* UniversalCompactionPicker::PickCompactionUniversalSizeAmp(
     c->inputs_[0].push_back(f);
     LogToBuffer(log_buffer,
                 "[%s] Universal: size amp picking file %lu[%d] with size %lu",
-                version->cfd_->GetName().c_str(), (unsigned long)f->number,
-                index, (unsigned long)f->file_size);
+                version->cfd_->GetName().c_str(),
+                (unsigned long)f->fd.GetNumber(), index,
+                (unsigned long)f->fd.GetFileSize());
   }
   return c;
 }
 
+Compaction* FIFOCompactionPicker::PickCompaction(Version* version,
+                                                 LogBuffer* log_buffer) {
+  assert(version->NumberLevels() == 1);
+  uint64_t total_size = 0;
+  for (const auto& file : version->files_[0]) {
+    total_size += file->fd.GetFileSize();
+  }
+
+  if (total_size <= options_->compaction_options_fifo.max_table_files_size ||
+      version->files_[0].size() == 0) {
+    // total size not exceeded
+    LogToBuffer(log_buffer,
+                "[%s] FIFO compaction: nothing to do. Total size %" PRIu64
+                ", max size %" PRIu64 "\n",
+                version->cfd_->GetName().c_str(), total_size,
+                options_->compaction_options_fifo.max_table_files_size);
+    return nullptr;
+  }
+
+  if (compactions_in_progress_[0].size() > 0) {
+    LogToBuffer(log_buffer,
+                "[%s] FIFO compaction: Already executing compaction. No need "
+                "to run parallel compactions since compactions are very fast",
+                version->cfd_->GetName().c_str());
+    return nullptr;
+  }
+
+  Compaction* c = new Compaction(version, 0, 0, 0, 0, false, false,
+                                 true /* is deletion compaction */);
+  // delete old files (FIFO)
+  for (auto ritr = version->files_[0].rbegin();
+       ritr != version->files_[0].rend(); ++ritr) {
+    auto f = *ritr;
+    total_size -= f->fd.GetFileSize();
+    c->inputs_[0].push_back(f);
+    char tmp_fsize[16];
+    AppendHumanBytes(f->fd.GetFileSize(), tmp_fsize, sizeof(tmp_fsize));
+    LogToBuffer(log_buffer, "[%s] FIFO compaction: picking file %" PRIu64
+                            " with size %s for deletion",
+                version->cfd_->GetName().c_str(), f->fd.GetNumber(), tmp_fsize);
+    if (total_size <= options_->compaction_options_fifo.max_table_files_size) {
+      break;
+    }
+  }
+
+  c->MarkFilesBeingCompacted(true);
+  compactions_in_progress_[0].insert(c);
+
+  return c;
+}
+
+Compaction* FIFOCompactionPicker::CompactRange(Version* version,
+                                               int input_level,
+                                               int output_level,
+                                               const InternalKey* begin,
+                                               const InternalKey* end,
+                                               InternalKey** compaction_end) {
+  assert(input_level == 0);
+  assert(output_level == 0);
+  *compaction_end = nullptr;
+  LogBuffer log_buffer(InfoLogLevel::INFO_LEVEL, options_->info_log.get());
+  auto c = PickCompaction(version, &log_buffer);
+  log_buffer.FlushBufferToLog();
+  return c;
+}
+
 }  // namespace rocksdb

db/compaction_picker.h

@@ -47,9 +47,10 @@ class CompactionPicker {
   // compaction_end will be set to nullptr.
   // Client is responsible for compaction_end storage -- when called,
   // *compaction_end should point to valid InternalKey!
-  Compaction* CompactRange(Version* version, int input_level, int output_level,
-                           const InternalKey* begin, const InternalKey* end,
-                           InternalKey** compaction_end);
+  virtual Compaction* CompactRange(Version* version, int input_level,
+                                   int output_level, const InternalKey* begin,
+                                   const InternalKey* end,
+                                   InternalKey** compaction_end);
 
   // Free up the files that participated in a compaction
   void ReleaseCompactionFiles(Compaction* c, Status status);
@@ -162,4 +163,19 @@ class LevelCompactionPicker : public CompactionPicker {
   Compaction* PickCompactionBySize(Version* version, int level, double score);
 };
 
+class FIFOCompactionPicker : public CompactionPicker {
+ public:
+  FIFOCompactionPicker(const Options* options,
+                       const InternalKeyComparator* icmp)
+      : CompactionPicker(options, icmp) {}
+
+  virtual Compaction* PickCompaction(Version* version,
+                                     LogBuffer* log_buffer) override;
+
+  virtual Compaction* CompactRange(Version* version, int input_level,
+                                   int output_level, const InternalKey* begin,
+                                   const InternalKey* end,
+                                   InternalKey** compaction_end) override;
+};
+
 }  // namespace rocksdb

db/db_bench.cc

@@ -8,6 +8,15 @@
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
 #define __STDC_FORMAT_MACROS
+
+#ifndef GFLAGS
+#include <cstdio>
+int main() {
+  fprintf(stderr, "Please install gflags to run rocksdb tools\n");
+  return 1;
+}
+#else
+
 #include <inttypes.h>
 #include <cstddef>
 #include <sys/types.h>
@@ -40,6 +49,9 @@
 #include "hdfs/env_hdfs.h"
 #include "utilities/merge_operators.h"
 
+using GFLAGS::ParseCommandLineFlags;
+using GFLAGS::RegisterFlagValidator;
+using GFLAGS::SetUsageMessage;
 
 DEFINE_string(benchmarks,
               "fillseq,"
@@ -147,14 +159,7 @@ DEFINE_int32(duration, 0, "Time in seconds for the random-ops tests to run."
 DEFINE_int32(value_size, 100, "Size of each value");
 
 
-// the maximum size of key in bytes
-static const int kMaxKeySize = 128;
 static bool ValidateKeySize(const char* flagname, int32_t value) {
-  if (value > kMaxKeySize) {
-    fprintf(stderr, "Invalid value for --%s: %d, must be < %d\n",
-            flagname, value, kMaxKeySize);
-    return false;
-  }
   return true;
 }
 
@@ -376,8 +381,7 @@ static bool ValidateCompressionLevel(const char* flagname, int32_t value) {
 }
 
 static const bool FLAGS_compression_level_dummy __attribute__((unused)) =
-    google::RegisterFlagValidator(&FLAGS_compression_level,
-                                  &ValidateCompressionLevel);
+    RegisterFlagValidator(&FLAGS_compression_level, &ValidateCompressionLevel);
 
 DEFINE_int32(min_level_to_compress, -1, "If non-negative, compression starts"
              " from this level. Levels with number < min_level_to_compress are"
@@ -461,6 +465,8 @@ static auto FLAGS_compaction_fadvice_e =
 
 DEFINE_bool(use_tailing_iterator, false,
             "Use tailing iterator to access a series of keys instead of get");
+DEFINE_int64(iter_refresh_interval_us, -1,
+             "How often to refresh iterators. Disable refresh when -1");
 
 DEFINE_bool(use_adaptive_mutex, rocksdb::Options().use_adaptive_mutex,
             "Use adaptive mutex");
@@ -529,33 +535,29 @@ DEFINE_string(merge_operator, "", "The merge operator to use with the database."
               " utilities/merge_operators.h");
 
 static const bool FLAGS_soft_rate_limit_dummy __attribute__((unused)) =
-  google::RegisterFlagValidator(&FLAGS_soft_rate_limit,
-                                &ValidateRateLimit);
+    RegisterFlagValidator(&FLAGS_soft_rate_limit, &ValidateRateLimit);
 
 static const bool FLAGS_hard_rate_limit_dummy __attribute__((unused)) =
-  google::RegisterFlagValidator(&FLAGS_hard_rate_limit, &ValidateRateLimit);
+    RegisterFlagValidator(&FLAGS_hard_rate_limit, &ValidateRateLimit);
 
 static const bool FLAGS_prefix_size_dummy __attribute__((unused)) =
-  google::RegisterFlagValidator(&FLAGS_prefix_size, &ValidatePrefixSize);
+    RegisterFlagValidator(&FLAGS_prefix_size, &ValidatePrefixSize);
 
 static const bool FLAGS_key_size_dummy __attribute__((unused)) =
-  google::RegisterFlagValidator(&FLAGS_key_size, &ValidateKeySize);
+    RegisterFlagValidator(&FLAGS_key_size, &ValidateKeySize);
 
 static const bool FLAGS_cache_numshardbits_dummy __attribute__((unused)) =
-  google::RegisterFlagValidator(&FLAGS_cache_numshardbits,
-                                &ValidateCacheNumshardbits);
+    RegisterFlagValidator(&FLAGS_cache_numshardbits,
+                          &ValidateCacheNumshardbits);
 
 static const bool FLAGS_readwritepercent_dummy __attribute__((unused)) =
-  google::RegisterFlagValidator(&FLAGS_readwritepercent,
-                                &ValidateInt32Percent);
+    RegisterFlagValidator(&FLAGS_readwritepercent, &ValidateInt32Percent);
 
 static const bool FLAGS_deletepercent_dummy __attribute__((unused)) =
-  google::RegisterFlagValidator(&FLAGS_deletepercent,
-                                &ValidateInt32Percent);
-static const bool
-  FLAGS_table_cache_numshardbits_dummy __attribute__((unused)) =
-  google::RegisterFlagValidator(&FLAGS_table_cache_numshardbits,
-                                &ValidateTableCacheNumshardbits);
+    RegisterFlagValidator(&FLAGS_deletepercent, &ValidateInt32Percent);
+static const bool FLAGS_table_cache_numshardbits_dummy __attribute__((unused)) =
+    RegisterFlagValidator(&FLAGS_table_cache_numshardbits,
+                          &ValidateTableCacheNumshardbits);
 
 namespace rocksdb {
 
@@ -1926,7 +1928,7 @@ class Benchmark {
     }
 
     char msg[100];
-    snprintf(msg, sizeof(msg), "(%" PRIu64 " of %" PRIu64 " found)",
+    snprintf(msg, sizeof(msg), "(%" PRIu64 " of %" PRIu64 " found)\n",
              found, read);
 
     thread->stats.AddMessage(msg);
@@ -2009,12 +2011,31 @@ class Benchmark {
         multi_iters.push_back(db->NewIterator(options));
       }
     }
+    uint64_t last_refresh = FLAGS_env->NowMicros();
 
     Slice key = AllocateKey();
     std::unique_ptr<const char[]> key_guard(key.data());
 
     Duration duration(FLAGS_duration, reads_);
     while (!duration.Done(1)) {
+      if (!FLAGS_use_tailing_iterator && FLAGS_iter_refresh_interval_us >= 0) {
+        uint64_t now = FLAGS_env->NowMicros();
+        if (now - last_refresh > (uint64_t)FLAGS_iter_refresh_interval_us) {
+          if (db_ != nullptr) {
+            delete single_iter;
+            single_iter = db_->NewIterator(options);
+          } else {
+            for (auto iter : multi_iters) {
+              delete iter;
+            }
+            multi_iters.clear();
+            for (DB* db : multi_dbs_) {
+              multi_iters.push_back(db->NewIterator(options));
+            }
+          }
+        }
+        last_refresh = now;
+      }
       // Pick a Iterator to use
       Iterator* iter_to_use = single_iter;
       if (single_iter == nullptr) {
@@ -2035,9 +2056,12 @@ class Benchmark {
     }
 
     char msg[100];
-    snprintf(msg, sizeof(msg), "(%" PRIu64 " of %" PRIu64 " found)",
+    snprintf(msg, sizeof(msg), "(%" PRIu64 " of %" PRIu64 " found)\n",
              found, read);
     thread->stats.AddMessage(msg);
+    if (FLAGS_perf_level > 0) {
+      thread->stats.AddMessage(perf_context.ToString());
+    }
   }
 
   void SeekRandomWhileWriting(ThreadState* thread) {
@@ -2561,9 +2585,9 @@ class Benchmark {
 
 int main(int argc, char** argv) {
   rocksdb::port::InstallStackTraceHandler();
-  google::SetUsageMessage(std::string("\nUSAGE:\n") + std::string(argv[0]) +
-                          " [OPTIONS]...");
-  google::ParseCommandLineFlags(&argc, &argv, true);
+  SetUsageMessage(std::string("\nUSAGE:\n") + std::string(argv[0]) +
+                  " [OPTIONS]...");
+  ParseCommandLineFlags(&argc, &argv, true);
 
   FLAGS_compaction_style_e = (rocksdb::CompactionStyle) FLAGS_compaction_style;
   if (FLAGS_statistics) {
@@ -2614,3 +2638,5 @@ int main(int argc, char** argv) {
   benchmark.Run();
   return 0;
 }
+
+#endif  // GFLAGS

db/db_filesnapshot.cc

@@ -29,8 +29,11 @@ Status DBImpl::DisableFileDeletions() {
   MutexLock l(&mutex_);
   ++disable_delete_obsolete_files_;
   if (disable_delete_obsolete_files_ == 1) {
-    // if not, it has already been disabled, so don't log anything
     Log(options_.info_log, "File Deletions Disabled");
+  } else {
+    Log(options_.info_log,
+        "File Deletions Disabled, but already disabled. Counter: %d",
+        disable_delete_obsolete_files_);
   }
   return Status::OK();
 }
@@ -50,6 +53,10 @@ Status DBImpl::EnableFileDeletions(bool force) {
       Log(options_.info_log, "File Deletions Enabled");
       should_purge_files = true;
       FindObsoleteFiles(deletion_state, true);
+    } else {
+      Log(options_.info_log,
+          "File Deletions Enable, but not really enabled. Counter: %d",
+          disable_delete_obsolete_files_);
     }
   }
   if (should_purge_files)  {

db/db_impl.cc

@@ -36,6 +36,7 @@
 #include "db/table_cache.h"
 #include "db/table_properties_collector.h"
 #include "db/tailing_iter.h"
+#include "db/forward_iterator.h"
 #include "db/transaction_log_impl.h"
 #include "db/version_set.h"
 #include "db/write_batch_internal.h"
@@ -487,7 +488,7 @@ Status DBImpl::NewDB() {
   }
   if (s.ok()) {
     // Make "CURRENT" file that points to the new manifest file.
-    s = SetCurrentFile(env_, dbname_, 1);
+    s = SetCurrentFile(env_, dbname_, 1, db_directory_.get());
   } else {
     env_->DeleteFile(manifest);
   }
@@ -643,8 +644,7 @@ void DBImpl::PurgeObsoleteFiles(DeletionState& state) {
   const char* kDumbDbName = "";
   for (auto file : state.sst_delete_files) {
     candidate_files.push_back(
-        TableFileName(kDumbDbName, file->number).substr(1)
-    );
+        TableFileName(kDumbDbName, file->fd.GetNumber()).substr(1));
     delete file;
   }
 
@@ -935,8 +935,19 @@ Status DBImpl::GetSortedWalsOfType(const std::string& path,
         continue;
       }
 
+      // Reproduce the race condition where a log file is moved
+      // to archived dir, between these two sync points, used in
+      // (DBTest,TransactionLogIteratorRace)
+      TEST_SYNC_POINT("DBImpl::GetSortedWalsOfType:1");
+      TEST_SYNC_POINT("DBImpl::GetSortedWalsOfType:2");
+
       uint64_t size_bytes;
       s = env_->GetFileSize(LogFileName(path, number), &size_bytes);
+      // re-try in case the alive log file has been moved to archive.
+      if (!s.ok() && log_type == kAliveLogFile &&
+          env_->FileExists(ArchivedLogFileName(path, number))) {
+        s = env_->GetFileSize(ArchivedLogFileName(path, number), &size_bytes);
+      }
       if (!s.ok()) {
         return s;
       }
@@ -1137,6 +1148,8 @@ Status DBImpl::Recover(
     SequenceNumber max_sequence(0);
     default_cf_handle_ = new ColumnFamilyHandleImpl(
         versions_->GetColumnFamilySet()->GetDefault(), this, &mutex_);
+    single_column_family_mode_ =
+        versions_->GetColumnFamilySet()->NumberOfColumnFamilies() == 1;
 
     // Recover from all newer log files than the ones named in the
     // descriptor (new log files may have been added by the previous
@@ -1356,14 +1369,14 @@ Status DBImpl::WriteLevel0TableForRecovery(ColumnFamilyData* cfd, MemTable* mem,
   mutex_.AssertHeld();
   const uint64_t start_micros = env_->NowMicros();
   FileMetaData meta;
-  meta.number = versions_->NewFileNumber();
-  pending_outputs_.insert(meta.number);
+  meta.fd.number = versions_->NewFileNumber();
+  pending_outputs_.insert(meta.fd.GetNumber());
   Iterator* iter = mem->NewIterator(ReadOptions(), true);
   const SequenceNumber newest_snapshot = snapshots_.GetNewest();
   const SequenceNumber earliest_seqno_in_memtable =
     mem->GetFirstSequenceNumber();
   Log(options_.info_log, "[%s] Level-0 table #%lu: started",
-      cfd->GetName().c_str(), (unsigned long)meta.number);
+      cfd->GetName().c_str(), (unsigned long)meta.fd.GetNumber());
 
   Status s;
   {
@@ -1377,27 +1390,28 @@ Status DBImpl::WriteLevel0TableForRecovery(ColumnFamilyData* cfd, MemTable* mem,
   }
 
   Log(options_.info_log, "[%s] Level-0 table #%lu: %lu bytes %s",
-      cfd->GetName().c_str(), (unsigned long)meta.number,
-      (unsigned long)meta.file_size, s.ToString().c_str());
+      cfd->GetName().c_str(), (unsigned long)meta.fd.GetNumber(),
+      (unsigned long)meta.fd.GetFileSize(), s.ToString().c_str());
   delete iter;
 
-  pending_outputs_.erase(meta.number);
+  pending_outputs_.erase(meta.fd.GetNumber());
 
   // Note that if file_size is zero, the file has been deleted and
   // should not be added to the manifest.
   int level = 0;
-  if (s.ok() && meta.file_size > 0) {
-    edit->AddFile(level, meta.number, meta.file_size,
-                  meta.smallest, meta.largest,
-                  meta.smallest_seqno, meta.largest_seqno);
+  if (s.ok() && meta.fd.GetFileSize() > 0) {
+    edit->AddFile(level, meta.fd.GetNumber(), meta.fd.GetFileSize(),
+                  meta.smallest, meta.largest, meta.smallest_seqno,
+                  meta.largest_seqno);
   }
 
   InternalStats::CompactionStats stats;
   stats.micros = env_->NowMicros() - start_micros;
-  stats.bytes_written = meta.file_size;
+  stats.bytes_written = meta.fd.GetFileSize();
   stats.files_out_levelnp1 = 1;
   cfd->internal_stats()->AddCompactionStats(level, stats);
-  RecordTick(options_.statistics.get(), COMPACT_WRITE_BYTES, meta.file_size);
+  RecordTick(options_.statistics.get(), COMPACT_WRITE_BYTES,
+             meta.fd.GetFileSize());
   return s;
 }
 
@@ -1407,9 +1421,9 @@ Status DBImpl::WriteLevel0Table(ColumnFamilyData* cfd,
   mutex_.AssertHeld();
   const uint64_t start_micros = env_->NowMicros();
   FileMetaData meta;
-  meta.number = versions_->NewFileNumber();
-  *filenumber = meta.number;
-  pending_outputs_.insert(meta.number);
+  meta.fd.number = versions_->NewFileNumber();
+  *filenumber = meta.fd.GetNumber();
+  pending_outputs_.insert(meta.fd.GetNumber());
 
   const SequenceNumber newest_snapshot = snapshots_.GetNewest();
   const SequenceNumber earliest_seqno_in_memtable =
@@ -1429,7 +1443,7 @@ Status DBImpl::WriteLevel0Table(ColumnFamilyData* cfd,
     Iterator* iter = NewMergingIterator(&cfd->internal_comparator(),
                                         &memtables[0], memtables.size());
     Log(options_.info_log, "[%s] Level-0 flush table #%lu: started",
-        cfd->GetName().c_str(), (unsigned long)meta.number);
+        cfd->GetName().c_str(), (unsigned long)meta.fd.GetNumber());
 
     s = BuildTable(dbname_, env_, *cfd->options(), storage_options_,
                    cfd->table_cache(), iter, &meta, cfd->internal_comparator(),
@@ -1438,8 +1452,8 @@ Status DBImpl::WriteLevel0Table(ColumnFamilyData* cfd,
     LogFlush(options_.info_log);
     delete iter;
     Log(options_.info_log, "[%s] Level-0 flush table #%lu: %lu bytes %s",
-        cfd->GetName().c_str(), (unsigned long)meta.number,
-        (unsigned long)meta.file_size, s.ToString().c_str());
+        cfd->GetName().c_str(), (unsigned long)meta.fd.GetFileSize(),
+        (unsigned long)meta.fd.GetFileSize(), s.ToString().c_str());
 
     if (!options_.disableDataSync) {
       db_directory_->Fsync();
@@ -1463,7 +1477,7 @@ Status DBImpl::WriteLevel0Table(ColumnFamilyData* cfd,
   // Note that if file_size is zero, the file has been deleted and
   // should not be added to the manifest.
   int level = 0;
-  if (s.ok() && meta.file_size > 0) {
+  if (s.ok() && meta.fd.GetFileSize() > 0) {
     const Slice min_user_key = meta.smallest.user_key();
     const Slice max_user_key = meta.largest.user_key();
     // if we have more than 1 background thread, then we cannot
@@ -1474,16 +1488,17 @@ Status DBImpl::WriteLevel0Table(ColumnFamilyData* cfd,
         cfd->options()->compaction_style == kCompactionStyleLevel) {
       level = base->PickLevelForMemTableOutput(min_user_key, max_user_key);
     }
-    edit->AddFile(level, meta.number, meta.file_size,
-                  meta.smallest, meta.largest,
-                  meta.smallest_seqno, meta.largest_seqno);
+    edit->AddFile(level, meta.fd.GetNumber(), meta.fd.GetFileSize(),
+                  meta.smallest, meta.largest, meta.smallest_seqno,
+                  meta.largest_seqno);
   }
 
   InternalStats::CompactionStats stats;
   stats.micros = env_->NowMicros() - start_micros;
-  stats.bytes_written = meta.file_size;
+  stats.bytes_written = meta.fd.GetFileSize();
   cfd->internal_stats()->AddCompactionStats(level, stats);
-  RecordTick(options_.statistics.get(), COMPACT_WRITE_BYTES, meta.file_size);
+  RecordTick(options_.statistics.get(), COMPACT_WRITE_BYTES,
+             meta.fd.GetFileSize());
   return s;
 }
 
@@ -1579,7 +1594,7 @@ Status DBImpl::CompactRange(ColumnFamilyHandle* column_family,
     return s;
   }
 
-  int max_level_with_files = 1;
+  int max_level_with_files = 0;
   {
     MutexLock l(&mutex_);
     Version* base = cfd->current();
@@ -1593,6 +1608,7 @@ Status DBImpl::CompactRange(ColumnFamilyHandle* column_family,
     // in case the compaction is unversal or if we're compacting the
     // bottom-most level, the output level will be the same as input one
     if (cfd->options()->compaction_style == kCompactionStyleUniversal ||
+        cfd->options()->compaction_style == kCompactionStyleFIFO ||
         level == max_level_with_files) {
       s = RunManualCompaction(cfd, level, level, begin, end);
     } else {
@@ -1673,9 +1689,10 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
     VersionEdit edit;
     edit.SetColumnFamily(cfd->GetID());
     for (const auto& f : cfd->current()->files_[level]) {
-      edit.DeleteFile(level, f->number);
-      edit.AddFile(to_level, f->number, f->file_size, f->smallest, f->largest,
-                   f->smallest_seqno, f->largest_seqno);
+      edit.DeleteFile(level, f->fd.GetNumber());
+      edit.AddFile(to_level, f->fd.GetNumber(), f->fd.GetFileSize(),
+                   f->smallest, f->largest, f->smallest_seqno,
+                   f->largest_seqno);
     }
     Log(options_.info_log, "[%s] Apply version edit:\n%s",
         cfd->GetName().c_str(), edit.DebugString().data());
@@ -1743,14 +1760,16 @@ Status DBImpl::RunManualCompaction(ColumnFamilyData* cfd, int input_level,
   // For universal compaction, we enforce every manual compaction to compact
   // all files.
   if (begin == nullptr ||
-      cfd->options()->compaction_style == kCompactionStyleUniversal) {
+      cfd->options()->compaction_style == kCompactionStyleUniversal ||
+      cfd->options()->compaction_style == kCompactionStyleFIFO) {
     manual.begin = nullptr;
   } else {
     begin_storage = InternalKey(*begin, kMaxSequenceNumber, kValueTypeForSeek);
     manual.begin = &begin_storage;
   }
   if (end == nullptr ||
-      cfd->options()->compaction_style == kCompactionStyleUniversal) {
+      cfd->options()->compaction_style == kCompactionStyleUniversal ||
+      cfd->options()->compaction_style == kCompactionStyleFIFO) {
     manual.end = nullptr;
   } else {
     end_storage = InternalKey(*end, 0, static_cast<ValueType>(0));
@@ -2053,7 +2072,15 @@ void DBImpl::BackgroundCallCompaction() {
     if (madeProgress || bg_schedule_needed_) {
       MaybeScheduleFlushOrCompaction();
     }
-    bg_cv_.SignalAll();
+    if (madeProgress || bg_compaction_scheduled_ == 0 || bg_manual_only_ > 0) {
+      // signal if
+      // * madeProgress -- need to wakeup MakeRoomForWrite
+      // * bg_compaction_scheduled_ == 0 -- need to wakeup ~DBImpl
+      // * bg_manual_only_ > 0 -- need to wakeup RunManualCompaction
+      // If none of this is true, there is no need to signal since nobody is
+      // waiting for it
+      bg_cv_.SignalAll();
+    }
     // IMPORTANT: there should be no code after calling SignalAll. This call may
     // signal the DB destructor that it's OK to proceed with destruction. In
     // that case, all DB variables will be dealloacated and referencing them
@@ -2139,25 +2166,43 @@ Status DBImpl::BackgroundCompaction(bool* madeProgress,
   if (!c) {
     // Nothing to do
     LogToBuffer(log_buffer, "Compaction nothing to do");
+  } else if (c->IsDeletionCompaction()) {
+    // TODO(icanadi) Do we want to honor snapshots here? i.e. not delete old
+    // file if there is alive snapshot pointing to it
+    assert(c->num_input_files(1) == 0);
+    assert(c->level() == 0);
+    assert(c->column_family_data()->options()->compaction_style ==
+           kCompactionStyleFIFO);
+    for (const auto& f : *c->inputs(0)) {
+      c->edit()->DeleteFile(c->level(), f->fd.GetNumber());
+    }
+    status = versions_->LogAndApply(c->column_family_data(), c->edit(), &mutex_,
+                                    db_directory_.get());
+    InstallSuperVersion(c->column_family_data(), deletion_state);
+    LogToBuffer(log_buffer, "[%s] Deleted %d files\n",
+                c->column_family_data()->GetName().c_str(),
+                c->num_input_files(0));
+    c->ReleaseCompactionFiles(status);
+    *madeProgress = true;
   } else if (!is_manual && c->IsTrivialMove()) {
     // Move file to next level
     assert(c->num_input_files(0) == 1);
     FileMetaData* f = c->input(0, 0);
-    c->edit()->DeleteFile(c->level(), f->number);
-    c->edit()->AddFile(c->level() + 1, f->number, f->file_size,
-                       f->smallest, f->largest,
-                       f->smallest_seqno, f->largest_seqno);
+    c->edit()->DeleteFile(c->level(), f->fd.GetNumber());
+    c->edit()->AddFile(c->level() + 1, f->fd.GetNumber(), f->fd.GetFileSize(),
+                       f->smallest, f->largest, f->smallest_seqno,
+                       f->largest_seqno);
     status = versions_->LogAndApply(c->column_family_data(), c->edit(), &mutex_,
                                     db_directory_.get());
     InstallSuperVersion(c->column_family_data(), deletion_state);
 
     Version::LevelSummaryStorage tmp;
-    LogToBuffer(log_buffer, "[%s] Moved #%lld to level-%d %lld bytes %s: %s\n",
-                c->column_family_data()->GetName().c_str(),
-                static_cast<unsigned long long>(f->number), c->level() + 1,
-                static_cast<unsigned long long>(f->file_size),
-                status.ToString().c_str(),
-                c->input_version()->LevelSummary(&tmp));
+    LogToBuffer(
+        log_buffer, "[%s] Moved #%lld to level-%d %lld bytes %s: %s\n",
+        c->column_family_data()->GetName().c_str(),
+        static_cast<unsigned long long>(f->fd.GetNumber()), c->level() + 1,
+        static_cast<unsigned long long>(f->fd.GetFileSize()),
+        status.ToString().c_str(), c->input_version()->LevelSummary(&tmp));
     c->ReleaseCompactionFiles(status);
     *madeProgress = true;
   } else {
@@ -2208,8 +2253,9 @@ Status DBImpl::BackgroundCompaction(bool* madeProgress,
     if (!m->done) {
       // We only compacted part of the requested range.  Update *m
       // to the range that is left to be compacted.
-      // Universal compaction should always compact the whole range
+      // Universal and FIFO compactions should always compact the whole range
       assert(m->cfd->options()->compaction_style != kCompactionStyleUniversal);
+      assert(m->cfd->options()->compaction_style != kCompactionStyleFIFO);
       m->tmp_storage = *manual_end;
       m->begin = &m->tmp_storage;
     }
@@ -2292,13 +2338,10 @@ Status DBImpl::OpenCompactionOutputFile(CompactionState* compact) {
   Status s = env_->NewWritableFile(fname, &compact->outfile, storage_options_);
 
   if (s.ok()) {
-    // Over-estimate slightly so we don't end up just barely crossing
-    // the threshold.
-    ColumnFamilyData* cfd = compact->compaction->column_family_data();
     compact->outfile->SetPreallocationBlockSize(
-        1.1 * cfd->compaction_picker()->MaxFileSizeForLevel(
-                  compact->compaction->output_level()));
+        compact->compaction->OutputFilePreallocationSize());
 
+    ColumnFamilyData* cfd = compact->compaction->column_family_data();
     CompressionType compression_type =
         GetCompressionType(*cfd->options(), compact->compaction->output_level(),
                            compact->compaction->enable_compression());
@@ -2353,7 +2396,7 @@ Status DBImpl::FinishCompactionOutputFile(CompactionState* compact,
   if (s.ok() && current_entries > 0) {
     // Verify that the table is usable
     ColumnFamilyData* cfd = compact->compaction->column_family_data();
-    FileMetaData meta(output_number, current_bytes);
+    FileDescriptor meta(output_number, current_bytes);
     Iterator* iter = cfd->table_cache()->NewIterator(
         ReadOptions(), storage_options_, cfd->internal_comparator(), meta);
     s = iter->status();
@@ -2415,9 +2458,6 @@ Status DBImpl::InstallCompactionResults(CompactionState* compact,
 inline SequenceNumber DBImpl::findEarliestVisibleSnapshot(
   SequenceNumber in, std::vector<SequenceNumber>& snapshots,
   SequenceNumber* prev_snapshot) {
-  if (!IsSnapshotSupported()) {
-    return 0;
-  }
   SequenceNumber prev __attribute__((unused)) = 0;
   for (const auto cur : snapshots) {
     assert(prev <= cur);
@@ -2456,6 +2496,7 @@ uint64_t DBImpl::CallFlushDuringCompaction(ColumnFamilyData* cfd,
 }
 
 Status DBImpl::ProcessKeyValueCompaction(
+    bool is_snapshot_supported,
     SequenceNumber visible_at_tip,
     SequenceNumber earliest_snapshot,
     SequenceNumber latest_snapshot,
@@ -2545,7 +2586,7 @@ Status DBImpl::ProcessKeyValueCompaction(
           cfd->user_comparator()->Compare(ikey.user_key,
                                           current_user_key.GetKey()) != 0) {
         // First occurrence of this user key
-        current_user_key.SetUserKey(ikey.user_key);
+        current_user_key.SetKey(ikey.user_key);
         has_current_user_key = true;
         last_sequence_for_key = kMaxSequenceNumber;
         visible_in_snapshot = kMaxSequenceNumber;
@@ -2584,11 +2625,10 @@ Status DBImpl::ProcessKeyValueCompaction(
       // Otherwise, search though all existing snapshots to find
       // the earlist snapshot that is affected by this kv.
       SequenceNumber prev_snapshot = 0; // 0 means no previous snapshot
-      SequenceNumber visible = visible_at_tip ?
-        visible_at_tip :
-        findEarliestVisibleSnapshot(ikey.sequence,
-            compact->existing_snapshots,
-            &prev_snapshot);
+      SequenceNumber visible = visible_at_tip ? visible_at_tip :
+        is_snapshot_supported ?  findEarliestVisibleSnapshot(ikey.sequence,
+                                  compact->existing_snapshots, &prev_snapshot)
+                              : 0;
 
       if (visible_in_snapshot == visible) {
         // If the earliest snapshot is which this key is visible in
@@ -2854,6 +2894,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact,
   // Allocate the output file numbers before we release the lock
   AllocateCompactionOutputFileNumbers(compact);
 
+  bool is_snapshot_supported = IsSnapshotSupported();
   // Release mutex while we're actually doing the compaction work
   mutex_.Unlock();
   log_buffer->FlushBufferToLog();
@@ -2940,6 +2981,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact,
       // Done buffering for the current prefix. Spit it out to disk
       // Now just iterate through all the kv-pairs
       status = ProcessKeyValueCompaction(
+          is_snapshot_supported,
           visible_at_tip,
           earliest_snapshot,
           latest_snapshot,
@@ -2975,6 +3017,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact,
         compact->MergeKeyValueSliceBuffer(&cfd->internal_comparator());
 
         status = ProcessKeyValueCompaction(
+            is_snapshot_supported,
             visible_at_tip,
             earliest_snapshot,
             latest_snapshot,
@@ -2996,6 +3039,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact,
     }
     compact->MergeKeyValueSliceBuffer(&cfd->internal_comparator());
     status = ProcessKeyValueCompaction(
+        is_snapshot_supported,
         visible_at_tip,
         earliest_snapshot,
         latest_snapshot,
@@ -3010,6 +3054,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact,
 
   if (!compaction_filter_v2) {
     status = ProcessKeyValueCompaction(
+      is_snapshot_supported,
       visible_at_tip,
       earliest_snapshot,
       latest_snapshot,
@@ -3053,15 +3098,15 @@ Status DBImpl::DoCompactionWork(CompactionState* compact,
   stats.files_out_levelnp1 = num_output_files;
 
   for (int i = 0; i < compact->compaction->num_input_files(0); i++) {
-    stats.bytes_readn += compact->compaction->input(0, i)->file_size;
+    stats.bytes_readn += compact->compaction->input(0, i)->fd.GetFileSize();
     RecordTick(options_.statistics.get(), COMPACT_READ_BYTES,
-               compact->compaction->input(0, i)->file_size);
+               compact->compaction->input(0, i)->fd.GetFileSize());
   }
 
   for (int i = 0; i < compact->compaction->num_input_files(1); i++) {
-    stats.bytes_readnp1 += compact->compaction->input(1, i)->file_size;
+    stats.bytes_readnp1 += compact->compaction->input(1, i)->fd.GetFileSize();
     RecordTick(options_.statistics.get(), COMPACT_READ_BYTES,
-               compact->compaction->input(1, i)->file_size);
+               compact->compaction->input(1, i)->fd.GetFileSize());
   }
 
   for (int i = 0; i < num_output_files; i++) {
@@ -3137,18 +3182,34 @@ static void CleanupIteratorState(void* arg1, void* arg2) {
 
 Iterator* DBImpl::NewInternalIterator(const ReadOptions& options,
                                       ColumnFamilyData* cfd,
-                                      SuperVersion* super_version) {
-  std::vector<Iterator*> iterator_list;
-  // Collect iterator for mutable mem
-  iterator_list.push_back(super_version->mem->NewIterator(options));
-  // Collect all needed child iterators for immutable memtables
-  super_version->imm->AddIterators(options, &iterator_list);
-  // Collect iterators for files in L0 - Ln
-  super_version->current->AddIterators(options, storage_options_,
-                                       &iterator_list);
-  Iterator* internal_iter = NewMergingIterator(
-      &cfd->internal_comparator(), &iterator_list[0], iterator_list.size());
-
+                                      SuperVersion* super_version,
+                                      Arena* arena) {
+  Iterator* internal_iter;
+  if (arena != nullptr) {
+    // Need to create internal iterator from the arena.
+    MergeIteratorBuilder merge_iter_builder(&cfd->internal_comparator(), arena);
+    // Collect iterator for mutable mem
+    merge_iter_builder.AddIterator(
+        super_version->mem->NewIterator(options, false, arena));
+    // Collect all needed child iterators for immutable memtables
+    super_version->imm->AddIterators(options, &merge_iter_builder);
+    // Collect iterators for files in L0 - Ln
+    super_version->current->AddIterators(options, storage_options_,
+                                         &merge_iter_builder);
+    internal_iter = merge_iter_builder.Finish();
+  } else {
+    // Need to create internal iterator using malloc.
+    std::vector<Iterator*> iterator_list;
+    // Collect iterator for mutable mem
+    iterator_list.push_back(super_version->mem->NewIterator(options));
+    // Collect all needed child iterators for immutable memtables
+    super_version->imm->AddIterators(options, &iterator_list);
+    // Collect iterators for files in L0 - Ln
+    super_version->current->AddIterators(options, storage_options_,
+                                         &iterator_list);
+    internal_iter = NewMergingIterator(&cfd->internal_comparator(),
+                                       &iterator_list[0], iterator_list.size());
+  }
   IterState* cleanup = new IterState(this, &mutex_, super_version);
   internal_iter->RegisterCleanup(CleanupIteratorState, cleanup, nullptr);
 
@@ -3422,6 +3483,7 @@ Status DBImpl::CreateColumnFamily(const ColumnFamilyOptions& options,
   Status s = versions_->LogAndApply(nullptr, &edit, &mutex_,
                                     db_directory_.get(), false, &options);
   if (s.ok()) {
+    single_column_family_mode_ = false;
     auto cfd =
         versions_->GetColumnFamilySet()->GetColumnFamily(column_family_name);
     assert(cfd != nullptr);
@@ -3499,30 +3561,77 @@ Iterator* DBImpl::NewIterator(const ReadOptions& options,
   auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
   auto cfd = cfh->cfd();
 
-  Iterator* iter;
   if (options.tailing) {
 #ifdef ROCKSDB_LITE
     // not supported in lite version
     return nullptr;
 #else
-    iter = new TailingIterator(env_, this, options, cfd);
+    // TODO(ljin): remove tailing iterator
+    auto iter = new ForwardIterator(this, options, cfd);
+    return NewDBIterator(env_, *cfd->options(), cfd->user_comparator(), iter,
+                         kMaxSequenceNumber);
+// return new TailingIterator(env_, this, options, cfd);
 #endif
   } else {
     SequenceNumber latest_snapshot = versions_->LastSequence();
     SuperVersion* sv = nullptr;
     sv = cfd->GetReferencedSuperVersion(&mutex_);
 
-    iter = NewInternalIterator(options, cfd, sv);
-
     auto snapshot =
         options.snapshot != nullptr
             ? reinterpret_cast<const SnapshotImpl*>(options.snapshot)->number_
             : latest_snapshot;
-    iter = NewDBIterator(env_, *cfd->options(),
-                         cfd->user_comparator(), iter, snapshot);
-  }
 
-  return iter;
+    // Try to generate a DB iterator tree in continuous memory area to be
+    // cache friendly. Here is an example of result:
+    // +-------------------------------+
+    // |                               |
+    // | ArenaWrappedDBIter            |
+    // |  +                            |
+    // |  +---> Inner Iterator   ------------+
+    // |  |                            |     |
+    // |  |    +-- -- -- -- -- -- -- --+     |
+    // |  +--- | Arena                 |     |
+    // |       |                       |     |
+    // |          Allocated Memory:    |     |
+    // |       |   +-------------------+     |
+    // |       |   | DBIter            | <---+
+    // |           |  +                |
+    // |       |   |  +-> iter_  ------------+
+    // |       |   |                   |     |
+    // |       |   +-------------------+     |
+    // |       |   | MergingIterator   | <---+
+    // |           |  +                |
+    // |       |   |  +->child iter1  ------------+
+    // |       |   |  |                |          |
+    // |           |  +->child iter2  ----------+ |
+    // |       |   |  |                |        | |
+    // |       |   |  +->child iter3  --------+ | |
+    // |           |                   |      | | |
+    // |       |   +-------------------+      | | |
+    // |       |   | Iterator1         | <--------+
+    // |       |   +-------------------+      | |
+    // |       |   | Iterator2         | <------+
+    // |       |   +-------------------+      |
+    // |       |   | Iterator3         | <----+
+    // |       |   +-------------------+
+    // |       |                       |
+    // +-------+-----------------------+
+    //
+    // ArenaWrappedDBIter inlines an arena area where all the iterartor in the
+    // the iterator tree is allocated in the order of being accessed when
+    // querying.
+    // Laying out the iterators in the order of being accessed makes it more
+    // likely that any iterator pointer is close to the iterator it points to so
+    // that they are likely to be in the same cache line and/or page.
+    ArenaWrappedDBIter* db_iter = NewArenaWrappedDbIterator(
+        env_, *cfd->options(), cfd->user_comparator(), snapshot);
+    Iterator* internal_iter =
+        NewInternalIterator(options, cfd, sv, db_iter->GetArena());
+    db_iter->SetIterUnderDBIter(internal_iter);
+
+    return db_iter;
+  }
 }
 
 Status DBImpl::NewIterators(
@@ -3585,9 +3694,9 @@ bool DBImpl::IsSnapshotSupported() const {
 }
 
 const Snapshot* DBImpl::GetSnapshot() {
+  MutexLock l(&mutex_);
   // returns null if the underlying memtable does not support snapshot.
   if (!IsSnapshotSupported()) return nullptr;
-  MutexLock l(&mutex_);
   return snapshots_.New(versions_->LastSequence());
 }
 
@@ -3644,42 +3753,55 @@ Status DBImpl::Write(const WriteOptions& options, WriteBatch* my_batch) {
     RecordTick(options_.statistics.get(), WRITE_DONE_BY_SELF, 1);
   }
 
+  assert(!single_column_family_mode_ ||
+         versions_->GetColumnFamilySet()->NumberOfColumnFamilies() == 1);
+
   uint64_t flush_column_family_if_log_file = 0;
   uint64_t max_total_wal_size = (options_.max_total_wal_size == 0)
-                                    ? 2 * max_total_in_memory_state_
+                                    ? 4 * max_total_in_memory_state_
                                     : options_.max_total_wal_size;
-  if (alive_log_files_.begin()->getting_flushed == false &&
+  if (UNLIKELY(!single_column_family_mode_) &&
+      alive_log_files_.begin()->getting_flushed == false &&
       total_log_size_ > max_total_wal_size) {
     flush_column_family_if_log_file = alive_log_files_.begin()->number;
     alive_log_files_.begin()->getting_flushed = true;
     Log(options_.info_log,
-        "Flushing all column families with data in WAL number %" PRIu64,
-        flush_column_family_if_log_file);
+        "Flushing all column families with data in WAL number %" PRIu64
+        ". Total log size is %" PRIu64 " while max_total_wal_size is %" PRIu64,
+        flush_column_family_if_log_file, total_log_size_, max_total_wal_size);
   }
 
   Status status;
-  // refcounting cfd in iteration
-  bool dead_cfd = false;
   autovector<SuperVersion*> superversions_to_free;
   autovector<log::Writer*> logs_to_free;
-  for (auto cfd : *versions_->GetColumnFamilySet()) {
-    cfd->Ref();
-    bool force_flush = my_batch == nullptr ||
-                       (flush_column_family_if_log_file != 0 &&
-                        cfd->GetLogNumber() <= flush_column_family_if_log_file);
-    // May temporarily unlock and wait.
-    status = MakeRoomForWrite(cfd, force_flush, &superversions_to_free,
-                              &logs_to_free);
-    if (cfd->Unref()) {
-      dead_cfd = true;
+
+  if (LIKELY(single_column_family_mode_)) {
+    // fast path
+    status = MakeRoomForWrite(default_cf_handle_->cfd(), my_batch == nullptr,
+                              &superversions_to_free, &logs_to_free);
+  } else {
+    // refcounting cfd in iteration
+    bool dead_cfd = false;
+    for (auto cfd : *versions_->GetColumnFamilySet()) {
+      cfd->Ref();
+      bool force_flush =
+          my_batch == nullptr ||
+          (flush_column_family_if_log_file != 0 &&
+           cfd->GetLogNumber() <= flush_column_family_if_log_file);
+      // May temporarily unlock and wait.
+      status = MakeRoomForWrite(cfd, force_flush, &superversions_to_free,
+                                &logs_to_free);
+      if (cfd->Unref()) {
+        dead_cfd = true;
+      }
+      if (!status.ok()) {
+        break;
+      }
     }
-    if (!status.ok()) {
-      break;
+    if (dead_cfd) {
+      versions_->GetColumnFamilySet()->FreeDeadColumnFamilies();
     }
   }
-  if (dead_cfd) {
-    versions_->GetColumnFamilySet()->FreeDeadColumnFamilies();
-  }
 
   uint64_t last_sequence = versions_->LastSequence();
   Writer* last_writer = &w;
@@ -3740,19 +3862,19 @@ Status DBImpl::Write(const WriteOptions& options, WriteBatch* my_batch) {
       }
       if (status.ok()) {
         PERF_TIMER_START(write_memtable_time);
+
         status = WriteBatchInternal::InsertInto(
             updates, column_family_memtables_.get(), false, 0, this, false);
+        // A non-OK status here indicates iteration failure (either in-memory
+        // writebatch corruption (very bad), or the client specified invalid
+        // column family).  This will later on trigger bg_error_.
+        //
+        // Note that existing logic was not sound. Any partial failure writing
+        // into the memtable would result in a state that some write ops might
+        // have succeeded in memtable but Status reports error for all writes.
+
         PERF_TIMER_STOP(write_memtable_time);
 
-        if (!status.ok()) {
-          // Iteration failed (either in-memory writebatch corruption (very
-          // bad), or the client specified invalid column family). Return
-          // failure.
-          // Note that existing logic was not sound. Any partial failure writing
-          // into the memtable would result in a state that some write ops might
-          // have succeeded in memtable but Status reports error for all writes.
-          return status;
-        }
         SetTickerCount(options_.statistics.get(), SEQUENCE_NUMBER,
                        last_sequence);
       }
@@ -3890,6 +4012,10 @@ Status DBImpl::MakeRoomForWrite(
   uint64_t rate_limit_delay_millis = 0;
   Status s;
   double score;
+  // Once we schedule background work, we shouldn't schedule it again, since it
+  // might generate a tight feedback loop, constantly scheduling more background
+  // work, even if additional background work is not needed
+  bool schedule_background_work = true;
 
   while (true) {
     if (!bg_error_.ok()) {
@@ -3933,7 +4059,10 @@ Status DBImpl::MakeRoomForWrite(
       DelayLoggingAndReset();
       Log(options_.info_log, "[%s] wait for memtable flush...\n",
           cfd->GetName().c_str());
-      MaybeScheduleFlushOrCompaction();
+      if (schedule_background_work) {
+        MaybeScheduleFlushOrCompaction();
+        schedule_background_work = false;
+      }
       uint64_t stall;
       {
         StopWatch sw(env_, options_.statistics.get(),
@@ -4434,12 +4563,26 @@ Status DB::Open(const DBOptions& db_options, const std::string& dbname,
       for (auto cf : column_families) {
         auto cfd =
             impl->versions_->GetColumnFamilySet()->GetColumnFamily(cf.name);
-        if (cfd == nullptr) {
-          s = Status::InvalidArgument("Column family not found: ", cf.name);
-          break;
+        if (cfd != nullptr) {
+          handles->push_back(
+              new ColumnFamilyHandleImpl(cfd, impl, &impl->mutex_));
+        } else {
+          if (db_options.create_missing_column_families) {
+            // missing column family, create it
+            ColumnFamilyHandle* handle;
+            impl->mutex_.Unlock();
+            s = impl->CreateColumnFamily(cf.options, cf.name, &handle);
+            impl->mutex_.Lock();
+            if (s.ok()) {
+              handles->push_back(handle);
+            } else {
+              break;
+            }
+          } else {
+            s = Status::InvalidArgument("Column family not found: ", cf.name);
+            break;
+          }
         }
-        handles->push_back(
-            new ColumnFamilyHandleImpl(cfd, impl, &impl->mutex_));
       }
     }
     if (s.ok()) {
@@ -4457,13 +4600,15 @@ Status DB::Open(const DBOptions& db_options, const std::string& dbname,
 
   if (s.ok()) {
     for (auto cfd : *impl->versions_->GetColumnFamilySet()) {
-      if (cfd->options()->compaction_style == kCompactionStyleUniversal) {
+      if (cfd->options()->compaction_style == kCompactionStyleUniversal ||
+          cfd->options()->compaction_style == kCompactionStyleFIFO) {
         Version* current = cfd->current();
         for (int i = 1; i < current->NumberLevels(); ++i) {
           int num_files = current->NumLevelFiles(i);
           if (num_files > 0) {
-            s = Status::InvalidArgument("Not all files are at level 0. Cannot "
-                "open with universal compaction style.");
+            s = Status::InvalidArgument(
+                "Not all files are at level 0. Cannot "
+                "open with universal or FIFO compaction style.");
             break;
           }
         }

db/db_impl.h

@@ -39,6 +39,7 @@ class Version;
 class VersionEdit;
 class VersionSet;
 class CompactionFilterV2;
+class Arena;
 
 class DBImpl : public DB {
  public:
@@ -278,13 +279,15 @@ class DBImpl : public DB {
   const DBOptions options_;
 
   Iterator* NewInternalIterator(const ReadOptions&, ColumnFamilyData* cfd,
-                                SuperVersion* super_version);
+                                SuperVersion* super_version,
+                                Arena* arena = nullptr);
 
  private:
   friend class DB;
   friend class InternalStats;
 #ifndef ROCKSDB_LITE
   friend class TailingIterator;
+  friend class ForwardIterator;
 #endif
   friend struct SuperVersion;
   struct CompactionState;
@@ -373,6 +376,7 @@ class DBImpl : public DB {
   // Call compaction filter if is_compaction_v2 is not true. Then iterate
   // through input and compact the kv-pairs
   Status ProcessKeyValueCompaction(
+    bool is_snapshot_supported,
     SequenceNumber visible_at_tip,
     SequenceNumber earliest_snapshot,
     SequenceNumber latest_snapshot,
@@ -449,7 +453,16 @@ class DBImpl : public DB {
   // State below is protected by mutex_
   port::Mutex mutex_;
   port::AtomicPointer shutting_down_;
-  port::CondVar bg_cv_;          // Signalled when background work finishes
+  // This condition variable is signaled on these conditions:
+  // * whenever bg_compaction_scheduled_ goes down to 0
+  // * if bg_manual_only_ > 0, whenever a compaction finishes, even if it hasn't
+  // made any progress
+  // * whenever a compaction made any progress
+  // * whenever bg_flush_scheduled_ value decreases (i.e. whenever a flush is
+  // done, even if it didn't make any progress)
+  // * whenever there is an error in background flush or compaction
+  // * whenever bg_logstats_scheduled_ turns to false
+  port::CondVar bg_cv_;
   uint64_t logfile_number_;
   unique_ptr<log::Writer> log_;
   bool log_empty_;
@@ -468,6 +481,9 @@ class DBImpl : public DB {
   // only used for dynamically adjusting max_total_wal_size. it is a sum of
   // [write_buffer_size * max_write_buffer_number] over all column families
   uint64_t max_total_in_memory_state_;
+  // If true, we have only one (default) column family. We use this to optimize
+  // some code-paths
+  bool single_column_family_mode_;
 
   std::string host_name_;
 

db/db_impl_debug.cc

@@ -81,7 +81,8 @@ Status DBImpl::TEST_CompactRange(int level, const Slice* begin,
     cfd = cfh->cfd();
   }
   int output_level =
-      (cfd->options()->compaction_style == kCompactionStyleUniversal)
+      (cfd->options()->compaction_style == kCompactionStyleUniversal ||
+       cfd->options()->compaction_style == kCompactionStyleFIFO)
           ? level
           : level + 1;
   return RunManualCompaction(cfd, level, output_level, begin, end);

db/db_iter.cc

@@ -18,6 +18,7 @@
 #include "rocksdb/iterator.h"
 #include "rocksdb/merge_operator.h"
 #include "port/port.h"
+#include "util/arena.h"
 #include "util/logging.h"
 #include "util/mutexlock.h"
 #include "util/perf_context_imp.h"
@@ -37,8 +38,6 @@ static void DumpInternalIter(Iterator* iter) {
 }
 #endif
 
-namespace {
-
 // Memtables and sstables that make the DB representation contain
 // (userkey,seq,type) => uservalue entries.  DBIter
 // combines multiple entries for the same userkey found in the DB
@@ -57,9 +56,10 @@ class DBIter: public Iterator {
     kReverse
   };
 
-  DBIter(Env* env, const Options& options,
-         const Comparator* cmp, Iterator* iter, SequenceNumber s)
-      : env_(env),
+  DBIter(Env* env, const Options& options, const Comparator* cmp,
+         Iterator* iter, SequenceNumber s, bool arena_mode)
+      : arena_mode_(arena_mode),
+        env_(env),
         logger_(options.info_log.get()),
         user_comparator_(cmp),
         user_merge_operator_(options.merge_operator.get()),
@@ -74,7 +74,15 @@ class DBIter: public Iterator {
   }
   virtual ~DBIter() {
     RecordTick(statistics_, NO_ITERATORS, -1);
-    delete iter_;
+    if (!arena_mode_) {
+      delete iter_;
+    } else {
+      iter_->~Iterator();
+    }
+  }
+  virtual void SetIter(Iterator* iter) {
+    assert(iter_ == nullptr);
+    iter_ = iter;
   }
   virtual bool Valid() const { return valid_; }
   virtual Slice key() const {
@@ -116,17 +124,17 @@ class DBIter: public Iterator {
     }
   }
 
+  bool arena_mode_;
   Env* const env_;
   Logger* logger_;
   const Comparator* const user_comparator_;
   const MergeOperator* const user_merge_operator_;
-  Iterator* const iter_;
+  Iterator* iter_;
   SequenceNumber const sequence_;
 
   Status status_;
   IterKey saved_key_;   // == current key when direction_==kReverse
   std::string saved_value_;   // == current raw value when direction_==kReverse
-  std::string skip_key_;
   Direction direction_;
   bool valid_;
   bool current_entry_is_merged_;
@@ -212,18 +220,18 @@ void DBIter::FindNextUserEntryInternal(bool skipping) {
           case kTypeDeletion:
             // Arrange to skip all upcoming entries for this key since
             // they are hidden by this deletion.
-            saved_key_.SetUserKey(ikey.user_key);
+            saved_key_.SetKey(ikey.user_key);
             skipping = true;
             num_skipped = 0;
             PERF_COUNTER_ADD(internal_delete_skipped_count, 1);
             break;
           case kTypeValue:
             valid_ = true;
-            saved_key_.SetUserKey(ikey.user_key);
+            saved_key_.SetKey(ikey.user_key);
             return;
           case kTypeMerge:
             // By now, we are sure the current ikey is going to yield a value
-            saved_key_.SetUserKey(ikey.user_key);
+            saved_key_.SetKey(ikey.user_key);
             current_entry_is_merged_ = true;
             valid_ = true;
             MergeValuesNewToOld();  // Go to a different state machine
@@ -332,7 +340,7 @@ void DBIter::Prev() {
     // iter_ is pointing at the current entry.  Scan backwards until
     // the key changes so we can use the normal reverse scanning code.
     assert(iter_->Valid());  // Otherwise valid_ would have been false
-    saved_key_.SetUserKey(ExtractUserKey(iter_->key()));
+    saved_key_.SetKey(ExtractUserKey(iter_->key()));
     while (true) {
       iter_->Prev();
       if (!iter_->Valid()) {
@@ -378,7 +386,7 @@ void DBIter::FindPrevUserEntry() {
             std::string empty;
             swap(empty, saved_value_);
           }
-          saved_key_.SetUserKey(ExtractUserKey(iter_->key()));
+          saved_key_.SetKey(ExtractUserKey(iter_->key()));
           saved_value_.assign(raw_value.data(), raw_value.size());
         }
       } else {
@@ -462,16 +470,48 @@ void DBIter::SeekToLast() {
   FindPrevUserEntry();
 }
 
-}  // anonymous namespace
+Iterator* NewDBIterator(Env* env, const Options& options,
+                        const Comparator* user_key_comparator,
+                        Iterator* internal_iter,
+                        const SequenceNumber& sequence) {
+  return new DBIter(env, options, user_key_comparator, internal_iter, sequence,
+                    false);
+}
 
-Iterator* NewDBIterator(
-    Env* env,
-    const Options& options,
-    const Comparator *user_key_comparator,
-    Iterator* internal_iter,
+ArenaWrappedDBIter::~ArenaWrappedDBIter() { db_iter_->~DBIter(); }
+
+void ArenaWrappedDBIter::SetDBIter(DBIter* iter) { db_iter_ = iter; }
+
+void ArenaWrappedDBIter::SetIterUnderDBIter(Iterator* iter) {
+  static_cast<DBIter*>(db_iter_)->SetIter(iter);
+}
+
+inline bool ArenaWrappedDBIter::Valid() const { return db_iter_->Valid(); }
+inline void ArenaWrappedDBIter::SeekToFirst() { db_iter_->SeekToFirst(); }
+inline void ArenaWrappedDBIter::SeekToLast() { db_iter_->SeekToLast(); }
+inline void ArenaWrappedDBIter::Seek(const Slice& target) {
+  db_iter_->Seek(target);
+}
+inline void ArenaWrappedDBIter::Next() { db_iter_->Next(); }
+inline void ArenaWrappedDBIter::Prev() { db_iter_->Prev(); }
+inline Slice ArenaWrappedDBIter::key() const { return db_iter_->key(); }
+inline Slice ArenaWrappedDBIter::value() const { return db_iter_->value(); }
+inline Status ArenaWrappedDBIter::status() const { return db_iter_->status(); }
+void ArenaWrappedDBIter::RegisterCleanup(CleanupFunction function, void* arg1,
+                                         void* arg2) {
+  db_iter_->RegisterCleanup(function, arg1, arg2);
+}
+
+ArenaWrappedDBIter* NewArenaWrappedDbIterator(
+    Env* env, const Options& options, const Comparator* user_key_comparator,
     const SequenceNumber& sequence) {
-  return new DBIter(env, options, user_key_comparator,
-                    internal_iter, sequence);
+  ArenaWrappedDBIter* iter = new ArenaWrappedDBIter();
+  Arena* arena = iter->GetArena();
+  auto mem = arena->AllocateAligned(sizeof(DBIter));
+  DBIter* db_iter = new (mem)
+      DBIter(env, options, user_key_comparator, nullptr, sequence, true);
+  iter->SetDBIter(db_iter);
+  return iter;
 }
 
 }  // namespace rocksdb

db/db_iter.h

@@ -11,9 +11,14 @@
 #include <stdint.h>
 #include "rocksdb/db.h"
 #include "db/dbformat.h"
+#include "util/arena.h"
+#include "util/autovector.h"
 
 namespace rocksdb {
 
+class Arena;
+class DBIter;
+
 // Return a new iterator that converts internal keys (yielded by
 // "*internal_iter") that were live at the specified "sequence" number
 // into appropriate user keys.
@@ -24,4 +29,45 @@ extern Iterator* NewDBIterator(
     Iterator* internal_iter,
     const SequenceNumber& sequence);
 
+// A wrapper iterator which wraps DB Iterator and the arena, with which the DB
+// iterator is supposed be allocated. This class is used as an entry point of
+// a iterator hierarchy whose memory can be allocated inline. In that way,
+// accessing the iterator tree can be more cache friendly. It is also faster
+// to allocate.
+class ArenaWrappedDBIter : public Iterator {
+ public:
+  virtual ~ArenaWrappedDBIter();
+
+  // Get the arena to be used to allocate memory for DBIter to be wrapped,
+  // as well as child iterators in it.
+  virtual Arena* GetArena() { return &arena_; }
+
+  // Set the DB Iterator to be wrapped
+
+  virtual void SetDBIter(DBIter* iter);
+
+  // Set the internal iterator wrapped inside the DB Iterator. Usually it is
+  // a merging iterator.
+  virtual void SetIterUnderDBIter(Iterator* iter);
+  virtual bool Valid() const override;
+  virtual void SeekToFirst() override;
+  virtual void SeekToLast() override;
+  virtual void Seek(const Slice& target) override;
+  virtual void Next() override;
+  virtual void Prev() override;
+  virtual Slice key() const override;
+  virtual Slice value() const override;
+  virtual Status status() const override;
+  void RegisterCleanup(CleanupFunction function, void* arg1, void* arg2);
+
+ private:
+  DBIter* db_iter_;
+  Arena arena_;
+};
+
+// Generate the arena wrapped iterator class.
+extern ArenaWrappedDBIter* NewArenaWrappedDbIterator(
+    Env* env, const Options& options, const Comparator* user_key_comparator,
+    const SequenceNumber& sequence);
+
 }  // namespace rocksdb

db/db_test.cc

@@ -317,6 +317,7 @@ class DBTest {
     kCompressedBlockCache,
     kInfiniteMaxOpenFiles,
     kxxHashChecksum,
+    kFIFOCompaction,
     kEnd
   };
   int option_config_;
@@ -339,7 +340,8 @@ class DBTest {
     kSkipPlainTable = 8,
     kSkipHashIndex = 16,
     kSkipNoSeekToLast = 32,
-    kSkipHashCuckoo = 64
+    kSkipHashCuckoo = 64,
+    kSkipFIFOCompaction = 128,
   };
 
   DBTest() : option_config_(kDefault),
@@ -391,6 +393,10 @@ class DBTest {
       if ((skip_mask & kSkipHashCuckoo) && (option_config_ == kHashCuckoo)) {
         continue;
       }
+      if ((skip_mask & kSkipFIFOCompaction) &&
+          option_config_ == kFIFOCompaction) {
+        continue;
+      }
       break;
     }
 
@@ -503,6 +509,10 @@ class DBTest {
         options.table_factory.reset(NewBlockBasedTableFactory(table_options));
         break;
       }
+      case kFIFOCompaction: {
+        options.compaction_style = kCompactionStyleFIFO;
+        break;
+      }
       case kBlockBasedTableWithPrefixHashIndex: {
         BlockBasedTableOptions table_options;
         table_options.index_type = BlockBasedTableOptions::kHashSearch;
@@ -1394,7 +1404,7 @@ TEST(DBTest, GetEncountersEmptyLevel) {
     env_->SleepForMicroseconds(1000000);
 
     ASSERT_EQ(NumTableFilesAtLevel(0, 1), 1);  // XXX
-  } while (ChangeOptions(kSkipUniversalCompaction));
+  } while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction));
 }
 
 // KeyMayExist can lead to a few false positives, but not false negatives.
@@ -1460,7 +1470,8 @@ TEST(DBTest, KeyMayExist) {
 
     // KeyMayExist function only checks data in block caches, which is not used
     // by plain table format.
-  } while (ChangeOptions(kSkipPlainTable | kSkipHashIndex));
+  } while (
+      ChangeOptions(kSkipPlainTable | kSkipHashIndex | kSkipFIFOCompaction));
 }
 
 TEST(DBTest, NonBlockingIteration) {
@@ -2365,9 +2376,9 @@ TEST(DBTest, NumImmutableMemTable) {
     ASSERT_EQ(num, "0");
     ASSERT_TRUE(dbfull()->GetProperty(
         handles_[1], "rocksdb.cur-size-active-mem-table", &num));
-    // "208" is the size of the metadata of an empty skiplist, this would
+    // "200" is the size of the metadata of an empty skiplist, this would
     // break if we change the default skiplist implementation
-    ASSERT_EQ(num, "208");
+    ASSERT_EQ(num, "200");
     SetPerfLevel(kDisable);
   } while (ChangeCompactOptions());
 }
@@ -3841,6 +3852,38 @@ TEST(DBTest, CompactionFilter) {
   delete iter;
 }
 
+// Tests the edge case where compaction does not produce any output -- all
+// entries are deleted. The compaction should create bunch of 'DeleteFile'
+// entries in VersionEdit, but none of the 'AddFile's.
+TEST(DBTest, CompactionFilterDeletesAll) {
+  Options options;
+  options.compaction_filter_factory = std::make_shared<DeleteFilterFactory>();
+  options.disable_auto_compactions = true;
+  options.create_if_missing = true;
+  DestroyAndReopen(&options);
+
+  // put some data
+  for (int table = 0; table < 4; ++table) {
+    for (int i = 0; i < 10 + table; ++i) {
+      Put(std::to_string(table * 100 + i), "val");
+    }
+    Flush();
+  }
+
+  // this will produce empty file (delete compaction filter)
+  ASSERT_OK(db_->CompactRange(nullptr, nullptr));
+  ASSERT_EQ(0, CountLiveFiles());
+
+  Reopen(&options);
+
+  Iterator* itr = db_->NewIterator(ReadOptions());
+  itr->SeekToFirst();
+  // empty db
+  ASSERT_TRUE(!itr->Valid());
+
+  delete itr;
+}
+
 TEST(DBTest, CompactionFilterWithValueChange) {
   do {
     Options options;
@@ -4355,7 +4398,8 @@ TEST(DBTest, ApproximateSizes) {
       ASSERT_GT(NumTableFilesAtLevel(1, 1), 0);
     }
     // ApproximateOffsetOf() is not yet implemented in plain table format.
-  } while (ChangeOptions(kSkipUniversalCompaction | kSkipPlainTable));
+  } while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction |
+                         kSkipPlainTable));
 }
 
 TEST(DBTest, ApproximateSizes_MixOfSmallAndLarge) {
@@ -4499,8 +4543,8 @@ TEST(DBTest, HiddenValuesAreRemoved) {
     // ApproximateOffsetOf() is not yet implemented in plain table format,
     // which is used by Size().
     // skip HashCuckooRep as it does not support snapshot
-  } while (ChangeOptions(kSkipUniversalCompaction | kSkipPlainTable |
-                         kSkipHashCuckoo));
+  } while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction |
+                         kSkipPlainTable | kSkipHashCuckoo));
 }
 
 TEST(DBTest, CompactBetweenSnapshots) {
@@ -4556,7 +4600,7 @@ TEST(DBTest, CompactBetweenSnapshots) {
     ASSERT_EQ("sixth", Get(1, "foo"));
     ASSERT_EQ(AllEntriesFor("foo", 1), "[ sixth ]");
     // skip HashCuckooRep as it does not support snapshot
-  } while (ChangeOptions(kSkipHashCuckoo));
+  } while (ChangeOptions(kSkipHashCuckoo | kSkipFIFOCompaction));
 }
 
 TEST(DBTest, DeletionMarkers1) {
@@ -4662,7 +4706,7 @@ TEST(DBTest, OverlapInLevel0) {
     Flush(1);
     ASSERT_EQ("3", FilesPerLevel(1));
     ASSERT_EQ("NOT_FOUND", Get(1, "600"));
-  } while (ChangeOptions(kSkipUniversalCompaction));
+  } while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction));
 }
 
 TEST(DBTest, L0_CompactionBug_Issue44_a) {
@@ -5577,48 +5621,56 @@ TEST(DBTest, TransactionLogIterator) {
 
 #ifndef NDEBUG // sync point is not included with DNDEBUG build
 TEST(DBTest, TransactionLogIteratorRace) {
-  // Setup sync point dependency to reproduce the race condition of
-  // a log file moved to archived dir, in the middle of GetSortedWalFiles
-  rocksdb::SyncPoint::GetInstance()->LoadDependency(
-    { { "DBImpl::GetSortedWalFiles:1", "DBImpl::PurgeObsoleteFiles:1" },
-      { "DBImpl::PurgeObsoleteFiles:2", "DBImpl::GetSortedWalFiles:2" },
-    });
+  static const int LOG_ITERATOR_RACE_TEST_COUNT = 2;
+  static const char* sync_points[LOG_ITERATOR_RACE_TEST_COUNT][4] =
+    { { "DBImpl::GetSortedWalFiles:1", "DBImpl::PurgeObsoleteFiles:1",
+        "DBImpl::PurgeObsoleteFiles:2", "DBImpl::GetSortedWalFiles:2" },
+      { "DBImpl::GetSortedWalsOfType:1", "DBImpl::PurgeObsoleteFiles:1",
+        "DBImpl::PurgeObsoleteFiles:2", "DBImpl::GetSortedWalsOfType:2" }};
+  for (int test = 0; test < LOG_ITERATOR_RACE_TEST_COUNT; ++test) {
+    // Setup sync point dependency to reproduce the race condition of
+    // a log file moved to archived dir, in the middle of GetSortedWalFiles
+    rocksdb::SyncPoint::GetInstance()->LoadDependency(
+      { { sync_points[test][0], sync_points[test][1] },
+        { sync_points[test][2], sync_points[test][3] },
+      });
 
-  do {
-    rocksdb::SyncPoint::GetInstance()->ClearTrace();
-    rocksdb::SyncPoint::GetInstance()->DisableProcessing();
-    Options options = OptionsForLogIterTest();
-    DestroyAndReopen(&options);
-    Put("key1", DummyString(1024));
-    dbfull()->Flush(FlushOptions());
-    Put("key2", DummyString(1024));
-    dbfull()->Flush(FlushOptions());
-    Put("key3", DummyString(1024));
-    dbfull()->Flush(FlushOptions());
-    Put("key4", DummyString(1024));
-    ASSERT_EQ(dbfull()->GetLatestSequenceNumber(), 4U);
+    do {
+      rocksdb::SyncPoint::GetInstance()->ClearTrace();
+      rocksdb::SyncPoint::GetInstance()->DisableProcessing();
+      Options options = OptionsForLogIterTest();
+      DestroyAndReopen(&options);
+      Put("key1", DummyString(1024));
+      dbfull()->Flush(FlushOptions());
+      Put("key2", DummyString(1024));
+      dbfull()->Flush(FlushOptions());
+      Put("key3", DummyString(1024));
+      dbfull()->Flush(FlushOptions());
+      Put("key4", DummyString(1024));
+      ASSERT_EQ(dbfull()->GetLatestSequenceNumber(), 4U);
 
-    {
-      auto iter = OpenTransactionLogIter(0);
-      ExpectRecords(4, iter);
-    }
+      {
+        auto iter = OpenTransactionLogIter(0);
+        ExpectRecords(4, iter);
+      }
 
-    rocksdb::SyncPoint::GetInstance()->EnableProcessing();
-    // trigger async flush, and log move. Well, log move will
-    // wait until the GetSortedWalFiles:1 to reproduce the race
-    // condition
-    FlushOptions flush_options;
-    flush_options.wait = false;
-    dbfull()->Flush(flush_options);
+      rocksdb::SyncPoint::GetInstance()->EnableProcessing();
+      // trigger async flush, and log move. Well, log move will
+      // wait until the GetSortedWalFiles:1 to reproduce the race
+      // condition
+      FlushOptions flush_options;
+      flush_options.wait = false;
+      dbfull()->Flush(flush_options);
 
-    // "key5" would be written in a new memtable and log
-    Put("key5", DummyString(1024));
-    {
-      // this iter would miss "key4" if not fixed
-      auto iter = OpenTransactionLogIter(0);
-      ExpectRecords(5, iter);
-    }
-  } while (ChangeCompactOptions());
+      // "key5" would be written in a new memtable and log
+      Put("key5", DummyString(1024));
+      {
+        // this iter would miss "key4" if not fixed
+        auto iter = OpenTransactionLogIter(0);
+        ExpectRecords(5, iter);
+      }
+    } while (ChangeCompactOptions());
+  }
 }
 #endif
 
@@ -6652,6 +6704,53 @@ TEST(DBTest, TailingIteratorKeepAdding) {
   }
 }
 
+TEST(DBTest, TailingIteratorSeekToNext) {
+  CreateAndReopenWithCF({"pikachu"});
+  ReadOptions read_options;
+  read_options.tailing = true;
+
+  std::unique_ptr<Iterator> iter(db_->NewIterator(read_options, handles_[1]));
+  std::string value(1024, 'a');
+
+  const int num_records = 1000;
+  for (int i = 1; i < num_records; ++i) {
+    char buf1[32];
+    char buf2[32];
+    snprintf(buf1, sizeof(buf1), "00a0%016d", i * 5);
+
+    Slice key(buf1, 20);
+    ASSERT_OK(Put(1, key, value));
+
+    if (i % 100 == 99) {
+      ASSERT_OK(Flush(1));
+    }
+
+    snprintf(buf2, sizeof(buf2), "00a0%016d", i * 5 - 2);
+    Slice target(buf2, 20);
+    iter->Seek(target);
+    ASSERT_TRUE(iter->Valid());
+    ASSERT_EQ(iter->key().compare(key), 0);
+  }
+  for (int i = 2 * num_records; i > 0; --i) {
+    char buf1[32];
+    char buf2[32];
+    snprintf(buf1, sizeof(buf1), "00a0%016d", i * 5);
+
+    Slice key(buf1, 20);
+    ASSERT_OK(Put(1, key, value));
+
+    if (i % 100 == 99) {
+      ASSERT_OK(Flush(1));
+    }
+
+    snprintf(buf2, sizeof(buf2), "00a0%016d", i * 5 - 2);
+    Slice target(buf2, 20);
+    iter->Seek(target);
+    ASSERT_TRUE(iter->Valid());
+    ASSERT_EQ(iter->key().compare(key), 0);
+  }
+}
+
 TEST(DBTest, TailingIteratorDeletes) {
   CreateAndReopenWithCF({"pikachu"});
   ReadOptions read_options;
@@ -6723,6 +6822,31 @@ TEST(DBTest, TailingIteratorPrefixSeek) {
   ASSERT_TRUE(!iter->Valid());
 }
 
+TEST(DBTest, BlockBasedTablePrefixIndexTest) {
+  // create a DB with block prefix index
+  BlockBasedTableOptions table_options;
+  Options options = CurrentOptions();
+  table_options.index_type = BlockBasedTableOptions::kHashSearch;
+  options.table_factory.reset(NewBlockBasedTableFactory(table_options));
+  options.prefix_extractor.reset(NewFixedPrefixTransform(1));
+
+
+  Reopen(&options);
+  ASSERT_OK(Put("k1", "v1"));
+  Flush();
+  ASSERT_OK(Put("k2", "v2"));
+
+  // Reopen it without prefix extractor, make sure everything still works.
+  // RocksDB should just fall back to the binary index.
+  table_options.index_type = BlockBasedTableOptions::kBinarySearch;
+  options.table_factory.reset(NewBlockBasedTableFactory(table_options));
+  options.prefix_extractor.reset();
+
+  Reopen(&options);
+  ASSERT_EQ("v1", Get("k1"));
+  ASSERT_EQ("v2", Get("k2"));
+}
+
 TEST(DBTest, ChecksumTest) {
   BlockBasedTableOptions table_options;
   Options options = CurrentOptions();
@@ -6757,6 +6881,42 @@ TEST(DBTest, ChecksumTest) {
   ASSERT_EQ("f", Get("e"));
   ASSERT_EQ("h", Get("g"));
 }
+
+TEST(DBTest, FIFOCompactionTest) {
+  for (int iter = 0; iter < 2; ++iter) {
+    // first iteration -- auto compaction
+    // second iteration -- manual compaction
+    Options options;
+    options.compaction_style = kCompactionStyleFIFO;
+    options.write_buffer_size = 100 << 10;                             // 100KB
+    options.compaction_options_fifo.max_table_files_size = 500 << 10;  // 500KB
+    options.compression = kNoCompression;
+    options.create_if_missing = true;
+    if (iter == 1) {
+      options.disable_auto_compactions = true;
+    }
+    DestroyAndReopen(&options);
+
+    Random rnd(301);
+    for (int i = 0; i < 6; ++i) {
+      for (int j = 0; j < 100; ++j) {
+        ASSERT_OK(Put(std::to_string(i * 100 + j), RandomString(&rnd, 1024)));
+      }
+      // flush should happen here
+    }
+    if (iter == 0) {
+      ASSERT_OK(dbfull()->TEST_WaitForCompact());
+    } else {
+      ASSERT_OK(db_->CompactRange(nullptr, nullptr));
+    }
+    // only 5 files should survive
+    ASSERT_EQ(NumTableFilesAtLevel(0), 5);
+    for (int i = 0; i < 50; ++i) {
+      // these keys should be deleted in previous compaction
+      ASSERT_EQ("NOT_FOUND", Get(std::to_string(i)));
+    }
+  }
+}
 }  // namespace rocksdb
 
 int main(int argc, char** argv) {

db/dbformat.h

@@ -146,6 +146,11 @@ class InternalKey {
     AppendInternalKey(&rep_, ParsedInternalKey(user_key, s, t));
   }
 
+  bool Valid() const {
+    ParsedInternalKey parsed;
+    return ParseInternalKey(Slice(rep_), &parsed);
+  }
+
   void DecodeFrom(const Slice& s) { rep_.assign(s.data(), s.size()); }
   Slice Encode() const {
     assert(!rep_.empty());
@@ -251,24 +256,45 @@ class IterKey {
 
   void Clear() { key_size_ = 0; }
 
-  void SetUserKey(const Slice& user_key) {
-    size_t size = user_key.size();
+  void SetKey(const Slice& key) {
+    size_t size = key.size();
     EnlargeBufferIfNeeded(size);
-    memcpy(key_, user_key.data(), size);
+    memcpy(key_, key.data(), size);
     key_size_ = size;
   }
 
+  void SetInternalKey(const Slice& key_prefix, const Slice& user_key,
+                      SequenceNumber s,
+                      ValueType value_type = kValueTypeForSeek) {
+    size_t psize = key_prefix.size();
+    size_t usize = user_key.size();
+    EnlargeBufferIfNeeded(psize + usize + sizeof(uint64_t));
+    if (psize > 0) {
+      memcpy(key_, key_prefix.data(), psize);
+    }
+    memcpy(key_ + psize, user_key.data(), usize);
+    EncodeFixed64(key_ + usize + psize, PackSequenceAndType(s, value_type));
+    key_size_ = psize + usize + sizeof(uint64_t);
+  }
+
   void SetInternalKey(const Slice& user_key, SequenceNumber s,
                       ValueType value_type = kValueTypeForSeek) {
-    size_t usize = user_key.size();
-    EnlargeBufferIfNeeded(usize + sizeof(uint64_t));
-    memcpy(key_, user_key.data(), usize);
-    EncodeFixed64(key_ + usize, PackSequenceAndType(s, value_type));
-    key_size_ = usize + sizeof(uint64_t);
+    SetInternalKey(Slice(), user_key, s, value_type);
+  }
+
+  void Reserve(size_t size) {
+    EnlargeBufferIfNeeded(size);
+    key_size_ = size;
   }
 
   void SetInternalKey(const ParsedInternalKey& parsed_key) {
-    SetInternalKey(parsed_key.user_key, parsed_key.sequence, parsed_key.type);
+    SetInternalKey(Slice(), parsed_key);
+  }
+
+  void SetInternalKey(const Slice& key_prefix,
+                      const ParsedInternalKey& parsed_key_suffix) {
+    SetInternalKey(key_prefix, parsed_key_suffix.user_key,
+                   parsed_key_suffix.sequence, parsed_key_suffix.type);
   }
 
  private:

db/filename.cc

@@ -226,7 +226,8 @@ bool ParseFileName(const std::string& fname,
 }
 
 Status SetCurrentFile(Env* env, const std::string& dbname,
-                      uint64_t descriptor_number) {
+                      uint64_t descriptor_number,
+                      Directory* directory_to_fsync) {
   // Remove leading "dbname/" and add newline to manifest file name
   std::string manifest = DescriptorFileName(dbname, descriptor_number);
   Slice contents = manifest;
@@ -237,7 +238,11 @@ Status SetCurrentFile(Env* env, const std::string& dbname,
   if (s.ok()) {
     s = env->RenameFile(tmp, CurrentFileName(dbname));
   }
-  if (!s.ok()) {
+  if (s.ok()) {
+    if (directory_to_fsync != nullptr) {
+      directory_to_fsync->Fsync();
+    }
+  } else {
     env->DeleteFile(tmp);
   }
   return s;

db/filename.h

@@ -20,6 +20,7 @@
 namespace rocksdb {
 
 class Env;
+class Directory;
 
 enum FileType {
   kLogFile,
@@ -100,7 +101,8 @@ extern bool ParseFileName(const std::string& filename,
 // Make the CURRENT file point to the descriptor file with the
 // specified number.
 extern Status SetCurrentFile(Env* env, const std::string& dbname,
-                             uint64_t descriptor_number);
+                             uint64_t descriptor_number,
+                             Directory* directory_to_fsync);
 
 // Make the IDENTITY file for the db
 extern Status SetIdentityFile(Env* env, const std::string& dbname);

db/forward_iterator.cc

@@ -0,0 +1,383 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+
+#ifndef ROCKSDB_LITE
+#include "db/forward_iterator.h"
+
+#include <string>
+#include <utility>
+#include <limits>
+#include "db/db_impl.h"
+#include "db/db_iter.h"
+#include "db/column_family.h"
+#include "rocksdb/env.h"
+#include "rocksdb/slice.h"
+#include "rocksdb/slice_transform.h"
+#include "table/merger.h"
+#include "db/dbformat.h"
+
+namespace rocksdb {
+
+// Usage:
+//     LevelIterator iter;
+//     iter.SetFileIndex(file_index);
+//     iter.Seek(target);
+//     iter.Next()
+class LevelIterator : public Iterator {
+ public:
+  LevelIterator(const ColumnFamilyData* const cfd,
+      const ReadOptions& read_options,
+      const std::vector<FileMetaData*>& files)
+    : cfd_(cfd), read_options_(read_options), files_(files), valid_(false),
+      file_index_(std::numeric_limits<uint32_t>::max()) {}
+
+  void SetFileIndex(uint32_t file_index) {
+    assert(file_index < files_.size());
+    if (file_index != file_index_) {
+      file_index_ = file_index;
+      file_iter_.reset(cfd_->table_cache()->NewIterator(
+          read_options_, *(cfd_->soptions()), cfd_->internal_comparator(),
+          files_[file_index_]->fd, nullptr /* table_reader_ptr */, false));
+    }
+    valid_ = false;
+  }
+  void SeekToLast() override {
+    status_ = Status::NotSupported("LevelIterator::SeekToLast()");
+    valid_ = false;
+  }
+  void Prev() {
+    status_ = Status::NotSupported("LevelIterator::Prev()");
+    valid_ = false;
+  }
+  bool Valid() const override {
+    return valid_;
+  }
+  void SeekToFirst() override {
+    SetFileIndex(0);
+    file_iter_->SeekToFirst();
+    valid_ = file_iter_->Valid();
+  }
+  void Seek(const Slice& internal_key) override {
+    assert(file_iter_ != nullptr);
+    file_iter_->Seek(internal_key);
+    valid_ = file_iter_->Valid();
+    assert(valid_);
+  }
+  void Next() override {
+    assert(valid_);
+    file_iter_->Next();
+    while (!file_iter_->Valid()) {
+      if (file_index_ + 1 >= files_.size()) {
+        valid_ = false;
+        return;
+      }
+      SetFileIndex(file_index_ + 1);
+      file_iter_->SeekToFirst();
+    }
+    valid_ = file_iter_->Valid();
+  }
+  Slice key() const override {
+    assert(valid_);
+    return file_iter_->key();
+  }
+  Slice value() const override {
+    assert(valid_);
+    return file_iter_->value();
+  }
+  Status status() const override {
+    return status_;
+  }
+
+ private:
+  const ColumnFamilyData* const cfd_;
+  const ReadOptions& read_options_;
+  const std::vector<FileMetaData*>& files_;
+
+  bool valid_;
+  uint32_t file_index_;
+  Status status_;
+  std::unique_ptr<Iterator> file_iter_;
+};
+
+ForwardIterator::ForwardIterator(DBImpl* db, const ReadOptions& read_options,
+                                 ColumnFamilyData* cfd)
+    : db_(db),
+      read_options_(read_options),
+      cfd_(cfd),
+      prefix_extractor_(cfd->options()->prefix_extractor.get()),
+      user_comparator_(cfd->user_comparator()),
+      immutable_min_heap_(MinIterComparator(&cfd_->internal_comparator())),
+      sv_(nullptr),
+      mutable_iter_(nullptr),
+      current_(nullptr),
+      valid_(false),
+      is_prev_set_(false) {}
+
+ForwardIterator::~ForwardIterator() {
+  Cleanup();
+}
+
+void ForwardIterator::Cleanup() {
+  delete mutable_iter_;
+  for (auto* m : imm_iters_) {
+    delete m;
+  }
+  imm_iters_.clear();
+  for (auto* f : l0_iters_) {
+    delete f;
+  }
+  l0_iters_.clear();
+  for (auto* l : level_iters_) {
+    delete l;
+  }
+  level_iters_.clear();
+
+  if (sv_ != nullptr && sv_->Unref()) {
+    DBImpl::DeletionState deletion_state;
+    db_->mutex_.Lock();
+    sv_->Cleanup();
+    db_->FindObsoleteFiles(deletion_state, false, true);
+    db_->mutex_.Unlock();
+    delete sv_;
+    if (deletion_state.HaveSomethingToDelete()) {
+      db_->PurgeObsoleteFiles(deletion_state);
+    }
+  }
+}
+
+bool ForwardIterator::Valid() const {
+  return valid_;
+}
+
+void ForwardIterator::SeekToFirst() {
+  if (sv_ == nullptr ||
+      sv_ ->version_number != cfd_->GetSuperVersionNumber()) {
+    RebuildIterators();
+  }
+  SeekInternal(Slice(), true);
+}
+
+void ForwardIterator::Seek(const Slice& internal_key) {
+  if (sv_ == nullptr ||
+      sv_ ->version_number != cfd_->GetSuperVersionNumber()) {
+    RebuildIterators();
+  }
+  SeekInternal(internal_key, false);
+}
+
+void ForwardIterator::SeekInternal(const Slice& internal_key,
+                                   bool seek_to_first) {
+  // mutable
+  seek_to_first ? mutable_iter_->SeekToFirst() :
+                  mutable_iter_->Seek(internal_key);
+
+  // immutable
+  // TODO(ljin): NeedToSeekImmutable has negative impact on performance
+  // if it turns to need to seek immutable often. We probably want to have
+  // an option to turn it off.
+  if (seek_to_first || NeedToSeekImmutable(internal_key)) {
+    {
+      auto tmp = MinIterHeap(MinIterComparator(&cfd_->internal_comparator()));
+      immutable_min_heap_.swap(tmp);
+    }
+    for (auto* m : imm_iters_) {
+      seek_to_first ? m->SeekToFirst() : m->Seek(internal_key);
+      if (m->Valid()) {
+        immutable_min_heap_.push(m);
+      }
+    }
+
+    auto* files = sv_->current->files_;
+    for (uint32_t i = 0; i < files[0].size(); ++i) {
+      if (seek_to_first) {
+        l0_iters_[i]->SeekToFirst();
+      } else {
+        // If the target key passes over the larget key, we are sure Next()
+        // won't go over this file.
+        if (user_comparator_->Compare(ExtractUserKey(internal_key),
+              files[0][i]->largest.user_key()) > 0) {
+          continue;
+        }
+        l0_iters_[i]->Seek(internal_key);
+      }
+      if (l0_iters_[i]->Valid()) {
+        immutable_min_heap_.push(l0_iters_[i]);
+      }
+    }
+    for (int32_t level = 1; level < sv_->current->NumberLevels(); ++level) {
+      if (files[level].empty()) {
+        continue;
+      }
+      assert(level_iters_[level - 1] != nullptr);
+      uint32_t f_idx = 0;
+      if (!seek_to_first) {
+        f_idx = FindFileInRange(
+            files[level], internal_key, 0, files[level].size());
+      }
+      if (f_idx < files[level].size()) {
+        level_iters_[level - 1]->SetFileIndex(f_idx);
+        seek_to_first ? level_iters_[level - 1]->SeekToFirst() :
+                        level_iters_[level - 1]->Seek(internal_key);
+        if (level_iters_[level - 1]->Valid()) {
+          immutable_min_heap_.push(level_iters_[level - 1]);
+        }
+      }
+    }
+
+    if (seek_to_first || immutable_min_heap_.empty()) {
+      is_prev_set_ = false;
+    } else {
+      prev_key_.SetKey(internal_key);
+      is_prev_set_ = true;
+    }
+  }
+
+  UpdateCurrent();
+}
+
+void ForwardIterator::Next() {
+  assert(valid_);
+
+  if (sv_ == nullptr ||
+      sv_ ->version_number != cfd_->GetSuperVersionNumber()) {
+    std::string current_key = key().ToString();
+    Slice old_key(current_key.data(), current_key.size());
+
+    RebuildIterators();
+    SeekInternal(old_key, false);
+    if (!valid_ || key().compare(old_key) != 0) {
+      return;
+    }
+  } else if (current_ != mutable_iter_) {
+    // It is going to advance immutable iterator
+    prev_key_.SetKey(current_->key());
+    is_prev_set_ = true;
+  }
+
+  current_->Next();
+  if (current_->Valid() && current_ != mutable_iter_) {
+    immutable_min_heap_.push(current_);
+  }
+  UpdateCurrent();
+}
+
+Slice ForwardIterator::key() const {
+  assert(valid_);
+  return current_->key();
+}
+
+Slice ForwardIterator::value() const {
+  assert(valid_);
+  return current_->value();
+}
+
+Status ForwardIterator::status() const {
+  if (!status_.ok()) {
+    return status_;
+  } else if (!mutable_iter_->status().ok()) {
+    return mutable_iter_->status();
+  }
+  return Status::OK();
+}
+
+void ForwardIterator::RebuildIterators() {
+  // Clean up
+  Cleanup();
+  // New
+  sv_ = cfd_->GetReferencedSuperVersion(&(db_->mutex_));
+  mutable_iter_ = sv_->mem->NewIterator(read_options_);
+  sv_->imm->AddIterators(read_options_, &imm_iters_);
+  const auto& l0_files = sv_->current->files_[0];
+  l0_iters_.reserve(l0_files.size());
+  for (const auto* l0 : l0_files) {
+    l0_iters_.push_back(cfd_->table_cache()->NewIterator(
+        read_options_, *cfd_->soptions(), cfd_->internal_comparator(), l0->fd));
+  }
+  level_iters_.reserve(sv_->current->NumberLevels() - 1);
+  for (int32_t level = 1; level < sv_->current->NumberLevels(); ++level) {
+    if (sv_->current->files_[level].empty()) {
+      level_iters_.push_back(nullptr);
+    } else {
+      level_iters_.push_back(new LevelIterator(cfd_, read_options_,
+          sv_->current->files_[level]));
+    }
+  }
+
+  current_ = nullptr;
+  is_prev_set_ = false;
+}
+
+void ForwardIterator::UpdateCurrent() {
+  if (immutable_min_heap_.empty() && !mutable_iter_->Valid()) {
+    current_ = nullptr;
+  } else if (immutable_min_heap_.empty()) {
+    current_ = mutable_iter_;
+  } else if (!mutable_iter_->Valid()) {
+    current_ = immutable_min_heap_.top();
+    immutable_min_heap_.pop();
+  } else {
+    current_ = immutable_min_heap_.top();
+    assert(current_ != nullptr);
+    assert(current_->Valid());
+    int cmp = cfd_->internal_comparator().InternalKeyComparator::Compare(
+        mutable_iter_->key(), current_->key());
+    assert(cmp != 0);
+    if (cmp > 0) {
+      immutable_min_heap_.pop();
+    } else {
+      current_ = mutable_iter_;
+    }
+  }
+  valid_ = (current_ != nullptr);
+  if (!status_.ok()) {
+    status_ = Status::OK();
+  }
+}
+
+bool ForwardIterator::NeedToSeekImmutable(const Slice& target) {
+  if (!is_prev_set_) {
+    return true;
+  }
+  Slice prev_key = prev_key_.GetKey();
+  if (prefix_extractor_ && prefix_extractor_->Transform(target).compare(
+    prefix_extractor_->Transform(prev_key)) != 0) {
+    return true;
+  }
+  if (cfd_->internal_comparator().InternalKeyComparator::Compare(
+        prev_key, target) >= 0) {
+    return true;
+  }
+  if (immutable_min_heap_.empty() ||
+      cfd_->internal_comparator().InternalKeyComparator::Compare(
+          target, current_ == mutable_iter_ ? immutable_min_heap_.top()->key()
+                                            : current_->key()) > 0) {
+    return true;
+  }
+  return false;
+}
+
+uint32_t ForwardIterator::FindFileInRange(
+    const std::vector<FileMetaData*>& files, const Slice& internal_key,
+    uint32_t left, uint32_t right) {
+  while (left < right) {
+    uint32_t mid = (left + right) / 2;
+    const FileMetaData* f = files[mid];
+    if (cfd_->internal_comparator().InternalKeyComparator::Compare(
+          f->largest.Encode(), internal_key) < 0) {
+      // Key at "mid.largest" is < "target".  Therefore all
+      // files at or before "mid" are uninteresting.
+      left = mid + 1;
+    } else {
+      // Key at "mid.largest" is >= "target".  Therefore all files
+      // after "mid" are uninteresting.
+      right = mid;
+    }
+  }
+  return right;
+}
+
+}  // namespace rocksdb
+
+#endif  // ROCKSDB_LITE

db/forward_iterator.h

@@ -0,0 +1,105 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+#pragma once
+
+#ifndef ROCKSDB_LITE
+
+#include <string>
+#include <vector>
+#include <queue>
+
+#include "rocksdb/db.h"
+#include "rocksdb/iterator.h"
+#include "rocksdb/options.h"
+#include "db/dbformat.h"
+
+namespace rocksdb {
+
+class DBImpl;
+class Env;
+struct SuperVersion;
+class ColumnFamilyData;
+class LevelIterator;
+struct FileMetaData;
+
+class MinIterComparator {
+ public:
+  explicit MinIterComparator(const Comparator* comparator) :
+    comparator_(comparator) {}
+
+  bool operator()(Iterator* a, Iterator* b) {
+    return comparator_->Compare(a->key(), b->key()) > 0;
+  }
+ private:
+  const Comparator* comparator_;
+};
+
+typedef std::priority_queue<Iterator*,
+          std::vector<Iterator*>,
+          MinIterComparator> MinIterHeap;
+
+/**
+ * ForwardIterator is a special type of iterator that only supports Seek()
+ * and Next(). It is expected to perform better than TailingIterator by
+ * removing the encapsulation and making all information accessible within
+ * the iterator. At the current implementation, snapshot is taken at the
+ * time Seek() is called. The Next() followed do not see new values after.
+ */
+class ForwardIterator : public Iterator {
+ public:
+  ForwardIterator(DBImpl* db, const ReadOptions& read_options,
+                  ColumnFamilyData* cfd);
+  virtual ~ForwardIterator();
+
+  void SeekToLast() override {
+    status_ = Status::NotSupported("ForwardIterator::SeekToLast()");
+    valid_ = false;
+  }
+  void Prev() {
+    status_ = Status::NotSupported("ForwardIterator::Prev");
+    valid_ = false;
+  }
+
+  virtual bool Valid() const override;
+  void SeekToFirst() override;
+  virtual void Seek(const Slice& target) override;
+  virtual void Next() override;
+  virtual Slice key() const override;
+  virtual Slice value() const override;
+  virtual Status status() const override;
+
+ private:
+  void Cleanup();
+  void RebuildIterators();
+  void SeekInternal(const Slice& internal_key, bool seek_to_first);
+  void UpdateCurrent();
+  bool NeedToSeekImmutable(const Slice& internal_key);
+  uint32_t FindFileInRange(
+    const std::vector<FileMetaData*>& files, const Slice& internal_key,
+    uint32_t left, uint32_t right);
+
+  DBImpl* const db_;
+  const ReadOptions read_options_;
+  ColumnFamilyData* const cfd_;
+  const SliceTransform* const prefix_extractor_;
+  const Comparator* user_comparator_;
+  MinIterHeap immutable_min_heap_;
+
+  SuperVersion* sv_;
+  Iterator* mutable_iter_;
+  std::vector<Iterator*> imm_iters_;
+  std::vector<Iterator*> l0_iters_;
+  std::vector<LevelIterator*> level_iters_;
+  Iterator* current_;
+  // internal iterator status
+  Status status_;
+  bool valid_;
+
+  IterKey prev_key_;
+  bool is_prev_set_;
+};
+
+}  // namespace rocksdb
+#endif  // ROCKSDB_LITE

db/memtable.cc

@@ -20,6 +20,7 @@
 #include "rocksdb/iterator.h"
 #include "rocksdb/merge_operator.h"
 #include "rocksdb/slice_transform.h"
+#include "table/merger.h"
 #include "util/arena.h"
 #include "util/coding.h"
 #include "util/murmurhash.h"
@@ -37,7 +38,8 @@ MemTable::MemTable(const InternalKeyComparator& cmp, const Options& options)
       kWriteBufferSize(options.write_buffer_size),
       arena_(options.arena_block_size),
       table_(options.memtable_factory->CreateMemTableRep(
-          comparator_, &arena_, options.prefix_extractor.get())),
+          comparator_, &arena_, options.prefix_extractor.get(),
+          options.info_log.get())),
       num_entries_(0),
       flush_in_progress_(false),
       flush_completed_(false),
@@ -55,7 +57,8 @@ MemTable::MemTable(const InternalKeyComparator& cmp, const Options& options)
     prefix_bloom_.reset(new DynamicBloom(
         options.memtable_prefix_bloom_bits, options.bloom_locality,
         options.memtable_prefix_bloom_probes, nullptr,
-        options.memtable_prefix_bloom_huge_page_tlb_size));
+        options.memtable_prefix_bloom_huge_page_tlb_size,
+        options.info_log.get()));
   }
 }
 
@@ -171,15 +174,24 @@ const char* EncodeKey(std::string* scratch, const Slice& target) {
 class MemTableIterator: public Iterator {
  public:
   MemTableIterator(const MemTable& mem, const ReadOptions& options,
-                   bool enforce_total_order)
+                   bool enforce_total_order, Arena* arena)
       : bloom_(nullptr),
         prefix_extractor_(mem.prefix_extractor_),
-        valid_(false) {
+        valid_(false),
+        arena_mode_(arena != nullptr) {
     if (prefix_extractor_ != nullptr && !enforce_total_order) {
       bloom_ = mem.prefix_bloom_.get();
-      iter_.reset(mem.table_->GetDynamicPrefixIterator());
+      iter_ = mem.table_->GetDynamicPrefixIterator(arena);
     } else {
-      iter_.reset(mem.table_->GetIterator());
+      iter_ = mem.table_->GetIterator(arena);
+    }
+  }
+
+  ~MemTableIterator() {
+    if (arena_mode_) {
+      iter_->~Iterator();
+    } else {
+      delete iter_;
     }
   }
 
@@ -226,8 +238,9 @@ class MemTableIterator: public Iterator {
  private:
   DynamicBloom* bloom_;
   const SliceTransform* const prefix_extractor_;
-  std::unique_ptr<MemTableRep::Iterator> iter_;
+  MemTableRep::Iterator* iter_;
   bool valid_;
+  bool arena_mode_;
 
   // No copying allowed
   MemTableIterator(const MemTableIterator&);
@@ -235,8 +248,14 @@ class MemTableIterator: public Iterator {
 };
 
 Iterator* MemTable::NewIterator(const ReadOptions& options,
-    bool enforce_total_order) {
-  return new MemTableIterator(*this, options, enforce_total_order);
+                                bool enforce_total_order, Arena* arena) {
+  if (arena == nullptr) {
+    return new MemTableIterator(*this, options, enforce_total_order, nullptr);
+  } else {
+    auto mem = arena->AllocateAligned(sizeof(MemTableIterator));
+    return new (mem)
+        MemTableIterator(*this, options, enforce_total_order, arena);
+  }
 }
 
 port::RWMutex* MemTable::GetLock(const Slice& key) {
@@ -347,7 +366,7 @@ static bool SaveValue(void* arg, const char* entry) {
           s->value->assign(v.data(), v.size());
         }
         if (s->inplace_update_support) {
-          s->mem->GetLock(s->key->user_key())->Unlock();
+          s->mem->GetLock(s->key->user_key())->ReadUnlock();
         }
         *(s->found_final_value) = true;
         return false;

db/memtable.h

@@ -21,6 +21,7 @@
 
 namespace rocksdb {
 
+class Arena;
 class Mutex;
 class MemTableIterator;
 class MergeContext;
@@ -77,8 +78,12 @@ class MemTable {
   //
   // By default, it returns an iterator for prefix seek if prefix_extractor
   // is configured in Options.
+  // arena: If not null, the arena needs to be used to allocate the Iterator.
+  //        Calling ~Iterator of the iterator will destroy all the states but
+  //        those allocated in arena.
   Iterator* NewIterator(const ReadOptions& options,
-                        bool enforce_total_order = false);
+                        bool enforce_total_order = false,
+                        Arena* arena = nullptr);
 
   // Add an entry into memtable that maps key to value at the
   // specified sequence number and with the specified type.

db/memtable_list.cc

@@ -11,6 +11,7 @@
 #include "db/version_set.h"
 #include "rocksdb/env.h"
 #include "rocksdb/iterator.h"
+#include "table/merger.h"
 #include "util/coding.h"
 #include "util/log_buffer.h"
 
@@ -78,6 +79,14 @@ void MemTableListVersion::AddIterators(const ReadOptions& options,
   }
 }
 
+void MemTableListVersion::AddIterators(
+    const ReadOptions& options, MergeIteratorBuilder* merge_iter_builder) {
+  for (auto& m : memlist_) {
+    merge_iter_builder->AddIterator(
+        m->NewIterator(options, merge_iter_builder->GetArena()));
+  }
+}
+
 uint64_t MemTableListVersion::GetTotalNumEntries() const {
   uint64_t total_num = 0;
   for (auto& m : memlist_) {

db/memtable_list.h

@@ -28,6 +28,7 @@ namespace rocksdb {
 class ColumnFamilyData;
 class InternalKeyComparator;
 class Mutex;
+class MergeIteratorBuilder;
 
 // keeps a list of immutable memtables in a vector. the list is immutable
 // if refcount is bigger than one. It is used as a state for Get() and
@@ -49,6 +50,9 @@ class MemTableListVersion {
   void AddIterators(const ReadOptions& options,
                     std::vector<Iterator*>* iterator_list);
 
+  void AddIterators(const ReadOptions& options,
+                    MergeIteratorBuilder* merge_iter_builder);
+
   uint64_t GetTotalNumEntries() const;
 
  private:

db/plain_table_db_test.cc

@@ -61,7 +61,8 @@ class PlainTableDBTest {
   // Return the current option configuration.
   Options CurrentOptions() {
     Options options;
-    options.table_factory.reset(NewPlainTableFactory(16, 2, 0.8, 3));
+    options.table_factory.reset(NewPlainTableFactory(0, 2, 0.8, 3, 0, kPrefix));
+    options.memtable_factory.reset(NewHashLinkListRepFactory(4, 0, 3, true));
     options.prefix_extractor.reset(NewFixedPrefixTransform(8));
     options.allow_mmap_reads = true;
     return options;
@@ -178,16 +179,21 @@ class TestPlainTableReader : public PlainTableReader {
  public:
   TestPlainTableReader(const EnvOptions& storage_options,
                        const InternalKeyComparator& icomparator,
-                       uint64_t file_size, int bloom_bits_per_key,
-                       double hash_table_ratio, size_t index_sparseness,
+                       EncodingType encoding_type, uint64_t file_size,
+                       int bloom_bits_per_key, double hash_table_ratio,
+                       size_t index_sparseness,
                        const TableProperties* table_properties,
                        unique_ptr<RandomAccessFile>&& file,
                        const Options& options, bool* expect_bloom_not_match)
       : PlainTableReader(options, std::move(file), storage_options, icomparator,
-                         file_size, bloom_bits_per_key, hash_table_ratio,
-                         index_sparseness, table_properties, 2 * 1024 * 1024),
+                         encoding_type, file_size, table_properties),
         expect_bloom_not_match_(expect_bloom_not_match) {
-    Status s = PopulateIndex(const_cast<TableProperties*>(table_properties));
+    Status s = MmapDataFile();
+    ASSERT_TRUE(s.ok());
+
+    s = PopulateIndex(const_cast<TableProperties*>(table_properties),
+                      bloom_bits_per_key, hash_table_ratio, index_sparseness,
+                      2 * 1024 * 1024);
     ASSERT_TRUE(s.ok());
   }
 
@@ -209,9 +215,10 @@ class TestPlainTableFactory : public PlainTableFactory {
                                  uint32_t user_key_len, int bloom_bits_per_key,
                                  double hash_table_ratio,
                                  size_t index_sparseness,
-                                 size_t huge_page_tlb_size)
-      : PlainTableFactory(user_key_len, user_key_len, hash_table_ratio,
-                          index_sparseness, huge_page_tlb_size),
+                                 size_t huge_page_tlb_size,
+                                 EncodingType encoding_type)
+      : PlainTableFactory(user_key_len, bloom_bits_per_key, hash_table_ratio,
+                          index_sparseness, huge_page_tlb_size, encoding_type),
         bloom_bits_per_key_(bloom_bits_per_key),
         hash_table_ratio_(hash_table_ratio),
         index_sparseness_(index_sparseness),
@@ -226,10 +233,17 @@ class TestPlainTableFactory : public PlainTableFactory {
                                  options.env, options.info_log.get(), &props);
     ASSERT_TRUE(s.ok());
 
+    auto& user_props = props->user_collected_properties;
+    auto encoding_type_prop =
+        user_props.find(PlainTablePropertyNames::kEncodingType);
+    assert(encoding_type_prop != user_props.end());
+    EncodingType encoding_type = static_cast<EncodingType>(
+        DecodeFixed32(encoding_type_prop->second.c_str()));
+
     std::unique_ptr<PlainTableReader> new_reader(new TestPlainTableReader(
-        soptions, internal_comparator, file_size, bloom_bits_per_key_,
-        hash_table_ratio_, index_sparseness_, props, std::move(file), options,
-        expect_bloom_not_match_));
+        soptions, internal_comparator, encoding_type, file_size,
+        bloom_bits_per_key_, hash_table_ratio_, index_sparseness_, props,
+        std::move(file), options, expect_bloom_not_match_));
 
     *table = std::move(new_reader);
     return s;
@@ -245,18 +259,22 @@ class TestPlainTableFactory : public PlainTableFactory {
 TEST(PlainTableDBTest, Flush) {
   for (size_t huge_page_tlb_size = 0; huge_page_tlb_size <= 2 * 1024 * 1024;
        huge_page_tlb_size += 2 * 1024 * 1024) {
+    for (EncodingType encoding_type : {kPlain, kPrefix}) {
     for (int bloom_bits = 0; bloom_bits <= 117; bloom_bits += 117) {
       for (int total_order = 0; total_order <= 1; total_order++) {
+        if (encoding_type == kPrefix && total_order == 1) {
+          continue;
+        }
         Options options = CurrentOptions();
         options.create_if_missing = true;
         // Set only one bucket to force bucket conflict.
         // Test index interval for the same prefix to be 1, 2 and 4
         if (total_order) {
           options.table_factory.reset(NewTotalOrderPlainTableFactory(
-              16, bloom_bits, 2, huge_page_tlb_size));
+              0, bloom_bits, 2, huge_page_tlb_size));
         } else {
           options.table_factory.reset(NewPlainTableFactory(
-              16, bloom_bits, 0.75, 16, huge_page_tlb_size));
+              0, bloom_bits, 0.75, 16, huge_page_tlb_size, encoding_type));
         }
         DestroyAndReopen(&options);
 
@@ -279,14 +297,19 @@ TEST(PlainTableDBTest, Flush) {
         ASSERT_EQ("v2", Get("0000000000000bar"));
       }
     }
+    }
   }
 }
 
 TEST(PlainTableDBTest, Flush2) {
   for (size_t huge_page_tlb_size = 0; huge_page_tlb_size <= 2 * 1024 * 1024;
        huge_page_tlb_size += 2 * 1024 * 1024) {
+    for (EncodingType encoding_type : {kPlain, kPrefix}) {
     for (int bloom_bits = 0; bloom_bits <= 117; bloom_bits += 117) {
       for (int total_order = 0; total_order <= 1; total_order++) {
+        if (encoding_type == kPrefix && total_order == 1) {
+          continue;
+        }
         bool expect_bloom_not_match = false;
         Options options = CurrentOptions();
         options.create_if_missing = true;
@@ -294,13 +317,13 @@ TEST(PlainTableDBTest, Flush2) {
         // Test index interval for the same prefix to be 1, 2 and 4
         if (total_order) {
           options.prefix_extractor = nullptr;
-          options.table_factory.reset(
-              new TestPlainTableFactory(&expect_bloom_not_match, 16, bloom_bits,
-                                        0, 2, huge_page_tlb_size));
+          options.table_factory.reset(new TestPlainTableFactory(
+              &expect_bloom_not_match, 0, bloom_bits, 0, 2, huge_page_tlb_size,
+              encoding_type));
         } else {
-          options.table_factory.reset(
-              new TestPlainTableFactory(&expect_bloom_not_match, 16, bloom_bits,
-                                        0.75, 16, huge_page_tlb_size));
+          options.table_factory.reset(new TestPlainTableFactory(
+              &expect_bloom_not_match, 0, bloom_bits, 0.75, 16,
+              huge_page_tlb_size, encoding_type));
         }
         DestroyAndReopen(&options);
         ASSERT_OK(Put("0000000000000bar", "b"));
@@ -339,14 +362,19 @@ TEST(PlainTableDBTest, Flush2) {
         }
       }
     }
+    }
   }
 }
 
 TEST(PlainTableDBTest, Iterator) {
   for (size_t huge_page_tlb_size = 0; huge_page_tlb_size <= 2 * 1024 * 1024;
        huge_page_tlb_size += 2 * 1024 * 1024) {
+    for (EncodingType encoding_type : {kPlain, kPrefix}) {
     for (int bloom_bits = 0; bloom_bits <= 117; bloom_bits += 117) {
       for (int total_order = 0; total_order <= 1; total_order++) {
+        if (encoding_type == kPrefix && total_order == 1) {
+          continue;
+        }
         bool expect_bloom_not_match = false;
         Options options = CurrentOptions();
         options.create_if_missing = true;
@@ -354,13 +382,13 @@ TEST(PlainTableDBTest, Iterator) {
         // Test index interval for the same prefix to be 1, 2 and 4
         if (total_order) {
           options.prefix_extractor = nullptr;
-          options.table_factory.reset(
-              new TestPlainTableFactory(&expect_bloom_not_match, 16, bloom_bits,
-                                        0, 2, huge_page_tlb_size));
+          options.table_factory.reset(new TestPlainTableFactory(
+              &expect_bloom_not_match, 16, bloom_bits, 0, 2, huge_page_tlb_size,
+              encoding_type));
         } else {
-          options.table_factory.reset(
-              new TestPlainTableFactory(&expect_bloom_not_match, 16, bloom_bits,
-                                        0.75, 16, huge_page_tlb_size));
+          options.table_factory.reset(new TestPlainTableFactory(
+              &expect_bloom_not_match, 16, bloom_bits, 0.75, 16,
+              huge_page_tlb_size, encoding_type));
         }
         DestroyAndReopen(&options);
 
@@ -447,6 +475,7 @@ TEST(PlainTableDBTest, Iterator) {
         delete iter;
       }
     }
+    }
   }
 }
 
@@ -458,7 +487,7 @@ std::string MakeLongKey(size_t length, char c) {
 
 TEST(PlainTableDBTest, IteratorLargeKeys) {
   Options options = CurrentOptions();
-  options.table_factory.reset(NewTotalOrderPlainTableFactory(0, 0, 16));
+  options.table_factory.reset(NewTotalOrderPlainTableFactory(0, 0, 16, 0));
   options.create_if_missing = true;
   options.prefix_extractor.reset();
   DestroyAndReopen(&options);
@@ -494,6 +523,45 @@ TEST(PlainTableDBTest, IteratorLargeKeys) {
   delete iter;
 }
 
+namespace {
+std::string MakeLongKeyWithPrefix(size_t length, char c) {
+  return "00000000" + std::string(length - 8, c);
+}
+}  // namespace
+
+TEST(PlainTableDBTest, IteratorLargeKeysWithPrefix) {
+  Options options = CurrentOptions();
+  options.table_factory.reset(NewPlainTableFactory(16, 0, 0.8, 3, 0, kPrefix));
+  options.create_if_missing = true;
+  DestroyAndReopen(&options);
+
+  std::string key_list[] = {
+      MakeLongKeyWithPrefix(30, '0'), MakeLongKeyWithPrefix(16, '1'),
+      MakeLongKeyWithPrefix(32, '2'), MakeLongKeyWithPrefix(60, '3'),
+      MakeLongKeyWithPrefix(90, '4'), MakeLongKeyWithPrefix(50, '5'),
+      MakeLongKeyWithPrefix(26, '6')};
+
+  for (size_t i = 0; i < 7; i++) {
+    ASSERT_OK(Put(key_list[i], std::to_string(i)));
+  }
+
+  dbfull()->TEST_FlushMemTable();
+
+  Iterator* iter = dbfull()->NewIterator(ReadOptions());
+  iter->Seek(key_list[0]);
+
+  for (size_t i = 0; i < 7; i++) {
+    ASSERT_TRUE(iter->Valid());
+    ASSERT_EQ(key_list[i], iter->key().ToString());
+    ASSERT_EQ(std::to_string(i), iter->value().ToString());
+    iter->Next();
+  }
+
+  ASSERT_TRUE(!iter->Valid());
+
+  delete iter;
+}
+
 // A test comparator which compare two strings in this way:
 // (1) first compare prefix of 8 bytes in alphabet order,
 // (2) if two strings share the same prefix, sort the other part of the string
@@ -846,6 +914,49 @@ TEST(PlainTableDBTest, CompactionTrigger) {
   ASSERT_EQ(NumTableFilesAtLevel(1), 1);
 }
 
+TEST(PlainTableDBTest, AdaptiveTable) {
+  Options options = CurrentOptions();
+  options.create_if_missing = true;
+
+  options.table_factory.reset(NewPlainTableFactory());
+  DestroyAndReopen(&options);
+
+  ASSERT_OK(Put("1000000000000foo", "v1"));
+  ASSERT_OK(Put("0000000000000bar", "v2"));
+  ASSERT_OK(Put("1000000000000foo", "v3"));
+  dbfull()->TEST_FlushMemTable();
+
+  options.create_if_missing = false;
+  std::shared_ptr<TableFactory> dummy_factory;
+  std::shared_ptr<TableFactory> block_based_factory(
+      NewBlockBasedTableFactory());
+  options.table_factory.reset(NewAdaptiveTableFactory(
+      block_based_factory, dummy_factory, dummy_factory));
+  Reopen(&options);
+  ASSERT_EQ("v3", Get("1000000000000foo"));
+  ASSERT_EQ("v2", Get("0000000000000bar"));
+
+  ASSERT_OK(Put("2000000000000foo", "v4"));
+  ASSERT_OK(Put("3000000000000bar", "v5"));
+  dbfull()->TEST_FlushMemTable();
+  ASSERT_EQ("v4", Get("2000000000000foo"));
+  ASSERT_EQ("v5", Get("3000000000000bar"));
+
+  Reopen(&options);
+  ASSERT_EQ("v3", Get("1000000000000foo"));
+  ASSERT_EQ("v2", Get("0000000000000bar"));
+  ASSERT_EQ("v4", Get("2000000000000foo"));
+  ASSERT_EQ("v5", Get("3000000000000bar"));
+
+  options.table_factory.reset(NewBlockBasedTableFactory());
+  Reopen(&options);
+  ASSERT_NE("v3", Get("1000000000000foo"));
+
+  options.table_factory.reset(NewPlainTableFactory());
+  Reopen(&options);
+  ASSERT_NE("v5", Get("3000000000000bar"));
+}
+
 }  // namespace rocksdb
 
 int main(int argc, char** argv) {

db/prefix_test.cc

@@ -3,6 +3,14 @@
 //  LICENSE file in the root directory of this source tree. An additional grant
 //  of patent rights can be found in the PATENTS file in the same directory.
 
+#ifndef GFLAGS
+#include <cstdio>
+int main() {
+  fprintf(stderr, "Please install gflags to run rocksdb tools\n");
+  return 1;
+}
+#else
+
 #include <algorithm>
 #include <iostream>
 #include <vector>
@@ -17,6 +25,8 @@
 #include "util/stop_watch.h"
 #include "util/testharness.h"
 
+using GFLAGS::ParseCommandLineFlags;
+
 DEFINE_bool(trigger_deadlock, false,
             "issue delete in range scan to trigger PrefixHashMap deadlock");
 DEFINE_uint64(bucket_count, 100000, "number of buckets");
@@ -479,9 +489,11 @@ TEST(PrefixTest, DynamicPrefixIterator) {
 }
 
 int main(int argc, char** argv) {
-  google::ParseCommandLineFlags(&argc, &argv, true);
+  ParseCommandLineFlags(&argc, &argv, true);
   std::cout << kDbName << "\n";
 
   rocksdb::test::RunAllTests();
   return 0;
 }
+
+#endif  // GFLAGS

db/repair.cc

@@ -84,7 +84,7 @@ class Repairer {
     if (status.ok()) {
       unsigned long long bytes = 0;
       for (size_t i = 0; i < tables_.size(); i++) {
-        bytes += tables_[i].meta.file_size;
+        bytes += tables_[i].meta.fd.GetFileSize();
       }
       Log(options_.info_log,
           "**** Repaired rocksdb %s; "
@@ -230,7 +230,7 @@ class Repairer {
     // Do not record a version edit for this conversion to a Table
     // since ExtractMetaData() will also generate edits.
     FileMetaData meta;
-    meta.number = next_file_number_++;
+    meta.fd.number = next_file_number_++;
     ReadOptions ro;
     Iterator* iter = mem->NewIterator(ro, true /* enforce_total_order */);
     status = BuildTable(dbname_, env_, options_, storage_options_, table_cache_,
@@ -240,22 +240,20 @@ class Repairer {
     delete cf_mems_default;
     mem = nullptr;
     if (status.ok()) {
-      if (meta.file_size > 0) {
-        table_numbers_.push_back(meta.number);
+      if (meta.fd.GetFileSize() > 0) {
+        table_numbers_.push_back(meta.fd.GetNumber());
       }
     }
     Log(options_.info_log, "Log #%llu: %d ops saved to Table #%llu %s",
-        (unsigned long long) log,
-        counter,
-        (unsigned long long) meta.number,
-        status.ToString().c_str());
+        (unsigned long long)log, counter,
+        (unsigned long long)meta.fd.GetNumber(), status.ToString().c_str());
     return status;
   }
 
   void ExtractMetaData() {
     for (size_t i = 0; i < table_numbers_.size(); i++) {
       TableInfo t;
-      t.meta.number = table_numbers_[i];
+      t.meta.fd.number = table_numbers_[i];
       Status status = ScanTable(&t);
       if (!status.ok()) {
         std::string fname = TableFileName(dbname_, table_numbers_[i]);
@@ -270,13 +268,12 @@ class Repairer {
   }
 
   Status ScanTable(TableInfo* t) {
-    std::string fname = TableFileName(dbname_, t->meta.number);
+    std::string fname = TableFileName(dbname_, t->meta.fd.GetNumber());
     int counter = 0;
-    Status status = env_->GetFileSize(fname, &t->meta.file_size);
+    Status status = env_->GetFileSize(fname, &t->meta.fd.file_size);
     if (status.ok()) {
-      FileMetaData dummy_meta(t->meta.number, t->meta.file_size);
       Iterator* iter = table_cache_->NewIterator(
-          ReadOptions(), storage_options_, icmp_, dummy_meta);
+          ReadOptions(), storage_options_, icmp_, t->meta.fd);
       bool empty = true;
       ParsedInternalKey parsed;
       t->min_sequence = 0;
@@ -285,7 +282,7 @@ class Repairer {
         Slice key = iter->key();
         if (!ParseInternalKey(key, &parsed)) {
           Log(options_.info_log, "Table #%llu: unparsable key %s",
-              (unsigned long long) t->meta.number,
+              (unsigned long long)t->meta.fd.GetNumber(),
               EscapeString(key).c_str());
           continue;
         }
@@ -309,8 +306,7 @@ class Repairer {
       delete iter;
     }
     Log(options_.info_log, "Table #%llu: %d entries %s",
-        (unsigned long long) t->meta.number,
-        counter,
+        (unsigned long long)t->meta.fd.GetNumber(), counter,
         status.ToString().c_str());
     return status;
   }
@@ -339,9 +335,9 @@ class Repairer {
     for (size_t i = 0; i < tables_.size(); i++) {
       // TODO(opt): separate out into multiple levels
       const TableInfo& t = tables_[i];
-      edit_->AddFile(0, t.meta.number, t.meta.file_size,
-                    t.meta.smallest, t.meta.largest,
-                    t.min_sequence, t.max_sequence);
+      edit_->AddFile(0, t.meta.fd.GetNumber(), t.meta.fd.GetFileSize(),
+                     t.meta.smallest, t.meta.largest, t.min_sequence,
+                     t.max_sequence);
     }
 
     //fprintf(stderr, "NewDescriptor:\n%s\n", edit_.DebugString().c_str());
@@ -363,7 +359,7 @@ class Repairer {
       // Install new manifest
       status = env_->RenameFile(tmp, DescriptorFileName(dbname_, 1));
       if (status.ok()) {
-        status = SetCurrentFile(env_, dbname_, 1);
+        status = SetCurrentFile(env_, dbname_, 1, nullptr);
       } else {
         env_->DeleteFile(tmp);
       }

db/simple_table_db_test.cc

@@ -83,7 +83,7 @@ public:
                      unique_ptr<RandomAccessFile> && file, uint64_t file_size,
                      unique_ptr<TableReader>* table_reader);
 
-  Iterator* NewIterator(const ReadOptions&) override;
+  Iterator* NewIterator(const ReadOptions&, Arena* arena) override;
 
   Status Get(const ReadOptions&, const Slice& key, void* arg,
              bool (*handle_result)(void* arg, const ParsedInternalKey& k,
@@ -218,8 +218,14 @@ std::shared_ptr<const TableProperties> SimpleTableReader::GetTableProperties()
   return rep_->table_properties;
 }
 
-Iterator* SimpleTableReader::NewIterator(const ReadOptions& options) {
-  return new SimpleTableIterator(this);
+Iterator* SimpleTableReader::NewIterator(const ReadOptions& options,
+                                         Arena* arena) {
+  if (arena == nullptr) {
+    return new SimpleTableIterator(this);
+  } else {
+    auto mem = arena->AllocateAligned(sizeof(SimpleTableIterator));
+    return new (mem) SimpleTableIterator(this);
+  }
 }
 
 Status SimpleTableReader::GetOffset(const Slice& target, uint64_t* offset) {

db/table_cache.cc

@@ -13,6 +13,7 @@
 #include "db/version_edit.h"
 
 #include "rocksdb/statistics.h"
+#include "table/iterator_wrapper.h"
 #include "table/table_reader.h"
 #include "util/coding.h"
 #include "util/stop_watch.h"
@@ -30,7 +31,7 @@ static void UnrefEntry(void* arg1, void* arg2) {
   cache->Release(h);
 }
 
-static Slice GetSliceForFileNumber(uint64_t* file_number) {
+static Slice GetSliceForFileNumber(const uint64_t* file_number) {
   return Slice(reinterpret_cast<const char*>(file_number),
                sizeof(*file_number));
 }
@@ -56,11 +57,10 @@ void TableCache::ReleaseHandle(Cache::Handle* handle) {
 
 Status TableCache::FindTable(const EnvOptions& toptions,
                              const InternalKeyComparator& internal_comparator,
-                             uint64_t file_number, uint64_t file_size,
-                             Cache::Handle** handle, bool* table_io,
-                             const bool no_io) {
+                             const FileDescriptor& fd, Cache::Handle** handle,
+                             bool* table_io, const bool no_io) {
   Status s;
-  Slice key = GetSliceForFileNumber(&file_number);
+  Slice key = GetSliceForFileNumber(&fd.number);
   *handle = cache_->Lookup(key);
   if (*handle == nullptr) {
     if (no_io) { // Dont do IO and return a not-found status
@@ -69,7 +69,7 @@ Status TableCache::FindTable(const EnvOptions& toptions,
     if (table_io != nullptr) {
       *table_io = true;    // we had to do IO from storage
     }
-    std::string fname = TableFileName(dbname_, file_number);
+    std::string fname = TableFileName(dbname_, fd.GetNumber());
     unique_ptr<RandomAccessFile> file;
     unique_ptr<TableReader> table_reader;
     s = env_->NewRandomAccessFile(fname, &file, toptions);
@@ -80,8 +80,8 @@ Status TableCache::FindTable(const EnvOptions& toptions,
       }
       StopWatch sw(env_, options_->statistics.get(), TABLE_OPEN_IO_MICROS);
       s = options_->table_factory->NewTableReader(
-          *options_, toptions, internal_comparator, std::move(file), file_size,
-          &table_reader);
+          *options_, toptions, internal_comparator, std::move(file),
+          fd.GetFileSize(), &table_reader);
     }
 
     if (!s.ok()) {
@@ -100,25 +100,25 @@ Status TableCache::FindTable(const EnvOptions& toptions,
 Iterator* TableCache::NewIterator(const ReadOptions& options,
                                   const EnvOptions& toptions,
                                   const InternalKeyComparator& icomparator,
-                                  const FileMetaData& file_meta,
+                                  const FileDescriptor& fd,
                                   TableReader** table_reader_ptr,
-                                  bool for_compaction) {
+                                  bool for_compaction, Arena* arena) {
   if (table_reader_ptr != nullptr) {
     *table_reader_ptr = nullptr;
   }
-  TableReader* table_reader = file_meta.table_reader;
+  TableReader* table_reader = fd.table_reader;
   Cache::Handle* handle = nullptr;
   Status s;
   if (table_reader == nullptr) {
-    s = FindTable(toptions, icomparator, file_meta.number, file_meta.file_size,
-                  &handle, nullptr, options.read_tier == kBlockCacheTier);
+    s = FindTable(toptions, icomparator, fd, &handle, nullptr,
+                  options.read_tier == kBlockCacheTier);
     if (!s.ok()) {
-      return NewErrorIterator(s);
+      return NewErrorIterator(s, arena);
     }
     table_reader = GetTableReaderFromHandle(handle);
   }
 
-  Iterator* result = table_reader->NewIterator(options);
+  Iterator* result = table_reader->NewIterator(options, arena);
   if (handle != nullptr) {
     result->RegisterCleanup(&UnrefEntry, cache_, handle);
   }
@@ -135,16 +135,15 @@ Iterator* TableCache::NewIterator(const ReadOptions& options,
 
 Status TableCache::Get(const ReadOptions& options,
                        const InternalKeyComparator& internal_comparator,
-                       const FileMetaData& file_meta, const Slice& k, void* arg,
+                       const FileDescriptor& fd, const Slice& k, void* arg,
                        bool (*saver)(void*, const ParsedInternalKey&,
                                      const Slice&, bool),
                        bool* table_io, void (*mark_key_may_exist)(void*)) {
-  TableReader* t = file_meta.table_reader;
+  TableReader* t = fd.table_reader;
   Status s;
   Cache::Handle* handle = nullptr;
   if (!t) {
-    s = FindTable(storage_options_, internal_comparator, file_meta.number,
-                  file_meta.file_size, &handle, table_io,
+    s = FindTable(storage_options_, internal_comparator, fd, &handle, table_io,
                   options.read_tier == kBlockCacheTier);
     if (s.ok()) {
       t = GetTableReaderFromHandle(handle);
@@ -164,11 +163,10 @@ Status TableCache::Get(const ReadOptions& options,
 }
 Status TableCache::GetTableProperties(
     const EnvOptions& toptions,
-    const InternalKeyComparator& internal_comparator,
-    const FileMetaData& file_meta,
+    const InternalKeyComparator& internal_comparator, const FileDescriptor& fd,
     std::shared_ptr<const TableProperties>* properties, bool no_io) {
   Status s;
-  auto table_reader = file_meta.table_reader;
+  auto table_reader = fd.table_reader;
   // table already been pre-loaded?
   if (table_reader) {
     *properties = table_reader->GetTableProperties();
@@ -178,8 +176,8 @@ Status TableCache::GetTableProperties(
 
   bool table_io;
   Cache::Handle* table_handle = nullptr;
-  s = FindTable(toptions, internal_comparator, file_meta.number,
-                file_meta.file_size, &table_handle, &table_io, no_io);
+  s = FindTable(toptions, internal_comparator, fd, &table_handle, &table_io,
+                no_io);
   if (!s.ok()) {
     return s;
   }

db/table_cache.h

@@ -23,10 +23,9 @@
 namespace rocksdb {
 
 class Env;
-struct FileMetaData;
+class Arena;
+struct FileDescriptor;
 
-// TODO(sdong): try to come up with a better API to pass the file information
-//              other than simply passing FileMetaData.
 class TableCache {
  public:
   TableCache(const std::string& dbname, const Options* options,
@@ -42,16 +41,16 @@ class TableCache {
   // returned iterator is live.
   Iterator* NewIterator(const ReadOptions& options, const EnvOptions& toptions,
                         const InternalKeyComparator& internal_comparator,
-                        const FileMetaData& file_meta,
+                        const FileDescriptor& file_fd,
                         TableReader** table_reader_ptr = nullptr,
-                        bool for_compaction = false);
+                        bool for_compaction = false, Arena* arena = nullptr);
 
   // If a seek to internal key "k" in specified file finds an entry,
   // call (*handle_result)(arg, found_key, found_value) repeatedly until
   // it returns false.
   Status Get(const ReadOptions& options,
              const InternalKeyComparator& internal_comparator,
-             const FileMetaData& file_meta, const Slice& k, void* arg,
+             const FileDescriptor& file_fd, const Slice& k, void* arg,
              bool (*handle_result)(void*, const ParsedInternalKey&,
                                    const Slice&, bool),
              bool* table_io, void (*mark_key_may_exist)(void*) = nullptr);
@@ -62,7 +61,7 @@ class TableCache {
   // Find table reader
   Status FindTable(const EnvOptions& toptions,
                    const InternalKeyComparator& internal_comparator,
-                   uint64_t file_number, uint64_t file_size, Cache::Handle**,
+                   const FileDescriptor& file_fd, Cache::Handle**,
                    bool* table_io = nullptr, const bool no_io = false);
 
   // Get TableReader from a cache handle.
@@ -76,7 +75,7 @@ class TableCache {
   //            we set `no_io` to be true.
   Status GetTableProperties(const EnvOptions& toptions,
                             const InternalKeyComparator& internal_comparator,
-                            const FileMetaData& file_meta,
+                            const FileDescriptor& file_meta,
                             std::shared_ptr<const TableProperties>* properties,
                             bool no_io = false);
 

db/table_properties_collector.h

@@ -36,6 +36,18 @@ class InternalKeyPropertiesCollector : public TablePropertiesCollector {
   uint64_t deleted_keys_ = 0;
 };
 
+class InternalKeyPropertiesCollectorFactory
+    : public TablePropertiesCollectorFactory {
+ public:
+  virtual TablePropertiesCollector* CreateTablePropertiesCollector() {
+    return new InternalKeyPropertiesCollector();
+  }
+
+  virtual const char* Name() const override {
+    return "InternalKeyPropertiesCollectorFactory";
+  }
+};
+
 // When rocksdb creates a new table, it will encode all "user keys" into
 // "internal keys", which contains meta information of a given entry.
 //
@@ -43,19 +55,11 @@ class InternalKeyPropertiesCollector : public TablePropertiesCollector {
 // invoked.
 class UserKeyTablePropertiesCollector : public TablePropertiesCollector {
  public:
-  explicit UserKeyTablePropertiesCollector(
-      TablePropertiesCollector* collector) :
-      UserKeyTablePropertiesCollector(
-        std::shared_ptr<TablePropertiesCollector>(collector)
-    ) {
-  }
+  // transfer of ownership
+  explicit UserKeyTablePropertiesCollector(TablePropertiesCollector* collector)
+      : collector_(collector) {}
 
-  explicit UserKeyTablePropertiesCollector(
-      std::shared_ptr<TablePropertiesCollector> collector) :
-      collector_(collector) {
-  }
-
-  virtual ~UserKeyTablePropertiesCollector() { }
+  virtual ~UserKeyTablePropertiesCollector() {}
 
   virtual Status Add(const Slice& key, const Slice& value) override;
 
@@ -66,7 +70,26 @@ class UserKeyTablePropertiesCollector : public TablePropertiesCollector {
   UserCollectedProperties GetReadableProperties() const override;
 
  protected:
-  std::shared_ptr<TablePropertiesCollector> collector_;
+  std::unique_ptr<TablePropertiesCollector> collector_;
+};
+
+class UserKeyTablePropertiesCollectorFactory
+    : public TablePropertiesCollectorFactory {
+ public:
+  explicit UserKeyTablePropertiesCollectorFactory(
+      std::shared_ptr<TablePropertiesCollectorFactory> user_collector_factory)
+      : user_collector_factory_(user_collector_factory) {}
+  virtual TablePropertiesCollector* CreateTablePropertiesCollector() {
+    return new UserKeyTablePropertiesCollector(
+        user_collector_factory_->CreateTablePropertiesCollector());
+  }
+
+  virtual const char* Name() const override {
+    return user_collector_factory_->Name();
+  }
+
+ private:
+  std::shared_ptr<TablePropertiesCollectorFactory> user_collector_factory_;
 };
 
 }  // namespace rocksdb

db/table_properties_collector_test.cc

@@ -121,11 +121,18 @@ class RegularKeysStartWithA: public TablePropertiesCollector {
     return UserCollectedProperties{};
   }
 
-
  private:
   uint32_t count_ = 0;
 };
 
+class RegularKeysStartWithAFactory : public TablePropertiesCollectorFactory {
+ public:
+  virtual TablePropertiesCollector* CreateTablePropertiesCollector() {
+    return new RegularKeysStartWithA();
+  }
+  const char* Name() const { return "RegularKeysStartWithA"; }
+};
+
 extern uint64_t kBlockBasedTableMagicNumber;
 extern uint64_t kPlainTableMagicNumber;
 namespace {
@@ -188,14 +195,14 @@ TEST(TablePropertiesTest, CustomizedTablePropertiesCollector) {
   // for block based table
   for (bool encode_as_internal : { true, false }) {
     Options options;
-    auto collector = new RegularKeysStartWithA();
+    std::shared_ptr<TablePropertiesCollectorFactory> collector_factory(
+        new RegularKeysStartWithAFactory());
     if (encode_as_internal) {
-      options.table_properties_collectors = {
-        std::make_shared<UserKeyTablePropertiesCollector>(collector)
-      };
+      options.table_properties_collector_factories.emplace_back(
+          new UserKeyTablePropertiesCollectorFactory(collector_factory));
     } else {
-      options.table_properties_collectors.resize(1);
-      options.table_properties_collectors[0].reset(collector);
+      options.table_properties_collector_factories.resize(1);
+      options.table_properties_collector_factories[0] = collector_factory;
     }
     test::PlainInternalKeyComparator ikc(options.comparator);
     TestCustomizedTablePropertiesCollector(kBlockBasedTableMagicNumber,
@@ -204,9 +211,8 @@ TEST(TablePropertiesTest, CustomizedTablePropertiesCollector) {
 
   // test plain table
   Options options;
-  options.table_properties_collectors.push_back(
-      std::make_shared<RegularKeysStartWithA>()
-  );
+  options.table_properties_collector_factories.emplace_back(
+      new RegularKeysStartWithAFactory());
   options.table_factory = std::make_shared<PlainTableFactory>(8, 8, 0);
   test::PlainInternalKeyComparator ikc(options.comparator);
   TestCustomizedTablePropertiesCollector(kPlainTableMagicNumber, true, options,
@@ -235,9 +241,8 @@ void TestInternalKeyPropertiesCollector(
 
   options.table_factory = table_factory;
   if (sanitized) {
-    options.table_properties_collectors = {
-      std::make_shared<RegularKeysStartWithA>()
-    };
+    options.table_properties_collector_factories.emplace_back(
+        new RegularKeysStartWithAFactory());
     // with sanitization, even regular properties collector will be able to
     // handle internal keys.
     auto comparator = options.comparator;
@@ -249,40 +254,36 @@ void TestInternalKeyPropertiesCollector(
                               options);
     options.comparator = comparator;
   } else {
-    options.table_properties_collectors = {
-      std::make_shared<InternalKeyPropertiesCollector>()
-    };
+    options.table_properties_collector_factories = {
+        std::make_shared<InternalKeyPropertiesCollectorFactory>()};
   }
 
-  MakeBuilder(options, pikc, &writable, &builder);
-  for (const auto& k : keys) {
-    builder->Add(k.Encode(), "val");
-  }
+  for (int iter = 0; iter < 2; ++iter) {
+    MakeBuilder(options, pikc, &writable, &builder);
+    for (const auto& k : keys) {
+      builder->Add(k.Encode(), "val");
+    }
 
-  ASSERT_OK(builder->Finish());
+    ASSERT_OK(builder->Finish());
 
-  FakeRandomeAccessFile readable(writable->contents());
-  TableProperties* props;
-  Status s = ReadTableProperties(
-      &readable,
-      writable->contents().size(),
-      magic_number,
-      Env::Default(),
-      nullptr,
-      &props
-  );
-  ASSERT_OK(s);
+    FakeRandomeAccessFile readable(writable->contents());
+    TableProperties* props;
+    Status s =
+        ReadTableProperties(&readable, writable->contents().size(),
+                            magic_number, Env::Default(), nullptr, &props);
+    ASSERT_OK(s);
 
-  std::unique_ptr<TableProperties> props_guard(props);
-  auto user_collected = props->user_collected_properties;
-  uint64_t deleted = GetDeletedKeys(user_collected);
-  ASSERT_EQ(4u, deleted);
+    std::unique_ptr<TableProperties> props_guard(props);
+    auto user_collected = props->user_collected_properties;
+    uint64_t deleted = GetDeletedKeys(user_collected);
+    ASSERT_EQ(4u, deleted);
 
-  if (sanitized) {
-    uint32_t starts_with_A = 0;
-    Slice key(user_collected.at("Count"));
-    ASSERT_TRUE(GetVarint32(&key, &starts_with_A));
-    ASSERT_EQ(1u, starts_with_A);
+    if (sanitized) {
+      uint32_t starts_with_A = 0;
+      Slice key(user_collected.at("Count"));
+      ASSERT_TRUE(GetVarint32(&key, &starts_with_A));
+      ASSERT_EQ(1u, starts_with_A);
+    }
   }
 }
 }  // namespace

db/transaction_log_impl.cc

@@ -93,6 +93,7 @@ void TransactionLogIteratorImpl::SeekToStartSequence(
   Status s = OpenLogReader(files_->at(startFileIndex).get());
   if (!s.ok()) {
     currentStatus_ = s;
+    reporter_.Info(currentStatus_.ToString().c_str());
     return;
   }
   while (RestrictedRead(&record, &scratch)) {

db/version_edit.cc

@@ -95,8 +95,8 @@ void VersionEdit::EncodeTo(std::string* dst) const {
     const FileMetaData& f = new_files_[i].second;
     PutVarint32(dst, kNewFile2);
     PutVarint32(dst, new_files_[i].first);  // level
-    PutVarint64(dst, f.number);
-    PutVarint64(dst, f.file_size);
+    PutVarint64(dst, f.fd.GetNumber());
+    PutVarint64(dst, f.fd.GetFileSize());
     PutLengthPrefixedSlice(dst, f.smallest.Encode());
     PutLengthPrefixedSlice(dst, f.largest.Encode());
     PutVarint64(dst, f.smallest_seqno);
@@ -123,7 +123,7 @@ static bool GetInternalKey(Slice* input, InternalKey* dst) {
   Slice str;
   if (GetLengthPrefixedSlice(input, &str)) {
     dst->DecodeFrom(str);
-    return true;
+    return dst->Valid();
   } else {
     return false;
   }
@@ -230,12 +230,14 @@ Status VersionEdit::DecodeFrom(const Slice& src) {
         }
         break;
 
-      case kNewFile:
-        if (GetLevel(&input, &level, &msg) &&
-            GetVarint64(&input, &f.number) &&
-            GetVarint64(&input, &f.file_size) &&
+      case kNewFile: {
+        uint64_t number;
+        uint64_t file_size;
+        if (GetLevel(&input, &level, &msg) && GetVarint64(&input, &number) &&
+            GetVarint64(&input, &file_size) &&
             GetInternalKey(&input, &f.smallest) &&
             GetInternalKey(&input, &f.largest)) {
+          f.fd = FileDescriptor(number, file_size);
           new_files_.push_back(std::make_pair(level, f));
         } else {
           if (!msg) {
@@ -243,15 +245,17 @@ Status VersionEdit::DecodeFrom(const Slice& src) {
           }
         }
         break;
-
-      case kNewFile2:
-        if (GetLevel(&input, &level, &msg) &&
-            GetVarint64(&input, &f.number) &&
-            GetVarint64(&input, &f.file_size) &&
+      }
+      case kNewFile2: {
+        uint64_t number;
+        uint64_t file_size;
+        if (GetLevel(&input, &level, &msg) && GetVarint64(&input, &number) &&
+            GetVarint64(&input, &file_size) &&
             GetInternalKey(&input, &f.smallest) &&
             GetInternalKey(&input, &f.largest) &&
             GetVarint64(&input, &f.smallest_seqno) &&
-            GetVarint64(&input, &f.largest_seqno) ) {
+            GetVarint64(&input, &f.largest_seqno)) {
+          f.fd = FileDescriptor(number, file_size);
           new_files_.push_back(std::make_pair(level, f));
         } else {
           if (!msg) {
@@ -259,6 +263,7 @@ Status VersionEdit::DecodeFrom(const Slice& src) {
           }
         }
         break;
+      }
 
       case kColumnFamily:
         if (!GetVarint32(&input, &column_family_)) {
@@ -336,9 +341,9 @@ std::string VersionEdit::DebugString(bool hex_key) const {
     r.append("\n  AddFile: ");
     AppendNumberTo(&r, new_files_[i].first);
     r.append(" ");
-    AppendNumberTo(&r, f.number);
+    AppendNumberTo(&r, f.fd.GetNumber());
     r.append(" ");
-    AppendNumberTo(&r, f.file_size);
+    AppendNumberTo(&r, f.fd.GetFileSize());
     r.append(" ");
     r.append(f.smallest.DebugString(hex_key));
     r.append(" .. ");

db/version_edit.h

@@ -19,11 +19,28 @@ namespace rocksdb {
 
 class VersionSet;
 
+// A copyable structure contains information needed to read data from an SST
+// file. It can contains a pointer to a table reader opened for the file, or
+// file number and size, which can be used to create a new table reader for it.
+// The behavior is undefined when a copied of the structure is used when the
+// file is not in any live version any more.
+struct FileDescriptor {
+  uint64_t number;
+  uint64_t file_size;  // File size in bytes
+  // Table reader in table_reader_handle
+  TableReader* table_reader;
+
+  FileDescriptor(uint64_t number, uint64_t file_size)
+      : number(number), file_size(file_size), table_reader(nullptr) {}
+
+  uint64_t GetNumber() const { return number; }
+  uint64_t GetFileSize() const { return file_size; }
+};
+
 struct FileMetaData {
   int refs;
+  FileDescriptor fd;
   int allowed_seeks;          // Seeks allowed until compaction
-  uint64_t number;
-  uint64_t file_size;         // File size in bytes
   InternalKey smallest;       // Smallest internal key served by table
   InternalKey largest;        // Largest internal key served by table
   bool being_compacted;       // Is this file undergoing compaction?
@@ -32,18 +49,13 @@ struct FileMetaData {
 
   // Needs to be disposed when refs becomes 0.
   Cache::Handle* table_reader_handle;
-  // Table reader in table_reader_handle
-  TableReader* table_reader;
 
-  FileMetaData(uint64_t number, uint64_t file_size)
+  FileMetaData()
       : refs(0),
+        fd(0, 0),
         allowed_seeks(1 << 30),
-        number(number),
-        file_size(file_size),
         being_compacted(false),
-        table_reader_handle(nullptr),
-        table_reader(nullptr) {}
-  FileMetaData() : FileMetaData(0, 0) {}
+        table_reader_handle(nullptr) {}
 };
 
 class VersionEdit {
@@ -89,8 +101,7 @@ class VersionEdit {
                const SequenceNumber& largest_seqno) {
     assert(smallest_seqno <= largest_seqno);
     FileMetaData f;
-    f.number = file;
-    f.file_size = file_size;
+    f.fd = FileDescriptor(file, file_size);
     f.smallest = smallest;
     f.largest = largest;
     f.smallest_seqno = smallest_seqno;

db/version_set.cc

@@ -7,9 +7,9 @@
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#define __STDC_FORMAT_MACROS
 #include "db/version_set.h"
 
+#define __STDC_FORMAT_MACROS
 #include <inttypes.h>
 #include <algorithm>
 #include <map>
@@ -42,7 +42,7 @@ namespace rocksdb {
 static uint64_t TotalFileSize(const std::vector<FileMetaData*>& files) {
   uint64_t sum = 0;
   for (size_t i = 0; i < files.size() && files[i]; i++) {
-    sum += files[i]->file_size;
+    sum += files[i]->fd.GetFileSize();
   }
   return sum;
 }
@@ -150,18 +150,6 @@ bool SomeFileOverlapsRange(
   return !BeforeFile(ucmp, largest_user_key, files[index]);
 }
 
-namespace {
-// Used for LevelFileNumIterator to pass "block handle" value,
-// which actually means file information in this iterator.
-// It contains subset of fields of FileMetaData, that is sufficient
-// for table cache to use.
-struct EncodedFileMetaData {
-  uint64_t number;   // file number
-  uint64_t file_size;   // file size
-  TableReader* table_reader;   // cached table reader
-};
-}  // namespace
-
 // An internal iterator.  For a given version/level pair, yields
 // information about the files in the level.  For a given entry, key()
 // is the largest key that occurs in the file, and value() is an
@@ -173,7 +161,8 @@ class Version::LevelFileNumIterator : public Iterator {
                        const std::vector<FileMetaData*>* flist)
       : icmp_(icmp),
         flist_(flist),
-        index_(flist->size()) {        // Marks as invalid
+        index_(flist->size()),
+        current_value_(0, 0) {  // Marks as invalid
   }
   virtual bool Valid() const {
     return index_ < flist_->size();
@@ -204,18 +193,16 @@ class Version::LevelFileNumIterator : public Iterator {
   Slice value() const {
     assert(Valid());
     auto* file_meta = (*flist_)[index_];
-    current_value_.number = file_meta->number;
-    current_value_.file_size = file_meta->file_size;
-    current_value_.table_reader = file_meta->table_reader;
+    current_value_ = file_meta->fd;
     return Slice(reinterpret_cast<const char*>(&current_value_),
-                 sizeof(EncodedFileMetaData));
+                 sizeof(FileDescriptor));
   }
   virtual Status status() const { return Status::OK(); }
  private:
   const InternalKeyComparator icmp_;
   const std::vector<FileMetaData*>* const flist_;
   uint32_t index_;
-  mutable EncodedFileMetaData current_value_;
+  mutable FileDescriptor current_value_;
 };
 
 class Version::LevelFileIteratorState : public TwoLevelIteratorState {
@@ -230,17 +217,15 @@ class Version::LevelFileIteratorState : public TwoLevelIteratorState {
       for_compaction_(for_compaction) {}
 
   Iterator* NewSecondaryIterator(const Slice& meta_handle) override {
-    if (meta_handle.size() != sizeof(EncodedFileMetaData)) {
+    if (meta_handle.size() != sizeof(FileDescriptor)) {
       return NewErrorIterator(
           Status::Corruption("FileReader invoked with unexpected value"));
     } else {
-      const EncodedFileMetaData* encoded_meta =
-          reinterpret_cast<const EncodedFileMetaData*>(meta_handle.data());
-      FileMetaData meta(encoded_meta->number, encoded_meta->file_size);
-      meta.table_reader = encoded_meta->table_reader;
-      return table_cache_->NewIterator(read_options_, env_options_,
-          icomparator_, meta, nullptr /* don't need reference to table*/,
-          for_compaction_);
+      const FileDescriptor* fd =
+          reinterpret_cast<const FileDescriptor*>(meta_handle.data());
+      return table_cache_->NewIterator(
+          read_options_, env_options_, icomparator_, *fd,
+          nullptr /* don't need reference to table*/, for_compaction_);
     }
   }
 
@@ -261,12 +246,12 @@ Status Version::GetPropertiesOfAllTables(TablePropertiesCollection* props) {
   auto options = cfd_->options();
   for (int level = 0; level < num_levels_; level++) {
     for (const auto& file_meta : files_[level]) {
-      auto fname = TableFileName(vset_->dbname_, file_meta->number);
+      auto fname = TableFileName(vset_->dbname_, file_meta->fd.GetNumber());
       // 1. If the table is already present in table cache, load table
       // properties from there.
       std::shared_ptr<const TableProperties> table_properties;
       Status s = table_cache->GetTableProperties(
-          vset_->storage_options_, cfd_->internal_comparator(), *file_meta,
+          vset_->storage_options_, cfd_->internal_comparator(), file_meta->fd,
           &table_properties, true /* no io */);
       if (s.ok()) {
         props->insert({fname, table_properties});
@@ -292,7 +277,7 @@ Status Version::GetPropertiesOfAllTables(TablePropertiesCollection* props) {
       // By setting the magic number to kInvalidTableMagicNumber, we can by
       // pass the magic number check in the footer.
       s = ReadTableProperties(
-          file.get(), file_meta->file_size,
+          file.get(), file_meta->fd.GetFileSize(),
           Footer::kInvalidTableMagicNumber /* table's magic number */,
           vset_->env_, options->info_log.get(), &raw_table_properties);
       if (!s.ok()) {
@@ -315,7 +300,7 @@ void Version::AddIterators(const ReadOptions& read_options,
   // Merge all level zero files together since they may overlap
   for (const FileMetaData* file : files_[0]) {
     iters->push_back(cfd_->table_cache()->NewIterator(
-        read_options, soptions, cfd_->internal_comparator(), *file));
+        read_options, soptions, cfd_->internal_comparator(), file->fd));
   }
 
   // For levels > 0, we can use a concatenating iterator that sequentially
@@ -332,6 +317,32 @@ void Version::AddIterators(const ReadOptions& read_options,
   }
 }
 
+void Version::AddIterators(const ReadOptions& read_options,
+                           const EnvOptions& soptions,
+                           MergeIteratorBuilder* merge_iter_builder) {
+  // Merge all level zero files together since they may overlap
+  for (const FileMetaData* file : files_[0]) {
+    merge_iter_builder->AddIterator(cfd_->table_cache()->NewIterator(
+        read_options, soptions, cfd_->internal_comparator(), file->fd, nullptr,
+        false, merge_iter_builder->GetArena()));
+  }
+
+  // For levels > 0, we can use a concatenating iterator that sequentially
+  // walks through the non-overlapping files in the level, opening them
+  // lazily.
+  for (int level = 1; level < num_levels_; level++) {
+    if (!files_[level].empty()) {
+      merge_iter_builder->AddIterator(NewTwoLevelIterator(
+          new LevelFileIteratorState(
+              cfd_->table_cache(), read_options, soptions,
+              cfd_->internal_comparator(), false /* for_compaction */,
+              cfd_->options()->prefix_extractor != nullptr),
+          new LevelFileNumIterator(cfd_->internal_comparator(), &files_[level]),
+          merge_iter_builder->GetArena()));
+    }
+  }
+}
+
 // Callback from TableCache::Get()
 namespace {
 enum SaverState {
@@ -435,7 +446,7 @@ static bool SaveValue(void* arg, const ParsedInternalKey& parsed_key,
 
 namespace {
 bool NewestFirst(FileMetaData* a, FileMetaData* b) {
-  return a->number > b->number;
+  return a->fd.GetNumber() > b->fd.GetNumber();
 }
 bool NewestFirstBySeqNo(FileMetaData* a, FileMetaData* b) {
   if (a->smallest_seqno != b->smallest_seqno) {
@@ -454,7 +465,7 @@ bool BySmallestKey(FileMetaData* a, FileMetaData* b,
     return (r < 0);
   }
   // Break ties by file number
-  return (a->number < b->number);
+  return (a->fd.GetNumber() < b->fd.GetNumber());
 }
 }  // anonymous namespace
 
@@ -472,12 +483,13 @@ Version::Version(ColumnFamilyData* cfd, VersionSet* vset,
       info_log_((cfd == nullptr) ? nullptr : cfd->options()->info_log.get()),
       db_statistics_((cfd == nullptr) ? nullptr
                                       : cfd->options()->statistics.get()),
+      // cfd is nullptr if Version is dummy
+      num_levels_(cfd == nullptr ? 0 : cfd->NumberLevels()),
+      num_non_empty_levels_(num_levels_),
       vset_(vset),
       next_(this),
       prev_(this),
       refs_(0),
-      // cfd is nullptr if Version is dummy
-      num_levels_(cfd == nullptr ? 0 : cfd->NumberLevels()),
       files_(new std::vector<FileMetaData*>[num_levels_]),
       files_by_size_(num_levels_),
       next_file_to_compact_by_size_(num_levels_),
@@ -525,7 +537,7 @@ void Version::Get(const ReadOptions& options,
 
   int32_t search_left_bound = 0;
   int32_t search_right_bound = FileIndexer::kLevelMaxIndex;
-  for (int level = 0; level < num_levels_; ++level) {
+  for (int level = 0; level < num_non_empty_levels_; ++level) {
     int num_files = files_[level].size();
     if (num_files == 0) {
       // When current level is empty, the search bound generated from upper
@@ -541,6 +553,16 @@ void Version::Get(const ReadOptions& options,
       continue;
     }
 
+    // Prefetch table data to avoid cache miss if possible
+    if (level == 0) {
+      for (int i = 0; i < num_files; ++i) {
+        auto* r = files_[0][i]->fd.table_reader;
+        if (r) {
+          r->Prepare(ikey);
+        }
+      }
+    }
+
     // Get the list of files to search in this level
     FileMetaData* const* files = &files_[level][0];
 
@@ -581,30 +603,46 @@ void Version::Get(const ReadOptions& options,
 
     for (int32_t i = start_index; i < num_files;) {
       FileMetaData* f = files[i];
-      // Check if key is within a file's range. If search left bound and right
-      // bound point to the same find, we are sure key falls in range.
-      assert(level == 0 || i == start_index ||
-             user_comparator_->Compare(user_key, f->smallest.user_key()) <= 0);
-
-      int cmp_smallest = user_comparator_->Compare(user_key, f->smallest.user_key());
       int cmp_largest = -1;
-      if (cmp_smallest >= 0) {
-        cmp_largest = user_comparator_->Compare(user_key, f->largest.user_key());
-      }
 
-      // Setup file search bound for the next level based on the comparison
-      // results
-      if (level > 0) {
-        file_indexer_.GetNextLevelIndex(level, i, cmp_smallest, cmp_largest,
-            &search_left_bound, &search_right_bound);
-      }
-      // Key falls out of current file's range
-      if (cmp_smallest < 0 || cmp_largest > 0) {
-        if (level == 0) {
-          ++i;
-          continue;
-        } else {
-          break;
+      // Do key range filtering of files or/and fractional cascading if:
+      // (1) not all the files are in level 0, or
+      // (2) there are more than 3 Level 0 files
+      // If there are only 3 or less level 0 files in the system, we skip the
+      // key range filtering. In this case, more likely, the system is highly
+      // tuned to minimize number of tables queried by each query, so it is
+      // unlikely that key range filtering is more efficient than querying the
+      // files.
+      if (num_non_empty_levels_ > 1 || num_files > 3) {
+        // Check if key is within a file's range. If search left bound and right
+        // bound point to the same find, we are sure key falls in range.
+        assert(
+            level == 0 || i == start_index
+                || user_comparator_->Compare(user_key, f->smallest.user_key())
+                    <= 0);
+
+        int cmp_smallest = user_comparator_->Compare(user_key,
+                                                     f->smallest.user_key());
+        if (cmp_smallest >= 0) {
+          cmp_largest = user_comparator_->Compare(user_key,
+                                                  f->largest.user_key());
+        }
+
+        // Setup file search bound for the next level based on the comparison
+        // results
+        if (level > 0) {
+          file_indexer_.GetNextLevelIndex(level, i, cmp_smallest, cmp_largest,
+                                          &search_left_bound,
+                                          &search_right_bound);
+        }
+        // Key falls out of current file's range
+        if (cmp_smallest < 0 || cmp_largest > 0) {
+          if (level == 0) {
+            ++i;
+            continue;
+          } else {
+            break;
+          }
         }
       }
 
@@ -627,7 +665,7 @@ void Version::Get(const ReadOptions& options,
       prev_file = f;
 #endif
       bool tableIO = false;
-      *status = table_cache_->Get(options, *internal_comparator_, *f, ikey,
+      *status = table_cache_->Get(options, *internal_comparator_, f->fd, ikey,
                                   &saver, SaveValue, &tableIO, MarkKeyMayExist);
       // TODO: examine the behavior for corrupted key
       if (!status->ok()) {
@@ -705,13 +743,20 @@ bool Version::UpdateStats(const GetStats& stats) {
   return false;
 }
 
+void Version::PrepareApply(std::vector<uint64_t>& size_being_compacted) {
+  ComputeCompactionScore(size_being_compacted);
+  UpdateFilesBySize();
+  UpdateNumNonEmptyLevels();
+}
+
 void Version::ComputeCompactionScore(
     std::vector<uint64_t>& size_being_compacted) {
   double max_score = 0;
   int max_score_level = 0;
 
   int num_levels_to_check =
-      (cfd_->options()->compaction_style != kCompactionStyleUniversal)
+      (cfd_->options()->compaction_style != kCompactionStyleUniversal &&
+       cfd_->options()->compaction_style != kCompactionStyleFIFO)
           ? NumberLevels() - 1
           : 1;
 
@@ -730,14 +775,18 @@ void Version::ComputeCompactionScore(
       // setting, or very high compression ratios, or lots of
       // overwrites/deletions).
       int numfiles = 0;
+      uint64_t total_size = 0;
       for (unsigned int i = 0; i < files_[level].size(); i++) {
         if (!files_[level][i]->being_compacted) {
+          total_size += files_[level][i]->fd.GetFileSize();
           numfiles++;
         }
       }
-
-      // If we are slowing down writes, then we better compact that first
-      if (numfiles >= cfd_->options()->level0_stop_writes_trigger) {
+      if (cfd_->options()->compaction_style == kCompactionStyleFIFO) {
+        score = static_cast<double>(total_size) /
+                cfd_->options()->compaction_options_fifo.max_table_files_size;
+      } else if (numfiles >= cfd_->options()->level0_stop_writes_trigger) {
+        // If we are slowing down writes, then we better compact that first
         score = 1000000;
       } else if (numfiles >= cfd_->options()->level0_slowdown_writes_trigger) {
         score = 10000;
@@ -786,7 +835,7 @@ namespace {
 // In normal mode: descending size
 bool CompareSizeDescending(const Version::Fsize& first,
                            const Version::Fsize& second) {
-  return (first.file->file_size > second.file->file_size);
+  return (first.file->fd.GetFileSize() > second.file->fd.GetFileSize());
 }
 // A static compator used to sort files based on their seqno
 // In universal style : descending seqno
@@ -802,7 +851,22 @@ bool CompareSeqnoDescending(const Version::Fsize& first,
 
 } // anonymous namespace
 
+void Version::UpdateNumNonEmptyLevels() {
+  num_non_empty_levels_ = num_levels_;
+  for (int i = num_levels_ - 1; i >= 0; i--) {
+    if (files_[i].size() != 0) {
+      return;
+    } else {
+      num_non_empty_levels_ = i;
+    }
+  }
+}
+
 void Version::UpdateFilesBySize() {
+  if (cfd_->options()->compaction_style == kCompactionStyleFIFO) {
+    // don't need this
+    return;
+  }
   // No need to sort the highest level because it is never compacted.
   int max_level =
       (cfd_->options()->compaction_style == kCompactionStyleUniversal)
@@ -871,7 +935,8 @@ bool Version::NeedsCompaction() const {
   // TODO(sdong): improve this function to be accurate for universal
   //              compactions.
   int num_levels_to_check =
-      (cfd_->options()->compaction_style != kCompactionStyleUniversal)
+      (cfd_->options()->compaction_style != kCompactionStyleUniversal &&
+       cfd_->options()->compaction_style != kCompactionStyleFIFO)
           ? NumberLevels() - 1
           : 1;
   for (int i = 0; i < num_levels_to_check; i++) {
@@ -1151,6 +1216,10 @@ const char* Version::LevelSummary(LevelSummaryStorage* scratch) const {
     if (ret < 0 || ret >= sz) break;
     len += ret;
   }
+  if (len > 0) {
+    // overwrite the last space
+    --len;
+  }
   snprintf(scratch->buffer + len, sizeof(scratch->buffer) - len, "]");
   return scratch->buffer;
 }
@@ -1160,16 +1229,20 @@ const char* Version::LevelFileSummary(FileSummaryStorage* scratch,
   int len = snprintf(scratch->buffer, sizeof(scratch->buffer), "files_size[");
   for (const auto& f : files_[level]) {
     int sz = sizeof(scratch->buffer) - len;
+    char sztxt[16];
+    AppendHumanBytes(f->fd.GetFileSize(), sztxt, 16);
     int ret = snprintf(scratch->buffer + len, sz,
-                       "#%lu(seq=%lu,sz=%lu,%lu) ",
-                       (unsigned long)f->number,
-                       (unsigned long)f->smallest_seqno,
-                       (unsigned long)f->file_size,
-                       (unsigned long)f->being_compacted);
+                       "#%" PRIu64 "(seq=%" PRIu64 ",sz=%s,%d) ",
+                       f->fd.GetNumber(), f->smallest_seqno, sztxt,
+                       static_cast<int>(f->being_compacted));
     if (ret < 0 || ret >= sz)
       break;
     len += ret;
   }
+  // overwrite the last space (only if files_[level].size() is non-zero)
+  if (files_[level].size() && len > 0) {
+    --len;
+  }
   snprintf(scratch->buffer + len, sizeof(scratch->buffer) - len, "]");
   return scratch->buffer;
 }
@@ -1193,7 +1266,7 @@ void Version::AddLiveFiles(std::set<uint64_t>* live) {
   for (int level = 0; level < NumberLevels(); level++) {
     const std::vector<FileMetaData*>& files = files_[level];
     for (const auto& file : files) {
-      live->insert(file->number);
+      live->insert(file->fd.GetNumber());
     }
   }
 }
@@ -1213,9 +1286,9 @@ std::string Version::DebugString(bool hex) const {
     const std::vector<FileMetaData*>& files = files_[level];
     for (size_t i = 0; i < files.size(); i++) {
       r.push_back(' ');
-      AppendNumberTo(&r, files[i]->number);
+      AppendNumberTo(&r, files[i]->fd.GetNumber());
       r.push_back(':');
-      AppendNumberTo(&r, files[i]->file_size);
+      AppendNumberTo(&r, files[i]->fd.GetFileSize());
       r.append("[");
       r.append(files[i]->smallest.DebugString(hex));
       r.append(" .. ");
@@ -1245,7 +1318,7 @@ struct VersionSet::ManifestWriter {
 class VersionSet::Builder {
  private:
   // Helper to sort v->files_
-  // kLevel0LevelCompaction -- NewestFirst
+  // kLevel0LevelCompaction -- NewestFirst (also used for FIFO compaction)
   // kLevel0UniversalCompaction -- NewestFirstBySeqNo
   // kLevelNon0 -- BySmallestKey
   struct FileComparator {
@@ -1364,7 +1437,7 @@ class VersionSet::Builder {
         const std::vector<FileMetaData*>& base_files = base_->files_[l];
         for (unsigned int i = 0; i < base_files.size(); i++) {
           FileMetaData* f = base_files[i];
-          if (f->number == number) {
+          if (f->fd.GetNumber() == number) {
             found =  true;
             break;
           }
@@ -1378,7 +1451,7 @@ class VersionSet::Builder {
         for (FileSet::const_iterator added_iter = added->begin();
              added_iter != added->end(); ++added_iter) {
           FileMetaData* f = *added_iter;
-          if (f->number == number) {
+          if (f->fd.GetNumber() == number) {
             found = true;
             break;
           }
@@ -1391,7 +1464,7 @@ class VersionSet::Builder {
         for (FileSet::const_iterator added_iter = added->begin();
              added_iter != added->end(); ++added_iter) {
           FileMetaData* f = *added_iter;
-          if (f->number == number) {
+          if (f->fd.GetNumber() == number) {
             found = true;
             break;
           }
@@ -1433,10 +1506,10 @@ class VersionSet::Builder {
       // same as the compaction of 40KB of data.  We are a little
       // conservative and allow approximately one seek for every 16KB
       // of data before triggering a compaction.
-      f->allowed_seeks = (f->file_size / 16384);
+      f->allowed_seeks = (f->fd.GetFileSize() / 16384);
       if (f->allowed_seeks < 100) f->allowed_seeks = 100;
 
-      levels_[level].deleted_files.erase(f->number);
+      levels_[level].deleted_files.erase(f->fd.GetNumber());
       levels_[level].added_files->insert(f);
     }
   }
@@ -1485,11 +1558,10 @@ class VersionSet::Builder {
         bool table_io;
         cfd_->table_cache()->FindTable(
             base_->vset_->storage_options_, cfd_->internal_comparator(),
-            file_meta->number, file_meta->file_size,
-            &file_meta->table_reader_handle, &table_io, false);
+            file_meta->fd, &file_meta->table_reader_handle, &table_io, false);
         if (file_meta->table_reader_handle != nullptr) {
           // Load table_reader
-          file_meta->table_reader =
+          file_meta->fd.table_reader =
               cfd_->table_cache()->GetTableReaderFromHandle(
                   file_meta->table_reader_handle);
         }
@@ -1498,7 +1570,7 @@ class VersionSet::Builder {
   }
 
   void MaybeAddFile(Version* v, int level, FileMetaData* f) {
-    if (levels_[level].deleted_files.count(f->number) > 0) {
+    if (levels_[level].deleted_files.count(f->fd.GetNumber()) > 0) {
       // File is deleted: do nothing
     } else {
       auto* files = &v->files_[level];
@@ -1680,10 +1752,8 @@ Status VersionSet::LogAndApply(ColumnFamilyData* column_family_data,
     }
 
     if (!edit->IsColumnFamilyManipulation()) {
-      // The calls to ComputeCompactionScore and UpdateFilesBySize are cpu-heavy
-      // and is best called outside the mutex.
-      v->ComputeCompactionScore(size_being_compacted);
-      v->UpdateFilesBySize();
+      // This is cpu-heavy operations, which should be called outside mutex.
+      v->PrepareApply(size_being_compacted);
     }
 
     // Write new record to MANIFEST log
@@ -1731,7 +1801,8 @@ Status VersionSet::LogAndApply(ColumnFamilyData* column_family_data,
     // If we just created a new descriptor file, install it by writing a
     // new CURRENT file that points to it.
     if (s.ok() && new_descriptor_log) {
-      s = SetCurrentFile(env_, dbname_, pending_manifest_file_number_);
+      s = SetCurrentFile(env_, dbname_, pending_manifest_file_number_,
+                         db_directory);
       if (s.ok() && pending_manifest_file_number_ > manifest_file_number_) {
         // delete old manifest file
         Log(options_->info_log,
@@ -1741,9 +1812,6 @@ Status VersionSet::LogAndApply(ColumnFamilyData* column_family_data,
         // of it later
         env_->DeleteFile(DescriptorFileName(dbname_, manifest_file_number_));
       }
-      if (!options_->disableDataSync && db_directory != nullptr) {
-        db_directory->Fsync();
-      }
     }
 
     if (s.ok()) {
@@ -2102,8 +2170,7 @@ Status VersionSet::Recover(
       // Install recovered version
       std::vector<uint64_t> size_being_compacted(v->NumberLevels() - 1);
       cfd->compaction_picker()->SizeBeingCompacted(size_being_compacted);
-      v->ComputeCompactionScore(size_being_compacted);
-      v->UpdateFilesBySize();
+      v->PrepareApply(size_being_compacted);
       AppendVersion(cfd, v);
     }
 
@@ -2436,8 +2503,7 @@ Status VersionSet::DumpManifest(Options& options, std::string& dscname,
       builder->SaveTo(v);
       std::vector<uint64_t> size_being_compacted(v->NumberLevels() - 1);
       cfd->compaction_picker()->SizeBeingCompacted(size_being_compacted);
-      v->ComputeCompactionScore(size_being_compacted);
-      v->UpdateFilesBySize();
+      v->PrepareApply(size_being_compacted);
       delete builder;
 
       printf("--------------- Column family \"%s\"  (ID %u) --------------\n",
@@ -2510,12 +2576,8 @@ Status VersionSet::WriteSnapshot(log::Writer* log) {
 
       for (int level = 0; level < cfd->NumberLevels(); level++) {
         for (const auto& f : cfd->current()->files_[level]) {
-          edit.AddFile(level,
-                       f->number,
-                       f->file_size,
-                       f->smallest,
-                       f->largest,
-                       f->smallest_seqno,
+          edit.AddFile(level, f->fd.GetNumber(), f->fd.GetFileSize(),
+                       f->smallest, f->largest, f->smallest_seqno,
                        f->largest_seqno);
         }
       }
@@ -2571,7 +2633,7 @@ uint64_t VersionSet::ApproximateOffsetOf(Version* v, const InternalKey& ikey) {
       if (v->cfd_->internal_comparator().Compare(files[i]->largest, ikey) <=
           0) {
         // Entire file is before "ikey", so just add the file size
-        result += files[i]->file_size;
+        result += files[i]->fd.GetFileSize();
       } else if (v->cfd_->internal_comparator().Compare(files[i]->smallest,
                                                         ikey) > 0) {
         // Entire file is after "ikey", so ignore
@@ -2587,7 +2649,7 @@ uint64_t VersionSet::ApproximateOffsetOf(Version* v, const InternalKey& ikey) {
         TableReader* table_reader_ptr;
         Iterator* iter = v->cfd_->table_cache()->NewIterator(
             ReadOptions(), storage_options_, v->cfd_->internal_comparator(),
-            *(files[i]), &table_reader_ptr);
+            files[i]->fd, &table_reader_ptr);
         if (table_reader_ptr != nullptr) {
           result += table_reader_ptr->ApproximateOffsetOf(ikey.Encode());
         }
@@ -2620,7 +2682,7 @@ void VersionSet::AddLiveFiles(std::vector<uint64_t>* live_list) {
          v = v->next_) {
       for (int level = 0; level < v->NumberLevels(); level++) {
         for (const auto& f : v->files_[level]) {
-          live_list->push_back(f->number);
+          live_list->push_back(f->fd.GetNumber());
         }
       }
     }
@@ -2646,7 +2708,7 @@ Iterator* VersionSet::MakeInputIterator(Compaction* c) {
         for (const auto& file : *c->inputs(which)) {
           list[num++] = cfd->table_cache()->NewIterator(
               read_options, storage_options_compactions_,
-              cfd->internal_comparator(), *file, nullptr,
+              cfd->internal_comparator(), file->fd, nullptr,
               true /* for compaction */);
         }
       } else {
@@ -2681,13 +2743,13 @@ bool VersionSet::VerifyCompactionFileConsistency(Compaction* c) {
   // verify files in level
   int level = c->level();
   for (int i = 0; i < c->num_input_files(0); i++) {
-    uint64_t number = c->input(0,i)->number;
+    uint64_t number = c->input(0, i)->fd.GetNumber();
 
     // look for this file in the current version
     bool found = false;
     for (unsigned int j = 0; j < version->files_[level].size(); j++) {
       FileMetaData* f = version->files_[level][j];
-      if (f->number == number) {
+      if (f->fd.GetNumber() == number) {
         found = true;
         break;
       }
@@ -2699,13 +2761,13 @@ bool VersionSet::VerifyCompactionFileConsistency(Compaction* c) {
   // verify level+1 files
   level++;
   for (int i = 0; i < c->num_input_files(1); i++) {
-    uint64_t number = c->input(1,i)->number;
+    uint64_t number = c->input(1, i)->fd.GetNumber();
 
     // look for this file in the current version
     bool found = false;
     for (unsigned int j = 0; j < version->files_[level].size(); j++) {
       FileMetaData* f = version->files_[level][j];
-      if (f->number == number) {
+      if (f->fd.GetNumber() == number) {
         found = true;
         break;
       }
@@ -2725,7 +2787,7 @@ Status VersionSet::GetMetadataForFile(uint64_t number, int* filelevel,
     Version* version = cfd_iter->current();
     for (int level = 0; level < version->NumberLevels(); level++) {
       for (const auto& file : version->files_[level]) {
-        if (file->number == number) {
+        if (file->fd.GetNumber() == number) {
           *meta = file;
           *filelevel = level;
           *cfd = cfd_iter;
@@ -2743,9 +2805,9 @@ void VersionSet::GetLiveFilesMetaData(std::vector<LiveFileMetaData>* metadata) {
       for (const auto& file : cfd->current()->files_[level]) {
         LiveFileMetaData filemetadata;
         filemetadata.column_family_name = cfd->GetName();
-        filemetadata.name = TableFileName("", file->number);
+        filemetadata.name = TableFileName("", file->fd.GetNumber());
         filemetadata.level = level;
-        filemetadata.size = file->file_size;
+        filemetadata.size = file->fd.GetFileSize();
         filemetadata.smallestkey = file->smallest.user_key().ToString();
         filemetadata.largestkey = file->largest.user_key().ToString();
         filemetadata.smallest_seqno = file->smallest_seqno;

db/version_set.h

@@ -51,6 +51,7 @@ class MergeContext;
 class ColumnFamilyData;
 class ColumnFamilySet;
 class TableCache;
+class MergeIteratorBuilder;
 
 // Return the smallest index i such that files[i]->largest >= key.
 // Return files.size() if there is no such file.
@@ -80,6 +81,9 @@ class Version {
   void AddIterators(const ReadOptions&, const EnvOptions& soptions,
                     std::vector<Iterator*>* iters);
 
+  void AddIterators(const ReadOptions&, const EnvOptions& soptions,
+                    MergeIteratorBuilder* merger_iter_builder);
+
   // Lookup the value for key.  If found, store it in *val and
   // return OK.  Else return a non-OK status.  Fills *stats.
   // Uses *operands to store merge_operator operations to apply later
@@ -88,6 +92,7 @@ class Version {
     FileMetaData* seek_file;
     int seek_file_level;
   };
+
   void Get(const ReadOptions&, const LookupKey& key, std::string* val,
            Status* status, MergeContext* merge_context, GetStats* stats,
            bool* value_found = nullptr);
@@ -103,6 +108,10 @@ class Version {
   // a lock. Once a version is saved to current_, call only with mutex held
   void ComputeCompactionScore(std::vector<uint64_t>& size_being_compacted);
 
+  // Update scores, pre-calculated variables. It needs to be called before
+  // applying the version to the version set.
+  void PrepareApply(std::vector<uint64_t>& size_being_compacted);
+
   // Reference count management (so Versions do not disappear out from
   // under live iterators)
   void Ref();
@@ -217,6 +226,8 @@ class Version {
   friend class CompactionPicker;
   friend class LevelCompactionPicker;
   friend class UniversalCompactionPicker;
+  friend class FIFOCompactionPicker;
+  friend class ForwardIterator;
 
   class LevelFileNumIterator;
   class LevelFileIteratorState;
@@ -224,6 +235,9 @@ class Version {
   bool PrefixMayMatch(const ReadOptions& options, Iterator* level_iter,
                       const Slice& internal_prefix) const;
 
+  // Update num_non_empty_levels_.
+  void UpdateNumNonEmptyLevels();
+
   // Sort all files for this version based on their file size and
   // record results in files_by_size_. The largest files are listed first.
   void UpdateFilesBySize();
@@ -235,11 +249,13 @@ class Version {
   const MergeOperator* merge_operator_;
   Logger* info_log_;
   Statistics* db_statistics_;
+  int num_levels_;              // Number of levels
+  int num_non_empty_levels_;    // Number of levels. Any level larger than it
+                                // is guaranteed to be empty.
   VersionSet* vset_;            // VersionSet to which this Version belongs
   Version* next_;               // Next version in linked list
   Version* prev_;               // Previous version in linked list
   int refs_;                    // Number of live refs to this version
-  int num_levels_;              // Number of levels
 
   // List of files per level, files in each level are arranged
   // in increasing order of keys

db/version_set_test.cc

@@ -31,7 +31,7 @@ class FindFileTest {
            SequenceNumber smallest_seq = 100,
            SequenceNumber largest_seq = 100) {
     FileMetaData* f = new FileMetaData;
-    f->number = files_.size() + 1;
+    f->fd = FileDescriptor(files_.size() + 1, 0);
     f->smallest = InternalKey(smallest, smallest_seq, kTypeValue);
     f->largest = InternalKey(largest, largest_seq, kTypeValue);
     files_.push_back(f);

examples/.gitignore

@@ -0,0 +1,2 @@
+column_families_example
+simple_example

examples/Makefile

@@ -0,0 +1,9 @@
+include ../build_config.mk
+
+all: simple_example column_families_example
+
+simple_example: simple_example.cc
+	$(CXX) $(CXXFLAGS) $@.cc -o$@ ../librocksdb.a -I../include -O2 -std=c++11 $(PLATFORM_LDFLAGS) $(PLATFORM_CXXFLAGS) $(EXEC_LDFLAGS)
+
+column_families_example: column_families_example.cc
+	$(CXX) $(CXXFLAGS) $@.cc -o$@ ../librocksdb.a -I../include -O2 -std=c++11 $(PLATFORM_LDFLAGS) $(PLATFORM_CXXFLAGS) $(EXEC_LDFLAGS)

examples/README.md

@@ -0,0 +1 @@
+Compile RocksDB first by executing `make static_lib` in parent dir

examples/column_families_example.cc

@@ -0,0 +1,72 @@
+// Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree. An additional grant
+// of patent rights can be found in the PATENTS file in the same directory.
+#include <cstdio>
+#include <string>
+#include <vector>
+
+#include "rocksdb/db.h"
+#include "rocksdb/slice.h"
+#include "rocksdb/options.h"
+
+using namespace rocksdb;
+
+std::string kDBPath = "/tmp/rocksdb_column_families_example";
+
+int main() {
+  // open DB
+  Options options;
+  options.create_if_missing = true;
+  DB* db;
+  Status s = DB::Open(options, kDBPath, &db);
+  assert(s.ok());
+
+  // create column family
+  ColumnFamilyHandle* cf;
+  s = db->CreateColumnFamily(ColumnFamilyOptions(), "new_cf", &cf);
+  assert(s.ok());
+
+  // close DB
+  delete cf;
+  delete db;
+
+  // open DB with two column families
+  std::vector<ColumnFamilyDescriptor> column_families;
+  // have to open default column familiy
+  column_families.push_back(ColumnFamilyDescriptor(
+      kDefaultColumnFamilyName, ColumnFamilyOptions()));
+  // open the new one, too
+  column_families.push_back(ColumnFamilyDescriptor(
+      "new_cf", ColumnFamilyOptions()));
+  std::vector<ColumnFamilyHandle*> handles;
+  s = DB::Open(DBOptions(), kDBPath, column_families, &handles, &db);
+  assert(s.ok());
+
+  // put and get from non-default column family
+  s = db->Put(WriteOptions(), handles[1], Slice("key"), Slice("value"));
+  assert(s.ok());
+  std::string value;
+  s = db->Get(ReadOptions(), handles[1], Slice("key"), &value);
+  assert(s.ok());
+
+  // atomic write
+  WriteBatch batch;
+  batch.Put(handles[0], Slice("key2"), Slice("value2"));
+  batch.Put(handles[1], Slice("key3"), Slice("value3"));
+  batch.Delete(handles[0], Slice("key"));
+  s = db->Write(WriteOptions(), &batch);
+  assert(s.ok());
+
+  // drop column family
+  s = db->DropColumnFamily(handles[1]);
+  assert(s.ok());
+
+  // close db
+  for (auto handle : handles) {
+    delete handle;
+  }
+  delete db;
+
+  return 0;
+}

examples/simple_example.cc

@@ -0,0 +1,41 @@
+// Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree. An additional grant
+// of patent rights can be found in the PATENTS file in the same directory.
+#include <cstdio>
+#include <string>
+
+#include "rocksdb/db.h"
+#include "rocksdb/slice.h"
+#include "rocksdb/options.h"
+
+using namespace rocksdb;
+
+std::string kDBPath = "/tmp/rocksdb_simple_example";
+
+int main() {
+  DB* db;
+  Options options;
+  // Optimize RocksDB. This is the easiest way to get RocksDB to perform well
+  options.IncreaseParallelism();
+  options.OptimizeLevelStyleCompaction();
+  // create the DB if it's not already present
+  options.create_if_missing = true;
+
+  // open DB
+  Status s = DB::Open(options, kDBPath, &db);
+  assert(s.ok());
+
+  // Put key-value
+  s = db->Put(WriteOptions(), "key", "value");
+  assert(s.ok());
+  std::string value;
+  // get value
+  s = db->Get(ReadOptions(), "key", &value);
+  assert(s.ok());
+  assert(value == "value");
+
+  delete db;
+
+  return 0;
+}

hdfs/README

@@ -1,19 +1,16 @@
 This directory contains the hdfs extensions needed to make rocksdb store
 files in HDFS.
 
-The hdfs.h file is copied from the Apache Hadoop 1.0 source code. 
-It defines the libhdfs library
-(http://hadoop.apache.org/common/docs/r0.20.2/libhdfs.html) to access 
-data in HDFS.  The libhdfs.a is copied from the Apache Hadoop 1.0 build. 
-It implements the API defined in hdfs.h. If your hadoop cluster is running
-a different hadoop release, then install these two files manually from your
-hadoop distribution and then recompile rocksdb.
+It has been compiled and testing against CDH 4.4 (2.0.0+1475-1.cdh4.4.0.p0.23~precise-cdh4.4.0).
+
+The configuration assumes that packages libhdfs0, libhdfs0-dev are 
+installed which basically means that hdfs.h is in /usr/include and libhdfs in /usr/lib
 
 The env_hdfs.h file defines the rocksdb objects that are needed to talk to an
 underlying filesystem. 
 
 If you want to compile rocksdb with hdfs support, please set the following
-enviroment variables appropriately:
+enviroment variables appropriately (also defined in setup.sh for convenience)
    USE_HDFS=1
    JAVA_HOME=/usr/local/jdk-6u22-64
    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/jdk-6u22-64/jre/lib/amd64/server:/usr/local/jdk-6u22-64/jre/lib/amd64/:./snappy/libs

hdfs/env_hdfs.h

@@ -14,13 +14,10 @@
 #include "rocksdb/status.h"
 
 #ifdef USE_HDFS
-#include "hdfs/hdfs.h"
+#include <hdfs.h>
 
 namespace rocksdb {
 
-static const std::string kProto = "hdfs://";
-static const std::string pathsep = "/";
-
 // Thrown during execution when there is an issue with the supplied
 // arguments.
 class HdfsUsageException : public std::exception { };
@@ -58,20 +55,23 @@ class HdfsEnv : public Env {
   }
 
   virtual Status NewSequentialFile(const std::string& fname,
-                                   SequentialFile** result);
+                                   std::unique_ptr<SequentialFile>* result,
+                                   const EnvOptions& options);
 
   virtual Status NewRandomAccessFile(const std::string& fname,
-                                     RandomAccessFile** result);
+                                     std::unique_ptr<RandomAccessFile>* result,
+                                     const EnvOptions& options);
 
   virtual Status NewWritableFile(const std::string& fname,
-                                 WritableFile** result);
+                                 std::unique_ptr<WritableFile>* result,
+                                 const EnvOptions& options);
 
   virtual Status NewRandomRWFile(const std::string& fname,
-                                 unique_ptr<RandomRWFile>* result,
+                                 std::unique_ptr<RandomRWFile>* result,
                                  const EnvOptions& options);
 
   virtual Status NewDirectory(const std::string& name,
-                              unique_ptr<Directory>* result);
+                              std::unique_ptr<Directory>* result);
 
   virtual bool FileExists(const std::string& fname);
 
@@ -97,7 +97,8 @@ class HdfsEnv : public Env {
 
   virtual Status UnlockFile(FileLock* lock);
 
-  virtual Status NewLogger(const std::string& fname, Logger** result);
+  virtual Status NewLogger(const std::string& fname,
+                           std::shared_ptr<Logger>* result);
 
   virtual void Schedule(void (*function)(void* arg), void* arg,
                         Priority pri = LOW) {
@@ -161,6 +162,9 @@ class HdfsEnv : public Env {
                         // object here so that we can use posix timers,
                         // posix threads, etc.
 
+  static const std::string kProto;
+  static const std::string pathsep;
+
   /**
    * If the URI is specified of the form hdfs://server:port/path,
    * then connect to the specified cluster

hdfs/hdfs.h

@@ -1,477 +0,0 @@
-//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under the BSD-style license found in the
-//  LICENSE file in the root directory of this source tree. An additional grant
-//  of patent rights can be found in the PATENTS file in the same directory.
-//
-#ifndef LIBHDFS_HDFS_H
-#define LIBHDFS_HDFS_H
-
-#include <sys/types.h>
-#include <sys/stat.h>
-
-#include <fcntl.h>
-#include <stdio.h>
-#include <stdint.h>
-#include <string.h>
-#include <stdlib.h>
-#include <time.h>
-#include <errno.h>
-
-#include <jni.h>
-
-#ifndef O_RDONLY
-#define O_RDONLY 1
-#endif
-
-#ifndef O_WRONLY 
-#define O_WRONLY 2
-#endif
-
-#ifndef EINTERNAL
-#define EINTERNAL 255 
-#endif
-
-
-/** All APIs set errno to meaningful values */
-#ifdef __cplusplus
-extern  "C" {
-#endif
-
-    /**
-     * Some utility decls used in libhdfs.
-     */
-
-    typedef int32_t   tSize; /// size of data for read/write io ops 
-    typedef time_t    tTime; /// time type in seconds
-    typedef int64_t   tOffset;/// offset within the file
-    typedef uint16_t  tPort; /// port
-    typedef enum tObjectKind {
-        kObjectKindFile = 'F',
-        kObjectKindDirectory = 'D',
-    } tObjectKind;
-
-
-    /**
-     * The C reflection of org.apache.org.hadoop.FileSystem .
-     */
-    typedef void* hdfsFS;
-
-    
-    /**
-     * The C equivalent of org.apache.org.hadoop.FSData(Input|Output)Stream .
-     */
-    enum hdfsStreamType
-    {
-        UNINITIALIZED = 0,
-        INPUT = 1,
-        OUTPUT = 2,
-    };
-
-    
-    /**
-     * The 'file-handle' to a file in hdfs.
-     */
-    struct hdfsFile_internal {
-        void* file;
-        enum hdfsStreamType type;
-    };
-    typedef struct hdfsFile_internal* hdfsFile;
-      
-
-    /** 
-     * hdfsConnectAsUser - Connect to a hdfs file system as a specific user
-     * Connect to the hdfs.
-     * @param host A string containing either a host name, or an ip address
-     * of the namenode of a hdfs cluster. 'host' should be passed as NULL if
-     * you want to connect to local filesystem. 'host' should be passed as
-     * 'default' (and port as 0) to used the 'configured' filesystem
-     * (core-site/core-default.xml).
-     * @param port The port on which the server is listening.
-     * @param user the user name (this is hadoop domain user). Or NULL is equivelant to hhdfsConnect(host, port)
-     * @param groups the groups (these are hadoop domain groups)
-     * @return Returns a handle to the filesystem or NULL on error.
-     */
-     hdfsFS hdfsConnectAsUser(const char* host, tPort port, const char *user , const char *groups[], int groups_size );
-
-
-    /** 
-     * hdfsConnect - Connect to a hdfs file system.
-     * Connect to the hdfs.
-     * @param host A string containing either a host name, or an ip address
-     * of the namenode of a hdfs cluster. 'host' should be passed as NULL if
-     * you want to connect to local filesystem. 'host' should be passed as
-     * 'default' (and port as 0) to used the 'configured' filesystem
-     * (core-site/core-default.xml).
-     * @param port The port on which the server is listening.
-     * @return Returns a handle to the filesystem or NULL on error.
-     */
-     hdfsFS hdfsConnect(const char* host, tPort port);
-
-
-    /**
-     * This are the same as hdfsConnectAsUser except that every invocation returns a new FileSystem handle.
-     * Applications should call a hdfsDisconnect for every call to hdfsConnectAsUserNewInstance.
-     */
-     hdfsFS hdfsConnectAsUserNewInstance(const char* host, tPort port, const char *user , const char *groups[], int groups_size );
-     hdfsFS hdfsConnectNewInstance(const char* host, tPort port);
-     hdfsFS hdfsConnectPath(const char* uri);
-
-    /** 
-     * hdfsDisconnect - Disconnect from the hdfs file system.
-     * Disconnect from hdfs.
-     * @param fs The configured filesystem handle.
-     * @return Returns 0 on success, -1 on error.  
-     */
-    int hdfsDisconnect(hdfsFS fs);
-        
-
-    /** 
-     * hdfsOpenFile - Open a hdfs file in given mode.
-     * @param fs The configured filesystem handle.
-     * @param path The full path to the file.
-     * @param flags - an | of bits/fcntl.h file flags - supported flags are O_RDONLY, O_WRONLY (meaning create or overwrite i.e., implies O_TRUNCAT), 
-     * O_WRONLY|O_APPEND. Other flags are generally ignored other than (O_RDWR || (O_EXCL & O_CREAT)) which return NULL and set errno equal ENOTSUP.
-     * @param bufferSize Size of buffer for read/write - pass 0 if you want
-     * to use the default configured values.
-     * @param replication Block replication - pass 0 if you want to use
-     * the default configured values.
-     * @param blocksize Size of block - pass 0 if you want to use the
-     * default configured values.
-     * @return Returns the handle to the open file or NULL on error.
-     */
-    hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags,
-                          int bufferSize, short replication, tSize blocksize);
-
-
-    /** 
-     * hdfsCloseFile - Close an open file. 
-     * @param fs The configured filesystem handle.
-     * @param file The file handle.
-     * @return Returns 0 on success, -1 on error.  
-     */
-    int hdfsCloseFile(hdfsFS fs, hdfsFile file);
-
-
-    /** 
-     * hdfsExists - Checks if a given path exsits on the filesystem 
-     * @param fs The configured filesystem handle.
-     * @param path The path to look for
-     * @return Returns 0 on exists, 1 on non-exists, -1/-2 on error.  
-     */
-    int hdfsExists(hdfsFS fs, const char *path);
-
-
-    /** 
-     * hdfsSeek - Seek to given offset in file. 
-     * This works only for files opened in read-only mode. 
-     * @param fs The configured filesystem handle.
-     * @param file The file handle.
-     * @param desiredPos Offset into the file to seek into.
-     * @return Returns 0 on success, -1 on error.  
-     */
-    int hdfsSeek(hdfsFS fs, hdfsFile file, tOffset desiredPos); 
-
-
-    /** 
-     * hdfsTell - Get the current offset in the file, in bytes.
-     * @param fs The configured filesystem handle.
-     * @param file The file handle.
-     * @return Current offset, -1 on error.
-     */
-    tOffset hdfsTell(hdfsFS fs, hdfsFile file);
-
-
-    /** 
-     * hdfsRead - Read data from an open file.
-     * @param fs The configured filesystem handle.
-     * @param file The file handle.
-     * @param buffer The buffer to copy read bytes into.
-     * @param length The length of the buffer.
-     * @return Returns the number of bytes actually read, possibly less
-     * than than length;-1 on error.
-     */
-    tSize hdfsRead(hdfsFS fs, hdfsFile file, void* buffer, tSize length);
-
-
-    /** 
-     * hdfsPread - Positional read of data from an open file.
-     * @param fs The configured filesystem handle.
-     * @param file The file handle.
-     * @param position Position from which to read
-     * @param buffer The buffer to copy read bytes into.
-     * @param length The length of the buffer.
-     * @return Returns the number of bytes actually read, possibly less than
-     * than length;-1 on error.
-     */
-    tSize hdfsPread(hdfsFS fs, hdfsFile file, tOffset position,
-                    void* buffer, tSize length);
-
-
-    /** 
-     * hdfsWrite - Write data into an open file.
-     * @param fs The configured filesystem handle.
-     * @param file The file handle.
-     * @param buffer The data.
-     * @param length The no. of bytes to write. 
-     * @return Returns the number of bytes written, -1 on error.
-     */
-    tSize hdfsWrite(hdfsFS fs, hdfsFile file, const void* buffer,
-                    tSize length);
-
-
-    /** 
-     * hdfsWrite - Flush the data. 
-     * @param fs The configured filesystem handle.
-     * @param file The file handle.
-     * @return Returns 0 on success, -1 on error. 
-     */
-    int hdfsFlush(hdfsFS fs, hdfsFile file);
-
-    /**
-     * hdfsSync - Sync the data to persistent store.
-     * @param fs The configured filesystem handle.
-     * @param file The file handle.
-     * @return Returns 0 on success, -1 on error.
-     */
-    int hdfsSync(hdfsFS fs, hdfsFile file);
-
-    /**
-     * hdfsGetNumReplicasInPipeline - get number of remaining replicas in 
-     * pipeline
-     * @param fs The configured filesystem handle
-     * @param file the file handle
-     * @return returns the # of datanodes in the write pipeline; -1 on error
-     */
-   int hdfsGetNumCurrentReplicas(hdfsFS, hdfsFile file);
-
-    /**
-     * hdfsAvailable - Number of bytes that can be read from this
-     * input stream without blocking.
-     * @param fs The configured filesystem handle.
-     * @param file The file handle.
-     * @return Returns available bytes; -1 on error. 
-     */
-    int hdfsAvailable(hdfsFS fs, hdfsFile file);
-
-
-    /**
-     * hdfsCopy - Copy file from one filesystem to another.
-     * @param srcFS The handle to source filesystem.
-     * @param src The path of source file. 
-     * @param dstFS The handle to destination filesystem.
-     * @param dst The path of destination file. 
-     * @return Returns 0 on success, -1 on error. 
-     */
-    int hdfsCopy(hdfsFS srcFS, const char* src, hdfsFS dstFS, const char* dst);
-
-
-    /**
-     * hdfsMove - Move file from one filesystem to another.
-     * @param srcFS The handle to source filesystem.
-     * @param src The path of source file. 
-     * @param dstFS The handle to destination filesystem.
-     * @param dst The path of destination file. 
-     * @return Returns 0 on success, -1 on error. 
-     */
-    int hdfsMove(hdfsFS srcFS, const char* src, hdfsFS dstFS, const char* dst);
-
-
-    /**
-     * hdfsDelete - Delete file. 
-     * @param fs The configured filesystem handle.
-     * @param path The path of the file. 
-     * @return Returns 0 on success, -1 on error. 
-     */
-    int hdfsDelete(hdfsFS fs, const char* path);
-
-
-    /**
-     * hdfsRename - Rename file. 
-     * @param fs The configured filesystem handle.
-     * @param oldPath The path of the source file. 
-     * @param newPath The path of the destination file. 
-     * @return Returns 0 on success, -1 on error. 
-     */
-    int hdfsRename(hdfsFS fs, const char* oldPath, const char* newPath);
-
-
-    /** 
-     * hdfsGetWorkingDirectory - Get the current working directory for
-     * the given filesystem.
-     * @param fs The configured filesystem handle.
-     * @param buffer The user-buffer to copy path of cwd into. 
-     * @param bufferSize The length of user-buffer.
-     * @return Returns buffer, NULL on error.
-     */
-    char* hdfsGetWorkingDirectory(hdfsFS fs, char *buffer, size_t bufferSize);
-
-
-    /** 
-     * hdfsSetWorkingDirectory - Set the working directory. All relative
-     * paths will be resolved relative to it.
-     * @param fs The configured filesystem handle.
-     * @param path The path of the new 'cwd'. 
-     * @return Returns 0 on success, -1 on error. 
-     */
-    int hdfsSetWorkingDirectory(hdfsFS fs, const char* path);
-
-
-    /** 
-     * hdfsCreateDirectory - Make the given file and all non-existent
-     * parents into directories.
-     * @param fs The configured filesystem handle.
-     * @param path The path of the directory. 
-     * @return Returns 0 on success, -1 on error. 
-     */
-    int hdfsCreateDirectory(hdfsFS fs, const char* path);
-
-
-    /** 
-     * hdfsSetReplication - Set the replication of the specified
-     * file to the supplied value
-     * @param fs The configured filesystem handle.
-     * @param path The path of the file. 
-     * @return Returns 0 on success, -1 on error. 
-     */
-    int hdfsSetReplication(hdfsFS fs, const char* path, int16_t replication);
-
-
-    /** 
-     * hdfsFileInfo - Information about a file/directory.
-     */
-    typedef struct  {
-        tObjectKind mKind;   /* file or directory */
-        char *mName;         /* the name of the file */
-        tTime mLastMod;      /* the last modification time for the file in seconds */
-        tOffset mSize;       /* the size of the file in bytes */
-        short mReplication;    /* the count of replicas */
-        tOffset mBlockSize;  /* the block size for the file */
-        char *mOwner;        /* the owner of the file */
-        char *mGroup;        /* the group associated with the file */
-        short mPermissions;  /* the permissions associated with the file */
-        tTime mLastAccess;    /* the last access time for the file in seconds */
-    } hdfsFileInfo;
-
-
-    /** 
-     * hdfsListDirectory - Get list of files/directories for a given
-     * directory-path. hdfsFreeFileInfo should be called to deallocate memory if
-     * the function returns non-NULL value.
-     * @param fs The configured filesystem handle.
-     * @param path The path of the directory. 
-     * @param numEntries Set to the number of files/directories in path.
-     * @return Returns a dynamically-allocated array of hdfsFileInfo
-     * objects; NULL if empty or on error.
-     * on error, numEntries will be -1.
-     */
-    hdfsFileInfo *hdfsListDirectory(hdfsFS fs, const char* path,
-                                    int *numEntries);
-
-
-    /** 
-     * hdfsGetPathInfo - Get information about a path as a (dynamically
-     * allocated) single hdfsFileInfo struct. hdfsFreeFileInfo should be
-     * called when the pointer is no longer needed.
-     * @param fs The configured filesystem handle.
-     * @param path The path of the file. 
-     * @return Returns a dynamically-allocated hdfsFileInfo object;
-     * NULL on error.
-     */
-    hdfsFileInfo *hdfsGetPathInfo(hdfsFS fs, const char* path);
-
-
-    /** 
-     * hdfsFreeFileInfo - Free up the hdfsFileInfo array (including fields) 
-     * @param hdfsFileInfo The array of dynamically-allocated hdfsFileInfo
-     * objects.
-     * @param numEntries The size of the array.
-     */
-    void hdfsFreeFileInfo(hdfsFileInfo *hdfsFileInfo, int numEntries);
-
-
-    /** 
-     * hdfsGetHosts - Get hostnames where a particular block (determined by
-     * pos & blocksize) of a file is stored. The last element in the array
-     * is NULL. Due to replication, a single block could be present on
-     * multiple hosts.
-     * @param fs The configured filesystem handle.
-     * @param path The path of the file. 
-     * @param start The start of the block.
-     * @param length The length of the block.
-     * @return Returns a dynamically-allocated 2-d array of blocks-hosts;
-     * NULL on error.
-     */
-    char*** hdfsGetHosts(hdfsFS fs, const char* path, 
-            tOffset start, tOffset length);
-
-
-    /** 
-     * hdfsFreeHosts - Free up the structure returned by hdfsGetHosts
-     * @param hdfsFileInfo The array of dynamically-allocated hdfsFileInfo
-     * objects.
-     * @param numEntries The size of the array.
-     */
-    void hdfsFreeHosts(char ***blockHosts);
-
-
-    /** 
-     * hdfsGetDefaultBlockSize - Get the optimum blocksize.
-     * @param fs The configured filesystem handle.
-     * @return Returns the blocksize; -1 on error. 
-     */
-    tOffset hdfsGetDefaultBlockSize(hdfsFS fs);
-
-
-    /** 
-     * hdfsGetCapacity - Return the raw capacity of the filesystem.  
-     * @param fs The configured filesystem handle.
-     * @return Returns the raw-capacity; -1 on error. 
-     */
-    tOffset hdfsGetCapacity(hdfsFS fs);
-
-
-    /** 
-     * hdfsGetUsed - Return the total raw size of all files in the filesystem.
-     * @param fs The configured filesystem handle.
-     * @return Returns the total-size; -1 on error. 
-     */
-    tOffset hdfsGetUsed(hdfsFS fs);
-
-    /** 
-     * hdfsChown 
-     * @param fs The configured filesystem handle.
-     * @param path the path to the file or directory
-     * @param owner this is a string in Hadoop land. Set to null or "" if only setting group
-     * @param group  this is a string in Hadoop land. Set to null or "" if only setting user
-     * @return 0 on success else -1
-     */
-    int hdfsChown(hdfsFS fs, const char* path, const char *owner, const char *group);
-
-    /** 
-     * hdfsChmod
-     * @param fs The configured filesystem handle.
-     * @param path the path to the file or directory
-     * @param mode the bitmask to set it to
-     * @return 0 on success else -1
-     */
-      int hdfsChmod(hdfsFS fs, const char* path, short mode);
-
-    /** 
-     * hdfsUtime
-     * @param fs The configured filesystem handle.
-     * @param path the path to the file or directory
-     * @param mtime new modification time or 0 for only set access time in seconds
-     * @param atime new access time or 0 for only set modification time in seconds
-     * @return 0 on success else -1
-     */
-    int hdfsUtime(hdfsFS fs, const char* path, tTime mtime, tTime atime);
-    
-#ifdef __cplusplus
-}
-#endif
-
-#endif /*LIBHDFS_HDFS_H*/
-
-/**
- * vim: ts=4: sw=4: et
- */

hdfs/setup.sh

@@ -0,0 +1,7 @@
+export USE_HDFS=1
+export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:$JAVA_HOME/jre/lib/amd64:/usr/lib/hadoop/lib/native
+
+export CLASSPATH=
+for f in `find /usr/lib/hadoop-hdfs | grep jar`; do export CLASSPATH=$CLASSPATH:$f; done
+for f in `find /usr/lib/hadoop | grep jar`; do export CLASSPATH=$CLASSPATH:$f; done
+for f in `find /usr/lib/hadoop/client | grep jar`; do export CLASSPATH=$CLASSPATH:$f; done

include/rocksdb/c.h

@@ -56,6 +56,7 @@ extern "C" {
 
 typedef struct rocksdb_t                 rocksdb_t;
 typedef struct rocksdb_cache_t           rocksdb_cache_t;
+typedef struct rocksdb_compactionfilter_t rocksdb_compactionfilter_t;
 typedef struct rocksdb_comparator_t      rocksdb_comparator_t;
 typedef struct rocksdb_env_t             rocksdb_env_t;
 typedef struct rocksdb_filelock_t        rocksdb_filelock_t;
@@ -229,6 +230,9 @@ extern const char* rocksdb_writebatch_data(rocksdb_writebatch_t*, size_t *size);
 
 extern rocksdb_options_t* rocksdb_options_create();
 extern void rocksdb_options_destroy(rocksdb_options_t*);
+extern void rocksdb_options_set_compaction_filter(
+    rocksdb_options_t*,
+    rocksdb_compactionfilter_t*);
 extern void rocksdb_options_set_comparator(
     rocksdb_options_t*,
     rocksdb_comparator_t*);
@@ -401,6 +405,22 @@ enum {
 };
 extern void rocksdb_options_set_compaction_style(rocksdb_options_t*, int);
 extern void rocksdb_options_set_universal_compaction_options(rocksdb_options_t*, rocksdb_universal_compaction_options_t*);
+
+/* Compaction Filter */
+
+extern rocksdb_compactionfilter_t* rocksdb_compactionfilter_create(
+    void* state,
+    void (*destructor)(void*),
+    unsigned char (*filter)(
+        void*,
+        int level,
+        const char* key, size_t key_length,
+        const char* existing_value, size_t value_length,
+        char** new_value, size_t *new_value_length,
+        unsigned char* value_changed),
+    const char* (*name)(void*));
+extern void rocksdb_compactionfilter_destroy(rocksdb_compactionfilter_t*);
+
 /* Comparator */
 
 extern rocksdb_comparator_t* rocksdb_comparator_create(

include/rocksdb/db.h

@@ -396,7 +396,7 @@ class DB {
   // times have the same effect as calling it once.
   virtual Status DisableFileDeletions() = 0;
 
-  // Allow compactions to delete obselete files.
+  // Allow compactions to delete obsolete files.
   // If force == true, the call to EnableFileDeletions() will guarantee that
   // file deletions are enabled after the call, even if DisableFileDeletions()
   // was called multiple times before.

include/rocksdb/memtablerep.h

@@ -44,6 +44,7 @@ class Arena;
 class LookupKey;
 class Slice;
 class SliceTransform;
+class Logger;
 
 typedef void* KeyHandle;
 
@@ -141,16 +142,28 @@ class MemTableRep {
   };
 
   // Return an iterator over the keys in this representation.
-  virtual Iterator* GetIterator() = 0;
+  // arena: If not null, the arena needs to be used to allocate the Iterator.
+  //        When destroying the iterator, the caller will not call "delete"
+  //        but Iterator::~Iterator() directly. The destructor needs to destroy
+  //        all the states but those allocated in arena.
+  virtual Iterator* GetIterator(Arena* arena = nullptr) = 0;
 
   // Return an iterator over at least the keys with the specified user key. The
   // iterator may also allow access to other keys, but doesn't have to. Default:
   // GetIterator().
-  virtual Iterator* GetIterator(const Slice& user_key) { return GetIterator(); }
+  virtual Iterator* GetIterator(const Slice& user_key) {
+    return GetIterator(nullptr);
+  }
 
   // Return an iterator that has a special Seek semantics. The result of
   // a Seek might only include keys with the same prefix as the target key.
-  virtual Iterator* GetDynamicPrefixIterator() { return GetIterator(); }
+  // arena: If not null, the arena needs to be used to allocate the Iterator.
+  //        When destroying the iterator, the caller will not call "delete"
+  //        but Iterator::~Iterator() directly. The destructor needs to destroy
+  //        all the states but those allocated in arena.
+  virtual Iterator* GetDynamicPrefixIterator(Arena* arena = nullptr) {
+    return GetIterator(arena);
+  }
 
   // Return true if the current MemTableRep supports merge operator.
   // Default: true
@@ -174,7 +187,8 @@ class MemTableRepFactory {
  public:
   virtual ~MemTableRepFactory() {}
   virtual MemTableRep* CreateMemTableRep(const MemTableRep::KeyComparator&,
-      Arena*, const SliceTransform*) = 0;
+                                         Arena*, const SliceTransform*,
+                                         Logger* logger) = 0;
   virtual const char* Name() const = 0;
 };
 
@@ -182,8 +196,8 @@ class MemTableRepFactory {
 class SkipListFactory : public MemTableRepFactory {
  public:
   virtual MemTableRep* CreateMemTableRep(const MemTableRep::KeyComparator&,
-                                         Arena*,
-                                         const SliceTransform*) override;
+                                         Arena*, const SliceTransform*,
+                                         Logger* logger) override;
   virtual const char* Name() const override { return "SkipListFactory"; }
 };
 
@@ -201,9 +215,9 @@ class VectorRepFactory : public MemTableRepFactory {
 
  public:
   explicit VectorRepFactory(size_t count = 0) : count_(count) { }
-  virtual MemTableRep* CreateMemTableRep(
-      const MemTableRep::KeyComparator&, Arena*,
-      const SliceTransform*) override;
+  virtual MemTableRep* CreateMemTableRep(const MemTableRep::KeyComparator&,
+                                         Arena*, const SliceTransform*,
+                                         Logger* logger) override;
   virtual const char* Name() const override {
     return "VectorRepFactory";
   }
@@ -229,8 +243,14 @@ extern MemTableRepFactory* NewHashSkipListRepFactory(
 //                      huge pages for it to be allocated, like:
 //                          sysctl -w vm.nr_hugepages=20
 //                      See linux doc Documentation/vm/hugetlbpage.txt
+// @bucket_entries_logging_threshold: if number of entries in one bucket
+//                                    exceeds this number, log about it.
+// @if_log_bucket_dist_when_flash: if true, log distribution of number of
+//                                 entries when flushing.
 extern MemTableRepFactory* NewHashLinkListRepFactory(
-    size_t bucket_count = 50000, size_t huge_page_tlb_size = 0);
+    size_t bucket_count = 50000, size_t huge_page_tlb_size = 0,
+    int bucket_entries_logging_threshold = 4096,
+    bool if_log_bucket_dist_when_flash = true);
 
 // This factory creates a cuckoo-hashing based mem-table representation.
 // Cuckoo-hash is a closed-hash strategy, in which all key/value pairs

include/rocksdb/options.h

@@ -33,7 +33,7 @@ class MergeOperator;
 class Snapshot;
 class TableFactory;
 class MemTableRepFactory;
-class TablePropertiesCollector;
+class TablePropertiesCollectorFactory;
 class Slice;
 class SliceTransform;
 class Statistics;
@@ -53,8 +53,18 @@ enum CompressionType : char {
 };
 
 enum CompactionStyle : char {
-  kCompactionStyleLevel = 0x0,     // level based compaction style
-  kCompactionStyleUniversal = 0x1  // Universal compaction style
+  kCompactionStyleLevel = 0x0,      // level based compaction style
+  kCompactionStyleUniversal = 0x1,  // Universal compaction style
+  kCompactionStyleFIFO = 0x2,       // FIFO compaction style
+};
+
+struct CompactionOptionsFIFO {
+  // once the total sum of table files reaches this, we will delete the oldest
+  // table file
+  // Default: 1GB
+  uint64_t max_table_files_size;
+
+  CompactionOptionsFIFO() : max_table_files_size(1 * 1024 * 1024 * 1024) {}
 };
 
 // Compression options for different compression algorithms like Zlib
@@ -76,6 +86,31 @@ enum UpdateStatus {    // Return status For inplace update callback
 struct Options;
 
 struct ColumnFamilyOptions {
+  // Some functions that make it easier to optimize RocksDB
+
+  // Use this if you don't need to keep the data sorted, i.e. you'll never use
+  // an iterator, only Put() and Get() API calls
+  ColumnFamilyOptions* OptimizeForPointLookup();
+
+  // Default values for some parameters in ColumnFamilyOptions are not
+  // optimized for heavy workloads and big datasets, which means you might
+  // observe write stalls under some conditions. As a starting point for tuning
+  // RocksDB options, use the following two functions:
+  // * OptimizeLevelStyleCompaction -- optimizes level style compaction
+  // * OptimizeUniversalStyleCompaction -- optimizes universal style compaction
+  // Universal style compaction is focused on reducing Write Amplification
+  // Factor for big data sets, but increases Space Amplification. You can learn
+  // more about the different styles here:
+  // https://github.com/facebook/rocksdb/wiki/Rocksdb-Architecture-Guide
+  // Make sure to also call IncreaseParallelism(), which will provide the
+  // biggest performance gains.
+  // Note: we might use more memory than memtable_memory_budget during high
+  // write rate period
+  ColumnFamilyOptions* OptimizeLevelStyleCompaction(
+      uint64_t memtable_memory_budget = 512 * 1024 * 1024);
+  ColumnFamilyOptions* OptimizeUniversalStyleCompaction(
+      uint64_t memtable_memory_budget = 512 * 1024 * 1024);
+
   // -------------------
   // Parameters that affect behavior
 
@@ -149,8 +184,9 @@ struct ColumnFamilyOptions {
   size_t write_buffer_size;
 
   // The maximum number of write buffers that are built up in memory.
-  // The default is 2, so that when 1 write buffer is being flushed to
-  // storage, new writes can continue to the other write buffer.
+  // The default and the minimum number is 2, so that when 1 write buffer
+  // is being flushed to storage, new writes can continue to the other
+  // write buffer.
   // Default: 2
   int max_write_buffer_number;
 
@@ -336,6 +372,7 @@ struct ColumnFamilyOptions {
   // With bloomfilter and fast storage, a miss on one level
   // is very cheap if the file handle is cached in table cache
   // (which is true if max_open_files is large).
+  // Default: true
   bool disable_seek_compaction;
 
   // Puts are delayed 0-1 ms when any level has a compaction score that exceeds
@@ -403,6 +440,9 @@ struct ColumnFamilyOptions {
   // The options needed to support Universal Style compactions
   CompactionOptionsUniversal compaction_options_universal;
 
+  // The options for FIFO compaction style
+  CompactionOptionsFIFO compaction_options_fifo;
+
   // Use KeyMayExist API to filter deletes when this is true.
   // If KeyMayExist returns false, i.e. the key definitely does not exist, then
   // the delete is a noop. KeyMayExist only incurs in-memory look up.
@@ -429,11 +469,11 @@ struct ColumnFamilyOptions {
 
   // This option allows user to to collect their own interested statistics of
   // the tables.
-  // Default: emtpy vector -- no user-defined statistics collection will be
+  // Default: empty vector -- no user-defined statistics collection will be
   // performed.
-  typedef std::vector<std::shared_ptr<TablePropertiesCollector>>
-      TablePropertiesCollectors;
-  TablePropertiesCollectors table_properties_collectors;
+  typedef std::vector<std::shared_ptr<TablePropertiesCollectorFactory>>
+      TablePropertiesCollectorFactories;
+  TablePropertiesCollectorFactories table_properties_collector_factories;
 
   // Allows thread-safe inplace updates.
   // If inplace_callback function is not set,
@@ -508,12 +548,9 @@ struct ColumnFamilyOptions {
 
   // Control locality of bloom filter probes to improve cache miss rate.
   // This option only applies to memtable prefix bloom and plaintable
-  // prefix bloom. It essentially limits the max number of cache lines each
-  // bloom filter check can touch.
-  // This optimization is turned off when set to 0. The number should never
-  // be greater than number of probes. This option can boost performance
-  // for in-memory workload but should use with care since it can cause
-  // higher false positive rate.
+  // prefix bloom. It essentially limits every bloom checking to one cache line.
+  // This optimization is turned off when set to 0, and positive number to turn
+  // it on.
   // Default: 0
   uint32_t bloom_locality;
 
@@ -546,10 +583,23 @@ struct ColumnFamilyOptions {
 };
 
 struct DBOptions {
+  // Some functions that make it easier to optimize RocksDB
+
+  // By default, RocksDB uses only one background thread for flush and
+  // compaction. Calling this function will set it up such that total of
+  // `total_threads` is used. Good value for `total_threads` is the number of
+  // cores. You almost definitely want to call this function if your system is
+  // bottlenecked by RocksDB.
+  DBOptions* IncreaseParallelism(int total_threads = 16);
+
   // If true, the database will be created if it is missing.
   // Default: false
   bool create_if_missing;
 
+  // If true, missing column families will be automatically created.
+  // Default: false
+  bool create_missing_column_families;
+
   // If true, an error is raised if the database already exists.
   // Default: false
   bool error_if_exists;

include/rocksdb/statistics.h

@@ -125,6 +125,7 @@ enum Tickers {
   NUMBER_SUPERVERSION_ACQUIRES,
   NUMBER_SUPERVERSION_RELEASES,
   NUMBER_SUPERVERSION_CLEANUPS,
+  NUMBER_BLOCK_NOT_COMPRESSED,
   TICKER_ENUM_MAX
 };
 
@@ -183,6 +184,7 @@ const std::vector<std::pair<Tickers, std::string>> TickersNameMap = {
     {NUMBER_SUPERVERSION_ACQUIRES, "rocksdb.number.superversion_acquires"},
     {NUMBER_SUPERVERSION_RELEASES, "rocksdb.number.superversion_releases"},
     {NUMBER_SUPERVERSION_CLEANUPS, "rocksdb.number.superversion_cleanups"},
+    {NUMBER_BLOCK_NOT_COMPRESSED, "rocksdb.number.block.not_compressed"},
 };
 
 /**

include/rocksdb/status.h

@@ -97,7 +97,6 @@ class Status {
   // Returns the string "OK" for success.
   std::string ToString() const;
 
- private:
   enum Code {
     kOk = 0,
     kNotFound = 1,
@@ -110,6 +109,10 @@ class Status {
     kShutdownInProgress = 8
   };
 
+  Code code() const {
+    return code_;
+  }
+ private:
   // A nullptr state_ (which is always the case for OK) means the message
   // is empty.
   // of the following form:
@@ -118,9 +121,6 @@ class Status {
   Code code_;
   const char* state_;
 
-  Code code() const {
-    return code_;
-  }
   explicit Status(Code code) : code_(code), state_(nullptr) { }
   Status(Code code, const Slice& msg, const Slice& msg2);
   static const char* CopyState(const char* s);

include/rocksdb/table.h

@@ -74,6 +74,12 @@ struct BlockBasedTableOptions {
 
   IndexType index_type = kBinarySearch;
 
+  // Influence the behavior when kHashSearch is used.
+  // if false, stores a precise prefix to block range mapping
+  // if true, does not store prefix and allows prefix hash collision
+  // (less memory consumption)
+  bool hash_index_allow_collision = true;
+
   // Use the specified checksum type. Newly created table files will be
   // protected with this checksum type. Old table files will still be readable,
   // even though they have different checksum type.
@@ -91,6 +97,30 @@ extern TableFactory* NewBlockBasedTableFactory(
     const BlockBasedTableOptions& table_options = BlockBasedTableOptions());
 
 #ifndef ROCKSDB_LITE
+
+enum EncodingType : char {
+  // Always write full keys without any special encoding.
+  kPlain,
+  // Find opportunity to write the same prefix once for multiple rows.
+  // In some cases, when a key follows a previous key with the same prefix,
+  // instead of writing out the full key, it just writes out the size of the
+  // shared prefix, as well as other bytes, to save some bytes.
+  //
+  // When using this option, the user is required to use the same prefix
+  // extractor to make sure the same prefix will be extracted from the same key.
+  // The Name() value of the prefix extractor will be stored in the file. When
+  // reopening the file, the name of the options.prefix_extractor given will be
+  // bitwise compared to the prefix extractors stored in the file. An error
+  // will be returned if the two don't match.
+  kPrefix,
+};
+
+// Table Properties that are specific to plain table properties.
+struct PlainTablePropertyNames {
+  static const std::string kPrefixExtractorName;
+  static const std::string kEncodingType;
+};
+
 // -- Plain Table with prefix-only seek
 // For this factory, you need to set Options.prefix_extrator properly to make it
 // work. Look-up will starts with prefix hash lookup for key prefix. Inside the
@@ -107,11 +137,22 @@ extern TableFactory* NewBlockBasedTableFactory(
 //                    in the hash table
 // @index_sparseness: inside each prefix, need to build one index record for how
 //                    many keys for binary search inside each hash bucket.
+//                    For encoding type kPrefix, the value will be used when
+//                    writing to determine an interval to rewrite the full key.
+//                    It will also be used as a suggestion and satisfied when
+//                    possible.
 // @huge_page_tlb_size: if <=0, allocate hash indexes and blooms from malloc.
 //                      Otherwise from huge page TLB. The user needs to reserve
 //                      huge pages for it to be allocated, like:
 //                          sysctl -w vm.nr_hugepages=20
 //                      See linux doc Documentation/vm/hugetlbpage.txt
+// @encoding_type: how to encode the keys. See enum EncodingType above for
+//                 the choices. The value will determine how to encode keys
+//                 when writing to a new SST file. This value will be stored
+//                 inside the SST file which will be used when reading from the
+//                 file, which makes it possible for users to choose different
+//                 encoding type when reopening a DB. Files with different
+//                 encoding types can co-exist in the same DB and can be read.
 
 const uint32_t kPlainTableVariableLength = 0;
 extern TableFactory* NewPlainTableFactory(uint32_t user_key_len =
@@ -119,7 +160,8 @@ extern TableFactory* NewPlainTableFactory(uint32_t user_key_len =
                                           int bloom_bits_per_prefix = 10,
                                           double hash_table_ratio = 0.75,
                                           size_t index_sparseness = 16,
-                                          size_t huge_page_tlb_size = 0);
+                                          size_t huge_page_tlb_size = 0,
+                                          EncodingType encoding_type = kPlain);
 
 // -- Plain Table
 // This factory of plain table ignores Options.prefix_extractor and assumes no
@@ -141,7 +183,7 @@ extern TableFactory* NewPlainTableFactory(uint32_t user_key_len =
 extern TableFactory* NewTotalOrderPlainTableFactory(
     uint32_t user_key_len = kPlainTableVariableLength,
     int bloom_bits_per_key = 0, size_t index_sparseness = 16,
-    size_t huge_page_tlb_size = 0);
+    size_t huge_page_tlb_size = 0, bool full_scan_mode = false);
 
 #endif  // ROCKSDB_LITE
 
@@ -203,4 +245,19 @@ class TableFactory {
       WritableFile* file, CompressionType compression_type) const = 0;
 };
 
+#ifndef ROCKSDB_LITE
+// Create a special table factory that can open both of block based table format
+// and plain table, based on setting inside the SST files. It should be used to
+// convert a DB from one table format to another.
+// @table_factory_to_write: the table factory used when writing to new files.
+// @block_based_table_factory:  block based table factory to use. If NULL, use
+//                              a default one.
+// @plain_table_factory: plain table factory to use. If NULL, use a default one.
+extern TableFactory* NewAdaptiveTableFactory(
+    std::shared_ptr<TableFactory> table_factory_to_write = nullptr,
+    std::shared_ptr<TableFactory> block_based_table_factory = nullptr,
+    std::shared_ptr<TableFactory> plain_table_factory = nullptr);
+
+#endif  // ROCKSDB_LITE
+
 }  // namespace rocksdb

include/rocksdb/table_properties.h

@@ -79,7 +79,10 @@ extern const std::string kPropertiesBlock;
 
 // `TablePropertiesCollector` provides the mechanism for users to collect
 // their own interested properties. This class is essentially a collection
-//  of callback functions that will be invoked during table building.
+// of callback functions that will be invoked during table building.
+// It is construced with TablePropertiesCollectorFactory. The methods don't
+// need to be thread-safe, as we will create exactly one
+// TablePropertiesCollector object per table and then call it sequentially
 class TablePropertiesCollector {
  public:
   virtual ~TablePropertiesCollector() {}
@@ -95,12 +98,24 @@ class TablePropertiesCollector {
   // `properties`.
   virtual Status Finish(UserCollectedProperties* properties) = 0;
 
-  // The name of the properties collector can be used for debugging purpose.
-  virtual const char* Name() const = 0;
-
   // Return the human-readable properties, where the key is property name and
   // the value is the human-readable form of value.
   virtual UserCollectedProperties GetReadableProperties() const = 0;
+
+  // The name of the properties collector can be used for debugging purpose.
+  virtual const char* Name() const = 0;
+};
+
+// Constructs TablePropertiesCollector. Internals create a new
+// TablePropertiesCollector for each new table
+class TablePropertiesCollectorFactory {
+ public:
+  virtual ~TablePropertiesCollectorFactory() {}
+  // has to be thread-safe
+  virtual TablePropertiesCollector* CreateTablePropertiesCollector() = 0;
+
+  // The name of the properties collector can be used for debugging purpose.
+  virtual const char* Name() const = 0;
 };
 
 // Extra properties

include/rocksdb/version.h

@@ -1,6 +1,17 @@
+// Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree. An additional grant
+// of patent rights can be found in the PATENTS file in the same directory.
 #pragma once
 
 // Also update Makefile if you change these
-#define __ROCKSDB_MAJOR__ 3
-#define __ROCKSDB_MINOR__ 0
-#define __ROCKSDB_PATCH__ 0
+#define ROCKSDB_MAJOR 3
+#define ROCKSDB_MINOR 2
+#define ROCKSDB_PATCH 0
+
+// Do not use these. We made the mistake of declaring macros starting with
+// double underscore. Now we have to live with our choice. We'll deprecate these
+// at some point
+#define __ROCKSDB_MAJOR__ ROCKSDB_MAJOR
+#define __ROCKSDB_MINOR__ ROCKSDB_MINOR
+#define __ROCKSDB_PATCH__ ROCKSDB_PATCH

java/Makefile

@@ -1,4 +1,5 @@
-NATIVE_JAVA_CLASSES = org.rocksdb.RocksDB org.rocksdb.Options org.rocksdb.WriteBatch org.rocksdb.WriteBatchInternal org.rocksdb.WriteBatchTest org.rocksdb.WriteOptions org.rocksdb.BackupableDB org.rocksdb.BackupableDBOptions org.rocksdb.Statistics org.rocksdb.Iterator org.rocksdb.VectorMemTableConfig org.rocksdb.SkipListMemTableConfig org.rocksdb.HashLinkedListMemTableConfig org.rocksdb.HashSkipListMemTableConfig org.rocksdb.PlainTableConfig org.rocksdb.ReadOptions org.rocksdb.Filter org.rocksdb.BloomFilter
+NATIVE_JAVA_CLASSES = org.rocksdb.RocksDB org.rocksdb.Options org.rocksdb.WriteBatch org.rocksdb.WriteBatchInternal org.rocksdb.WriteBatchTest org.rocksdb.WriteOptions org.rocksdb.BackupableDB org.rocksdb.BackupableDBOptions org.rocksdb.Statistics org.rocksdb.RocksIterator org.rocksdb.VectorMemTableConfig org.rocksdb.SkipListMemTableConfig org.rocksdb.HashLinkedListMemTableConfig org.rocksdb.HashSkipListMemTableConfig org.rocksdb.PlainTableConfig org.rocksdb.ReadOptions org.rocksdb.Filter org.rocksdb.BloomFilter org.rocksdb.RestoreOptions org.rocksdb.RestoreBackupableDB org.rocksdb.RocksEnv
+
 NATIVE_INCLUDE = ./include
 ROCKSDB_JAR = rocksdbjni.jar
 

java/RocksDBSample.java

@@ -186,7 +186,7 @@ public class RocksDBSample {
         assert(false); //Should never reach here.
       }
 
-      Iterator iterator = db.newIterator();
+      RocksIterator iterator = db.newIterator();
 
       boolean seekToFirstPassed = false;
       for (iterator.seekToFirst(); iterator.isValid(); iterator.next()) {

java/org/rocksdb/BackupableDB.java

@@ -25,11 +25,14 @@ public class BackupableDB extends RocksDB {
   public static BackupableDB open(
       Options opt, BackupableDBOptions bopt, String db_path)
       throws RocksDBException {
-    // since BackupableDB c++ will handle the life cycle of
-    // the returned RocksDB of RocksDB.open(), here we store
-    // it as a BackupableDB member variable to avoid GC.
-    BackupableDB bdb = new BackupableDB(RocksDB.open(opt, db_path));
-    bdb.open(bdb.db_.nativeHandle_, bopt.nativeHandle_);
+
+    RocksDB db = RocksDB.open(opt, db_path);
+    BackupableDB bdb = new BackupableDB();
+    bdb.open(db.nativeHandle_, bopt.nativeHandle_);
+
+    // Prevent the RocksDB object from attempting to delete
+    // the underly C++ DB object.
+    db.disOwnNativeHandle();
 
     return bdb;
   }
@@ -64,9 +67,8 @@ public class BackupableDB extends RocksDB {
    * A protected construction that will be used in the static factory
    * method BackupableDB.open().
    */
-  protected BackupableDB(RocksDB db) {
+  protected BackupableDB() {
     super();
-    db_ = db;
   }
 
   @Override protected void finalize() {
@@ -75,6 +77,4 @@ public class BackupableDB extends RocksDB {
 
   protected native void open(long rocksDBHandle, long backupDBOptionsHandle);
   protected native void createNewBackup(long handle, boolean flag);
-
-  private final RocksDB db_;
 }

java/org/rocksdb/BackupableDBOptions.java

@@ -11,11 +11,39 @@ package org.rocksdb;
  *
  * Note that dispose() must be called before an Options instance
  * become out-of-scope to release the allocated memory in c++.
+ *
+ * @param path Where to keep the backup files. Has to be different than dbname.
+       Best to set this to dbname_ + "/backups"
+ * @param shareTableFiles If share_table_files == true, backup will assume that
+ *     table files with same name have the same contents. This enables
+ *     incremental backups and avoids unnecessary data copies. If
+ *     share_table_files == false, each backup will be on its own and will not
+ *     share any data with other backups. default: true
+ * @param sync If sync == true, we can guarantee you'll get consistent backup
+ *     even on a machine crash/reboot. Backup process is slower with sync
+ *     enabled. If sync == false, we don't guarantee anything on machine reboot.
+ *     However, chances are some of the backups are consistent. Default: true
+ * @param destroyOldData If true, it will delete whatever backups there are
+ *     already. Default: false
+ * @param backupLogFiles If false, we won't backup log files. This option can be
+ *     useful for backing up in-memory databases where log file are persisted,
+ *     but table files are in memory. Default: true
+ * @param backupRateLimit Max bytes that can be transferred in a second during
+ *     backup. If 0 or negative, then go as fast as you can. Default: 0
+ * @param restoreRateLimit Max bytes that can be transferred in a second during
+ *     restore. If 0 or negative, then go as fast as you can. Default: 0
  */
 public class BackupableDBOptions extends RocksObject {
-  public BackupableDBOptions(String path) {
+  public BackupableDBOptions(String path, boolean shareTableFiles, boolean sync,
+      boolean destroyOldData, boolean backupLogFiles, long backupRateLimit,
+      long restoreRateLimit) {
     super();
-    newBackupableDBOptions(path);
+    
+    backupRateLimit = (backupRateLimit <= 0) ? 0 : backupRateLimit;
+    restoreRateLimit = (restoreRateLimit <= 0) ? 0 : restoreRateLimit;
+    
+    newBackupableDBOptions(path, shareTableFiles, sync, destroyOldData,
+        backupLogFiles, backupRateLimit, restoreRateLimit);
   }
 
   /**
@@ -32,13 +60,14 @@ public class BackupableDBOptions extends RocksObject {
    * Release the memory allocated for the current instance
    * in the c++ side.
    */
-  @Override public synchronized void dispose() {
-    if (isInitialized()) {
-      dispose(nativeHandle_);
-    }
+  @Override protected void disposeInternal() {
+    assert(isInitialized());
+    disposeInternal(nativeHandle_);
   }
 
-  private native void newBackupableDBOptions(String path);
+  private native void newBackupableDBOptions(String path,
+      boolean shareTableFiles, boolean sync, boolean destroyOldData,
+      boolean backupLogFiles, long backupRateLimit, long restoreRateLimit);
   private native String backupDir(long handle);
-  private native void dispose(long handle);
+  private native void disposeInternal(long handle);
 }

java/org/rocksdb/Filter.java

@@ -22,11 +22,10 @@ public abstract class Filter extends RocksObject {
    * RocksDB instances referencing the filter are closed.
    * Otherwise an undefined behavior will occur.
    */
-  @Override public synchronized void dispose() {
-    if (isInitialized()) {
-      dispose0(nativeHandle_);
-    }
+  @Override protected void disposeInternal() {
+    assert(isInitialized());
+    disposeInternal(nativeHandle_);
   }
 
-  private native void dispose0(long handle);
+  private native void disposeInternal(long handle);
 }

java/org/rocksdb/Options.java

@@ -24,6 +24,7 @@ public class Options extends RocksObject {
     super();
     cacheSize_ = DEFAULT_CACHE_SIZE;
     newOptions();
+    env_ = RocksEnv.getDefault();
   }
 
   /**
@@ -42,6 +43,24 @@ public class Options extends RocksObject {
     return this;
   }
 
+  /**
+   * Use the specified object to interact with the environment,
+   * e.g. to read/write files, schedule background work, etc.
+   * Default: RocksEnv.getDefault()
+   */
+  public Options setEnv(RocksEnv env) {
+    assert(isInitialized());
+    setEnv(nativeHandle_, env.nativeHandle_);
+    env_ = env;
+    return this;
+  }
+  private native void setEnv(long optHandle, long envHandle);
+
+  public RocksEnv getEnv() {
+    return env_;
+  }
+  private native long getEnvHandle(long handle);
+
   /**
    * Return true if the create_if_missing flag is set to true.
    * If true, the database will be created if it is missing.
@@ -2311,10 +2330,9 @@ public class Options extends RocksObject {
    * Release the memory allocated for the current instance
    * in the c++ side.
    */
-  @Override public synchronized void dispose() {
-    if (isInitialized()) {
-      dispose0();
-    }
+  @Override protected void disposeInternal() {
+    assert(isInitialized());
+    disposeInternal(nativeHandle_);
   }
 
   static final int DEFAULT_PLAIN_TABLE_BLOOM_BITS_PER_KEY = 10;
@@ -2322,7 +2340,7 @@ public class Options extends RocksObject {
   static final int DEFAULT_PLAIN_TABLE_INDEX_SPARSENESS = 16;
 
   private native void newOptions();
-  private native void dispose0();
+  private native void disposeInternal(long handle);
   private native void setCreateIfMissing(long handle, boolean flag);
   private native boolean createIfMissing(long handle);
   private native void setWriteBufferSize(long handle, long writeBufferSize);
@@ -2352,4 +2370,5 @@ public class Options extends RocksObject {
 
   long cacheSize_;
   Filter filter_;
+  RocksEnv env_;
 }

java/org/rocksdb/ReadOptions.java

@@ -18,19 +18,6 @@ public class ReadOptions extends RocksObject {
   }
   private native void newReadOptions();
 
-  /**
-   * Release the memory allocated for the current instance
-   * in the c++ side.
-   *
-   * Calling other methods after dispose() leads to undefined behavior.
-   */
-  @Override public synchronized void dispose() {
-    if (isInitialized()) {
-      dispose(nativeHandle_);
-    }
-  }
-  private native void dispose(long handle);
-
   /**
    * If true, all data read from underlying storage will be
    * verified against corresponding checksums.
@@ -127,4 +114,12 @@ public class ReadOptions extends RocksObject {
   }
   private native void setTailing(
       long handle, boolean tailing);
+
+
+  @Override protected void disposeInternal() {
+    assert(isInitialized());
+    disposeInternal(nativeHandle_);
+  }
+  private native void disposeInternal(long handle);
+
 }

java/org/rocksdb/RestoreBackupableDB.java

@@ -0,0 +1,84 @@
+// Copyright (c) 2014, Facebook, Inc.  All rights reserved.
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree. An additional grant
+// of patent rights can be found in the PATENTS file in the same directory.
+
+package org.rocksdb;
+
+/**
+ * This class is used to access information about backups and restore from them.
+ *
+ * Note that dispose() must be called before this instance become out-of-scope
+ * to release the allocated memory in c++.
+ *
+ * @param options Instance of BackupableDBOptions.
+ */
+public class RestoreBackupableDB extends RocksObject {
+  public RestoreBackupableDB(BackupableDBOptions options) {
+    super();
+    nativeHandle_ = newRestoreBackupableDB(options.nativeHandle_);
+  }
+
+  /**
+   * Restore from backup with backup_id
+   * IMPORTANT -- if options_.share_table_files == true and you restore DB
+   * from some backup that is not the latest, and you start creating new
+   * backups from the new DB, they will probably fail.
+   *
+   * Example: Let's say you have backups 1, 2, 3, 4, 5 and you restore 3.
+   * If you add new data to the DB and try creating a new backup now, the
+   * database will diverge from backups 4 and 5 and the new backup will fail.
+   * If you want to create new backup, you will first have to delete backups 4
+   * and 5.
+   */
+  public void restoreDBFromBackup(long backupId, String dbDir, String walDir,
+      RestoreOptions restoreOptions) throws RocksDBException {
+    restoreDBFromBackup0(nativeHandle_, backupId, dbDir, walDir,
+        restoreOptions.nativeHandle_);
+  }
+
+  /**
+   * Restore from the latest backup.
+   */
+  public void restoreDBFromLatestBackup(String dbDir, String walDir,
+      RestoreOptions restoreOptions) throws RocksDBException {
+    restoreDBFromLatestBackup0(nativeHandle_, dbDir, walDir,
+        restoreOptions.nativeHandle_);
+  }
+
+  /**
+   * Deletes old backups, keeping latest numBackupsToKeep alive.
+   *
+   * @param Number of latest backups to keep
+   */
+  public void purgeOldBackups(int numBackupsToKeep) throws RocksDBException {
+    purgeOldBackups0(nativeHandle_, numBackupsToKeep);
+  }
+
+  /**
+   * Deletes a specific backup.
+   *
+   * @param ID of backup to delete.
+   */
+  public void deleteBackup(long backupId) throws RocksDBException {
+    deleteBackup0(nativeHandle_, backupId);
+  }
+
+  /**
+   * Release the memory allocated for the current instance
+   * in the c++ side.
+   */
+  @Override public synchronized void disposeInternal() {
+    assert(isInitialized());
+    dispose(nativeHandle_);
+  }
+
+  private native long newRestoreBackupableDB(long options);
+  private native void restoreDBFromBackup0(long nativeHandle, long backupId,
+      String dbDir, String walDir, long restoreOptions) throws RocksDBException;
+  private native void restoreDBFromLatestBackup0(long nativeHandle,
+      String dbDir, String walDir, long restoreOptions) throws RocksDBException;
+  private native void purgeOldBackups0(long nativeHandle, int numBackupsToKeep);
+  private native void deleteBackup0(long nativeHandle, long backupId);
+  private native void dispose(long nativeHandle);
+}

java/org/rocksdb/RestoreOptions.java

@@ -0,0 +1,37 @@
+// Copyright (c) 2014, Facebook, Inc.  All rights reserved.
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree. An additional grant
+// of patent rights can be found in the PATENTS file in the same directory.
+
+package org.rocksdb;
+
+/**
+ * RestoreOptions to control the behavior of restore.
+ *
+ * Note that dispose() must be called before this instance become out-of-scope
+ * to release the allocated memory in c++.
+ *
+ * @param If true, restore won't overwrite the existing log files in wal_dir. It
+ *     will also move all log files from archive directory to wal_dir. Use this
+ *     option in combination with BackupableDBOptions::backup_log_files = false
+ *     for persisting in-memory databases.
+ *     Default: false
+ */
+public class RestoreOptions extends RocksObject {
+  public RestoreOptions(boolean keepLogFiles) {
+    super();
+    nativeHandle_ = newRestoreOptions(keepLogFiles);
+  }
+
+  /**
+   * Release the memory allocated for the current instance
+   * in the c++ side.
+   */
+  @Override public synchronized void disposeInternal() {
+    assert(isInitialized());
+    dispose(nativeHandle_);
+  }
+
+  private native long newRestoreOptions(boolean keepLogFiles);
+  private native void dispose(long handle);
+}

java/org/rocksdb/RocksDB.java

@@ -114,11 +114,9 @@ public class RocksDB extends RocksObject {
     return db;
   }
 
-  @Override public synchronized void dispose() {
-    if (isInitialized()) {
-      dispose(nativeHandle_);
-      nativeHandle_ = 0;
-    }
+  @Override protected void disposeInternal() {
+    assert(isInitialized());
+    disposeInternal(nativeHandle_);
   }
 
   /**
@@ -311,12 +309,8 @@ public class RocksDB extends RocksObject {
    *
    * @return instance of iterator object.
    */
-  public Iterator newIterator() {
-    return new Iterator(iterator0(nativeHandle_));
-  }
-
-  @Override protected void finalize() {
-    close();
+  public RocksIterator newIterator() {
+    return new RocksIterator(iterator0(nativeHandle_));
   }
 
   /**
@@ -370,7 +364,7 @@ public class RocksDB extends RocksObject {
       long handle, long writeOptHandle,
       byte[] key, int keyLen) throws RocksDBException;
   protected native long iterator0(long optHandle);
-  protected native void dispose(long handle);
+  private native void disposeInternal(long handle);
 
   protected Filter filter_;
 }

java/org/rocksdb/RocksEnv.java

@@ -0,0 +1,102 @@
+// Copyright (c) 2014, Facebook, Inc.  All rights reserved.
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree. An additional grant
+// of patent rights can be found in the PATENTS file in the same directory.
+
+package org.rocksdb;
+
+/**
+ * A RocksEnv is an interface used by the rocksdb implementation to access
+ * operating system functionality like the filesystem etc.
+ *
+ * All Env implementations are safe for concurrent access from
+ * multiple threads without any external synchronization.
+ */
+public class RocksEnv extends RocksObject {
+  public static final int FLUSH_POOL = 0;
+  public static final int COMPACTION_POOL = 1;
+
+  static {
+    default_env_ = new RocksEnv(getDefaultEnvInternal());
+  }
+  private static native long getDefaultEnvInternal();
+
+  /**
+   * Returns the default environment suitable for the current operating
+   * system.
+   *
+   * The result of getDefault() is a singleton whose ownership belongs
+   * to rocksdb c++.  As a result, the returned RocksEnv will not
+   * have the ownership of its c++ resource, and calling its dispose()
+   * will be no-op.
+   */
+  public static RocksEnv getDefault() {
+    return default_env_;
+  }
+
+  /**
+   * Sets the number of background worker threads of the flush pool
+   * for this environment.
+   * default number: 1
+   */
+  public RocksEnv setBackgroundThreads(int num) {
+    return setBackgroundThreads(num, FLUSH_POOL);
+  }
+
+  /**
+   * Sets the number of background worker threads of the specified thread
+   * pool for this environment.
+   *
+   * @param num the number of threads
+   * @param poolID the id to specified a thread pool.  Should be either
+   *     FLUSH_POOL or COMPACTION_POOL.
+   * Default number: 1
+   */
+  public RocksEnv setBackgroundThreads(int num, int poolID) {
+    setBackgroundThreads(nativeHandle_, num, poolID);
+    return this;
+  }
+  private native void setBackgroundThreads(
+      long handle, int num, int priority);
+
+  /**
+   * Returns the length of the queue associated with the specified
+   * thread pool.
+   *
+   * @param poolID the id to specified a thread pool.  Should be either
+   *     FLUSH_POOL or COMPACTION_POOL.
+   */
+  public int getThreadPoolQueueLen(int poolID) {
+    return getThreadPoolQueueLen(nativeHandle_, poolID);
+  }
+  private native int getThreadPoolQueueLen(long handle, int poolID);
+
+  /**
+   * Package-private constructor that uses the specified native handle
+   * to construct a RocksEnv.  Note that the ownership of the input handle
+   * belongs to the caller, and the newly created RocksEnv will not take
+   * the ownership of the input handle.  As a result, calling dispose()
+   * of the created RocksEnv will be no-op.
+   */
+  RocksEnv(long handle) {
+    super();
+    nativeHandle_ = handle;
+    disOwnNativeHandle();
+  }
+
+  /**
+   * The helper function of dispose() which all subclasses of RocksObject
+   * must implement to release their associated C++ resource.
+   */
+  protected void disposeInternal() {
+    disposeInternal(nativeHandle_);
+  }
+  private native void disposeInternal(long handle);
+
+  /**
+   * The static default RocksEnv.  The ownership of its native handle
+   * belongs to rocksdb c++ and is not able to be released on the Java
+   * side.
+   */
+  static RocksEnv default_env_;
+}

java/org/rocksdb/RocksIterator.java

@@ -11,13 +11,13 @@ package org.rocksdb;
  * are provided by this library.  In particular, iterators are provided
  * to access the contents of a Table or a DB.
  *
- * Multiple threads can invoke const methods on an Iterator without
+ * Multiple threads can invoke const methods on an RocksIterator without
  * external synchronization, but if any of the threads may call a
- * non-const method, all threads accessing the same Iterator must use
+ * non-const method, all threads accessing the same RocksIterator must use
  * external synchronization.
  */
-public class Iterator extends RocksObject {
-  public Iterator(long nativeHandle) {
+public class RocksIterator extends RocksObject {
+  public RocksIterator(long nativeHandle) {
     super();
     nativeHandle_ = nativeHandle;
   }
@@ -118,15 +118,13 @@ public class Iterator extends RocksObject {
   /**
    * Deletes underlying C++ iterator pointer.
    */
-  @Override public synchronized void dispose() {
-    if(isInitialized()) {
-      dispose(nativeHandle_);
-      nativeHandle_ = 0;
-    }
+  @Override protected void disposeInternal() {
+    assert(isInitialized());
+    disposeInternal(nativeHandle_);
   }
 
   private native boolean isValid0(long handle);
-  private native void dispose(long handle);
+  private native void disposeInternal(long handle);
   private native void seekToFirst0(long handle);
   private native void seekToLast0(long handle);
   private native void next0(long handle);

java/org/rocksdb/RocksObject.java

@@ -16,12 +16,48 @@ package org.rocksdb;
 public abstract class RocksObject {
   protected RocksObject() {
     nativeHandle_ = 0;
+    owningHandle_ = true;
   }
 
   /**
    * Release the c++ object pointed by the native handle.
+   *
+   * Note that once an instance of RocksObject has been disposed,
+   * calling its function will lead undefined behavior.
    */
-  public abstract void dispose();
+  public final synchronized void dispose() {
+    if (isOwningNativeHandle() && isInitialized()) {
+      disposeInternal();
+    }
+    nativeHandle_ = 0;
+    disOwnNativeHandle();
+  }
+
+  /**
+   * The helper function of dispose() which all subclasses of RocksObject
+   * must implement to release their associated C++ resource.
+   */
+  protected abstract void disposeInternal();
+
+  /**
+   * Revoke ownership of the native object.
+   *
+   * This will prevent the object from attempting to delete the underlying
+   * native object in its finalizer. This must be used when another object
+   * takes over ownership of the native object or both will attempt to delete
+   * the underlying object when garbage collected.
+   *
+   * When disOwnNativeHandle is called, dispose() will simply set nativeHandle_
+   * to 0 without releasing its associated C++ resource.  As a result,
+   * incorrectly use this function may cause memory leak.
+   */
+  protected void disOwnNativeHandle() {
+    owningHandle_ = false;
+  }
+
+  protected boolean isOwningNativeHandle() {
+    return owningHandle_;
+  }
 
   protected boolean isInitialized() {
     return (nativeHandle_ != 0);
@@ -32,4 +68,5 @@ public abstract class RocksObject {
   }
 
   protected long nativeHandle_;
+  private boolean owningHandle_;
 }

java/org/rocksdb/WriteBatch.java

@@ -86,10 +86,9 @@ public class WriteBatch extends RocksObject {
   /**
    * Delete the c++ side pointer.
    */
-  @Override public synchronized void dispose() {
-    if (isInitialized()) {
-      dispose0();
-    }
+  @Override protected void disposeInternal() {
+    assert(isInitialized());
+    disposeInternal(nativeHandle_);
   }
 
   private native void newWriteBatch(int reserved_bytes);
@@ -99,7 +98,7 @@ public class WriteBatch extends RocksObject {
                             byte[] value, int valueLen);
   private native void remove(byte[] key, int keyLen);
   private native void putLogData(byte[] blob, int blobLen);
-  private native void dispose0();
+  private native void disposeInternal(long handle);
 }
 
 /**

java/org/rocksdb/WriteOptions.java

@@ -17,10 +17,9 @@ public class WriteOptions extends RocksObject {
     newWriteOptions();
   }
 
-  @Override public synchronized void dispose() {
-    if (isInitialized()) {
-      dispose0(nativeHandle_);
-    }
+  @Override protected void disposeInternal() {
+    assert(isInitialized());
+    disposeInternal(nativeHandle_);
   }
 
   /**
@@ -96,5 +95,5 @@ public class WriteOptions extends RocksObject {
   private native boolean sync(long handle);
   private native void setDisableWAL(long handle, boolean flag);
   private native boolean disableWAL(long handle);
-  private native void dispose0(long handle);
+  private native void disposeInternal(long handle);
 }

java/org/rocksdb/benchmark/DbBenchmark.java

@@ -395,7 +395,7 @@ public class DbBenchmark {
       byte[] key = new byte[keySize_];
       byte[] value = new byte[valueSize_];
       for (long i = 0; i < numEntries_; i++) {
-        getRandomKey(key, numEntries_);
+        getRandomKey(key, keyRange_);
         int len = db_.get(key, value);
         if (len != RocksDB.NOT_FOUND) {
           stats_.found_++;
@@ -416,7 +416,7 @@ public class DbBenchmark {
       super(tid, randSeed, numEntries, keyRange);
     }
     @Override public void runTask() throws RocksDBException {
-      org.rocksdb.Iterator iter = db_.newIterator();
+      RocksIterator iter = db_.newIterator();
       long i;
       for (iter.seekToFirst(), i = 0;
            iter.isValid() && i < numEntries_;
@@ -526,6 +526,8 @@ public class DbBenchmark {
         (Integer)flags_.get(Flag.max_write_buffer_number));
     options.setMaxBackgroundCompactions(
         (Integer)flags_.get(Flag.max_background_compactions));
+    options.getEnv().setBackgroundThreads(
+        (Integer)flags_.get(Flag.max_background_compactions));
     options.setMaxBackgroundFlushes(
         (Integer)flags_.get(Flag.max_background_flushes));
     options.setCacheSize(
@@ -534,8 +536,6 @@ public class DbBenchmark {
         (Long)flags_.get(Flag.block_size));
     options.setMaxOpenFiles(
         (Integer)flags_.get(Flag.open_files));
-    options.setCreateIfMissing(
-        !(Boolean)flags_.get(Flag.use_existing_db));
     options.setTableCacheRemoveScanCountLimit(
         (Integer)flags_.get(Flag.cache_remove_scan_count_limit));
     options.setDisableDataSync(
@@ -800,11 +800,17 @@ public class DbBenchmark {
         }
       }
     }
+    String extra = "";
+    if (benchmark.indexOf("read") >= 0) {
+      extra = String.format(" %d / %d found; ", stats.found_, stats.done_);
+    } else {
+      extra = String.format(" %d ops done; ", stats.done_);
+    }
 
     System.out.printf(
-        "%-16s : %11.5f micros/op; %6.1f MB/s; %d / %d task(s) finished.\n",
+        "%-16s : %11.5f micros/op; %6.1f MB/s;%s %d / %d task(s) finished.\n",
         benchmark, (double) elapsedSeconds / stats.done_ * 1e6,
-        (stats.bytes_ / 1048576.0) / elapsedSeconds,
+        (stats.bytes_ / 1048576.0) / elapsedSeconds, extra,
         taskFinishedCount, concurrentThreads);
   }
 
@@ -933,7 +939,7 @@ public class DbBenchmark {
         "\tflag and also specify a benchmark that wants a fresh database,\n" +
         "\tthat benchmark will fail.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     num(1000000,
@@ -1022,7 +1028,7 @@ public class DbBenchmark {
     use_plain_table(false,
         "Use plain-table sst format.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     cache_size(-1L,
@@ -1079,7 +1085,7 @@ public class DbBenchmark {
     },
     histogram(false,"Print histogram of operation timings.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     min_write_buffer_number_to_merge(
@@ -1197,12 +1203,12 @@ public class DbBenchmark {
     verify_checksum(false,"Verify checksum for every block read\n" +
         "\tfrom storage.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     statistics(false,"Database statistics.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     writes(-1,"Number of write operations to do. If negative, do\n" +
@@ -1213,23 +1219,23 @@ public class DbBenchmark {
     },
     sync(false,"Sync all writes to disk.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     disable_data_sync(false,"If true, do not wait until data is\n" +
         "\tsynced to disk.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     use_fsync(false,"If true, issue fsync instead of fdatasync.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     disable_wal(false,"If true, do not write WAL for write.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     wal_dir("", "If not empty, use the given dir for WAL.") {
@@ -1306,7 +1312,7 @@ public class DbBenchmark {
     disable_seek_compaction(false,"Option to disable compaction\n" +
         "\ttriggered by read.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     delete_obsolete_files_period_micros(0,"Option to delete\n" +
@@ -1387,12 +1393,12 @@ public class DbBenchmark {
     },
     readonly(false,"Run read only benchmarks.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     disable_auto_compactions(false,"Do not auto trigger compactions.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     source_compaction_factor(1,"Cap the size of data in level-K for\n" +
@@ -1417,26 +1423,26 @@ public class DbBenchmark {
     bufferedio(rocksdb::EnvOptions().use_os_buffer,
         "Allow buffered io using OS buffers.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     */
     mmap_read(false,
         "Allow reads to occur via mmap-ing files.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     mmap_write(false,
         "Allow writes to occur via mmap-ing files.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     advise_random_on_open(defaultOptions_.adviseRandomOnOpen(),
         "Advise random access on table file open.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     compaction_fadvice("NORMAL",
@@ -1448,13 +1454,13 @@ public class DbBenchmark {
     use_tailing_iterator(false,
         "Use tailing iterator to access a series of keys instead of get.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     use_adaptive_mutex(defaultOptions_.useAdaptiveMutex(),
         "Use adaptive mutex.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     bytes_per_sync(defaultOptions_.bytesPerSync(),
@@ -1468,7 +1474,7 @@ public class DbBenchmark {
     filter_deletes(false," On true, deletes use bloom-filter and drop\n" +
         "\tthe delete if key not present.") {
       @Override public Object parseValue(String value) {
-        return Boolean.parseBoolean(value);
+        return parseBoolean(value);
       }
     },
     max_successive_merges(0,"Maximum number of successive merge\n" +
@@ -1489,8 +1495,6 @@ public class DbBenchmark {
       desc_ = desc;
     }
 
-    protected abstract Object parseValue(String value);
-
     public Object getDefaultValue() {
       return defaultValue_;
     }
@@ -1499,6 +1503,17 @@ public class DbBenchmark {
       return desc_;
     }
 
+    public boolean parseBoolean(String value) {
+      if (value.equals("1")) {
+        return true;
+      } else if (value.equals("0")) {
+        return false;
+      }
+      return Boolean.parseBoolean(value);
+    }
+
+    protected abstract Object parseValue(String value);
+
     private final Object defaultValue_;
     private final String desc_;
   }

java/org/rocksdb/test/BackupableDBTest.java

@@ -18,15 +18,37 @@ public class BackupableDBTest {
     Options opt = new Options();
     opt.setCreateIfMissing(true);
 
-    BackupableDBOptions bopt = new BackupableDBOptions(backup_path);
+    BackupableDBOptions bopt = new BackupableDBOptions(backup_path, false,
+        true, false, true, 0, 0);
     BackupableDB bdb = null;
 
     try {
       bdb = BackupableDB.open(opt, bopt, db_path);
-      bdb.put("hello".getBytes(), "BackupableDB".getBytes());
+
+      bdb.put("abc".getBytes(), "def".getBytes());
+      bdb.put("ghi".getBytes(), "jkl".getBytes());
       bdb.createNewBackup(true);
-      byte[] value = bdb.get("hello".getBytes());
-      assert(new String(value).equals("BackupableDB"));
+
+      // delete record after backup
+      bdb.remove("abc".getBytes());
+      byte[] value = bdb.get("abc".getBytes());
+      assert(value == null);
+      bdb.close();
+
+      // restore from backup
+      RestoreOptions ropt = new RestoreOptions(false);
+      RestoreBackupableDB rdb = new RestoreBackupableDB(bopt);
+      rdb.restoreDBFromLatestBackup(db_path, db_path,
+          ropt);
+      rdb.dispose();
+      ropt.dispose();
+
+      // verify that backed up data contains deleted record
+      bdb = BackupableDB.open(opt, bopt, db_path);
+      value = bdb.get("abc".getBytes());
+      assert(new String(value).equals("def"));
+
+      System.out.println("Backup and restore test passed");
     } catch (RocksDBException e) {
       System.err.format("[ERROR]: %s%n", e);
       e.printStackTrace();

java/rocksjni/backupablejni.cc

@@ -4,7 +4,8 @@
 // of patent rights can be found in the PATENTS file in the same directory.
 //
 // This file implements the "bridge" between Java and C++ and enables
-// calling c++ rocksdb::DB methods from Java side.
+// calling c++ rocksdb::BackupableDB and rocksdb::BackupableDBOptions methods
+// from Java side.
 
 #include <stdio.h>
 #include <stdlib.h>
@@ -51,9 +52,18 @@ void Java_org_rocksdb_BackupableDB_createNewBackup(
  * Signature: (Ljava/lang/String;)V
  */
 void Java_org_rocksdb_BackupableDBOptions_newBackupableDBOptions(
-    JNIEnv* env, jobject jobj, jstring jpath) {
+    JNIEnv* env, jobject jobj, jstring jpath, jboolean jshare_table_files,
+    jboolean jsync, jboolean jdestroy_old_data, jboolean jbackup_log_files,
+    jlong jbackup_rate_limit, jlong jrestore_rate_limit) {
+  jbackup_rate_limit = (jbackup_rate_limit <= 0) ? 0 : jbackup_rate_limit;
+  jrestore_rate_limit = (jrestore_rate_limit <= 0) ? 0 : jrestore_rate_limit;
+
   const char* cpath = env->GetStringUTFChars(jpath, 0);
-  auto bopt = new rocksdb::BackupableDBOptions(cpath);
+
+  auto bopt = new rocksdb::BackupableDBOptions(cpath, nullptr,
+      jshare_table_files, nullptr, jsync, jdestroy_old_data, jbackup_log_files,
+      jbackup_rate_limit, jrestore_rate_limit);
+
   env->ReleaseStringUTFChars(jpath, cpath);
 
   rocksdb::BackupableDBOptionsJni::setHandle(env, jobj, bopt);
@@ -72,10 +82,10 @@ jstring Java_org_rocksdb_BackupableDBOptions_backupDir(
 
 /*
  * Class:     org_rocksdb_BackupableDBOptions
- * Method:    dispose
+ * Method:    disposeInternal
  * Signature: (J)V
  */
-void Java_org_rocksdb_BackupableDBOptions_dispose(
+void Java_org_rocksdb_BackupableDBOptions_disposeInternal(
     JNIEnv* env, jobject jopt, jlong jhandle) {
   auto bopt = reinterpret_cast<rocksdb::BackupableDBOptions*>(jhandle);
   assert(bopt);

java/rocksjni/env.cc

@@ -0,0 +1,66 @@
+// Copyright (c) 2014, Facebook, Inc.  All rights reserved.
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree. An additional grant
+// of patent rights can be found in the PATENTS file in the same directory.
+//
+// This file implements the "bridge" between Java and C++ and enables
+// calling c++ rocksdb::Env methods from Java side.
+
+#include "include/org_rocksdb_RocksEnv.h"
+#include "rocksdb/env.h"
+
+/*
+ * Class:     org_rocksdb_RocksEnv
+ * Method:    getDefaultEnvInternal
+ * Signature: ()J
+ */
+jlong Java_org_rocksdb_RocksEnv_getDefaultEnvInternal(
+    JNIEnv* env, jclass jclass) {
+  return reinterpret_cast<jlong>(rocksdb::Env::Default());
+}
+
+/*
+ * Class:     org_rocksdb_RocksEnv
+ * Method:    setBackgroundThreads
+ * Signature: (JII)V
+ */
+void Java_org_rocksdb_RocksEnv_setBackgroundThreads(
+    JNIEnv* env, jobject jobj, jlong jhandle,
+    jint num, jint priority) {
+  auto* rocks_env = reinterpret_cast<rocksdb::Env*>(jhandle);
+  switch (priority) {
+    case org_rocksdb_RocksEnv_FLUSH_POOL:
+      rocks_env->SetBackgroundThreads(num, rocksdb::Env::Priority::LOW);
+      break;
+    case org_rocksdb_RocksEnv_COMPACTION_POOL:
+      rocks_env->SetBackgroundThreads(num, rocksdb::Env::Priority::HIGH);
+      break;
+  }
+}
+
+/*
+ * Class:     org_rocksdb_RocksEnv
+ * Method:    getThreadPoolQueueLen
+ * Signature: (JI)I
+ */
+jint Java_org_rocksdb_RocksEnv_getThreadPoolQueueLen(
+    JNIEnv* env, jobject jobj, jlong jhandle, jint pool_id) {
+  auto* rocks_env = reinterpret_cast<rocksdb::Env*>(jhandle);
+  switch (pool_id) {
+    case org_rocksdb_RocksEnv_FLUSH_POOL:
+      return rocks_env->GetThreadPoolQueueLen(rocksdb::Env::Priority::LOW);
+    case org_rocksdb_RocksEnv_COMPACTION_POOL:
+      return rocks_env->GetThreadPoolQueueLen(rocksdb::Env::Priority::HIGH);
+  }
+  return 0;
+}
+
+/*
+ * Class:     org_rocksdb_RocksEnv
+ * Method:    disposeInternal
+ * Signature: (J)V
+ */
+void Java_org_rocksdb_RocksEnv_disposeInternal(
+    JNIEnv* env, jobject jobj, jlong jhandle) {
+  delete reinterpret_cast<rocksdb::Env*>(jhandle);
+}

java/rocksjni/filter.cc

@@ -29,13 +29,10 @@ void Java_org_rocksdb_BloomFilter_createNewFilter0(
 
 /*
  * Class:     org_rocksdb_Filter
- * Method:    dispose0
+ * Method:    disposeInternal
  * Signature: (J)V
  */
-void Java_org_rocksdb_Filter_dispose0(
+void Java_org_rocksdb_Filter_disposeInternal(
     JNIEnv* env, jobject jobj, jlong handle) {
-  auto fp = reinterpret_cast<rocksdb::FilterPolicy*>(handle);
-  delete fp;
-
-  rocksdb::FilterJni::setHandle(env, jobj, nullptr);
+  delete reinterpret_cast<rocksdb::FilterPolicy*>(handle);
 }

java/rocksjni/iterator.cc

@@ -10,66 +10,66 @@
 #include <stdlib.h>
 #include <jni.h>
 
-#include "include/org_rocksdb_Iterator.h"
+#include "include/org_rocksdb_RocksIterator.h"
 #include "rocksjni/portal.h"
 #include "rocksdb/iterator.h"
 
 /*
- * Class:     org_rocksdb_Iterator
+ * Class:     org_rocksdb_RocksIterator
  * Method:    isValid0
  * Signature: (J)Z
  */
-jboolean Java_org_rocksdb_Iterator_isValid0(
+jboolean Java_org_rocksdb_RocksIterator_isValid0(
     JNIEnv* env, jobject jobj, jlong handle) {
   return reinterpret_cast<rocksdb::Iterator*>(handle)->Valid();
 }
 
 /*
- * Class:     org_rocksdb_Iterator
+ * Class:     org_rocksdb_RocksIterator
  * Method:    seekToFirst0
  * Signature: (J)V
  */
-void Java_org_rocksdb_Iterator_seekToFirst0(
+void Java_org_rocksdb_RocksIterator_seekToFirst0(
     JNIEnv* env, jobject jobj, jlong handle) {
   reinterpret_cast<rocksdb::Iterator*>(handle)->SeekToFirst();
 }
 
 /*
- * Class:     org_rocksdb_Iterator
+ * Class:     org_rocksdb_RocksIterator
  * Method:    seekToFirst0
  * Signature: (J)V
  */
-void Java_org_rocksdb_Iterator_seekToLast0(
+void Java_org_rocksdb_RocksIterator_seekToLast0(
     JNIEnv* env, jobject jobj, jlong handle) {
   reinterpret_cast<rocksdb::Iterator*>(handle)->SeekToLast();
 }
 
 /*
- * Class:     org_rocksdb_Iterator
+ * Class:     org_rocksdb_RocksIterator
  * Method:    seekToLast0
  * Signature: (J)V
  */
-void Java_org_rocksdb_Iterator_next0(
+void Java_org_rocksdb_RocksIterator_next0(
     JNIEnv* env, jobject jobj, jlong handle) {
   reinterpret_cast<rocksdb::Iterator*>(handle)->Next();
 }
 
 /*
- * Class:     org_rocksdb_Iterator
+ * Class:     org_rocksdb_RocksIterator
  * Method:    next0
  * Signature: (J)V
  */
-void Java_org_rocksdb_Iterator_prev0(
+void Java_org_rocksdb_RocksIterator_prev0(
     JNIEnv* env, jobject jobj, jlong handle) {
   reinterpret_cast<rocksdb::Iterator*>(handle)->Prev();
 }
 
 /*
- * Class:     org_rocksdb_Iterator
+ * Class:     org_rocksdb_RocksIterator
  * Method:    prev0
  * Signature: (J)V
  */
-jbyteArray Java_org_rocksdb_Iterator_key0(
+jbyteArray Java_org_rocksdb_RocksIterator_key0(
     JNIEnv* env, jobject jobj, jlong handle) {
   auto it = reinterpret_cast<rocksdb::Iterator*>(handle);
   rocksdb::Slice key_slice = it->key();
@@ -82,11 +82,11 @@ jbyteArray Java_org_rocksdb_Iterator_key0(
 }
 
 /*
- * Class:     org_rocksdb_Iterator
+ * Class:     org_rocksdb_RocksIterator
  * Method:    key0
  * Signature: (J)[B
  */
-jbyteArray Java_org_rocksdb_Iterator_value0(
+jbyteArray Java_org_rocksdb_RocksIterator_value0(
     JNIEnv* env, jobject jobj, jlong handle) {
   auto it = reinterpret_cast<rocksdb::Iterator*>(handle);
   rocksdb::Slice value_slice = it->value();
@@ -99,11 +99,11 @@ jbyteArray Java_org_rocksdb_Iterator_value0(
 }
 
 /*
- * Class:     org_rocksdb_Iterator
+ * Class:     org_rocksdb_RocksIterator
  * Method:    value0
  * Signature: (J)[B
  */
-void Java_org_rocksdb_Iterator_seek0(
+void Java_org_rocksdb_RocksIterator_seek0(
     JNIEnv* env, jobject jobj, jlong handle,
     jbyteArray jtarget, jint jtarget_len) {
   auto it = reinterpret_cast<rocksdb::Iterator*>(handle);
@@ -117,11 +117,11 @@ void Java_org_rocksdb_Iterator_seek0(
 }
 
 /*
- * Class:     org_rocksdb_Iterator
+ * Class:     org_rocksdb_RocksIterator
  * Method:    seek0
  * Signature: (J[BI)V
  */
-void Java_org_rocksdb_Iterator_status0(
+void Java_org_rocksdb_RocksIterator_status0(
     JNIEnv* env, jobject jobj, jlong handle) {
   auto it = reinterpret_cast<rocksdb::Iterator*>(handle);
   rocksdb::Status s = it->status();
@@ -134,11 +134,11 @@ void Java_org_rocksdb_Iterator_status0(
 }
 
 /*
- * Class:     org_rocksdb_Iterator
- * Method:    dispose
+ * Class:     org_rocksdb_RocksIterator
+ * Method:    disposeInternal
  * Signature: (J)V
  */
-void Java_org_rocksdb_Iterator_dispose(
+void Java_org_rocksdb_RocksIterator_disposeInternal(
     JNIEnv* env, jobject jobj, jlong handle) {
   auto it = reinterpret_cast<rocksdb::Iterator*>(handle);
   delete it;

java/rocksjni/options.cc

@@ -35,14 +35,12 @@ void Java_org_rocksdb_Options_newOptions(JNIEnv* env, jobject jobj) {
 
 /*
  * Class:     org_rocksdb_Options
- * Method:    dispose0
- * Signature: ()V
+ * Method:    disposeInternal
+ * Signature: (J)V
  */
-void Java_org_rocksdb_Options_dispose0(JNIEnv* env, jobject jobj) {
-  rocksdb::Options* op = rocksdb::OptionsJni::getHandle(env, jobj);
-  delete op;
-
-  rocksdb::OptionsJni::setHandle(env, jobj, nullptr);
+void Java_org_rocksdb_Options_disposeInternal(
+    JNIEnv* env, jobject jobj, jlong handle) {
+  delete reinterpret_cast<rocksdb::Options*>(handle);
 }
 
 /*
@@ -1665,10 +1663,10 @@ void Java_org_rocksdb_WriteOptions_newWriteOptions(
 
 /*
  * Class:     org_rocksdb_WriteOptions
- * Method:    dispose0
+ * Method:    disposeInternal
  * Signature: ()V
  */
-void Java_org_rocksdb_WriteOptions_dispose0(
+void Java_org_rocksdb_WriteOptions_disposeInternal(
     JNIEnv* env, jobject jwrite_options, jlong jhandle) {
   auto write_options = reinterpret_cast<rocksdb::WriteOptions*>(jhandle);
   delete write_options;
@@ -1732,10 +1730,10 @@ void Java_org_rocksdb_ReadOptions_newReadOptions(
 
 /*
  * Class:     org_rocksdb_ReadOptions
- * Method:    dispose
+ * Method:    disposeInternal
  * Signature: (J)V
  */
-void Java_org_rocksdb_ReadOptions_dispose(
+void Java_org_rocksdb_ReadOptions_disposeInternal(
     JNIEnv* env, jobject jobj, jlong jhandle) {
   delete reinterpret_cast<rocksdb::ReadOptions*>(jhandle);
   rocksdb::ReadOptionsJni::setHandle(env, jobj, nullptr);

java/rocksjni/portal.h

@@ -217,6 +217,7 @@ class HistogramDataJni {
     return mid;
   }
 };
+
 class BackupableDBOptionsJni {
  public:
   // Get the java class id of org.rocksdb.BackupableDBOptions.
@@ -254,7 +255,7 @@ class IteratorJni {
  public:
   // Get the java class id of org.rocksdb.Iteartor.
   static jclass getJClass(JNIEnv* env) {
-    static jclass jclazz = env->FindClass("org/rocksdb/Iterator");
+    static jclass jclazz = env->FindClass("org/rocksdb/RocksIterator");
     assert(jclazz != nullptr);
     return jclazz;
   }

java/rocksjni/restorejni.cc

@@ -0,0 +1,145 @@
+// Copyright (c) 2014, Facebook, Inc.  All rights reserved.
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree. An additional grant
+// of patent rights can be found in the PATENTS file in the same directory.
+//
+// This file implements the "bridge" between Java and C++ and enables
+// calling c++ rocksdb::RestoreBackupableDB and rocksdb::RestoreOptions methods
+// from Java side.
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <jni.h>
+#include <iostream>
+#include <string>
+
+#include "include/org_rocksdb_RestoreOptions.h"
+#include "include/org_rocksdb_RestoreBackupableDB.h"
+#include "rocksjni/portal.h"
+#include "utilities/backupable_db.h"
+/*
+ * Class:     org_rocksdb_RestoreOptions
+ * Method:    newRestoreOptions
+ * Signature: (Z)J
+ */
+jlong Java_org_rocksdb_RestoreOptions_newRestoreOptions(JNIEnv* env,
+    jobject jobj, jboolean keep_log_files) {
+  auto ropt = new rocksdb::RestoreOptions(keep_log_files);
+  return reinterpret_cast<jlong>(ropt);
+}
+
+/*
+ * Class:     org_rocksdb_RestoreOptions
+ * Method:    dispose
+ * Signature: (J)V
+ */
+void Java_org_rocksdb_RestoreOptions_dispose(JNIEnv* env, jobject jobj,
+    jlong jhandle) {
+  auto ropt = reinterpret_cast<rocksdb::RestoreOptions*>(jhandle);
+  assert(ropt);
+  delete ropt;
+}
+
+/*
+ * Class:     org_rocksdb_RestoreBackupableDB
+ * Method:    newRestoreBackupableDB
+ * Signature: (J)J
+ */
+jlong Java_org_rocksdb_RestoreBackupableDB_newRestoreBackupableDB(JNIEnv* env,
+    jobject jobj, jlong jopt_handle) {
+  auto opt = reinterpret_cast<rocksdb::BackupableDBOptions*>(jopt_handle);
+  auto rdb = new rocksdb::RestoreBackupableDB(rocksdb::Env::Default(), *opt);
+  return reinterpret_cast<jlong>(rdb);
+}
+
+/*
+ * Class:     org_rocksdb_RestoreBackupableDB
+ * Method:    restoreDBFromBackup0
+ * Signature: (JJLjava/lang/String;Ljava/lang/String;J)V
+ */
+void Java_org_rocksdb_RestoreBackupableDB_restoreDBFromBackup0(JNIEnv* env,
+    jobject jobj, jlong jhandle, jlong jbackup_id, jstring jdb_dir,
+    jstring jwal_dir, jlong jopt_handle) {
+  auto opt = reinterpret_cast<rocksdb::RestoreOptions*>(jopt_handle);
+
+  const char* cdb_dir = env->GetStringUTFChars(jdb_dir, 0);
+  const char* cwal_dir = env->GetStringUTFChars(jwal_dir, 0);
+
+  auto rdb = reinterpret_cast<rocksdb::RestoreBackupableDB*>(jhandle);
+  rocksdb::Status s =
+      rdb->RestoreDBFromBackup(jbackup_id, cdb_dir, cwal_dir, *opt);
+
+  env->ReleaseStringUTFChars(jdb_dir, cdb_dir);
+  env->ReleaseStringUTFChars(jwal_dir, cwal_dir);
+
+  if(!s.ok()) {
+    rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
+  }
+}
+
+/*
+ * Class:     org_rocksdb_RestoreBackupableDB
+ * Method:    restoreDBFromLatestBackup0
+ * Signature: (JLjava/lang/String;Ljava/lang/String;J)V
+ */
+void Java_org_rocksdb_RestoreBackupableDB_restoreDBFromLatestBackup0(
+    JNIEnv* env, jobject jobj, jlong jhandle, jstring jdb_dir, jstring jwal_dir,
+    jlong jopt_handle) {
+  auto opt = reinterpret_cast<rocksdb::RestoreOptions*>(jopt_handle);
+
+  const char* cdb_dir = env->GetStringUTFChars(jdb_dir, 0);
+  const char* cwal_dir = env->GetStringUTFChars(jwal_dir, 0);
+
+  auto rdb = reinterpret_cast<rocksdb::RestoreBackupableDB*>(jhandle);
+  rocksdb::Status s =
+      rdb->RestoreDBFromLatestBackup(cdb_dir, cwal_dir, *opt);
+
+  env->ReleaseStringUTFChars(jdb_dir, cdb_dir);
+  env->ReleaseStringUTFChars(jwal_dir, cwal_dir);
+
+  if(!s.ok()) {
+    rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
+  }
+}
+
+/*
+ * Class:     org_rocksdb_RestoreBackupableDB
+ * Method:    purgeOldBackups0
+ * Signature: (JI)V
+ */
+void Java_org_rocksdb_RestoreBackupableDB_purgeOldBackups0(JNIEnv* env,
+    jobject jobj, jlong jhandle, jint jnum_backups_to_keep) {
+  auto rdb = reinterpret_cast<rocksdb::RestoreBackupableDB*>(jhandle);
+  rocksdb::Status s = rdb->PurgeOldBackups(jnum_backups_to_keep);
+
+  if(!s.ok()) {
+    rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
+  }
+}
+
+/*
+ * Class:     org_rocksdb_RestoreBackupableDB
+ * Method:    deleteBackup0
+ * Signature: (JJ)V
+ */
+void Java_org_rocksdb_RestoreBackupableDB_deleteBackup0(JNIEnv* env,
+    jobject jobj, jlong jhandle, jlong jbackup_id) {
+  auto rdb = reinterpret_cast<rocksdb::RestoreBackupableDB*>(jhandle);
+  rocksdb::Status s = rdb->DeleteBackup(jbackup_id);
+
+  if(!s.ok()) {
+    rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
+  }
+}
+
+/*
+ * Class:     org_rocksdb_RestoreBackupableDB
+ * Method:    dispose
+ * Signature: (J)V
+ */
+void Java_org_rocksdb_RestoreBackupableDB_dispose(JNIEnv* env, jobject jobj,
+    jlong jhandle) {
+  auto ropt = reinterpret_cast<rocksdb::RestoreBackupableDB*>(jhandle);
+  assert(ropt);
+  delete ropt;
+}

java/rocksjni/rocksjni.cc

@@ -415,14 +415,12 @@ void Java_org_rocksdb_RocksDB_remove__JJ_3BI(
 
 /*
  * Class:     org_rocksdb_RocksDB
- * Method:    dispose
+ * Method:    disposeInternal
  * Signature: (J)V
  */
-void Java_org_rocksdb_RocksDB_dispose(
+void Java_org_rocksdb_RocksDB_disposeInternal(
     JNIEnv* env, jobject java_db, jlong jhandle) {
-  auto db = reinterpret_cast<rocksdb::DB*>(jhandle);
-  assert(db != nullptr);
-  delete db;
+  delete reinterpret_cast<rocksdb::DB*>(jhandle);
 }
 
 /*

java/rocksjni/write_batch.cc

@@ -134,15 +134,12 @@ void Java_org_rocksdb_WriteBatch_putLogData(
 
 /*
  * Class:     org_rocksdb_WriteBatch
- * Method:    dispose0
- * Signature: ()V
+ * Method:    disposeInternal
+ * Signature: (J)V
  */
-void Java_org_rocksdb_WriteBatch_dispose0(JNIEnv* env, jobject jobj) {
-  rocksdb::WriteBatch* wb = rocksdb::WriteBatchJni::getHandle(env, jobj);
-  assert(wb != nullptr);
-  delete wb;
-
-  rocksdb::WriteBatchJni::setHandle(env, jobj, nullptr);
+void Java_org_rocksdb_WriteBatch_disposeInternal(
+    JNIEnv* env, jobject jobj, jlong handle) {
+  delete reinterpret_cast<rocksdb::WriteBatch*>(handle);
 }
 
 /*