Dbid feature

Summary:
Create a new type of file on startup if it doesn't already exist called DBID.
This will store a unique number generated from boost library's uuid header file.
The use-case is to identify the case of a db losing all its data and coming back up either empty or from an image(backup/live replica's recovery)
the key point to note is that DBID is not stored in a backup or db snapshot
It's preferable to use Boost for uuid because:
1) A non-standard way of generating uuid is not good
2) /proc/sys/kernel/random/uuid generates a uuid but only on linux environments and the solution would not be clean
3) c++ doesn't have any direct way to get a uuid
4) Boost is a very good library that was already having linkage in rocksdb from third-party
Note: I had to update the TOOLCHAIN_REV in build files to get latest verison of boost from third-party as the older version had a bug.
I had to put Wno-uninitialized in Makefile because boost-1.51 has an unitialized variable and rocksdb would not comiple otherwise. Latet open-source for boost is 1.54 but is not there in third-party. I have notified the concerned people in fbcode about it.
@kailiu : While releasing to third-party, an additional dependency will need to be created for boost in TARGETS file. I can help identify.

Test Plan:
Expand db_test to test 2 cases
1) Restarting db with Id file present - verify that no change to Id
2)Restarting db with Id file deleted - verify that a different Id is there after reopen
Also run make all check

Reviewers: dhruba, haobo, kailiu, sdong

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D13587

LICENSE

@@ -1,3 +1,11 @@
+BSD License
+
+For rocksdb software
+
+Copyright (c) 2013, Facebook, Inc.
+All rights reserved.
+---------------------------------------------------------------------
+
 Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

Makefile

@@ -48,6 +48,7 @@ TESTS = \
 	db_test \
 	dbformat_test \
 	env_test \
+	blob_store_test \
 	filelock_test \
 	filename_test \
 	filter_block_test \
@@ -156,7 +157,7 @@ valgrind_check: all $(PROGRAMS) $(TESTS)
 	done
 
 clean:
-	-rm -f $(PROGRAMS) $(BENCHMARKS) $(LIBRARY) $(SHARED) $(MEMENVLIBRARY) $(THRIFTSERVER) build_config.mk
+	-rm -f $(PROGRAMS) $(BENCHMARKS) $(LIBRARY) $(SHARED) $(MEMENVLIBRARY) build_config.mk
 	-rm -rf ios-x86/* ios-arm/*
 	-find . -name "*.[od]" | xargs --no-run-if-empty rm
 	-find . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" | xargs --no-run-if-empty rm
@@ -204,6 +205,9 @@ cache_test: util/cache_test.o $(LIBOBJECTS) $(TESTHARNESS)
 coding_test: util/coding_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(CXX) util/coding_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
 
+blob_store_test: util/blob_store_test.o $(LIBOBJECTS) $(TESTHARNESS) $(TESTUTIL)
+	$(CXX) util/blob_store_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o$@ $(LDFLAGS) $(COVERAGEFLAGS)
+
 stringappend_test: utilities/merge_operators/string_append/stringappend_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(CXX) utilities/merge_operators/string_append/stringappend_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
 

PATENTS

@@ -0,0 +1,23 @@
+Additional Grant of Patent Rights
+
+“Software” means the rocksdb software distributed by Facebook, Inc.
+
+Facebook hereby grants you a perpetual, worldwide, royalty-free,
+non-exclusive, irrevocable (subject to the termination provision below)
+license under any rights in any patent claims owned by Facebook, to make,
+have made, use, sell, offer to sell, import, and otherwise transfer the
+Software. For avoidance of doubt, no license is granted under Facebook’s
+rights in any patent claims that are infringed by (i) modifications to the
+Software made by you or a third party, or (ii) the Software in combination
+with any software or other technology provided by you or a third party.
+
+The license granted hereunder will terminate, automatically and without
+notice, for anyone that makes any claim (including by filing any lawsuit,
+assertion or other action) alleging (a) direct, indirect, or contributory
+infringement or inducement to infringe any patent: (i) by Facebook or any
+of its subsidiaries or affiliates, whether or not such claim is related
+to the Software, (ii) by any party if such claim arises in whole or in
+part from any software, product or service of Facebook or any of its
+subsidiaries or affiliates, whether or not such claim is related to the
+Software, or (iii) by any party relating to the Software; or (b) that
+any right in any patent claim of Facebook is invalid or unenforceable.

README

@@ -1,6 +1,6 @@
 rocksdb: A persistent key-value store for flash storage
 Authors: * The Facebook Database Engineering Team
-         * Build on earlier work on leveldb by Sanjay Ghemawat 
+         * Build on earlier work on leveldb by Sanjay Ghemawat
            (sanjay@google.com) and Jeff Dean (jeff@google.com)
 
 This code is a library that forms the core building block for a fast
@@ -19,64 +19,64 @@ persistent key/value store.
 See doc/index.html for more explanation.
 See doc/impl.html for a brief overview of the implementation.
 
-The public interface is in include/*.h.  Callers should not include or
+The public interface is in include/*.  Callers should not include or
 rely on the details of any other header files in this package.  Those
 internal APIs may be changed without warning.
 
 Guide to header files:
 
-include/db.h
+include/rocksdb/db.h
     Main interface to the DB: Start here
 
-include/options.h
+include/rocksdb/options.h
     Control over the behavior of an entire database, and also
     control over the behavior of individual reads and writes.
 
-include/comparator.h
+include/rocksdb/comparator.h
     Abstraction for user-specified comparison function.  If you want
     just bytewise comparison of keys, you can use the default comparator,
     but clients can write their own comparator implementations if they
     want custom ordering (e.g. to handle different character
     encodings, etc.)
 
-include/iterator.h
+include/rocksdb/iterator.h
     Interface for iterating over data. You can get an iterator
     from a DB object.
 
-include/write_batch.h
+include/rocksdb/write_batch.h
     Interface for atomically applying multiple updates to a database.
 
-include/slice.h
+include/rocksdb/slice.h
     A simple module for maintaining a pointer and a length into some
     other byte array.
 
-include/status.h
+include/rocksdb/status.h
     Status is returned from many of the public interfaces and is used
     to report success and various kinds of errors.
 
-include/env.h
+include/rocksdb/env.h
     Abstraction of the OS environment.  A posix implementation of
     this interface is in util/env_posix.cc
 
-include/table_builder.h
+include/rocksdb/table_builder.h
     Lower-level modules that most clients probably won't use directly
 
-include/cache.h
+include/rocksdb/cache.h
     An API for the block cache.
 
-include/compaction_filter.h
+include/rocksdb/compaction_filter.h
     An API for a application filter invoked on every compaction.
 
-include/filter_policy.h
+include/rocksdb/filter_policy.h
     An API for configuring a bloom filter.
 
-include/memtablerep.h
+include/rocksdb/memtablerep.h
     An API for implementing a memtable.
 
-include/statistics.h
+include/rocksdb/statistics.h
     An API to retrieve various database statistics.
 
-include/transaction_log_iterator.h
+include/rocksdb/transaction_log.h
     An API to retrieve transaction logs from a database.
 
 

README.fb

@@ -1,3 +1,3 @@
 * Detailed instructions on how to compile using fbcode and jemalloc
 
-* Latest release is 2.3.fb
+* Latest release is 2.4.fb

build_tools/build_detect_platform

@@ -134,14 +134,6 @@ $PWD/build_tools/build_detect_version
 # of all files matching either rule, so we need to append -print to make the
 # prune take effect.
 DIRS="util db table utilities"
-if test "$USE_THRIFT"; then
-  DIRS="$DIRS thrift/server_utils.cpp thrift/gen-cpp "
-  THRIFTSERVER=leveldb_server
-fi
-
-if test "$USE_SCRIBE"; then
-  DIRS="$DIRS scribe "
-fi
 
 set -f # temporarily disable globbing so that our patterns arent expanded
 PRUNE_TEST="-name *test*.cc -prune"
@@ -227,14 +219,6 @@ if test "$USE_HDFS"; then
   PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS $HDFS_LDFLAGS"
 fi
 
-# shall we build thrift server or scribe logger
-if test "$USE_THRIFT" ||  test "$USE_SCRIBE" ; then
-    THRIFT_CCFLAGS=" -I./thrift -I./thrift/gen-cpp -I./thrift/lib/cpp -I/usr/include -std=gnu++0x"
-    THRIFT_LDFLAGS=" -lexample -lserver -lthrift_base -ltransport -lthrift_exception -lutil -L./thrift/libs "
-    COMMON_FLAGS="$COMMON_FLAGS $THRIFT_CCFLAGS"
-    PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS $THRIFT_LDFLAGS"
-fi
-
 # if Intel SSE instruction set is supported, set USE_SSE=" -msse -msse4.2 "
 COMMON_FLAGS="$COMMON_FLAGS $USE_SSE"
 
@@ -254,5 +238,4 @@ echo "PLATFORM_SHARED_CFLAGS=$PLATFORM_SHARED_CFLAGS" >> $OUTPUT
 echo "PLATFORM_SHARED_EXT=$PLATFORM_SHARED_EXT" >> $OUTPUT
 echo "PLATFORM_SHARED_LDFLAGS=$PLATFORM_SHARED_LDFLAGS" >> $OUTPUT
 echo "PLATFORM_SHARED_VERSIONED=$PLATFORM_SHARED_VERSIONED" >> $OUTPUT
-echo "THRIFTSERVER=$THRIFTSERVER" >> $OUTPUT
 echo "EXEC_LDFLAGS=$EXEC_LDFLAGS" >> $OUTPUT

build_tools/build_detect_version

@@ -20,7 +20,7 @@ if [ "$?" = 0 ]; then
   BEGIN {
     print "#include \"build_version.h\"\n"
   }
-  { print "const char* leveldb_build_git_sha = \"leveldb_build_git_sha:" $0"\";" }
+  { print "const char* rocksdb_build_git_sha = \"rocksdb_build_git_sha:" $0"\";" }
   ' >  ${VFILE}
 else
   echo "git not found" |
@@ -28,13 +28,13 @@ else
     BEGIN {
       print "#include \"build_version.h\""
     }
-    { print "const char* leveldb_build_git_sha = \"leveldb_build_git_sha:git not found\";" }
+    { print "const char* rocksdb_build_git_sha = \"rocksdb_build_git_sha:git not found\";" }
     ' > ${VFILE}
 fi
 
-echo "const char* leveldb_build_git_datetime = \"leveldb_build_git_datetime:$(date)\";" >> ${VFILE}
-echo "const char* leveldb_build_compile_date = __DATE__;" >> ${VFILE}
-echo "const char* leveldb_build_compile_time = __TIME__;" >> ${VFILE}
+echo "const char* rocksdb_build_git_datetime = \"rocksdb_build_git_datetime:$(date)\";" >> ${VFILE}
+echo "const char* rocksdb_build_compile_date = __DATE__;" >> ${VFILE}
+echo "const char* rocksdb_build_compile_time = __TIME__;" >> ${VFILE}
 
 OUTFILE=$ROCKSDB_ROOT/util/build_version.cc
 if [ ! -e $OUTFILE ] || ! cmp -s $VFILE $OUTFILE; then

build_tools/build_java.sh

@@ -1,86 +0,0 @@
-#!/bin/bash -e
-function print_usage {
-  echo "Usage: $0 [build | release | bump_version version ]"
-}
-
-function check_env {
-  if [ -z $LEVELDB_HOME ] ; then
-    echo "Expect LEVELDB_HOME to be SET"
-    exit 1
-  fi
-  if [ -z $JAVA_HOME ] ; then
-    export JAVA_HOME=/usr/local/jdk-6u14-64/
-    echo "JAVA_HOME not set. Assuming JAVA_HOME=$JAVA_HOME"
-  fi
-  if [ -z $LEVELDBJNI_HOME ] ; then
-    export LEVELDBJNI_HOME=$LEVELDB_HOME/java/leveldbjni/
-    echo "LEVELDBJNI_HOME not set. Assuming LEVELDBJNI_HOME=$LEVELDBJNI_HOME"
-  fi
-  if [ -z $SNAPPY_HOME ] ; then
-    export SNAPPY_HOME="/home/dhruba/snappy-1.0.5"
-    echo "SNAPPY_HOME not set. Assuming SNAPPY_HOME=$SNAPPY_HOME"
-  fi
-  if [ -z $LEVELDB_PATCH ] ; then 
-    LEVELDB_PATCH=$LEVELDB_HOME/java/leveldbjni/db.h.patch
-    echo "LEVELDB_PATCH not set. Assuming LEVELDB_PATCH=$LEVELDB_PATCH"
-  fi
-}
-
-function build {
-  cd $LEVELDB_HOME
-  git apply $LEVELDB_PATCH
-  make clean libleveldb.a
-  cd $LEVELDB_HOME/java/leveldb/leveldb-api
-  mvn clean package
-  cd $LEVELDBJNI_HOME
-  mvn clean install -P linux64
-  cd $LEVELDB_HOME
-  git checkout $LEVELDB_HOME/include/leveldb/db.h
-}
-
-function release {
-  cd $LEVELDB_HOME
-  git apply $LEVELDB_PATCH
-  make clean libleveldb.a
-  cd $LEVELDB_HOME/java/leveldb/leveldb-api
-  mvn clean package
-  cd $LEVELDBJNI_HOME
-  mvn deploy -P linux64 -DskipTests
-  cd $LEVELDB_HOME
-  git checkout $LEVELDB_HOME/include/leveldb/db.h
-}
-
-CMD=$1
-if [ -z $CMD ]; then
-  print_usage
-  exit 1
-fi
-
-case "$CMD" in 
-  build) 
-    check_env
-    build
-    ;;
-  bump_version)
-    if [ -z $2 ] ; then
-      echo "bump_version requires a version.no parameter at the end"
-      exit 1
-    fi 
-    check_env
-    VERSION=$2
-    pushd $LEVELDBJNI_HOME
-    mvn versions:set -DnewVersion="$VERSION"
-    popd
-    echo "VERSION SET TO $VERSION"
-    ;;
-  release)
-    check_env
-    pushd $LEVELDBJNI_HOME
-    release
-    popd
-    ;;
-  *)
-    print_usage
-    exit 1
-    ;;
-esac

build_tools/fbcode.clang31.sh

@@ -4,21 +4,16 @@
 # fbcode settings.  It uses the latest g++ compiler and also
 # uses jemalloc
 
-TOOLCHAIN_REV=f365dbeae46a30414a2874a6f45e73e10f1caf7d
+TOOLCHAIN_REV=fbe3b095a4cc4a3713730050d182b7b4a80c342f
 TOOLCHAIN_EXECUTABLES="/mnt/gvfs/third-party/$TOOLCHAIN_REV/centos5.2-native"
 TOOLCHAIN_LIB_BASE="/mnt/gvfs/third-party/$TOOLCHAIN_REV/gcc-4.7.1-glibc-2.14.1"
-TOOL_JEMALLOC=jemalloc-3.0.0/2f45f3a
+TOOL_JEMALLOC=jemalloc-3.3.1/9202ce3
 GLIBC_RUNTIME_PATH=/usr/local/fbcode/gcc-4.7.1-glibc-2.14.1
 
 # location of snappy headers and libraries
 SNAPPY_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/snappy/snappy-1.0.3/7518bbe/include"
 SNAPPY_LIBS=" $TOOLCHAIN_LIB_BASE/snappy/snappy-1.0.3/7518bbe/lib/libsnappy.a"
 
-# location of boost headers and libraries
-THRIFT_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/boost/boost-1.48.0/bef9365/include -std=gnu++0x"
-THRIFT_INCLUDE+=" -I./thrift -I./thrift/gen-cpp -I./thrift/lib/cpp"
-THRIFT_LIBS=" -L $TOOLCHAIN_LIB_BASE/boost/boost-1.48.0/bef9365/lib"
-
 # location of libevent
 LIBEVENT_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/include"
 LIBEVENT_LIBS=" -L $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/lib"
@@ -27,7 +22,7 @@ LIBEVENT_LIBS=" -L $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/lib"
 export USE_SSE=" -msse -msse4.2 "
 
 CC="$TOOLCHAIN_EXECUTABLES/clang/clang-3.1/6ca8a59/bin/clang $CLANG_INCLUDES"
-CXX="$TOOLCHAIN_EXECUTABLES/clang/clang-3.1/6ca8a59/bin/clang++ $CLANG_INCLUDES $JINCLUDE $SNAPPY_INCLUDE $THRIFT_INCLUDE $LIBEVENT_INCLUDE"
+CXX="$TOOLCHAIN_EXECUTABLES/clang/clang-3.1/6ca8a59/bin/clang++ $CLANG_INCLUDES $JINCLUDE $SNAPPY_INCLUDE $LIBEVENT_INCLUDE"
 AR=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ar
 RANLIB=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ranlib
 
@@ -37,8 +32,8 @@ CXXFLAGS="$CFLAGS -nostdinc++ -std=gnu++0x"
 CFLAGS+=" -I $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/include -DHAVE_JEMALLOC"
 
 EXEC_LDFLAGS=" -Wl,--whole-archive $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/lib/libjemalloc.a"
-EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/91ddd43/lib/libunwind.a"
-EXEC_LDFLAGS+=" $HDFSLIB $SNAPPY_LIBS $THRIFT_LIBS $LIBEVENT_LIBS"
+EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/350336c/lib/libunwind.a"
+EXEC_LDFLAGS+=" $HDFSLIB $SNAPPY_LIBS $LIBEVENT_LIBS"
 EXEC_LDFLAGS+=" -Wl,--dynamic-linker,$GLIBC_RUNTIME_PATH/lib/ld-linux-x86-64.so.2"
 EXEC_LDFLAGS+=" -B$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin"
 

build_tools/fbcode.gcc471.sh

@@ -4,10 +4,10 @@
 # fbcode settings.  It uses the latest g++ compiler and also
 # uses jemalloc
 
-TOOLCHAIN_REV=83eb773e262fa705eaebbf3de40db71d53fabf2e
+TOOLCHAIN_REV=fbe3b095a4cc4a3713730050d182b7b4a80c342f
 TOOLCHAIN_EXECUTABLES="/mnt/gvfs/third-party/$TOOLCHAIN_REV/centos5.2-native"
 TOOLCHAIN_LIB_BASE="/mnt/gvfs/third-party/$TOOLCHAIN_REV/gcc-4.7.1-glibc-2.14.1"
-TOOL_JEMALLOC=jemalloc-3.3.1/2f45f3a
+TOOL_JEMALLOC=jemalloc-3.3.1/9202ce3
 
 # location of libhdfs libraries
 if test "$USE_HDFS"; then
@@ -27,11 +27,6 @@ SNAPPY_LIBS=" $TOOLCHAIN_LIB_BASE/snappy/snappy-1.0.3/7518bbe/lib/libsnappy.a"
 ZLIB_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/zlib/zlib-1.2.5/91ddd43/include"
 ZLIB_LIBS=" $TOOLCHAIN_LIB_BASE/zlib/zlib-1.2.5/91ddd43/lib/libz.a"
 
-# location of boost headers and libraries
-THRIFT_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/boost/boost-1.48.0/2a0840d/include"
-THRIFT_INCLUDE+=" -I./thrift -I./thrift/gen-cpp -I./thrift/lib/cpp"
-THRIFT_LIBS=" -L $TOOLCHAIN_LIB_BASE/boost/boost-1.48.0/2a0840d/lib"
-
 # location of libevent
 LIBEVENT_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/include"
 LIBEVENT_LIBS=" -L $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/lib"
@@ -40,7 +35,7 @@ LIBEVENT_LIBS=" -L $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/lib"
 export USE_SSE=" -msse -msse4.2 "
 
 CC="$TOOLCHAIN_EXECUTABLES/gcc/gcc-4.7.1-glibc-2.14.1/bin/gcc"
-CXX="$TOOLCHAIN_EXECUTABLES/gcc/gcc-4.7.1-glibc-2.14.1/bin/g++ $JINCLUDE $SNAPPY_INCLUDE $ZLIB_INCLUDE $THRIFT_INCLUDE $LIBEVENT_INCLUDE"
+CXX="$TOOLCHAIN_EXECUTABLES/gcc/gcc-4.7.1-glibc-2.14.1/bin/g++ $JINCLUDE $SNAPPY_INCLUDE $ZLIB_INCLUDE $LIBEVENT_INCLUDE"
 AR=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ar
 RANLIB=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ranlib
 
@@ -48,8 +43,8 @@ CFLAGS="-B$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/bin/gold -m64 -mtune=g
 CFLAGS+=" -I $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/include -DHAVE_JEMALLOC"
 
 EXEC_LDFLAGS=" -Wl,--whole-archive $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/lib/libjemalloc.a"
-EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/91ddd43/lib/libunwind.a"
-EXEC_LDFLAGS+=" $HDFSLIB $SNAPPY_LIBS $ZLIB_LIBS $THRIFT_LIBS $LIBEVENT_LIBS"
+EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/350336c/lib/libunwind.a"
+EXEC_LDFLAGS+=" $HDFSLIB $SNAPPY_LIBS $ZLIB_LIBS $LIBEVENT_LIBS"
 
 PLATFORM_LDFLAGS="-L$TOOLCHAIN_LIB_BASE/libgcc/libgcc-4.7.1/afc21dc/lib -L$TOOLCHAIN_LIB_BASE/glibc/glibc-2.14.1/99df8fc/lib"
 

db/builder.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -14,7 +19,7 @@
 #include "rocksdb/iterator.h"
 #include "util/stop_watch.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 Status BuildTable(const std::string& dbname,
                   Env* env,
@@ -25,7 +30,8 @@ Status BuildTable(const std::string& dbname,
                   FileMetaData* meta,
                   const Comparator* user_comparator,
                   const SequenceNumber newest_snapshot,
-                  const SequenceNumber earliest_seqno_in_memtable) {
+                  const SequenceNumber earliest_seqno_in_memtable,
+                  const bool enable_compression) {
   Status s;
   meta->file_size = 0;
   meta->smallest_seqno = meta->largest_seqno = 0;
@@ -46,7 +52,8 @@ Status BuildTable(const std::string& dbname,
     if (!s.ok()) {
       return s;
     }
-    TableBuilder* builder = new TableBuilder(options, file.get(), 0);
+    TableBuilder* builder = new TableBuilder(options, file.get(), 0,
+                                             enable_compression);
 
     // the first key is the smallest key
     Slice key = iter->key();
@@ -204,4 +211,4 @@ Status BuildTable(const std::string& dbname,
   return s;
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/builder.h

@@ -1,15 +1,16 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
-
-#ifndef STORAGE_LEVELDB_DB_BUILDER_H_
-#define STORAGE_LEVELDB_DB_BUILDER_H_
-
+#pragma once
 #include "rocksdb/comparator.h"
 #include "rocksdb/status.h"
 #include "rocksdb/types.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 struct Options;
 struct FileMetaData;
@@ -34,8 +35,7 @@ extern Status BuildTable(const std::string& dbname,
                          FileMetaData* meta,
                          const Comparator* user_comparator,
                          const SequenceNumber newest_snapshot,
-                         const SequenceNumber earliest_seqno_in_memtable);
+                         const SequenceNumber earliest_seqno_in_memtable,
+                         const bool enable_compression);
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_BUILDER_H_
+}  // namespace rocksdb

db/c.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -16,28 +21,28 @@
 #include "rocksdb/status.h"
 #include "rocksdb/write_batch.h"
 
-using leveldb::Cache;
-using leveldb::Comparator;
-using leveldb::CompressionType;
-using leveldb::DB;
-using leveldb::Env;
-using leveldb::FileLock;
-using leveldb::FilterPolicy;
-using leveldb::Iterator;
-using leveldb::Logger;
-using leveldb::NewBloomFilterPolicy;
-using leveldb::NewLRUCache;
-using leveldb::Options;
-using leveldb::RandomAccessFile;
-using leveldb::Range;
-using leveldb::ReadOptions;
-using leveldb::SequentialFile;
-using leveldb::Slice;
-using leveldb::Snapshot;
-using leveldb::Status;
-using leveldb::WritableFile;
-using leveldb::WriteBatch;
-using leveldb::WriteOptions;
+using rocksdb::Cache;
+using rocksdb::Comparator;
+using rocksdb::CompressionType;
+using rocksdb::DB;
+using rocksdb::Env;
+using rocksdb::FileLock;
+using rocksdb::FilterPolicy;
+using rocksdb::Iterator;
+using rocksdb::Logger;
+using rocksdb::NewBloomFilterPolicy;
+using rocksdb::NewLRUCache;
+using rocksdb::Options;
+using rocksdb::RandomAccessFile;
+using rocksdb::Range;
+using rocksdb::ReadOptions;
+using rocksdb::SequentialFile;
+using rocksdb::Slice;
+using rocksdb::Snapshot;
+using rocksdb::Status;
+using rocksdb::WritableFile;
+using rocksdb::WriteBatch;
+using rocksdb::WriteOptions;
 
 using std::shared_ptr;
 

db/c_test.c

@@ -296,7 +296,7 @@ int main(int argc, char** argv) {
   {
     char* prop = leveldb_property_value(db, "nosuchprop");
     CheckCondition(prop == NULL);
-    prop = leveldb_property_value(db, "leveldb.stats");
+    prop = leveldb_property_value(db, "rocksdb.stats");
     CheckCondition(prop != NULL);
     Free(&prop);
   }

db/corruption_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -20,7 +25,7 @@
 #include "util/testharness.h"
 #include "util/testutil.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 static const int kValueSize = 1000;
 
@@ -68,7 +73,7 @@ class CorruptionTest {
   void RepairDB() {
     delete db_;
     db_ = nullptr;
-    ASSERT_OK(::leveldb::RepairDB(dbname_, options_));
+    ASSERT_OK(::rocksdb::RepairDB(dbname_, options_));
   }
 
   void Build(int n) {
@@ -230,7 +235,7 @@ TEST(CorruptionTest, NewFileErrorDuringWrite) {
 TEST(CorruptionTest, TableFile) {
   Build(100);
   DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
   dbi->TEST_CompactRange(0, nullptr, nullptr);
   dbi->TEST_CompactRange(1, nullptr, nullptr);
 
@@ -241,7 +246,7 @@ TEST(CorruptionTest, TableFile) {
 TEST(CorruptionTest, TableFileIndexData) {
   Build(10000);  // Enough to build multiple Tables
   DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
 
   Corrupt(kTableFile, -2000, 500);
   Reopen();
@@ -279,7 +284,7 @@ TEST(CorruptionTest, SequenceNumberRecovery) {
 TEST(CorruptionTest, CorruptedDescriptor) {
   ASSERT_OK(db_->Put(WriteOptions(), "foo", "hello"));
   DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
   dbi->TEST_CompactRange(0, nullptr, nullptr);
 
   Corrupt(kDescriptorFile, 0, 1000);
@@ -296,9 +301,9 @@ TEST(CorruptionTest, CorruptedDescriptor) {
 TEST(CorruptionTest, CompactionInputError) {
   Build(10);
   DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
   const int last = dbi->MaxMemCompactionLevel();
-  ASSERT_EQ(1, Property("leveldb.num-files-at-level" + NumberToString(last)));
+  ASSERT_EQ(1, Property("rocksdb.num-files-at-level" + NumberToString(last)));
 
   Corrupt(kTableFile, 100, 1);
   Check(9, 9);
@@ -319,12 +324,13 @@ TEST(CorruptionTest, CompactionInputErrorParanoid) {
   for (int level = 1; level < dbi->NumberLevels(); level++) {
     dbi->Put(WriteOptions(), "", "begin");
     dbi->Put(WriteOptions(), "~", "end");
-    dbi->TEST_CompactMemTable();
+    dbi->TEST_FlushMemTable();
   }
 
   Build(10);
-  dbi->TEST_CompactMemTable();
-  ASSERT_EQ(1, Property("leveldb.num-files-at-level0"));
+  dbi->TEST_FlushMemTable();
+  dbi->TEST_WaitForCompact();
+  ASSERT_EQ(1, Property("rocksdb.num-files-at-level0"));
 
   Corrupt(kTableFile, 100, 1);
   Check(9, 9);
@@ -341,7 +347,7 @@ TEST(CorruptionTest, CompactionInputErrorParanoid) {
 TEST(CorruptionTest, UnrelatedKeys) {
   Build(10);
   DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
   Corrupt(kTableFile, 100, 1);
 
   std::string tmp1, tmp2;
@@ -349,13 +355,13 @@ TEST(CorruptionTest, UnrelatedKeys) {
   std::string v;
   ASSERT_OK(db_->Get(ReadOptions(), Key(1000, &tmp1), &v));
   ASSERT_EQ(Value(1000, &tmp2).ToString(), v);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
   ASSERT_OK(db_->Get(ReadOptions(), Key(1000, &tmp1), &v));
   ASSERT_EQ(Value(1000, &tmp2).ToString(), v);
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }

db/db_bench.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -13,6 +18,7 @@
 #include "rocksdb/cache.h"
 #include "rocksdb/db.h"
 #include "rocksdb/env.h"
+#include "rocksdb/memtablerep.h"
 #include "rocksdb/write_batch.h"
 #include "rocksdb/statistics.h"
 #include "port/port.h"
@@ -79,6 +85,7 @@ static const char* FLAGS_benchmarks =
     "snappycomp,"
     "snappyuncomp,"
     "acquireload,"
+    "fillfromstdin,"
     ;
 // the maximum size of key in bytes
 static const int MAX_KEY_SIZE = 128;
@@ -151,15 +158,21 @@ static int FLAGS_min_write_buffer_number_to_merge = 0;
 static int FLAGS_max_background_compactions = 0;
 
 // style of compaction: level-based vs universal
-static leveldb::CompactionStyle FLAGS_compaction_style = leveldb::kCompactionStyleLevel;
+static rocksdb::CompactionStyle FLAGS_compaction_style = rocksdb::kCompactionStyleLevel;
 
 // Percentage flexibility while comparing file size
 // (for universal compaction only).
-static int FLAGS_universal_size_ratio = 1;
+static int FLAGS_universal_size_ratio = 0;
 
 // The minimum number of files in a single compaction run
 // (for universal compaction only).
-static int FLAGS_compaction_universal_min_merge_width = 2;
+static int FLAGS_universal_min_merge_width = 0;
+
+// The max number of files to compact in universal style compaction
+static unsigned int FLAGS_universal_max_merge_width = 0;
+
+// The max size amplification for universal style compaction
+static unsigned int FLAGS_universal_max_size_amplification_percent = 0;
 
 // Number of bytes to use as a cache of uncompressed data.
 // Negative means use default settings.
@@ -188,12 +201,14 @@ static const char* FLAGS_db = nullptr;
 // if FLAGS_cache_size is non-negative.
 static int FLAGS_cache_numshardbits = -1;
 
+static int FLAGS_cache_remove_scan_count_limit = 32;
+
 // Verify checksum for every block read from storage
 static bool FLAGS_verify_checksum = false;
 
 // Database statistics
 static bool FLAGS_statistics = false;
-static class std::shared_ptr<leveldb::Statistics> dbstats;
+static class std::shared_ptr<rocksdb::Statistics> dbstats;
 
 // Number of write operations to do.  If negative, do FLAGS_num reads.
 static long FLAGS_writes = -1;
@@ -271,8 +286,8 @@ static int FLAGS_disable_seek_compaction = false;
 static uint64_t FLAGS_delete_obsolete_files_period_micros = 0;
 
 // Algorithm used to compress the database
-static enum leveldb::CompressionType FLAGS_compression_type =
-    leveldb::kSnappyCompression;
+static enum rocksdb::CompressionType FLAGS_compression_type =
+    rocksdb::kSnappyCompression;
 
 // If non-negative, compression starts from this level. Levels with number
 // < FLAGS_min_level_to_compress are not compressed.
@@ -282,7 +297,7 @@ static int FLAGS_min_level_to_compress = -1;
 static int FLAGS_table_cache_numshardbits = 4;
 
 // posix or hdfs environment
-static leveldb::Env* FLAGS_env = leveldb::Env::Default();
+static rocksdb::Env* FLAGS_env = rocksdb::Env::Default();
 
 // Stats are reported every N operations when this is greater
 // than zero. When 0 the interval grows over time.
@@ -331,11 +346,11 @@ static bool FLAGS_use_mmap_writes;
 
 // Advise random access on table file open
 static bool FLAGS_advise_random_on_open =
-  leveldb::Options().advise_random_on_open;
+  rocksdb::Options().advise_random_on_open;
 
 // Access pattern advice when a file is compacted
 static auto FLAGS_compaction_fadvice =
-  leveldb::Options().access_hint_on_compaction_start;
+  rocksdb::Options().access_hint_on_compaction_start;
 
 // Use multiget to access a series of keys instead of get
 static bool FLAGS_use_multiget = false;
@@ -350,13 +365,13 @@ static bool FLAGS_warn_missing_keys = true;
 
 // Use adaptive mutex
 static auto FLAGS_use_adaptive_mutex =
-  leveldb::Options().use_adaptive_mutex;
+  rocksdb::Options().use_adaptive_mutex;
 
 // Allows OS to incrementally sync files to disk while they are being
 // written, in the background. Issue one request for every bytes_per_sync
 // written. 0 turns it off.
 static auto FLAGS_bytes_per_sync =
-  leveldb::Options().bytes_per_sync;
+  rocksdb::Options().bytes_per_sync;
 
 // On true, deletes use bloom-filter and drop the delete if key not present
 static bool FLAGS_filter_deletes = false;
@@ -378,7 +393,10 @@ static enum RepFactory FLAGS_rep_factory;
 // The possible merge operators are defined in utilities/merge_operators.h
 static std::string FLAGS_merge_operator = "";
 
-namespace leveldb {
+static auto FLAGS_purge_log_after_memtable_flush =
+  rocksdb::Options().purge_log_after_memtable_flush;
+
+namespace rocksdb {
 
 // Helper for quickly generating random data.
 class RandomGenerator {
@@ -535,7 +553,7 @@ class Stats {
 
         if (FLAGS_stats_per_interval) {
           std::string stats;
-          if (db && db->GetProperty("leveldb.stats", &stats))
+          if (db && db->GetProperty("rocksdb.stats", &stats))
             fprintf(stderr, "%s\n", stats.c_str());
         }
 
@@ -680,16 +698,16 @@ class Benchmark {
     fprintf(stdout, "Write rate limit: %d\n", FLAGS_writes_per_second);
 
     switch (FLAGS_compression_type) {
-      case leveldb::kNoCompression:
+      case rocksdb::kNoCompression:
         fprintf(stdout, "Compression: none\n");
         break;
-      case leveldb::kSnappyCompression:
+      case rocksdb::kSnappyCompression:
         fprintf(stdout, "Compression: snappy\n");
         break;
-      case leveldb::kZlibCompression:
+      case rocksdb::kZlibCompression:
         fprintf(stdout, "Compression: zlib\n");
         break;
-      case leveldb::kBZip2Compression:
+      case rocksdb::kBZip2Compression:
         fprintf(stdout, "Compression: bzip2\n");
         break;
     }
@@ -724,7 +742,7 @@ class Benchmark {
             "WARNING: Assertions are enabled; benchmarks unnecessarily slow\n");
 #endif
 
-    if (FLAGS_compression_type != leveldb::kNoCompression) {
+    if (FLAGS_compression_type != rocksdb::kNoCompression) {
       // The test string should not be too small.
       const int len = FLAGS_block_size;
       char* text = (char*) malloc(len+1);
@@ -805,7 +823,8 @@ class Benchmark {
   Benchmark()
   : cache_(FLAGS_cache_size >= 0 ?
            (FLAGS_cache_numshardbits >= 1 ?
-            NewLRUCache(FLAGS_cache_size, FLAGS_cache_numshardbits) :
+            NewLRUCache(FLAGS_cache_size, FLAGS_cache_numshardbits,
+                        FLAGS_cache_remove_scan_count_limit) :
             NewLRUCache(FLAGS_cache_size)) : nullptr),
     filter_policy_(FLAGS_bloom_bits >= 0
                    ? NewBloomFilterPolicy(FLAGS_bloom_bits)
@@ -897,6 +916,9 @@ class Benchmark {
       } else if (name == Slice("fillrandom")) {
         fresh_db = true;
         method = &Benchmark::WriteRandom;
+      } else if (name == Slice("fillfromstdin")) {
+        fresh_db = true;
+        method = &Benchmark::WriteFromStdin;
       } else if (name == Slice("filluniquerandom")) {
         fresh_db = true;
         if (num_threads > 1) {
@@ -971,11 +993,11 @@ class Benchmark {
       } else if (name == Slice("heapprofile")) {
         HeapProfile();
       } else if (name == Slice("stats")) {
-        PrintStats("leveldb.stats");
+        PrintStats("rocksdb.stats");
       } else if (name == Slice("levelstats")) {
-        PrintStats("leveldb.levelstats");
+        PrintStats("rocksdb.levelstats");
       } else if (name == Slice("sstables")) {
-        PrintStats("leveldb.sstables");
+        PrintStats("rocksdb.sstables");
       } else {
         if (name != Slice()) {  // No error message for empty name
           fprintf(stderr, "unknown benchmark '%s'\n", name.ToString().c_str());
@@ -1182,9 +1204,6 @@ class Benchmark {
       FLAGS_min_write_buffer_number_to_merge;
     options.max_background_compactions = FLAGS_max_background_compactions;
     options.compaction_style = FLAGS_compaction_style;
-    options.compaction_options_universal.size_ratio = FLAGS_universal_size_ratio;
-    options.compaction_options_universal.min_merge_width =
-      FLAGS_compaction_universal_min_merge_width;
     options.block_size = FLAGS_block_size;
     options.filter_policy = filter_policy_;
     options.prefix_extractor = FLAGS_use_prefix_blooms ? prefix_extractor_
@@ -1226,6 +1245,7 @@ class Benchmark {
         );
         break;
     }
+    options.purge_log_after_memtable_flush = FLAGS_purge_log_after_memtable_flush;
     if (FLAGS_max_bytes_for_level_multiplier_additional.size() > 0) {
       if (FLAGS_max_bytes_for_level_multiplier_additional.size() !=
           (unsigned int)FLAGS_num_levels) {
@@ -1286,6 +1306,24 @@ class Benchmark {
       exit(1);
     }
 
+    // set universal style compaction configurations, if applicable
+    if (FLAGS_universal_size_ratio != 0) {
+      options.compaction_options_universal.size_ratio =
+        FLAGS_universal_size_ratio;
+    }
+    if (FLAGS_universal_min_merge_width != 0) {
+      options.compaction_options_universal.min_merge_width =
+        FLAGS_universal_min_merge_width;
+    }
+    if (FLAGS_universal_max_merge_width != 0) {
+      options.compaction_options_universal.max_merge_width =
+        FLAGS_universal_max_merge_width;
+    }
+    if (FLAGS_universal_max_size_amplification_percent != 0) {
+      options.compaction_options_universal.max_size_amplification_percent =
+        FLAGS_universal_max_size_amplification_percent;
+    }
+
     Status s;
     if(FLAGS_read_only) {
       s = DB::OpenForReadOnly(options, FLAGS_db, &db_);
@@ -1317,6 +1355,54 @@ class Benchmark {
     DoWrite(thread, UNIQUE_RANDOM);
   }
 
+  void writeOrFail(WriteBatch& batch) {
+    Status s = db_->Write(write_options_, &batch);
+    if (!s.ok()) {
+      fprintf(stderr, "put error: %s\n", s.ToString().c_str());
+      exit(1);
+    }
+  }
+
+  void WriteFromStdin(ThreadState* thread) {
+    size_t count = 0;
+    WriteBatch batch;
+    const size_t bufferLen = 32 << 20;
+    unique_ptr<char[]> line = unique_ptr<char[]>(new char[bufferLen]);
+    char* linep = line.get();
+    const int batchSize = 100 << 10;
+    const char columnSeparator = '\t';
+    const char lineSeparator = '\n';
+
+    while (fgets(linep, bufferLen, stdin) != nullptr) {
+      ++count;
+      char* tab = std::find(linep, linep + bufferLen, columnSeparator);
+      if (tab == linep + bufferLen) {
+        fprintf(stderr, "[Error] No Key delimiter TAB at line %ld\n", count);
+        continue;
+      }
+      Slice key(linep, tab - linep);
+      tab++;
+      char* endLine = std::find(tab, linep + bufferLen, lineSeparator);
+      if (endLine  == linep + bufferLen) {
+        fprintf(stderr, "[Error] No ENTER at end of line # %ld\n", count);
+        continue;
+      }
+      Slice value(tab, endLine - tab);
+      thread->stats.FinishedSingleOp(db_);
+      thread->stats.AddBytes(endLine - linep - 1);
+
+      if (batch.Count() < batchSize) {
+        batch.Put(key, value);
+        continue;
+      }
+      writeOrFail(batch);
+      batch.Clear();
+    }
+    if (batch.Count() > 0) {
+      writeOrFail(batch);
+    }
+  }
+
   void DoWrite(ThreadState* thread, WriteMode write_mode) {
     const int test_duration = write_mode == RANDOM ? FLAGS_duration : 0;
     const int num_ops = writes_ == 0 ? num_ : writes_ ;
@@ -2225,28 +2311,24 @@ class Benchmark {
   }
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  leveldb::InstallStackTraceHandler();
+  rocksdb::InstallStackTraceHandler();
 
-  FLAGS_write_buffer_size = leveldb::Options().write_buffer_size;
-  FLAGS_max_write_buffer_number = leveldb::Options().max_write_buffer_number;
+  FLAGS_write_buffer_size = rocksdb::Options().write_buffer_size;
+  FLAGS_max_write_buffer_number = rocksdb::Options().max_write_buffer_number;
   FLAGS_min_write_buffer_number_to_merge =
-    leveldb::Options().min_write_buffer_number_to_merge;
-  FLAGS_open_files = leveldb::Options().max_open_files;
+    rocksdb::Options().min_write_buffer_number_to_merge;
+  FLAGS_open_files = rocksdb::Options().max_open_files;
   FLAGS_max_background_compactions =
-    leveldb::Options().max_background_compactions;
-  FLAGS_compaction_style = leveldb::Options().compaction_style;
-  FLAGS_universal_size_ratio =
-    leveldb::Options().compaction_options_universal.size_ratio;
-  FLAGS_compaction_universal_min_merge_width =
-    leveldb::Options().compaction_options_universal.min_merge_width;
+    rocksdb::Options().max_background_compactions;
+  FLAGS_compaction_style = rocksdb::Options().compaction_style;
   // Compression test code above refers to FLAGS_block_size
-  FLAGS_block_size = leveldb::Options().block_size;
-  FLAGS_use_os_buffer = leveldb::EnvOptions().use_os_buffer;
-  FLAGS_use_mmap_reads = leveldb::EnvOptions().use_mmap_reads;
-  FLAGS_use_mmap_writes = leveldb::EnvOptions().use_mmap_writes;
+  FLAGS_block_size = rocksdb::Options().block_size;
+  FLAGS_use_os_buffer = rocksdb::EnvOptions().use_os_buffer;
+  FLAGS_use_mmap_reads = rocksdb::EnvOptions().use_mmap_reads;
+  FLAGS_use_mmap_writes = rocksdb::EnvOptions().use_mmap_writes;
 
   std::string default_db_path;
 
@@ -2259,7 +2341,7 @@ int main(int argc, char** argv) {
     char buf[2048];
     char str[512];
 
-    if (leveldb::Slice(argv[i]).starts_with("--benchmarks=")) {
+    if (rocksdb::Slice(argv[i]).starts_with("--benchmarks=")) {
       FLAGS_benchmarks = argv[i] + strlen("--benchmarks=");
     } else if (sscanf(argv[i], "--compression_ratio=%lf%c", &d, &junk) == 1) {
       FLAGS_compression_ratio = d;
@@ -2299,8 +2381,8 @@ int main(int argc, char** argv) {
       } else {
         FLAGS_key_size = n;
       }
-    } else if (sscanf(argv[i], "--write_buffer_size=%d%c", &n, &junk) == 1) {
-      FLAGS_write_buffer_size = n;
+    } else if (sscanf(argv[i], "--write_buffer_size=%lld%c", &ll, &junk) == 1) {
+      FLAGS_write_buffer_size = ll;
     } else if (sscanf(argv[i], "--max_write_buffer_number=%d%c", &n, &junk) == 1) {
       FLAGS_max_write_buffer_number = n;
     } else if (sscanf(argv[i], "--min_write_buffer_number_to_merge=%d%c",
@@ -2310,12 +2392,7 @@ int main(int argc, char** argv) {
                == 1) {
       FLAGS_max_background_compactions = n;
     } else if (sscanf(argv[i], "--compaction_style=%d%c", &n, &junk) == 1) {
-      FLAGS_compaction_style = (leveldb::CompactionStyle)n;
-    } else if (sscanf(argv[i], "--universal_size_ratio=%d%c", &n, &junk) == 1) {
-      FLAGS_universal_size_ratio = n;
-    } else if (sscanf(argv[i], "--universal_min_merge_width=%d%c",
-               &n, &junk) == 1) {
-      FLAGS_compaction_universal_min_merge_width = n;
+      FLAGS_compaction_style = (rocksdb::CompactionStyle)n;
     } else if (sscanf(argv[i], "--cache_size=%ld%c", &l, &junk) == 1) {
       FLAGS_cache_size = l;
     } else if (sscanf(argv[i], "--block_size=%d%c", &n, &junk) == 1) {
@@ -2355,7 +2432,7 @@ int main(int argc, char** argv) {
     } else if (sscanf(argv[i], "--statistics=%d%c", &n, &junk) == 1 &&
                (n == 0 || n == 1)) {
       if (n == 1) {
-        dbstats = leveldb::CreateDBStatistics();
+        dbstats = rocksdb::CreateDBStatistics();
         FLAGS_statistics = true;
       }
     } else if (sscanf(argv[i], "--writes=%lld%c", &ll, &junk) == 1) {
@@ -2387,7 +2464,7 @@ int main(int argc, char** argv) {
         (n == 0 || n == 1)) {
       FLAGS_get_approx = n;
     } else if (sscanf(argv[i], "--hdfs=%s", buf) == 1) {
-      FLAGS_env  = new leveldb::HdfsEnv(buf);
+      FLAGS_env  = new rocksdb::HdfsEnv(buf);
     } else if (sscanf(argv[i], "--num_levels=%d%c",
         &n, &junk) == 1) {
       FLAGS_num_levels = n;
@@ -2409,7 +2486,7 @@ int main(int argc, char** argv) {
     } else if (sscanf(argv[i],
                 "--max_bytes_for_level_multiplier_additional=%s%c",
                 str, &junk) == 1) {
-      std::vector<std::string> fanout = leveldb::stringSplit(str, ',');
+      std::vector<std::string> fanout = rocksdb::stringSplit(str, ',');
       for (unsigned int j= 0; j < fanout.size(); j++) {
         FLAGS_max_bytes_for_level_multiplier_additional.push_back(
           std::stoi(fanout[j]));
@@ -2423,13 +2500,13 @@ int main(int argc, char** argv) {
     } else if (strncmp(argv[i], "--compression_type=", 19) == 0) {
       const char* ctype = argv[i] + 19;
       if (!strcasecmp(ctype, "none"))
-        FLAGS_compression_type = leveldb::kNoCompression;
+        FLAGS_compression_type = rocksdb::kNoCompression;
       else if (!strcasecmp(ctype, "snappy"))
-        FLAGS_compression_type = leveldb::kSnappyCompression;
+        FLAGS_compression_type = rocksdb::kSnappyCompression;
       else if (!strcasecmp(ctype, "zlib"))
-        FLAGS_compression_type = leveldb::kZlibCompression;
+        FLAGS_compression_type = rocksdb::kZlibCompression;
       else if (!strcasecmp(ctype, "bzip2"))
-        FLAGS_compression_type = leveldb::kBZip2Compression;
+        FLAGS_compression_type = rocksdb::kBZip2Compression;
       else {
         fprintf(stdout, "Cannot parse %s\n", argv[i]);
       }
@@ -2492,13 +2569,13 @@ int main(int argc, char** argv) {
       FLAGS_advise_random_on_open = n;
     } else if (sscanf(argv[i], "--compaction_fadvice=%s", buf) == 1) {
       if (!strcasecmp(buf, "NONE"))
-        FLAGS_compaction_fadvice = leveldb::Options::NONE;
+        FLAGS_compaction_fadvice = rocksdb::Options::NONE;
       else if (!strcasecmp(buf, "NORMAL"))
-        FLAGS_compaction_fadvice = leveldb::Options::NORMAL;
+        FLAGS_compaction_fadvice = rocksdb::Options::NORMAL;
       else if (!strcasecmp(buf, "SEQUENTIAL"))
-        FLAGS_compaction_fadvice = leveldb::Options::SEQUENTIAL;
+        FLAGS_compaction_fadvice = rocksdb::Options::SEQUENTIAL;
       else if (!strcasecmp(buf, "WILLNEED"))
-        FLAGS_compaction_fadvice = leveldb::Options::WILLNEED;
+        FLAGS_compaction_fadvice = rocksdb::Options::WILLNEED;
       else {
         fprintf(stdout, "Unknown compaction fadvice:%s\n", buf);
       }
@@ -2518,6 +2595,22 @@ int main(int argc, char** argv) {
       FLAGS_filter_deletes = n;
     } else if (sscanf(argv[i], "--merge_operator=%s", buf) == 1) {
       FLAGS_merge_operator = buf;
+    } else if (sscanf(argv[i], "--purge_log_after_memtable_flush=%d%c", &n, &junk)
+               == 1 && (n == 0 || n ==1 )) {
+      FLAGS_purge_log_after_memtable_flush = n;
+    } else if (sscanf(argv[i], "--universal_size_ratio=%d%c",
+                      &n, &junk) == 1) {
+      FLAGS_universal_size_ratio = n;
+    } else if (sscanf(argv[i], "--universal_min_merge_width=%d%c",
+                      &n, &junk) == 1) {
+      FLAGS_universal_min_merge_width = n;
+    } else if (sscanf(argv[i], "--universal_max_merge_width=%d%c",
+                      &n, &junk) == 1) {
+      FLAGS_universal_max_merge_width = n;
+    } else if (sscanf(argv[i],
+               "--universal_max_size_amplification_percent=%d%c",
+               &n, &junk) == 1) {
+      FLAGS_universal_max_size_amplification_percent = n;
     } else {
       fprintf(stderr, "Invalid flag '%s'\n", argv[i]);
       exit(1);
@@ -2530,12 +2623,12 @@ int main(int argc, char** argv) {
 
   // Choose a location for the test database if none given with --db=<path>
   if (FLAGS_db == nullptr) {
-      leveldb::Env::Default()->GetTestDirectory(&default_db_path);
+      rocksdb::Env::Default()->GetTestDirectory(&default_db_path);
       default_db_path += "/dbbench";
       FLAGS_db = default_db_path.c_str();
   }
 
-  leveldb::Benchmark benchmark;
+  rocksdb::Benchmark benchmark;
   benchmark.Run();
   return 0;
 }

db/db_filesnapshot.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2012 Facebook.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
@@ -13,7 +18,7 @@
 #include "port/port.h"
 #include "util/mutexlock.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 Status DBImpl::DisableFileDeletions() {
   MutexLock l(&mutex_);
@@ -30,16 +35,19 @@ Status DBImpl::EnableFileDeletions() {
 }
 
 Status DBImpl::GetLiveFiles(std::vector<std::string>& ret,
-                            uint64_t* manifest_file_size) {
+                            uint64_t* manifest_file_size,
+                            bool flush_memtable) {
 
   *manifest_file_size = 0;
 
-  // flush all dirty data to disk.
-  Status status =  Flush(FlushOptions());
-  if (!status.ok()) {
-    Log(options_.info_log, "Cannot Flush data %s\n",
-        status.ToString().c_str());
-    return status;
+  if (flush_memtable) {
+    // flush all dirty data to disk.
+    Status status =  Flush(FlushOptions());
+    if (!status.ok()) {
+      Log(options_.info_log, "Cannot Flush data %s\n",
+          status.ToString().c_str());
+      return status;
+    }
   }
 
   MutexLock l(&mutex_);
@@ -72,7 +80,7 @@ Status DBImpl::GetSortedWalFiles(VectorLogPtr& files) {
 
   // list wal files in archive dir.
   Status s;
-  std::string archivedir = ArchivalDirectory(dbname_);
+  std::string archivedir = ArchivalDirectory(options_.wal_dir);
   if (env_->FileExists(archivedir)) {
     s = AppendSortedWalsOfType(archivedir, files, kArchivedLogFile);
     if (!s.ok()) {
@@ -80,19 +88,19 @@ Status DBImpl::GetSortedWalFiles(VectorLogPtr& files) {
     }
   }
   // list wal files in main db dir.
-  return AppendSortedWalsOfType(dbname_, files, kAliveLogFile);
+  return AppendSortedWalsOfType(options_.wal_dir, files, kAliveLogFile);
 }
 
 Status DBImpl::DeleteWalFiles(const VectorLogPtr& files) {
   Status s;
-  std::string archivedir = ArchivalDirectory(dbname_);
+  std::string archivedir = ArchivalDirectory(options_.wal_dir);
   std::string files_not_deleted;
   for (const auto& wal : files) {
     /* Try deleting in the dir that pathname points to for the logfile.
        This may fail if we try to delete a log file which was live when captured
        but is archived now. Try deleting it from archive also
      */
-    Status st = env_->DeleteFile(dbname_ + "/" + wal->PathName());
+    Status st = env_->DeleteFile(options_.wal_dir + "/" + wal->PathName());
     if (!st.ok()) {
       if (wal->Type() == kAliveLogFile &&
           env_->DeleteFile(LogFileName(archivedir, wal->LogNumber())).ok()) {

db/db_impl.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -47,7 +52,7 @@
 #include "util/mutexlock.h"
 #include "util/stop_watch.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 void dumpLeveldbBuildVersion(Logger * log);
 
@@ -169,14 +174,21 @@ Options SanitizeOptions(const std::string& dbname,
     // function.
     auto factory = dynamic_cast<PrefixHashRepFactory*>(
       result.memtable_factory.get());
-    if (factory != nullptr &&
+    if (factory &&
         factory->GetTransform() != result.prefix_extractor) {
       Log(result.info_log, "A prefix hash representation factory was supplied "
           "whose prefix extractor does not match options.prefix_extractor. "
           "Falling back to skip list representation factory");
       result.memtable_factory = std::make_shared<SkipListFactory>();
+    } else if (factory) {
+      Log(result.info_log, "Prefix hash memtable rep is in use.");
     }
   }
+
+  if (result.wal_dir.empty()) {
+    // Use dbname as default
+    result.wal_dir = dbname;
+  }
   return result;
 }
 
@@ -198,6 +210,7 @@ DBImpl::DBImpl(const Options& options, const std::string& dbname)
       logfile_number_(0),
       tmp_batch_(),
       bg_compaction_scheduled_(0),
+      bg_flush_scheduled_(0),
       bg_logstats_scheduled_(false),
       manual_compaction_(nullptr),
       logger_(nullptr),
@@ -242,10 +255,6 @@ DBImpl::DBImpl(const Options& options, const std::string& dbname)
   dumpLeveldbBuildVersion(options_.info_log.get());
   options_.Dump(options_.info_log.get());
 
-#ifdef USE_SCRIBE
-  logger_.reset(new ScribeLogger("localhost", 1456));
-#endif
-
   char name[100];
   Status st = env_->GetHostName(name, 100L);
   if (st.ok()) {
@@ -265,7 +274,9 @@ DBImpl::~DBImpl() {
   }
   mutex_.Lock();
   shutting_down_.Release_Store(this);  // Any non-nullptr value is ok
-  while (bg_compaction_scheduled_ || bg_logstats_scheduled_) {
+  while (bg_compaction_scheduled_ ||
+         bg_flush_scheduled_ ||
+         bg_logstats_scheduled_) {
     bg_cv_.Wait();
   }
   mutex_.Unlock();
@@ -285,13 +296,17 @@ void DBImpl::TEST_Destroy_DBImpl() {
 
   // wait till all background compactions are done.
   mutex_.Lock();
-  while (bg_compaction_scheduled_ || bg_logstats_scheduled_) {
+  while (bg_compaction_scheduled_ ||
+         bg_flush_scheduled_ ||
+         bg_logstats_scheduled_) {
     bg_cv_.Wait();
   }
 
   // Prevent new compactions from occuring.
+  bg_work_gate_closed_ = true;
   const int LargeNumber = 10000000;
   bg_compaction_scheduled_ += LargeNumber;
+
   mutex_.Unlock();
 
   // force release the lock file.
@@ -348,7 +363,7 @@ void DBImpl::MaybeIgnoreError(Status* s) const {
 
 const Status DBImpl::CreateArchivalDirectory() {
   if (options_.WAL_ttl_seconds > 0) {
-    std::string archivalPath = ArchivalDirectory(dbname_);
+    std::string archivalPath = ArchivalDirectory(options_.wal_dir);
     return env_->CreateDirIfMissing(archivalPath);
   }
   return Status::OK();
@@ -377,7 +392,7 @@ void DBImpl::MaybeDumpStats() {
     // period in rare cases.
     last_stats_dump_time_microsec_ = now_micros;
     std::string stats;
-    GetProperty("leveldb.stats", &stats);
+    GetProperty("rocksdb.stats", &stats);
     Log(options_.info_log, "%s", stats.c_str());
     PrintStatistics();
   }
@@ -414,16 +429,47 @@ void DBImpl::FindObsoleteFiles(DeletionState& deletion_state) {
   // set of all files in the directory
   env_->GetChildren(dbname_, &deletion_state.allfiles); // Ignore errors
 
+  //Add log files in wal_dir
+  if (options_.wal_dir != dbname_) {
+    std::vector<std::string> log_files;
+    env_->GetChildren(options_.wal_dir, &log_files); // Ignore errors
+    deletion_state.allfiles.insert(
+      deletion_state.allfiles.end(),
+      log_files.begin(),
+      log_files.end()
+    );
+  }
+
   // store the current filenum, lognum, etc
   deletion_state.filenumber = versions_->ManifestFileNumber();
   deletion_state.lognumber = versions_->LogNumber();
   deletion_state.prevlognumber = versions_->PrevLogNumber();
 }
 
+Status DBImpl::DeleteLogFile(uint64_t number) {
+  Status s;
+  auto filename = LogFileName(options_.wal_dir, number);
+  if (options_.WAL_ttl_seconds > 0) {
+    s = env_->RenameFile(filename,
+                         ArchivedLogFileName(options_.wal_dir, number));
+
+    if (!s.ok()) {
+      Log(options_.info_log, "RenameFile logfile #%lu FAILED", number);
+    }
+  } else {
+    s = env_->DeleteFile(filename);
+    if(!s.ok()) {
+      Log(options_.info_log, "Delete logfile #%lu FAILED", number);
+    }
+  }
+
+  return s;
+}
+
 // Diffs the files listed in filenames and those that do not
 // belong to live files are posibly removed. If the removed file
 // is a sst file, then it returns the file number in files_to_evict.
-// It is not necesary to hold the mutex when invoking this method.
+// It is not necessary to hold the mutex when invoking this method.
 void DBImpl::PurgeObsoleteFiles(DeletionState& state) {
   uint64_t number;
   FileType type;
@@ -463,6 +509,7 @@ void DBImpl::PurgeObsoleteFiles(DeletionState& state) {
           break;
         case kCurrentFile:
         case kDBLockFile:
+        case kIdentityFile:
         case kMetaDatabase:
           keep = true;
           break;
@@ -474,19 +521,9 @@ void DBImpl::PurgeObsoleteFiles(DeletionState& state) {
           state.files_to_evict.push_back(number);
         }
         Log(options_.info_log, "Delete type=%d #%lu", int(type), number);
-        if (type == kLogFile && options_.WAL_ttl_seconds > 0) {
-          Status st = env_->RenameFile(
-            LogFileName(dbname_, number),
-            ArchivedLogFileName(dbname_, number)
-          );
 
-          if (!st.ok()) {
-            Log(
-              options_.info_log, "RenameFile type=%d #%lu FAILED",
-              int(type),
-              number
-            );
-          }
+        if (type == kLogFile) {
+          DeleteLogFile(number);
         } else {
           Status st = env_->DeleteFile(dbname_ + "/" + state.allfiles[i]);
           if (!st.ok()) {
@@ -499,7 +536,7 @@ void DBImpl::PurgeObsoleteFiles(DeletionState& state) {
     }
   }
 
-  // Delete old log files.
+  // Delete old info log files.
   size_t old_log_file_count = old_log_files.size();
   // NOTE: Currently we only support log purge when options_.db_log_dir is
   // located in `dbname` directory.
@@ -542,7 +579,7 @@ void DBImpl::PurgeObsoleteWALFiles() {
       return;
     }
     std::vector<std::string> wal_files;
-    std::string archival_dir = ArchivalDirectory(dbname_);
+    std::string archival_dir = ArchivalDirectory(options_.wal_dir);
     env_->GetChildren(archival_dir, &wal_files);
     for (const auto& f : wal_files) {
       uint64_t file_m_time;
@@ -603,6 +640,13 @@ Status DBImpl::Recover(VersionEdit* edit, MemTable* external_table,
             dbname_, "exists (error_if_exists is true)");
       }
     }
+    // Check for the IDENTITY file and create it if not there
+    if (!env_->FileExists(IdentityFileName(dbname_))) {
+      s = SetIdentityFile(env_, dbname_);
+      if (!s.ok()) {
+        return s;
+      }
+    }
   }
 
   Status s = versions_->Recover();
@@ -615,11 +659,11 @@ Status DBImpl::Recover(VersionEdit* edit, MemTable* external_table,
     //
     // Note that PrevLogNumber() is no longer used, but we pay
     // attention to it in case we are recovering a database
-    // produced by an older version of leveldb.
+    // produced by an older version of rocksdb.
     const uint64_t min_log = versions_->LogNumber();
     const uint64_t prev_log = versions_->PrevLogNumber();
     std::vector<std::string> filenames;
-    s = env_->GetChildren(dbname_, &filenames);
+    s = env_->GetChildren(options_.wal_dir, &filenames);
     if (!s.ok()) {
       return s;
     }
@@ -686,7 +730,7 @@ Status DBImpl::RecoverLogFile(uint64_t log_number,
   mutex_.AssertHeld();
 
   // Open the log file
-  std::string fname = LogFileName(dbname_, log_number);
+  std::string fname = LogFileName(options_.wal_dir, log_number);
   unique_ptr<SequentialFile> file;
   Status status = env_->NewSequentialFile(fname, &file, storage_options_);
   if (!status.ok()) {
@@ -785,7 +829,7 @@ Status DBImpl::WriteLevel0TableForRecovery(MemTable* mem, VersionEdit* edit) {
     s = BuildTable(dbname_, env_, options_, storage_options_,
                    table_cache_.get(), iter, &meta,
                    user_comparator(), newest_snapshot,
-                   earliest_seqno_in_memtable);
+                   earliest_seqno_in_memtable, true);
     mutex_.Lock();
   }
 
@@ -841,10 +885,15 @@ Status DBImpl::WriteLevel0Table(std::vector<MemTable*> &mems, VersionEdit* edit,
   Status s;
   {
     mutex_.Unlock();
+    // We skip compression if universal compression is used and the size
+    // threshold is set for compression.
+    bool enable_compression = (options_.compaction_style
+        != kCompactionStyleUniversal ||
+        options_.compaction_options_universal.compression_size_percent < 0);
     s = BuildTable(dbname_, env_, options_, storage_options_,
                    table_cache_.get(), iter, &meta,
                    user_comparator(), newest_snapshot,
-                   earliest_seqno_in_memtable);
+                   earliest_seqno_in_memtable, enable_compression);
     mutex_.Lock();
   }
   base->Unref();
@@ -893,18 +942,17 @@ Status DBImpl::WriteLevel0Table(std::vector<MemTable*> &mems, VersionEdit* edit,
   return s;
 }
 
-Status DBImpl::CompactMemTable(bool* madeProgress) {
+Status DBImpl::FlushMemTableToOutputFile(bool* madeProgress) {
   mutex_.AssertHeld();
   assert(imm_.size() != 0);
 
   if (!imm_.IsFlushPending(options_.min_write_buffer_number_to_merge)) {
-    Log(options_.info_log, "Memcompaction already in progress");
-    Status s = Status::IOError("Memcompaction already in progress");
+    Log(options_.info_log, "FlushMemTableToOutputFile already in progress");
+    Status s = Status::IOError("FlushMemTableToOutputFile already in progress");
     return s;
   }
 
   // Save the contents of the earliest memtable as a new Table
-  // This will release and re-acquire the mutex.
   uint64_t file_number;
   std::vector<MemTable*> mems;
   imm_.PickMemtablesToFlush(&mems);
@@ -918,8 +966,10 @@ Status DBImpl::CompactMemTable(bool* madeProgress) {
   MemTable* m = mems[0];
   VersionEdit* edit = m->GetEdits();
   edit->SetPrevLogNumber(0);
-  edit->SetLogNumber(m->GetLogNumber());  // Earlier logs no longer needed
+  edit->SetLogNumber(m->GetNextLogNumber());  // Earlier logs no longer needed
+  auto to_delete = m->GetLogNumber();
 
+  // This will release and re-acquire the mutex.
   Status s = WriteLevel0Table(mems, edit, &file_number);
 
   if (s.ok() && shutting_down_.Acquire_Load()) {
@@ -937,10 +987,19 @@ Status DBImpl::CompactMemTable(bool* madeProgress) {
     if (madeProgress) {
       *madeProgress = 1;
     }
+
     MaybeScheduleLogDBDeployStats();
-    // we could have deleted obsolete files here, but it is not
-    // absolutely necessary because it could be also done as part
-    // of other background compaction
+    // TODO: if log deletion failed for any reason, we probably
+    // should store the file number in the shared state, and retry
+    // However, for now, PurgeObsoleteFiles will take care of that
+    // anyways.
+    if (options_.purge_log_after_memtable_flush &&
+        !disable_delete_obsolete_files_ &&
+        to_delete > 0) {
+      mutex_.Unlock();
+      DeleteLogFile(to_delete);
+      mutex_.Lock();
+    }
   }
   return s;
 }
@@ -957,7 +1016,7 @@ void DBImpl::CompactRange(const Slice* begin, const Slice* end,
       }
     }
   }
-  TEST_CompactMemTable(); // TODO(sanjay): Skip if memtable does not overlap
+  TEST_FlushMemTable(); // TODO(sanjay): Skip if memtable does not overlap
   for (int level = 0; level < max_level_with_files; level++) {
     TEST_CompactRange(level, begin, end);
   }
@@ -997,10 +1056,10 @@ void DBImpl::ReFitLevel(int level, int target_level) {
 
   // wait for all background threads to stop
   bg_work_gate_closed_ = true;
-  while (bg_compaction_scheduled_ > 0) {
+  while (bg_compaction_scheduled_ > 0 || bg_flush_scheduled_) {
     Log(options_.info_log,
-        "RefitLevel: waiting for background threads to stop: %d",
-        bg_compaction_scheduled_);
+        "RefitLevel: waiting for background threads to stop: %d %d",
+        bg_compaction_scheduled_, bg_flush_scheduled_);
     bg_cv_.Wait();
   }
 
@@ -1063,6 +1122,7 @@ SequenceNumber DBImpl::GetLatestSequenceNumber() {
 Status DBImpl::GetUpdatesSince(SequenceNumber seq,
                                unique_ptr<TransactionLogIterator>* iter) {
 
+  RecordTick(options_.statistics, GET_UPDATES_SINCE_CALLS);
   if (seq > last_flushed_sequence_) {
     return Status::IOError("Requested sequence not yet written in the db");
   }
@@ -1075,24 +1135,18 @@ Status DBImpl::GetUpdatesSince(SequenceNumber seq,
     return s;
   }
 
-  if (wal_files->empty()) {
-    return Status::IOError(" NO WAL Files present in the db");
-  }
-  //  std::shared_ptr would have been useful here.
-
   s = RetainProbableWalFiles(*wal_files, seq);
   if (!s.ok()) {
     return s;
   }
   iter->reset(
-    new TransactionLogIteratorImpl(dbname_,
+    new TransactionLogIteratorImpl(options_.wal_dir,
                                    &options_,
                                    storage_options_,
                                    seq,
                                    std::move(wal_files),
                                    &last_flushed_sequence_));
-  iter->get()->Next();
-  return iter->get()->status();
+  return (*iter)->status();
 }
 
 Status DBImpl::RetainProbableWalFiles(VectorLogPtr& all_logs,
@@ -1121,7 +1175,8 @@ Status DBImpl::RetainProbableWalFiles(VectorLogPtr& all_logs,
 bool DBImpl::CheckWalFileExistsAndEmpty(const WalFileType type,
                                         const uint64_t number) {
   const std::string fname = (type == kAliveLogFile) ?
-    LogFileName(dbname_, number) : ArchivedLogFileName(dbname_, number);
+    LogFileName(options_.wal_dir, number) :
+    ArchivedLogFileName(options_.wal_dir, number);
   uint64_t file_size;
   Status s = env_->GetFileSize(fname, &file_size);
   return (s.ok() && (file_size == 0));
@@ -1131,23 +1186,24 @@ Status DBImpl::ReadFirstRecord(const WalFileType type, const uint64_t number,
                                WriteBatch* const result) {
 
   if (type == kAliveLogFile) {
-    std::string fname = LogFileName(dbname_, number);
+    std::string fname = LogFileName(options_.wal_dir, number);
     Status status = ReadFirstLine(fname, result);
     if (!status.ok()) {
       //  check if the file got moved to archive.
-      std::string archived_file = ArchivedLogFileName(dbname_, number);
+      std::string archived_file =
+        ArchivedLogFileName(options_.wal_dir, number);
       Status s = ReadFirstLine(archived_file, result);
       if (!s.ok()) {
-        return Status::IOError("Log File has been deleted");
+        return Status::IOError("Log File has been deleted: " + archived_file);
       }
     }
     return Status::OK();
   } else if (type == kArchivedLogFile) {
-    std::string fname = ArchivedLogFileName(dbname_, number);
+    std::string fname = ArchivedLogFileName(options_.wal_dir, number);
     Status status = ReadFirstLine(fname, result);
     return status;
   }
-  return Status::NotSupported("File Type Not Known");
+  return Status::NotSupported("File Type Not Known: " + type);
 }
 
 Status DBImpl::ReadFirstLine(const std::string& fname,
@@ -1301,7 +1357,7 @@ void DBImpl::TEST_CompactRange(int level, const Slice* begin,const Slice* end) {
     if (bg_compaction_scheduled_ == LargeNumber) {
       bg_compaction_scheduled_ = newvalue;
     }
-    MaybeScheduleCompaction();
+    MaybeScheduleFlushOrCompaction();
     while (manual_compaction_ == &manual) {
       bg_cv_.Wait();
     }
@@ -1322,12 +1378,12 @@ Status DBImpl::FlushMemTable(const FlushOptions& options) {
   Status s = Write(WriteOptions(), nullptr);
   if (s.ok() && options.wait) {
     // Wait until the compaction completes
-    s = WaitForCompactMemTable();
+    s = WaitForFlushMemTable();
   }
   return s;
 }
 
-Status DBImpl::WaitForCompactMemTable() {
+Status DBImpl::WaitForFlushMemTable() {
   Status s;
   // Wait until the compaction completes
   MutexLock l(&mutex_);
@@ -1340,50 +1396,110 @@ Status DBImpl::WaitForCompactMemTable() {
   return s;
 }
 
-Status DBImpl::TEST_CompactMemTable() {
+Status DBImpl::TEST_FlushMemTable() {
   return FlushMemTable(FlushOptions());
 }
 
-Status DBImpl::TEST_WaitForCompactMemTable() {
-  return WaitForCompactMemTable();
+Status DBImpl::TEST_WaitForFlushMemTable() {
+  return WaitForFlushMemTable();
 }
 
 Status DBImpl::TEST_WaitForCompact() {
   // Wait until the compaction completes
+
+  // TODO: a bug here. This function actually does not necessarily
+  // wait for compact. It actually waits for scheduled compaction
+  // OR flush to finish.
+
   MutexLock l(&mutex_);
-  while (bg_compaction_scheduled_ && bg_error_.ok()) {
+  while ((bg_compaction_scheduled_ || bg_flush_scheduled_) &&
+         bg_error_.ok()) {
     bg_cv_.Wait();
   }
   return bg_error_;
 }
 
-void DBImpl::MaybeScheduleCompaction() {
+void DBImpl::MaybeScheduleFlushOrCompaction() {
   mutex_.AssertHeld();
   if (bg_work_gate_closed_) {
     // gate closed for backgrond work
-  } else if (bg_compaction_scheduled_ >= options_.max_background_compactions) {
-    // Already scheduled
   } else if (shutting_down_.Acquire_Load()) {
     // DB is being deleted; no more background compactions
-  } else if (!imm_.IsFlushPending(options_.min_write_buffer_number_to_merge) &&
-             manual_compaction_ == nullptr &&
-             !versions_->NeedsCompaction()) {
-    // No work to be done
   } else {
-    bg_compaction_scheduled_++;
-    env_->Schedule(&DBImpl::BGWork, this);
+    bool is_flush_pending =
+      imm_.IsFlushPending(options_.min_write_buffer_number_to_merge);
+    if (is_flush_pending &&
+        (bg_flush_scheduled_ < options_.max_background_flushes)) {
+      // memtable flush needed
+      bg_flush_scheduled_++;
+      env_->Schedule(&DBImpl::BGWorkFlush, this, Env::Priority::HIGH);
+    }
+
+    if ((manual_compaction_ ||
+         versions_->NeedsCompaction() ||
+         (is_flush_pending && (options_.max_background_flushes <= 0))) &&
+        bg_compaction_scheduled_ < options_.max_background_compactions) {
+      // compaction needed, or memtable flush needed but HIGH pool not enabled.
+      bg_compaction_scheduled_++;
+      env_->Schedule(&DBImpl::BGWorkCompaction, this, Env::Priority::LOW);
+    }
   }
 }
 
-void DBImpl::BGWork(void* db) {
-  reinterpret_cast<DBImpl*>(db)->BackgroundCall();
+void DBImpl::BGWorkFlush(void* db) {
+  reinterpret_cast<DBImpl*>(db)->BackgroundCallFlush();
 }
 
+void DBImpl::BGWorkCompaction(void* db) {
+  reinterpret_cast<DBImpl*>(db)->BackgroundCallCompaction();
+}
+
+Status DBImpl::BackgroundFlush(bool* madeProgress) {
+  Status stat;
+  while (stat.ok() &&
+         imm_.IsFlushPending(options_.min_write_buffer_number_to_merge)) {
+    Log(options_.info_log,
+        "BackgroundCallFlush doing FlushMemTableToOutputFile, flush slots available %d",
+        options_.max_background_flushes - bg_flush_scheduled_);
+    stat = FlushMemTableToOutputFile(madeProgress);
+  }
+  return stat;
+}
+
+void DBImpl::BackgroundCallFlush() {
+  bool madeProgress = false;
+  assert(bg_flush_scheduled_);
+  MutexLock l(&mutex_);
+
+  if (!shutting_down_.Acquire_Load()) {
+    Status s = BackgroundFlush(&madeProgress);
+    if (!s.ok()) {
+      // Wait a little bit before retrying background compaction in
+      // case this is an environmental problem and we do not want to
+      // chew up resources for failed compactions for the duration of
+      // the problem.
+      bg_cv_.SignalAll();  // In case a waiter can proceed despite the error
+      Log(options_.info_log, "Waiting after background flush error: %s",
+          s.ToString().c_str());
+      mutex_.Unlock();
+      env_->SleepForMicroseconds(1000000);
+      mutex_.Lock();
+    }
+  }
+
+  bg_flush_scheduled_--;
+  if (madeProgress) {
+    MaybeScheduleFlushOrCompaction();
+  }
+  bg_cv_.SignalAll();
+}
+
+
 void DBImpl::TEST_PurgeObsoleteteWAL() {
   PurgeObsoleteWALFiles();
 }
 
-void DBImpl::BackgroundCall() {
+void DBImpl::BackgroundCallCompaction() {
   bool madeProgress = false;
   DeletionState deletion_state;
 
@@ -1425,7 +1541,7 @@ void DBImpl::BackgroundCall() {
   // So reschedule another compaction if we made progress in the
   // last compaction.
   if (madeProgress) {
-    MaybeScheduleCompaction();
+    MaybeScheduleFlushOrCompaction();
   }
   bg_cv_.SignalAll();
 
@@ -1436,11 +1552,13 @@ Status DBImpl::BackgroundCompaction(bool* madeProgress,
   *madeProgress = false;
   mutex_.AssertHeld();
 
+  // TODO: remove memtable flush from formal compaction
   while (imm_.IsFlushPending(options_.min_write_buffer_number_to_merge)) {
     Log(options_.info_log,
-        "BackgroundCompaction doing CompactMemTable, compaction slots available %d",
+        "BackgroundCompaction doing FlushMemTableToOutputFile, compaction slots "
+        "available %d",
         options_.max_background_compactions - bg_compaction_scheduled_);
-    Status stat = CompactMemTable(madeProgress);
+    Status stat = FlushMemTableToOutputFile(madeProgress);
     if (!stat.ok()) {
       return stat;
     }
@@ -1493,7 +1611,7 @@ Status DBImpl::BackgroundCompaction(bool* madeProgress,
     versions_->ReleaseCompactionFiles(c.get(), status);
     *madeProgress = true;
   } else {
-    MaybeScheduleCompaction(); // do more compaction work in parallel.
+    MaybeScheduleFlushOrCompaction(); // do more compaction work in parallel.
     CompactionState* compact = new CompactionState(c.get());
     status = DoCompactionWork(compact);
     CleanupCompaction(compact);
@@ -1619,7 +1737,8 @@ Status DBImpl::OpenCompactionOutputFile(CompactionState* compact) {
       1.1 * versions_->MaxFileSizeForLevel(compact->compaction->output_level()));
 
     compact->builder.reset(new TableBuilder(options_, compact->outfile.get(),
-                                            compact->compaction->output_level()));
+                                            compact->compaction->output_level(),
+                                            compact->compaction->enable_compression()));
   }
   return s;
 }
@@ -1812,11 +1931,12 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
   }
   for (; input->Valid() && !shutting_down_.Acquire_Load(); ) {
     // Prioritize immutable compaction work
+    // TODO: remove memtable flush from normal compaction work
     if (imm_.imm_flush_needed.NoBarrier_Load() != nullptr) {
       const uint64_t imm_start = env_->NowMicros();
       mutex_.Lock();
       if (imm_.IsFlushPending(options_.min_write_buffer_number_to_merge)) {
-        CompactMemTable();
+        FlushMemTableToOutputFile();
         bg_cv_.SignalAll();  // Wakeup MakeRoomForWrite() if necessary
       }
       mutex_.Unlock();
@@ -1868,9 +1988,9 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
           compaction_filter_value.clear();
           bool to_delete =
             compaction_filter->Filter(compact->compaction->level(),
-                                               ikey.user_key, value,
-                                               &compaction_filter_value,
-                                               &value_changed);
+                                      ikey.user_key, value,
+                                      &compaction_filter_value,
+                                      &value_changed);
           if (to_delete) {
             // make a copy of the original key
             delete_key.assign(key.data(), key.data() + key.size());
@@ -2108,7 +2228,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
   }
 
   mutex_.Lock();
-  stats_[compact->compaction->level() + 1].Add(stats);
+  stats_[compact->compaction->output_level()].Add(stats);
 
   // if there were any unused file number (mostly in case of
   // compaction error), free up the entry from pending_putputs
@@ -2258,7 +2378,7 @@ Status DBImpl::GetImpl(const ReadOptions& options,
 
   if (!options_.disable_seek_compaction &&
       have_stat_update && current->UpdateStats(stats)) {
-    MaybeScheduleCompaction();
+    MaybeScheduleFlushOrCompaction();
   }
   mem->Unref();
   imm.UnrefAll();
@@ -2336,7 +2456,7 @@ std::vector<Status> DBImpl::MultiGet(const ReadOptions& options,
   mutex_.Lock();
   if (!options_.disable_seek_compaction &&
       have_stat_update && current->UpdateStats(stats)) {
-    MaybeScheduleCompaction();
+    MaybeScheduleFlushOrCompaction();
   }
   mem->Unref();
   imm.UnrefAll();
@@ -2478,10 +2598,12 @@ Status DBImpl::Write(const WriteOptions& options, WriteBatch* my_batch) {
           throw std::runtime_error("In memory WriteBatch corruption!");
         }
         SetTickerCount(options_.statistics, SEQUENCE_NUMBER, last_sequence);
+      }
+      mutex_.Lock();
+      if (status.ok()) {
         versions_->SetLastSequence(last_sequence);
         last_flushed_sequence_ = current_sequence;
       }
-      mutex_.Lock();
     }
     if (updates == &tmp_batch_) tmp_batch_.Clear();
   }
@@ -2729,7 +2851,7 @@ Status DBImpl::MakeRoomForWrite(bool force) {
       EnvOptions soptions(storage_options_);
       soptions.use_mmap_writes = false;
       s = env_->NewWritableFile(
-            LogFileName(dbname_, new_log_number),
+            LogFileName(options_.wal_dir, new_log_number),
             &lfile,
             soptions
           );
@@ -2743,7 +2865,7 @@ Status DBImpl::MakeRoomForWrite(bool force) {
       lfile->SetPreallocationBlockSize(1.1 * options_.write_buffer_size);
       logfile_number_ = new_log_number;
       log_.reset(new log::Writer(std::move(lfile)));
-      mem_->SetLogNumber(logfile_number_);
+      mem_->SetNextLogNumber(logfile_number_);
       imm_.Add(mem_);
       if (force) {
         imm_.FlushRequested();
@@ -2751,8 +2873,9 @@ Status DBImpl::MakeRoomForWrite(bool force) {
       mem_ = new MemTable(internal_comparator_, mem_rep_factory_,
         NumberLevels(), options_);
       mem_->Ref();
+      mem_->SetLogNumber(logfile_number_);
       force = false;   // Do not force another compaction if have room
-      MaybeScheduleCompaction();
+      MaybeScheduleFlushOrCompaction();
     }
   }
   return s;
@@ -2763,7 +2886,7 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
 
   MutexLock l(&mutex_);
   Slice in = property;
-  Slice prefix("leveldb.");
+  Slice prefix("rocksdb.");
   if (!in.starts_with(prefix)) return false;
   in.remove_prefix(prefix.size());
 
@@ -2956,6 +3079,9 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
   } else if (in == "sstables") {
     *value = versions_->current()->DebugString();
     return true;
+  } else if (in == "num-immutable-mem-table") {
+    *value = std::to_string(imm_.size());
+    return true;
   }
 
   return false;
@@ -3090,8 +3216,15 @@ Status DB::Open(const Options& options, const std::string& dbname, DB** dbptr) {
     return Status::InvalidArgument(
         "no_block_cache is true while block_cache is not nullptr");
   }
+
   DBImpl* impl = new DBImpl(options, dbname);
-  Status s = impl->CreateArchivalDirectory();
+  Status s = impl->env_->CreateDirIfMissing(impl->options_.wal_dir);
+  if (!s.ok()) {
+    delete impl;
+    return s;
+  }
+
+  s = impl->CreateArchivalDirectory();
   if (!s.ok()) {
     delete impl;
     return s;
@@ -3103,8 +3236,11 @@ Status DB::Open(const Options& options, const std::string& dbname, DB** dbptr) {
     uint64_t new_log_number = impl->versions_->NewFileNumber();
     unique_ptr<WritableFile> lfile;
     soptions.use_mmap_writes = false;
-    s = options.env->NewWritableFile(LogFileName(dbname, new_log_number),
-                                     &lfile, soptions);
+    s = options.env->NewWritableFile(
+      LogFileName(impl->options_.wal_dir, new_log_number),
+      &lfile,
+      soptions
+    );
     if (s.ok()) {
       lfile->SetPreallocationBlockSize(1.1 * options.write_buffer_size);
       edit.SetLogNumber(new_log_number);
@@ -3113,15 +3249,15 @@ Status DB::Open(const Options& options, const std::string& dbname, DB** dbptr) {
       s = impl->versions_->LogAndApply(&edit, &impl->mutex_);
     }
     if (s.ok()) {
+      impl->mem_->SetLogNumber(impl->logfile_number_);
       impl->DeleteObsoleteFiles();
-      impl->MaybeScheduleCompaction();
+      impl->MaybeScheduleFlushOrCompaction();
       impl->MaybeScheduleLogDBDeployStats();
     }
   }
   impl->mutex_.Unlock();
 
   if (options.compaction_style == kCompactionStyleUniversal) {
-    std::string property;
     int num_files;
     for (int i = 1; i < impl->NumberLevels(); i++) {
       num_files = impl->versions_->NumLevelFiles(i);
@@ -3145,13 +3281,24 @@ Snapshot::~Snapshot() {
 }
 
 Status DestroyDB(const std::string& dbname, const Options& options) {
-  Env* env = options.env;
+  const InternalKeyComparator comparator(options.comparator);
+  const InternalFilterPolicy filter_policy(options.filter_policy);
+  const Options& soptions(SanitizeOptions(
+    dbname, &comparator, &filter_policy, options));
+  Env* env = soptions.env;
   std::vector<std::string> filenames;
   std::vector<std::string> archiveFiles;
 
+  std::string archivedir = ArchivalDirectory(dbname);
   // Ignore error in case directory does not exist
   env->GetChildren(dbname, &filenames);
-  env->GetChildren(ArchivalDirectory(dbname), &archiveFiles);
+
+  if (dbname != soptions.wal_dir) {
+    std::vector<std::string> logfilenames;
+    env->GetChildren(soptions.wal_dir, &logfilenames);
+    filenames.insert(filenames.end(), logfilenames.begin(), logfilenames.end());
+    archivedir = ArchivalDirectory(soptions.wal_dir);
+  }
 
   if (filenames.empty()) {
     return Status::OK();
@@ -3169,6 +3316,8 @@ Status DestroyDB(const std::string& dbname, const Options& options) {
         Status del;
         if (type == kMetaDatabase) {
           del = DestroyDB(dbname + "/" + filenames[i], options);
+        } else if (type == kLogFile) {
+          del = env->DeleteFile(soptions.wal_dir + "/" + filenames[i]);
         } else {
           del = env->DeleteFile(dbname + "/" + filenames[i]);
         }
@@ -3178,23 +3327,24 @@ Status DestroyDB(const std::string& dbname, const Options& options) {
       }
     }
 
+    env->GetChildren(archivedir, &archiveFiles);
     // Delete archival files.
     for (size_t i = 0; i < archiveFiles.size(); ++i) {
-      ParseFileName(archiveFiles[i], &number, &type);
-      if (type == kLogFile) {
-        Status del = env->DeleteFile(ArchivalDirectory(dbname) + "/" +
-                                     archiveFiles[i]);
+      if (ParseFileName(archiveFiles[i], &number, &type) &&
+          type == kLogFile) {
+        Status del = env->DeleteFile(archivedir + "/" + archiveFiles[i]);
         if (result.ok() && !del.ok()) {
           result = del;
         }
       }
     }
     // ignore case where no archival directory is present.
-    env->DeleteDir(ArchivalDirectory(dbname));
+    env->DeleteDir(archivedir);
 
     env->UnlockFile(lock);  // Ignore error since state is already gone
     env->DeleteFile(lockname);
     env->DeleteDir(dbname);  // Ignore error in case dir contains other files
+    env->DeleteDir(soptions.wal_dir);
   }
   return result;
 }
@@ -3202,9 +3352,9 @@ Status DestroyDB(const std::string& dbname, const Options& options) {
 //
 // A global method that can dump out the build version
 void dumpLeveldbBuildVersion(Logger * log) {
-  Log(log, "Git sha %s", leveldb_build_git_sha);
+  Log(log, "Git sha %s", rocksdb_build_git_sha);
   Log(log, "Compile time %s %s",
-      leveldb_build_compile_time, leveldb_build_compile_date);
+      rocksdb_build_compile_time, rocksdb_build_compile_date);
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/db_impl.h

@@ -1,10 +1,12 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
-
-#ifndef STORAGE_LEVELDB_DB_DB_IMPL_H_
-#define STORAGE_LEVELDB_DB_DB_IMPL_H_
-
+#pragma once
 #include <atomic>
 #include <deque>
 #include <set>
@@ -20,11 +22,7 @@
 #include "util/stats_logger.h"
 #include "memtablelist.h"
 
-#ifdef USE_SCRIBE
-#include "scribe/scribe_logger.h"
-#endif
-
-namespace leveldb {
+namespace rocksdb {
 
 class MemTable;
 class TableCache;
@@ -72,7 +70,8 @@ class DBImpl : public DB {
   virtual Status DisableFileDeletions();
   virtual Status EnableFileDeletions();
   virtual Status GetLiveFiles(std::vector<std::string>&,
-                              uint64_t* manifest_file_size);
+                              uint64_t* manifest_file_size,
+                              bool flush_memtable = true);
   virtual Status GetSortedWalFiles(VectorLogPtr& files);
   virtual Status DeleteWalFiles(const VectorLogPtr& files);
   virtual SequenceNumber GetLatestSequenceNumber();
@@ -88,11 +87,11 @@ class DBImpl : public DB {
   // Compact any files in the named level that overlap [*begin, *end]
   void TEST_CompactRange(int level, const Slice* begin, const Slice* end);
 
-  // Force current memtable contents to be compacted.
-  Status TEST_CompactMemTable();
+  // Force current memtable contents to be flushed.
+  Status TEST_FlushMemTable();
 
   // Wait for memtable compaction
-  Status TEST_WaitForCompactMemTable();
+  Status TEST_WaitForFlushMemTable();
 
   // Wait for any compaction
   Status TEST_WaitForCompact();
@@ -115,6 +114,9 @@ class DBImpl : public DB {
   // Trigger's a background call for testing.
   void TEST_PurgeObsoleteteWAL();
 
+  // get total level0 file size. Only for testing.
+  uint64_t TEST_GetLevel0TotalSize() { return versions_->NumLevelBytes(0);}
+
  protected:
   Env* const env_;
   const std::string dbname_;
@@ -154,9 +156,9 @@ class DBImpl : public DB {
   // Delete any unneeded files and stale in-memory entries.
   void DeleteObsoleteFiles();
 
-  // Compact the in-memory write buffer to disk.  Switches to a new
+  // Flush the in-memory write buffer to storage.  Switches to a new
   // log-file/memtable and writes a new descriptor iff successful.
-  Status CompactMemTable(bool* madeProgress = nullptr);
+  Status FlushMemTableToOutputFile(bool* madeProgress = nullptr);
 
   Status RecoverLogFile(uint64_t log_number,
                         VersionEdit* edit,
@@ -179,17 +181,20 @@ class DBImpl : public DB {
   // Force current memtable contents to be flushed.
   Status FlushMemTable(const FlushOptions& options);
 
-  // Wait for memtable compaction
-  Status WaitForCompactMemTable();
+  // Wait for memtable flushed
+  Status WaitForFlushMemTable();
 
   void MaybeScheduleLogDBDeployStats();
   static void BGLogDBDeployStats(void* db);
   void LogDBDeployStats();
 
-  void MaybeScheduleCompaction();
-  static void BGWork(void* db);
-  void BackgroundCall();
+  void MaybeScheduleFlushOrCompaction();
+  static void BGWorkCompaction(void* db);
+  static void BGWorkFlush(void* db);
+  void BackgroundCallCompaction();
+  void BackgroundCallFlush();
   Status BackgroundCompaction(bool* madeProgress,DeletionState& deletion_state);
+  Status BackgroundFlush(bool* madeProgress);
   void CleanupCompaction(CompactionState* compact);
   Status DoCompactionWork(CompactionState* compact);
 
@@ -212,6 +217,8 @@ class DBImpl : public DB {
   // Removes the file listed in files_to_evict from the table_cache
   void EvictObsoleteFiles(DeletionState& deletion_state);
 
+  Status DeleteLogFile(uint64_t number);
+
   void PurgeObsoleteWALFiles();
 
   Status AppendSortedWalsOfType(const std::string& path,
@@ -234,7 +241,7 @@ class DBImpl : public DB {
 
   void PrintStatistics();
 
-  // dump leveldb.stats to LOG
+  // dump rocksdb.stats to LOG
   void MaybeDumpStats();
 
   // Return the minimum empty level that could hold the total data in the
@@ -281,6 +288,9 @@ class DBImpl : public DB {
   // count how many background compaction been scheduled or is running?
   int bg_compaction_scheduled_;
 
+  // number of background memtable flush jobs, submitted to the HIGH pool
+  int bg_flush_scheduled_;
+
   // Has a background stats log thread scheduled?
   bool bg_logstats_scheduled_;
 
@@ -435,6 +445,4 @@ extern Options SanitizeOptions(const std::string& db,
                                const InternalFilterPolicy* ipolicy,
                                const Options& src);
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_DB_IMPL_H_
+}  // namespace rocksdb

db/db_impl_readonly.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2012 Facebook. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
@@ -34,7 +39,7 @@
 #include "util/logging.h"
 #include "util/build_version.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 DBImplReadOnly::DBImplReadOnly(const Options& options,
     const std::string& dbname)

db/db_impl_readonly.h

@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2012 Facebook. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
-#ifndef STORAGE_LEVELDB_DB_DB_IMPL_READONLY_H_
-#define STORAGE_LEVELDB_DB_DB_IMPL_READONLY_H_
-
+#pragma once
 #include "db/db_impl.h"
 
 #include <deque>
@@ -17,11 +20,7 @@
 #include "port/port.h"
 #include "util/stats_logger.h"
 
-#ifdef USE_SCRIBE
-#include "scribe/scribe_logger.h"
-#endif
-
-namespace leveldb {
+namespace rocksdb {
 
 class DBImplReadOnly : public DBImpl {
 public:
@@ -60,7 +59,8 @@ public:
    return Status::NotSupported("Not supported operation in read only mode.");
  }
  virtual Status GetLiveFiles(std::vector<std::string>&,
-                             uint64_t* manifest_file_size) {
+                             uint64_t* manifest_file_size,
+                             bool flush_memtable = true) {
    return Status::NotSupported("Not supported operation in read only mode.");
  }
  virtual Status Flush(const FlushOptions& options) {
@@ -76,5 +76,3 @@ private:
 };
 
 }
-
-#endif

db/db_iter.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -15,8 +20,9 @@
 #include "port/port.h"
 #include "util/logging.h"
 #include "util/mutexlock.h"
+#include "util/perf_context_imp.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 #if 0
 static void DumpInternalIter(Iterator* iter) {
@@ -197,6 +203,7 @@ void DBIter::FindNextUserEntry(bool skipping) {
       if (skipping &&
           user_comparator_->Compare(ikey.user_key, saved_key_) <= 0) {
         num_skipped++; // skip this entry
+        BumpPerfCount(&perf_context.internal_key_skipped_count);
       } else {
         skipping = false;
         switch (ikey.type) {
@@ -206,6 +213,7 @@ void DBIter::FindNextUserEntry(bool skipping) {
             SaveKey(ikey.user_key, &saved_key_);
             skipping = true;
             num_skipped = 0;
+            BumpPerfCount(&perf_context.internal_delete_skipped_count);
             break;
           case kTypeValue:
             valid_ = true;
@@ -365,6 +373,7 @@ void DBIter::FindPrevUserEntry() {
   if (iter_->Valid()) {
     do {
       ParsedInternalKey ikey;
+      bool saved_key_cleared = false;
       if (ParseKey(&ikey) && ikey.sequence <= sequence_) {
         if ((value_type != kTypeDeletion) &&
             user_comparator_->Compare(ikey.user_key, saved_key_) < 0) {
@@ -375,6 +384,7 @@ void DBIter::FindPrevUserEntry() {
         if (value_type == kTypeDeletion) {
           saved_key_.clear();
           ClearSavedValue();
+          saved_key_cleared = true;
         } else {
           Slice raw_value = iter_->value();
           if (saved_value_.capacity() > raw_value.size() + 1048576) {
@@ -390,7 +400,7 @@ void DBIter::FindPrevUserEntry() {
       // found the prev user-key, then it is better to seek so that we can
       // avoid too many key comparisons. We seek to the first occurence of
       // our current key by looking for max sequence number.
-      if (num_skipped > max_skip_) {
+      if (!saved_key_cleared && num_skipped > max_skip_) {
         num_skipped = 0;
         std::string last_key;
         AppendInternalKey(&last_key,
@@ -468,4 +478,4 @@ Iterator* NewDBIterator(
                     internal_iter, sequence);
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/db_iter.h

@@ -1,15 +1,18 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#ifndef STORAGE_LEVELDB_DB_DB_ITER_H_
-#define STORAGE_LEVELDB_DB_DB_ITER_H_
-
+#pragma once
 #include <stdint.h>
 #include "rocksdb/db.h"
 #include "db/dbformat.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 // Return a new iterator that converts internal keys (yielded by
 // "*internal_iter") that were live at the specified "sequence" number
@@ -22,6 +25,4 @@ extern Iterator* NewDBIterator(
     Iterator* internal_iter,
     const SequenceNumber& sequence);
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_DB_ITER_H_
+}  // namespace rocksdb

db/db_statistics.h

@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#ifndef LEVELDB_STORAGE_DB_DB_STATISTICS_H_
-#define LEVELDB_STORAGE_DB_DB_STATISTICS_H_
-
+#pragma once
 #include <cassert>
 #include <stdlib.h>
 #include <vector>
@@ -16,7 +19,7 @@
 #include "util/mutexlock.h"
 
 
-namespace leveldb {
+namespace rocksdb {
 
 class DBStatistics: public Statistics {
  public:
@@ -59,6 +62,4 @@ std::shared_ptr<Statistics> CreateDBStatistics() {
   return std::make_shared<DBStatistics>();
 }
 
-} // namespace leveldb
-
-#endif // LEVELDB_STORAGE_DB_DB_STATISTICS_H_
+} // namespace rocksdb

db/db_stats_logger.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -12,7 +17,7 @@
 #include "port/port.h"
 #include "util/mutexlock.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 void DBImpl::MaybeScheduleLogDBDeployStats() {
 

db/dbformat.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -8,7 +13,7 @@
 #include "util/coding.h"
 #include "util/perf_context_imp.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 static uint64_t PackSequenceAndType(uint64_t seq, ValueType t) {
   assert(seq <= kMaxSequenceNumber);
@@ -139,4 +144,4 @@ LookupKey::LookupKey(const Slice& user_key, SequenceNumber s) {
   end_ = dst;
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/dbformat.h

@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#ifndef STORAGE_LEVELDB_DB_FORMAT_H_
-#define STORAGE_LEVELDB_DB_FORMAT_H_
-
+#pragma once
 #include <stdio.h>
 #include "rocksdb/comparator.h"
 #include "rocksdb/db.h"
@@ -15,7 +18,7 @@
 #include "util/coding.h"
 #include "util/logging.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class InternalKey;
 
@@ -90,7 +93,7 @@ class InternalKeyComparator : public Comparator {
   std::string name_;
  public:
   explicit InternalKeyComparator(const Comparator* c) : user_comparator_(c),
-    name_("leveldb.InternalKeyComparator:" +
+    name_("rocksdb.InternalKeyComparator:" +
           std::string(user_comparator_->Name())) {
   }
 
@@ -223,6 +226,4 @@ inline LookupKey::~LookupKey() {
   if (start_ != space_) delete[] start_;
 }
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_FORMAT_H_
+}  // namespace rocksdb

db/dbformat_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -6,7 +11,7 @@
 #include "util/logging.h"
 #include "util/testharness.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 static std::string IKey(const std::string& user_key,
                         uint64_t seq,
@@ -105,8 +110,8 @@ TEST(FormatTest, InternalKeyShortestSuccessor) {
             ShortSuccessor(IKey("\xff\xff", 100, kTypeValue)));
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }

db/deletefile_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -9,14 +14,13 @@
 #include "db/write_batch_internal.h"
 #include "util/testharness.h"
 #include "util/testutil.h"
-#include "boost/lexical_cast.hpp"
 #include "rocksdb/env.h"
 #include <vector>
-#include <boost/algorithm/string.hpp>
 #include <stdlib.h>
 #include <map>
+#include <string>
 
-namespace leveldb {
+namespace rocksdb {
 
 class DeleteFileTest {
  public:
@@ -57,7 +61,7 @@ class DeleteFileTest {
     options.sync = false;
     ReadOptions roptions;
     for (int i = startkey; i < (numkeys + startkey) ; i++) {
-      std::string temp = boost::lexical_cast<std::string>(i);
+      std::string temp = std::to_string(i);
       Slice key(temp);
       Slice value(temp);
       ASSERT_OK(db_->Put(options, key, value));
@@ -92,12 +96,12 @@ class DeleteFileTest {
   void CreateTwoLevels() {
     AddKeys(50000, 10000);
     DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-    ASSERT_OK(dbi->TEST_CompactMemTable());
-    ASSERT_OK(dbi->TEST_WaitForCompactMemTable());
+    ASSERT_OK(dbi->TEST_FlushMemTable());
+    ASSERT_OK(dbi->TEST_WaitForFlushMemTable());
 
     AddKeys(50000, 10000);
-    ASSERT_OK(dbi->TEST_CompactMemTable());
-    ASSERT_OK(dbi->TEST_WaitForCompactMemTable());
+    ASSERT_OK(dbi->TEST_FlushMemTable());
+    ASSERT_OK(dbi->TEST_WaitForFlushMemTable());
   }
 
 };
@@ -180,9 +184,9 @@ TEST(DeleteFileTest, DeleteFileWithIterator) {
   delete it;
   CloseDB();
 }
-} //namespace leveldb
+} //namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }
 

db/filename.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -10,7 +15,7 @@
 #include "rocksdb/env.h"
 #include "util/logging.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 // Given a path, flatten the path name by replacing all chars not in
 // {[0-9,a-z,A-Z,-,_,.]} with _. And append '\0' at the end.
@@ -57,8 +62,8 @@ std::string LogFileName(const std::string& name, uint64_t number) {
   return MakeFileName(name, number, "log");
 }
 
-std::string ArchivalDirectory(const std::string& dbname) {
-  return dbname + "/" + ARCHIVAL_DIR;
+std::string ArchivalDirectory(const std::string& dir) {
+  return dir + "/" + ARCHIVAL_DIR;
 }
 std::string ArchivedLogFileName(const std::string& name, uint64_t number) {
   assert(number > 0);
@@ -122,7 +127,12 @@ std::string MetaDatabaseName(const std::string& dbname, uint64_t number) {
   return dbname + buf;
 }
 
+std::string IdentityFileName(const std::string& dbname) {
+  return dbname + "/IDENTITY";
+}
+
 // Owned filenames have the form:
+//    dbname/IDENTITY
 //    dbname/CURRENT
 //    dbname/LOCK
 //    dbname/LOG
@@ -138,7 +148,10 @@ bool ParseFileName(const std::string& fname,
   if (fname.length() > 1 && fname[0] == '/') {
     rest.remove_prefix(1);
   }
-  if (rest == "CURRENT") {
+  if (rest == "IDENTITY") {
+    *number = 0;
+    *type = kIdentityFile;
+  } else if (rest == "CURRENT") {
     *number = 0;
     *type = kCurrentFile;
   } else if (rest == "LOCK") {
@@ -218,4 +231,18 @@ Status SetCurrentFile(Env* env, const std::string& dbname,
   return s;
 }
 
-}  // namespace leveldb
+Status SetIdentityFile(Env* env, const std::string& dbname) {
+  std::string id = env->GenerateUniqueId();
+  assert(!id.empty());
+  std::string tmp = TempFileName(dbname, id.size());
+  Status s = WriteStringToFileSync(env, id, tmp);
+  if (s.ok()) {
+    s = env->RenameFile(tmp, IdentityFileName(dbname));
+  }
+  if (!s.ok()) {
+    env->DeleteFile(tmp);
+  }
+  return s;
+}
+
+}  // namespace rocksdb

db/filename.h

@@ -1,19 +1,22 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 //
 // File names used by DB code
 
-#ifndef STORAGE_LEVELDB_DB_FILENAME_H_
-#define STORAGE_LEVELDB_DB_FILENAME_H_
-
+#pragma once
 #include <stdint.h>
 #include <string>
 #include "rocksdb/slice.h"
 #include "rocksdb/status.h"
 #include "port/port.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class Env;
 
@@ -25,7 +28,8 @@ enum FileType {
   kCurrentFile,
   kTempFile,
   kInfoLogFile,  // Either the current one, or an old one
-  kMetaDatabase
+  kMetaDatabase,
+  kIdentityFile
 };
 
 // Return the name of the log file with the specified number
@@ -79,7 +83,12 @@ extern std::string OldInfoLogFileName(const std::string& dbname, uint64_t ts,
 extern std::string MetaDatabaseName(const std::string& dbname,
                                     uint64_t number);
 
-// If filename is a leveldb file, store the type of the file in *type.
+// Return the name of the Identity file which stores a unique number for the db
+// that will get regenerated if the db loses all its data and is recreated fresh
+// either from a backup-image or empty
+extern std::string IdentityFileName(const std::string& dbname);
+
+// If filename is a rocksdb file, store the type of the file in *type.
 // The number encoded in the filename is stored in *number.  If the
 // filename was successfully parsed, returns true.  Else return false.
 extern bool ParseFileName(const std::string& filename,
@@ -91,7 +100,7 @@ extern bool ParseFileName(const std::string& filename,
 extern Status SetCurrentFile(Env* env, const std::string& dbname,
                              uint64_t descriptor_number);
 
+// Make the IDENTITY file for the db
+extern Status SetIdentityFile(Env* env, const std::string& dbname);
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_FILENAME_H_
+}  // namespace rocksdb

db/filename_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -9,7 +14,7 @@
 #include "util/logging.h"
 #include "util/testharness.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class FileNameTest { };
 
@@ -128,8 +133,8 @@ TEST(FileNameTest, Construction) {
   ASSERT_EQ(kMetaDatabase, type);
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }

db/log_format.h

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -5,10 +10,8 @@
 // Log format information shared by reader and writer.
 // See ../doc/log_format.txt for more detail.
 
-#ifndef STORAGE_LEVELDB_DB_LOG_FORMAT_H_
-#define STORAGE_LEVELDB_DB_LOG_FORMAT_H_
-
-namespace leveldb {
+#pragma once
+namespace rocksdb {
 namespace log {
 
 enum RecordType {
@@ -30,6 +33,4 @@ static const unsigned int kBlockSize = 32768;
 static const int kHeaderSize = 4 + 1 + 2;
 
 }  // namespace log
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_LOG_FORMAT_H_
+}  // namespace rocksdb

db/log_reader.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -9,7 +14,7 @@
 #include "util/coding.h"
 #include "util/crc32c.h"
 
-namespace leveldb {
+namespace rocksdb {
 namespace log {
 
 Reader::Reporter::~Reporter() {
@@ -256,4 +261,4 @@ unsigned int Reader::ReadPhysicalRecord(Slice* result) {
 }
 
 }  // namespace log
-}  // namespace leveldb
+}  // namespace rocksdb

db/log_reader.h

@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#ifndef STORAGE_LEVELDB_DB_LOG_READER_H_
-#define STORAGE_LEVELDB_DB_LOG_READER_H_
-
+#pragma once
 #include <memory>
 #include <stdint.h>
 
@@ -12,7 +15,7 @@
 #include "rocksdb/slice.h"
 #include "rocksdb/status.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class SequentialFile;
 using std::unique_ptr;
@@ -118,6 +121,4 @@ class Reader {
 };
 
 }  // namespace log
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_LOG_READER_H_
+}  // namespace rocksdb

db/log_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -10,7 +15,7 @@
 #include "util/random.h"
 #include "util/testharness.h"
 
-namespace leveldb {
+namespace rocksdb {
 namespace log {
 
 // Construct a string of the specified length made out of the supplied
@@ -516,8 +521,8 @@ TEST(LogTest, ReadPastEnd) {
 }
 
 }  // namespace log
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }

db/log_writer.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -9,7 +14,7 @@
 #include "util/coding.h"
 #include "util/crc32c.h"
 
-namespace leveldb {
+namespace rocksdb {
 namespace log {
 
 Writer::Writer(unique_ptr<WritableFile>&& dest)
@@ -100,4 +105,4 @@ Status Writer::EmitPhysicalRecord(RecordType t, const char* ptr, size_t n) {
 }
 
 }  // namespace log
-}  // namespace leveldb
+}  // namespace rocksdb

db/log_writer.h

@@ -1,17 +1,20 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#ifndef STORAGE_LEVELDB_DB_LOG_WRITER_H_
-#define STORAGE_LEVELDB_DB_LOG_WRITER_H_
-
+#pragma once
 #include <memory>
 #include <stdint.h>
 #include "db/log_format.h"
 #include "rocksdb/slice.h"
 #include "rocksdb/status.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class WritableFile;
 
@@ -49,6 +52,4 @@ class Writer {
 };
 
 }  // namespace log
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_LOG_WRITER_H_
+}  // namespace rocksdb

db/memtable.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -14,7 +19,7 @@
 #include "util/coding.h"
 #include "util/murmurhash.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 MemTable::MemTable(const InternalKeyComparator& cmp,
                    std::shared_ptr<MemTableRepFactory> table_factory,
@@ -29,6 +34,7 @@ MemTable::MemTable(const InternalKeyComparator& cmp,
       file_number_(0),
       edit_(numlevel),
       first_seqno_(0),
+      mem_next_logfile_number_(0),
       mem_logfile_number_(0) { }
 
 MemTable::~MemTable() {
@@ -237,4 +243,4 @@ bool MemTable::Get(const LookupKey& key, std::string* value, Status* s,
   return false;
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/memtable.h

@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#ifndef STORAGE_LEVELDB_DB_MEMTABLE_H_
-#define STORAGE_LEVELDB_DB_MEMTABLE_H_
-
+#pragma once
 #include <string>
 #include <memory>
 #include <deque>
@@ -15,7 +18,7 @@
 #include "rocksdb/memtablerep.h"
 #include "util/arena_impl.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class Mutex;
 class MemTableIterator;
@@ -92,6 +95,14 @@ class MemTable {
   // into the memtable
   SequenceNumber GetFirstSequenceNumber() { return first_seqno_; }
 
+  // Returns the next active logfile number when this memtable is about to
+  // be flushed to storage
+  uint64_t GetNextLogNumber() { return mem_next_logfile_number_; }
+
+  // Sets the next active logfile number when this memtable is about to
+  // be flushed to storage
+  void SetNextLogNumber(uint64_t num) { mem_next_logfile_number_ = num; }
+
   // Returns the logfile number that can be safely deleted when this
   // memstore is flushed to storage
   uint64_t GetLogNumber() { return mem_logfile_number_; }
@@ -127,6 +138,10 @@ class MemTable {
   SequenceNumber first_seqno_;
 
   // The log files earlier than this number can be deleted.
+  uint64_t mem_next_logfile_number_;
+
+  // The log file that backs this memtable (to be deleted when
+  // memtable flush is done)
   uint64_t mem_logfile_number_;
 
   // No copying allowed
@@ -134,6 +149,4 @@ class MemTable {
   void operator=(const MemTable&);
 };
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_MEMTABLE_H_
+}  // namespace rocksdb

db/memtablelist.cc

@@ -1,6 +1,8 @@
-// Copyright (c) 2012 Facebook.
-
-
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #include "db/memtablelist.h"
 
 #include <string>
@@ -10,7 +12,7 @@
 #include "rocksdb/iterator.h"
 #include "util/coding.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class InternalKeyComparator;
 class Mutex;
@@ -210,4 +212,4 @@ void MemTableList::GetMemTables(std::vector<MemTable*>* output) {
   }
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/memtablelist.h

@@ -1,8 +1,10 @@
-// Copyright (c) 2012 Facebook.
-
-#ifndef STORAGE_LEVELDB_DB_MEMTABLELIST_H_
-#define STORAGE_LEVELDB_DB_MEMTABLELIST_H_
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 
+#pragma once
 #include <string>
 #include <list>
 #include <deque>
@@ -11,7 +13,7 @@
 #include "db/skiplist.h"
 #include "memtable.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class InternalKeyComparator;
 class Mutex;
@@ -99,6 +101,4 @@ class MemTableList {
 
 };
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_MEMTABLELIST_H_
+}  // namespace rocksdb

db/merge_helper.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #include "merge_helper.h"
 #include "db/dbformat.h"
 #include "rocksdb/comparator.h"
@@ -6,7 +11,7 @@
 #include <string>
 #include <stdio.h>
 
-namespace leveldb {
+namespace rocksdb {
 
 // PRE:  iter points to the first merge type entry
 // POST: iter points to the first entry beyond the merge process (or the end)
@@ -189,4 +194,4 @@ void MergeHelper::MergeUntil(Iterator* iter, SequenceNumber stop_before,
   }
 }
 
-} // namespace leveldb
+} // namespace rocksdb

db/merge_helper.h

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #ifndef MERGE_HELPER_H
 #define MERGE_HELPER_H
 
@@ -7,7 +12,7 @@
 #include <string>
 #include <deque>
 
-namespace leveldb {
+namespace rocksdb {
 
 class Comparator;
 class Iterator;
@@ -93,6 +98,6 @@ class MergeHelper {
   bool success_;
 };
 
-} // namespace leveldb
+} // namespace rocksdb
 
 #endif

db/merge_operator.cc

@@ -1,13 +1,15 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 /**
  * Back-end implementation details specific to the Merge Operator.
- *
- * @author Deon Nicholas (dnicholas@fb.com)
- * Copyright 2013 Facebook
  */
 
 #include "rocksdb/merge_operator.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 // Given a "real" merge from the library, call the user's
 // associative merge function one-by-one on each of the operands.
@@ -48,4 +50,4 @@ bool AssociativeMergeOperator::PartialMerge(
   return Merge(key, &left_operand, right_operand, new_value, logger);
 }
 
-} // namespace leveldb
+} // namespace rocksdb

db/merge_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #include <assert.h>
 #include <memory>
 #include <iostream>
@@ -14,7 +19,7 @@
 #include "utilities/utility_db.h"
 
 using namespace std;
-using namespace leveldb;
+using namespace rocksdb;
 
 
 std::shared_ptr<DB> OpenDb(const string& dbname, const bool ttl = false) {
@@ -85,7 +90,7 @@ class Counters {
     }
   }
 
-  // mapped to a leveldb Delete
+  // mapped to a rocksdb Delete
   bool remove(const string& key) {
     auto s = db_->Delete(delete_option_, key);
 
@@ -97,7 +102,7 @@ class Counters {
     }
   }
 
-  // mapped to a leveldb Get
+  // mapped to a rocksdb Get
   bool get(const string& key, uint64_t *value) {
     string str;
     auto s = db_->Get(get_option_, key, &str);
@@ -159,7 +164,7 @@ class MergeBasedCounters : public Counters {
         merge_option_() {
   }
 
-  // mapped to a leveldb Merge operation
+  // mapped to a rocksdb Merge operation
   virtual bool add(const string& key, uint64_t value) override {
     char encoded[sizeof(uint64_t)];
     EncodeFixed64(encoded, value);

db/perf_context_test.cc

@@ -1,5 +1,12 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
+#include <algorithm>
 #include <iostream>
 #include <vector>
+#include "/usr/include/valgrind/callgrind.h"
 
 #include "rocksdb/db.h"
 #include "rocksdb/perf_context.h"
@@ -8,16 +15,43 @@
 #include "util/testharness.h"
 
 
-namespace leveldb {
+bool FLAGS_random_key = false;
+bool FLAGS_use_set_based_memetable = false;
+int FLAGS_total_keys = 100;
+int FLAGS_write_buffer_size = 1000000000;
+int FLAGS_max_write_buffer_number = 8;
+int FLAGS_min_write_buffer_number_to_merge = 7;
 
 // Path to the database on file system
-const std::string kDbName = test::TmpDir() + "/perf_context_test";
+const std::string kDbName = rocksdb::test::TmpDir() + "/perf_context_test";
 
-std::shared_ptr<DB> OpenDb(size_t write_buffer_size) {
+void SeekToFirst(rocksdb::Iterator* iter) {
+//  std::cout << "Press a key to continue:";
+//  std::string s;
+//  std::cin >> s;
+  iter->SeekToFirst();
+//  std::cout << "Press a key to continue:";
+//  std::string s2;
+//  std::cin >> s2;
+}
+
+namespace rocksdb {
+
+std::shared_ptr<DB> OpenDb() {
     DB* db;
     Options options;
     options.create_if_missing = true;
-    options.write_buffer_size = write_buffer_size;
+    options.write_buffer_size = FLAGS_write_buffer_size;
+    options.max_write_buffer_number = FLAGS_max_write_buffer_number;
+    options.min_write_buffer_number_to_merge =
+      FLAGS_min_write_buffer_number_to_merge;
+
+    if (FLAGS_use_set_based_memetable) {
+      auto prefix_extractor = rocksdb::NewFixedPrefixTransform(0);
+      options.memtable_factory =
+        std::make_shared<rocksdb::PrefixHashRepFactory>(prefix_extractor);
+    }
+
     Status s = DB::Open(options, kDbName,  &db);
     ASSERT_OK(s);
     return std::shared_ptr<DB>(db);
@@ -25,7 +59,85 @@ std::shared_ptr<DB> OpenDb(size_t write_buffer_size) {
 
 class PerfContextTest { };
 
-int kTotalKeys = 100;
+TEST(PerfContextTest, SeekIntoDeletion) {
+  DestroyDB(kDbName, Options());
+  auto db = OpenDb();
+  WriteOptions write_options;
+  ReadOptions read_options;
+
+  for (int i = 0; i < FLAGS_total_keys; ++i) {
+    std::string key = "k" + std::to_string(i);
+    std::string value = "v" + std::to_string(i);
+
+    db->Put(write_options, key, value);
+  }
+
+  for (int i = 0; i < FLAGS_total_keys -1 ; ++i) {
+    std::string key = "k" + std::to_string(i);
+    db->Delete(write_options, key);
+  }
+
+  HistogramImpl hist_get;
+  HistogramImpl hist_get_time;
+  for (int i = 0; i < FLAGS_total_keys - 1; ++i) {
+    std::string key = "k" + std::to_string(i);
+    std::string value;
+
+    perf_context.Reset();
+    StopWatchNano timer(Env::Default(), true);
+    auto status = db->Get(read_options, key, &value);
+    auto elapsed_nanos = timer.ElapsedNanos();
+    ASSERT_TRUE(status.IsNotFound());
+    hist_get.Add(perf_context.user_key_comparison_count);
+    hist_get_time.Add(elapsed_nanos);
+  }
+
+  std::cout << "Get uesr key comparison: \n" << hist_get.ToString()
+            << "Get time: \n" << hist_get_time.ToString();
+
+  HistogramImpl hist_seek_to_first;
+  std::unique_ptr<Iterator> iter(db->NewIterator(read_options));
+
+  perf_context.Reset();
+  StopWatchNano timer(Env::Default(), true);
+  //CALLGRIND_ZERO_STATS;
+  SeekToFirst(iter.get());
+  //iter->SeekToFirst();
+  //CALLGRIND_DUMP_STATS;
+  hist_seek_to_first.Add(perf_context.user_key_comparison_count);
+  auto elapsed_nanos = timer.ElapsedNanos();
+
+  std::cout << "SeekToFirst uesr key comparison: \n" << hist_seek_to_first.ToString()
+            << "ikey skipped: " << perf_context.internal_key_skipped_count << "\n"
+            << "idelete skipped: " << perf_context.internal_delete_skipped_count << "\n"
+            << "elapsed: " << elapsed_nanos << "\n";
+
+  HistogramImpl hist_seek;
+  for (int i = 0; i < FLAGS_total_keys; ++i) {
+    std::unique_ptr<Iterator> iter(db->NewIterator(read_options));
+    std::string key = "k" + std::to_string(i);
+
+    perf_context.Reset();
+    StopWatchNano timer(Env::Default(), true);
+    iter->Seek(key);
+    auto elapsed_nanos = timer.ElapsedNanos();
+    hist_seek.Add(perf_context.user_key_comparison_count);
+    std::cout << "seek cmp: " << perf_context.user_key_comparison_count
+              << " ikey skipped " << perf_context.internal_key_skipped_count
+              << " idelete skipped " << perf_context.internal_delete_skipped_count
+              << " elapsed: " << elapsed_nanos << "ns\n";
+
+    perf_context.Reset();
+    ASSERT_TRUE(iter->Valid());
+    StopWatchNano timer2(Env::Default(), true);
+    iter->Next();
+    auto elapsed_nanos2 = timer2.ElapsedNanos();
+    std::cout << "next cmp: " << perf_context.user_key_comparison_count
+              << "elapsed: " << elapsed_nanos2 << "ns\n";
+  }
+
+  std::cout << "Seek uesr key comparison: \n" << hist_seek.ToString();
+}
 
 TEST(PerfContextTest, StopWatchNanoOverhead) {
   // profile the timer cost by itself!
@@ -68,62 +180,146 @@ TEST(PerfContextTest, StopWatchOverhead) {
 void ProfileKeyComparison() {
   DestroyDB(kDbName, Options());    // Start this test with a fresh DB
 
-  auto db = OpenDb(1000000000);
+  auto db = OpenDb();
 
   WriteOptions write_options;
   ReadOptions read_options;
 
-  uint64_t total_user_key_comparison_get = 0;
-  uint64_t total_user_key_comparison_put = 0;
-  uint64_t max_user_key_comparison_get = 0;
+  HistogramImpl hist_put;
+  HistogramImpl hist_get;
 
-  std::cout << "Inserting " << kTotalKeys << " key/value pairs\n...\n";
+  std::cout << "Inserting " << FLAGS_total_keys << " key/value pairs\n...\n";
 
-  for (int i = 0; i < kTotalKeys; ++i) {
+  std::vector<int> keys;
+  for (int i = 0; i < FLAGS_total_keys; ++i) {
+    keys.push_back(i);
+  }
+
+  if (FLAGS_random_key) {
+    std::random_shuffle(keys.begin(), keys.end());
+  }
+
+  for (const int i : keys) {
     std::string key = "k" + std::to_string(i);
     std::string value = "v" + std::to_string(i);
 
     perf_context.Reset();
     db->Put(write_options, key, value);
-    total_user_key_comparison_put += perf_context.user_key_comparison_count;
+    hist_put.Add(perf_context.user_key_comparison_count);
 
     perf_context.Reset();
     db->Get(read_options, key, &value);
-    total_user_key_comparison_get += perf_context.user_key_comparison_count;
-    max_user_key_comparison_get =
-      std::max(max_user_key_comparison_get,
-               perf_context.user_key_comparison_count);
+    hist_get.Add(perf_context.user_key_comparison_count);
   }
 
-  std::cout << "total user key comparison get: "
-            << total_user_key_comparison_get << "\n"
-            << "total user key comparison put: "
-            << total_user_key_comparison_put << "\n"
-            << "max user key comparison get: "
-            << max_user_key_comparison_get << "\n"
-            << "avg user key comparison get:"
-            << total_user_key_comparison_get/kTotalKeys << "\n";
+  std::cout << "Put uesr key comparison: \n" << hist_put.ToString()
+            << "Get uesr key comparison: \n" << hist_get.ToString();
+
 }
 
 TEST(PerfContextTest, KeyComparisonCount) {
-  SetPerfLevel(kDisable);
+  SetPerfLevel(kEnableCount);
   ProfileKeyComparison();
 
-  SetPerfLevel(kEnableCount);
+  SetPerfLevel(kDisable);
   ProfileKeyComparison();
 
   SetPerfLevel(kEnableTime);
   ProfileKeyComparison();
 }
 
+// make perf_context_test
+// export ROCKSDB_TESTS=PerfContextTest.SeekKeyComparison
+// For one memtable:
+// ./perf_context_test --write_buffer_size=500000 --total_keys=10000
+// For two memtables:
+// ./perf_context_test --write_buffer_size=250000 --total_keys=10000
+// Specify --random_key=1 to shuffle the key before insertion
+// Results show that, for sequential insertion, worst-case Seek Key comparison
+// is close to the total number of keys (linear), when there is only one
+// memtable. When there are two memtables, even the avg Seek Key comparison
+// starts to become linear to the input size.
+
+TEST(PerfContextTest, SeekKeyComparison) {
+  DestroyDB(kDbName, Options());
+  auto db = OpenDb();
+  WriteOptions write_options;
+  ReadOptions read_options;
+
+  std::cout << "Inserting " << FLAGS_total_keys << " key/value pairs\n...\n";
+
+  std::vector<int> keys;
+  for (int i = 0; i < FLAGS_total_keys; ++i) {
+    keys.push_back(i);
+  }
+
+  if (FLAGS_random_key) {
+    std::random_shuffle(keys.begin(), keys.end());
+  }
+
+  for (const int i : keys) {
+    std::string key = "k" + std::to_string(i);
+    std::string value = "v" + std::to_string(i);
+
+    db->Put(write_options, key, value);
+  }
+
+  HistogramImpl hist_seek;
+  HistogramImpl hist_next;
+
+  for (int i = 0; i < FLAGS_total_keys; ++i) {
+    std::string key = "k" + std::to_string(i);
+    std::string value = "v" + std::to_string(i);
+
+    std::unique_ptr<Iterator> iter(db->NewIterator(read_options));
+    perf_context.Reset();
+    iter->Seek(key);
+    ASSERT_TRUE(iter->Valid());
+    ASSERT_EQ(iter->value().ToString(), value);
+    hist_seek.Add(perf_context.user_key_comparison_count);
+  }
+
+  std::unique_ptr<Iterator> iter(db->NewIterator(read_options));
+  for (iter->SeekToFirst(); iter->Valid();) {
+    perf_context.Reset();
+    iter->Next();
+    hist_next.Add(perf_context.user_key_comparison_count);
+  }
+
+  std::cout << "Seek:\n" << hist_seek.ToString()
+            << "Next:\n" << hist_next.ToString();
+}
+
 }
 
 int main(int argc, char** argv) {
 
-  if (argc > 1) {
-    leveldb::kTotalKeys = std::stoi(argv[1]);
+  for (int i = 1; i < argc; i++) {
+    int n;
+    char junk;
+
+    if (sscanf(argv[i], "--write_buffer_size=%d%c", &n, &junk) == 1) {
+      FLAGS_write_buffer_size = n;
+    }
+
+    if (sscanf(argv[i], "--total_keys=%d%c", &n, &junk) == 1) {
+      FLAGS_total_keys = n;
+    }
+
+    if (sscanf(argv[i], "--random_key=%d%c", &n, &junk) == 1 &&
+        (n == 0 || n == 1)) {
+      FLAGS_random_key = n;
+    }
+
+    if (sscanf(argv[i], "--use_set_based_memetable=%d%c", &n, &junk) == 1 &&
+        (n == 0 || n == 1)) {
+      FLAGS_use_set_based_memetable = n;
+    }
+
   }
 
-  leveldb::test::RunAllTests();
+  std::cout << kDbName << "\n";
+
+  rocksdb::test::RunAllTests();
   return 0;
 }

db/prefix_filter_iterator.h

@@ -1,6 +1,7 @@
-// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file. See the AUTHORS file for names of contributors.
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
 //
 // Wrap an underlying iterator, but exclude any results not starting
 // with a given prefix.  Seeking to keys not beginning with the prefix
@@ -9,12 +10,10 @@
 // underlying iterator would if there happened to be no non-matching
 // keys in the dataset.
 
-#ifndef STORAGE_LEVELDB_DB_PREFIX_FILTER_ITERATOR_H_
-#define STORAGE_LEVELDB_DB_PREFIX_FILTER_ITERATOR_H_
-
+#pragma once
 #include "rocksdb/iterator.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class PrefixFilterIterator : public Iterator {
  private:
@@ -71,6 +70,4 @@ class PrefixFilterIterator : public Iterator {
   }
 };
 
-}  // namespace leveldb
-
-#endif
+}  // namespace rocksdb

db/repair.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -38,7 +43,7 @@
 #include "rocksdb/db.h"
 #include "rocksdb/env.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 namespace {
 
@@ -74,7 +79,7 @@ class Repairer {
         bytes += tables_[i].meta.file_size;
       }
       Log(options_.info_log,
-          "**** Repaired leveldb %s; "
+          "**** Repaired rocksdb %s; "
           "recovered %d files; %llu bytes. "
           "Some data may have been lost. "
           "****",
@@ -221,7 +226,7 @@ class Repairer {
     Iterator* iter = mem->NewIterator();
     status = BuildTable(dbname_, env_, options_, storage_options_,
                         table_cache_, iter, &meta,
-                        icmp_.user_comparator(), 0, 0);
+                        icmp_.user_comparator(), 0, 0, true);
     delete iter;
     mem->Unref();
     mem = nullptr;
@@ -383,4 +388,4 @@ Status RepairDB(const std::string& dbname, const Options& options) {
   return repairer.Run();
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/skiplist.h

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -25,15 +30,13 @@
 // ... prev vs. next pointer ordering ...
 //
 
-#ifndef STORAGE_LEVELDB_DB_SKIPLIST_H_
-#define STORAGE_LEVELDB_DB_SKIPLIST_H_
-
+#pragma once
 #include <assert.h>
 #include <stdlib.h>
 #include "port/port.h"
 #include "util/random.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 template<typename Key, class Comparator>
 class SkipList {
@@ -107,6 +110,7 @@ class SkipList {
 
   // Used for optimizing sequential insert patterns
   Node* prev_[kMaxHeight];
+  int   prev_height_;
 
   inline int GetMaxHeight() const {
     return static_cast<int>(
@@ -267,6 +271,10 @@ typename SkipList<Key,Comparator>::Node* SkipList<Key,Comparator>::FindGreaterOr
     Node* x = prev[0];
     Node* next = x->Next(0);
     if ((x == head_) || KeyIsAfterNode(key, x)) {
+      // Adjust all relevant insertion points to the previous entry
+      for (int i = 1; i < prev_height_; i++) {
+        prev[i] = x;
+      }
       return next;
     }
   }
@@ -275,6 +283,9 @@ typename SkipList<Key,Comparator>::Node* SkipList<Key,Comparator>::FindGreaterOr
   int level = GetMaxHeight() - 1;
   while (true) {
     Node* next = x->Next(level);
+    // Make sure the lists are sorted.
+    // If x points to head_ or next points nullptr, it is trivially satisfied.
+    assert((x == head_) || (next == nullptr) || KeyIsAfterNode(next->key, x));
     if (KeyIsAfterNode(key, next)) {
       // Keep searching in this list
       x = next;
@@ -337,6 +348,7 @@ SkipList<Key,Comparator>::SkipList(Comparator cmp, Arena* arena)
       arena_(arena),
       head_(NewNode(0 /* any key will do */, kMaxHeight)),
       max_height_(reinterpret_cast<void*>(1)),
+      prev_height_(1),
       rnd_(0xdeadbeef) {
   for (int i = 0; i < kMaxHeight; i++) {
     head_->SetNext(i, nullptr);
@@ -378,6 +390,7 @@ void SkipList<Key,Comparator>::Insert(const Key& key) {
     prev_[i]->SetNext(i, x);
   }
   prev_[0] = x;
+  prev_height_ = height;
 }
 
 template<typename Key, class Comparator>
@@ -390,6 +403,4 @@ bool SkipList<Key,Comparator>::Contains(const Key& key) const {
   }
 }
 
-}  // namespace leveldb
-
-#endif // STORAGE_LEVELDB_DB_SKIPLIST_H_
+}  // namespace rocksdb

db/skiplist_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -10,7 +15,7 @@
 #include "util/random.h"
 #include "util/testharness.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 typedef uint64_t Key;
 
@@ -371,8 +376,8 @@ TEST(SkipTest, Concurrent3) { RunConcurrent(3); }
 TEST(SkipTest, Concurrent4) { RunConcurrent(4); }
 TEST(SkipTest, Concurrent5) { RunConcurrent(5); }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }

db/snapshot.h

@@ -1,13 +1,16 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#ifndef STORAGE_LEVELDB_DB_SNAPSHOT_H_
-#define STORAGE_LEVELDB_DB_SNAPSHOT_H_
-
+#pragma once
 #include "rocksdb/db.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class SnapshotList;
 
@@ -80,6 +83,4 @@ class SnapshotList {
   SnapshotImpl list_;
 };
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_SNAPSHOT_H_
+}  // namespace rocksdb

db/table_cache.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -11,7 +16,7 @@
 #include "util/coding.h"
 #include "util/stop_watch.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 static void DeleteEntry(const Slice& key, void* value) {
   Table* table = reinterpret_cast<Table*>(value);
@@ -32,7 +37,10 @@ TableCache::TableCache(const std::string& dbname,
       dbname_(dbname),
       options_(options),
       storage_options_(storage_options),
-      cache_(NewLRUCache(entries, options->table_cache_numshardbits)) {}
+      cache_(
+        NewLRUCache(entries, options->table_cache_numshardbits,
+                    options->table_cache_remove_scan_count_limit)) {
+}
 
 TableCache::~TableCache() {
 }
@@ -160,4 +168,4 @@ void TableCache::Evict(uint64_t file_number) {
   cache_->Erase(Slice(buf, sizeof(buf)));
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/table_cache.h

@@ -1,12 +1,15 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 //
 // Thread-safe (provides internal synchronization)
 
-#ifndef STORAGE_LEVELDB_DB_TABLE_CACHE_H_
-#define STORAGE_LEVELDB_DB_TABLE_CACHE_H_
-
+#pragma once
 #include <string>
 #include <stdint.h>
 #include "db/dbformat.h"
@@ -15,7 +18,7 @@
 #include "port/port.h"
 #include "table/table.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class Env;
 
@@ -72,6 +75,4 @@ class TableCache {
                    const bool no_io = false);
 };
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_TABLE_CACHE_H_
+}  // namespace rocksdb

db/transaction_log_impl.cc

@@ -1,16 +1,21 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #include "db/transaction_log_impl.h"
 #include "db/write_batch_internal.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 TransactionLogIteratorImpl::TransactionLogIteratorImpl(
-                           const std::string& dbname,
+                           const std::string& dir,
                            const Options* options,
                            const EnvOptions& soptions,
                            const SequenceNumber seq,
                            std::unique_ptr<VectorLogPtr> files,
                            SequenceNumber const * const lastFlushedSequence) :
-    dbname_(dbname),
+    dir_(dir),
     options_(options),
     soptions_(soptions),
     startingSequenceNumber_(seq),
@@ -18,31 +23,34 @@ TransactionLogIteratorImpl::TransactionLogIteratorImpl(
     started_(false),
     isValid_(false),
     currentFileIndex_(0),
+    currentBatchSeq_(0),
+    currentBatchCount_(0),
     lastFlushedSequence_(lastFlushedSequence) {
   assert(startingSequenceNumber_ <= *lastFlushedSequence_);
-  assert(files_.get() != nullptr);
+  assert(files_ != nullptr);
 
   reporter_.env = options_->env;
   reporter_.info_log = options_->info_log.get();
+  SeekToStartSequence(); // Seek till starting sequence
 }
 
 Status TransactionLogIteratorImpl::OpenLogFile(
-  const LogFile* logFile,
-  unique_ptr<SequentialFile>* file) {
+    const LogFile* logFile,
+    unique_ptr<SequentialFile>* file) {
   Env* env = options_->env;
   if (logFile->Type() == kArchivedLogFile) {
-    std::string fname = ArchivedLogFileName(dbname_, logFile->LogNumber());
+    std::string fname = ArchivedLogFileName(dir_, logFile->LogNumber());
     return env->NewSequentialFile(fname, file, soptions_);
   } else {
-    std::string fname = LogFileName(dbname_, logFile->LogNumber());
+    std::string fname = LogFileName(dir_, logFile->LogNumber());
     Status status = env->NewSequentialFile(fname, file, soptions_);
     if (!status.ok()) {
       //  If cannot open file in DB directory.
       //  Try the archive dir, as it could have moved in the meanwhile.
-      fname = ArchivedLogFileName(dbname_, logFile->LogNumber());
+      fname = ArchivedLogFileName(dir_, logFile->LogNumber());
       status = env->NewSequentialFile(fname, file, soptions_);
       if (!status.ok()) {
-        return Status::IOError(" Requested file not present in the dir");
+        return Status::IOError("Requested file not present in the dir");
       }
     }
     return status;
@@ -52,7 +60,7 @@ Status TransactionLogIteratorImpl::OpenLogFile(
 BatchResult TransactionLogIteratorImpl::GetBatch()  {
   assert(isValid_);  //  cannot call in a non valid state.
   BatchResult result;
-  result.sequence = currentSequence_;
+  result.sequence = currentBatchSeq_;
   result.writeBatchPtr = std::move(currentBatch_);
   return result;
 }
@@ -65,92 +73,155 @@ bool TransactionLogIteratorImpl::Valid() {
   return started_ && isValid_;
 }
 
-void TransactionLogIteratorImpl::Next() {
-  LogFile* currentLogFile = files_.get()->at(currentFileIndex_).get();
+bool TransactionLogIteratorImpl::RestrictedRead(
+    Slice* record,
+    std::string* scratch) {
+  // Don't read if no more complete entries to read from logs
+  if (currentBatchSeq_ >= *lastFlushedSequence_) {
+    return false;
+  }
+  return currentLogReader_->ReadRecord(record, scratch);
+}
 
-//  First seek to the given seqNo. in the current file.
+void TransactionLogIteratorImpl::SeekToStartSequence(
+    uint64_t startFileIndex,
+    bool strict) {
   std::string scratch;
   Slice record;
-  if (!started_) {
-    started_ = true;  // this piece only runs onced.
-    isValid_ = false;
-    if (startingSequenceNumber_ > *lastFlushedSequence_) {
-      currentStatus_ = Status::IOError("Looking for a sequence, "
-                                        "which is not flushed yet.");
-      return;
+  started_ = false;
+  isValid_ = false;
+  if (startingSequenceNumber_ > *lastFlushedSequence_) {
+    currentStatus_ = Status::IOError("Looking for a sequence, "
+                                     "which is not flushed yet.");
+    return;
+  }
+  if (files_->size() <= startFileIndex) {
+    return;
+  }
+  Status s = OpenLogReader(files_->at(startFileIndex).get());
+  if (!s.ok()) {
+    currentStatus_ = s;
+    return;
+  }
+  while (RestrictedRead(&record, &scratch)) {
+    if (record.size() < 12) {
+      reporter_.Corruption(
+        record.size(), Status::Corruption("very small log record"));
+      continue;
     }
-    Status s = OpenLogReader(currentLogFile);
-    if (!s.ok()) {
-      currentStatus_ = s;
+    UpdateCurrentWriteBatch(record);
+    if (currentBatchSeq_ + currentBatchCount_ - 1 >=
+        startingSequenceNumber_) {
+      if (strict && currentBatchSeq_ != startingSequenceNumber_) {
+        currentStatus_ = Status::Corruption("Gap in sequence number. Could not "
+                                            "seek to required sequence number");
+        reporter_.Info(currentStatus_.ToString().c_str());
+        return;
+      } else if (strict) {
+        reporter_.Info("Could seek required sequence number. Iterator will "
+                       "continue.");
+      }
+      isValid_ = true;
+      started_ = true; // set started_ as we could seek till starting sequence
+      return;
+    } else {
       isValid_ = false;
-      return;
-    }
-    while (currentLogReader_->ReadRecord(&record, &scratch)) {
-      if (record.size() < 12) {
-        reporter_.Corruption(
-          record.size(), Status::Corruption("log record too small"));
-        continue;
-      }
-      UpdateCurrentWriteBatch(record);
-      if (currentSequence_ >= startingSequenceNumber_) {
-        assert(currentSequence_ <= *lastFlushedSequence_);
-        isValid_ = true;
-        break;
-      } else {
-        isValid_ = false;
-      }
-    }
-    if (isValid_) {
-      // Done for this iteration
-      return;
     }
   }
-  bool openNextFile = true;
-  while(openNextFile) {
+  // Could not find start sequence in first file. Normally this must be the
+  // only file. Otherwise log the error and let the iterator return next entry
+  if (files_->size() != 1) {
+    currentStatus_ = Status::Corruption("Start sequence was not found, "
+                                        "skipping to the next available");
+    reporter_.Corruption(0, currentStatus_);
+    started_ = true; // Let Next find next available entry
+    Next();
+  }
+}
+
+void TransactionLogIteratorImpl::Next() {
+  std::string scratch;
+  Slice record;
+  isValid_ = false;
+  if (!started_) {  // Runs every time until we can seek to the start sequence
+    return SeekToStartSequence();
+  }
+  while(true) {
     assert(currentLogReader_);
-    if (currentSequence_ < *lastFlushedSequence_) {
+    if (currentBatchSeq_ < *lastFlushedSequence_) {
       if (currentLogReader_->IsEOF()) {
         currentLogReader_->UnmarkEOF();
       }
       while (currentLogReader_->ReadRecord(&record, &scratch)) {
         if (record.size() < 12) {
           reporter_.Corruption(
-            record.size(), Status::Corruption("log record too small"));
+            record.size(), Status::Corruption("very small log record"));
           continue;
         } else {
-          UpdateCurrentWriteBatch(record);
-          openNextFile = false;
-          break;
+          return UpdateCurrentWriteBatch(record);
         }
       }
     }
 
-    if (openNextFile) {
-      if (currentFileIndex_ < files_.get()->size() - 1) {
-        ++currentFileIndex_;
-        Status status =OpenLogReader(files_.get()->at(currentFileIndex_).get());
-        if (!status.ok()) {
-          isValid_ = false;
-          currentStatus_ = status;
-          return;
-        }
-      } else {
+    // Open the next file
+    if (currentFileIndex_ < files_->size() - 1) {
+      ++currentFileIndex_;
+      Status status =OpenLogReader(files_->at(currentFileIndex_).get());
+      if (!status.ok()) {
         isValid_ = false;
-        openNextFile = false;
-        if (currentSequence_ == *lastFlushedSequence_) {
-          currentStatus_ = Status::OK();
-        } else {
-          currentStatus_ = Status::IOError(" NO MORE DATA LEFT");
-        }
+        currentStatus_ = status;
+        return;
       }
+    } else {
+      isValid_ = false;
+      if (currentBatchSeq_ == *lastFlushedSequence_) {
+        currentStatus_ = Status::OK();
+      } else {
+        currentStatus_ = Status::IOError(" NO MORE DATA LEFT");
+      }
+      return;
     }
   }
 }
 
+bool TransactionLogIteratorImpl::IsBatchContinuous(
+    const WriteBatch* batch,
+    const SequenceNumber expectedSeq) {
+  assert(batch);
+  SequenceNumber batchSeq = WriteBatchInternal::Sequence(batch);
+  if (started_ && batchSeq != expectedSeq) {
+    char buf[200];
+    snprintf(buf, sizeof(buf),
+             "Discontinuity in log records. Got seq=%lu, Expected seq=%lu, "
+             "Last flushed seq=%lu. Log iterator will seek the correct batch.",
+             batchSeq, expectedSeq, *lastFlushedSequence_);
+    reporter_.Info(buf);
+    return false;
+  }
+  return true;
+}
+
 void TransactionLogIteratorImpl::UpdateCurrentWriteBatch(const Slice& record) {
   WriteBatch* batch = new WriteBatch();
   WriteBatchInternal::SetContents(batch, record);
-  currentSequence_ = WriteBatchInternal::Sequence(batch);
+
+  SequenceNumber expectedSeq = currentBatchSeq_ + currentBatchCount_;
+  if (!IsBatchContinuous(batch, expectedSeq)) {
+    // Seek to the batch having expected sequence number
+    if (expectedSeq < files_->at(currentFileIndex_)->StartSequence()) {
+      // Expected batch must lie in the previous log file
+      currentFileIndex_--;
+      currentFileIndex_ = (currentFileIndex_ >= 0) ? currentFileIndex_ : 0;
+    }
+    startingSequenceNumber_ = expectedSeq;
+    return SeekToStartSequence(currentFileIndex_, true);
+  }
+
+  currentBatchSeq_ = WriteBatchInternal::Sequence(batch);
+  currentBatchCount_ = WriteBatchInternal::Count(batch);
+  // currentBatchSeq_ can only change here
+  assert(currentBatchSeq_ <= *lastFlushedSequence_);
+
   currentBatch_.reset(batch);
   isValid_ = true;
   currentStatus_ = Status::OK();
@@ -168,4 +239,4 @@ Status TransactionLogIteratorImpl::OpenLogReader(const LogFile* logFile) {
   );
   return Status::OK();
 }
-}  //  namespace leveldb
+}  //  namespace rocksdb

db/transaction_log_impl.h

@@ -1,7 +1,9 @@
-// Copyright 2008-present Facebook. All Rights Reserved.
-#ifndef STORAGE_LEVELDB_INCLUDE_WRITES_ITERATOR_IMPL_H_
-#define STORAGE_LEVELDB_INCLUDE_WRITES_ITERATOR_IMPL_H_
-
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
+#pragma once
 #include <vector>
 
 #include "rocksdb/env.h"
@@ -11,7 +13,7 @@
 #include "db/log_reader.h"
 #include "db/filename.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 struct LogReporter : public log::Reader::Reporter {
   Env* env;
@@ -19,6 +21,9 @@ struct LogReporter : public log::Reader::Reporter {
   virtual void Corruption(size_t bytes, const Status& s) {
     Log(info_log, "dropping %zu bytes; %s", bytes, s.ToString().c_str());
   }
+  virtual void Info(const char* s) {
+    Log(info_log, "%s", s);
+  }
 };
 
 class LogFileImpl : public LogFile {
@@ -60,7 +65,7 @@ class LogFileImpl : public LogFile {
 
 class TransactionLogIteratorImpl : public TransactionLogIterator {
  public:
-  TransactionLogIteratorImpl(const std::string& dbname,
+  TransactionLogIteratorImpl(const std::string& dir,
                              const Options* options,
                              const EnvOptions& soptions,
                              const SequenceNumber seqNum,
@@ -76,10 +81,10 @@ class TransactionLogIteratorImpl : public TransactionLogIterator {
   virtual BatchResult GetBatch();
 
  private:
-  const std::string& dbname_;
+  const std::string& dir_;
   const Options* options_;
   const EnvOptions& soptions_;
-  const SequenceNumber startingSequenceNumber_;
+  SequenceNumber startingSequenceNumber_;
   std::unique_ptr<VectorLogPtr> files_;
   bool started_;
   bool isValid_;  // not valid when it starts of.
@@ -89,15 +94,19 @@ class TransactionLogIteratorImpl : public TransactionLogIterator {
   unique_ptr<log::Reader> currentLogReader_;
   Status OpenLogFile(const LogFile* logFile, unique_ptr<SequentialFile>* file);
   LogReporter reporter_;
+  SequenceNumber currentBatchSeq_; // sequence number at start of current batch
+  uint64_t currentBatchCount_; // count in current batch
   SequenceNumber const * const lastFlushedSequence_;
-  // represents the sequence number being read currently.
-  SequenceNumber currentSequence_;
 
+  // Reads from transaction log only if the writebatch record has been written
+  bool RestrictedRead(Slice* record, std::string* scratch);
+  // Seeks to startingSequenceNumber reading from startFileIndex in files_.
+  // If strict is set,then must get a batch starting with startingSequenceNumber
+  void SeekToStartSequence(uint64_t startFileIndex = 0, bool strict = false);
+  // Check if batch is continuous starting from expectedSeq, else return false
+  bool IsBatchContinuous(const WriteBatch* batch, SequenceNumber expectedSeq);
+  // Update current batch if a continuous batch is found, else return false
   void UpdateCurrentWriteBatch(const Slice& record);
   Status OpenLogReader(const LogFile* file);
 };
-
-
-
-}  //  namespace leveldb
-#endif  //  STORAGE_LEVELDB_INCLUDE_WRITES_ITERATOR_IMPL_H_
+}  //  namespace rocksdb

db/version_edit.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -7,7 +12,7 @@
 #include "db/version_set.h"
 #include "util/coding.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 // Tag numbers for serialized VersionEdit.  These numbers are written to
 // disk and should not be changed.
@@ -293,4 +298,4 @@ std::string VersionEdit::DebugString(bool hex_key) const {
   return r;
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/version_edit.h

@@ -1,16 +1,19 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#ifndef STORAGE_LEVELDB_DB_VERSION_EDIT_H_
-#define STORAGE_LEVELDB_DB_VERSION_EDIT_H_
-
+#pragma once
 #include <set>
 #include <utility>
 #include <vector>
 #include "db/dbformat.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class VersionSet;
 
@@ -122,6 +125,4 @@ class VersionEdit {
   std::vector< std::pair<int, FileMetaData> > new_files_;
 };
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_VERSION_EDIT_H_
+}  // namespace rocksdb

db/version_edit_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -5,7 +10,7 @@
 #include "db/version_edit.h"
 #include "util/testharness.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 static void TestEncodeDecode(const VersionEdit& edit) {
   std::string encoded, encoded2;
@@ -41,8 +46,8 @@ TEST(VersionEditTest, EncodeDecode) {
   TestEncodeDecode(edit);
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }

db/version_set.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -21,7 +26,7 @@
 #include "util/logging.h"
 #include "util/stop_watch.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 static uint64_t TotalFileSize(const std::vector<FileMetaData*>& files) {
   uint64_t sum = 0;
@@ -1430,9 +1435,9 @@ Status VersionSet::Recover() {
       if (s.ok()) {
         if (edit.has_comparator_ &&
             edit.comparator_ != icmp_.user_comparator()->Name()) {
-          s = Status::InvalidArgument(
-              edit.comparator_ + "does not match existing comparator ",
-              icmp_.user_comparator()->Name());
+          s = Status::InvalidArgument(icmp_.user_comparator()->Name(),
+                                      "does not match existing comparator " +
+                                      edit.comparator_);
         }
       }
 
@@ -1548,9 +1553,9 @@ Status VersionSet::DumpManifest(Options& options, std::string& dscname,
       if (s.ok()) {
         if (edit.has_comparator_ &&
             edit.comparator_ != icmp_.user_comparator()->Name()) {
-          s = Status::InvalidArgument(
-              edit.comparator_ + "does not match existing comparator ",
-              icmp_.user_comparator()->Name());
+          s = Status::InvalidArgument(icmp_.user_comparator()->Name(),
+                                      "does not match existing comparator " +
+                                      edit.comparator_);
         }
       }
 
@@ -1645,7 +1650,12 @@ void VersionSet::Finalize(Version* v,
   double max_score = 0;
   int max_score_level = 0;
 
-  for (int level = 0; level < NumberLevels()-1; level++) {
+  int num_levels_to_check =
+      (options_->compaction_style != kCompactionStyleUniversal) ?
+          NumberLevels() - 1 : 1;
+
+  for (int level = 0; level < num_levels_to_check; level++) {
+
     double score;
     if (level == 0) {
       // We treat level-0 specially by bounding the number of files
@@ -2159,17 +2169,236 @@ void VersionSet::SizeBeingCompacted(std::vector<uint64_t>& sizes) {
   }
 }
 
-Compaction* VersionSet::PickCompactionUniversal(int level, double score) {
+//
+// Look at overall size amplification. If size amplification
+// exceeeds the configured value, then do a compaction
+// of the candidate files all the way upto the earliest
+// base file (overrides configured values of file-size ratios,
+// min_merge_width and max_merge_width).
+//
+Compaction* VersionSet::PickCompactionUniversalSizeAmp(
+    int level, double score) {
   assert (level == 0);
 
-  // percentage flexibilty while comparing file sizes
-  uint64_t ratio = options_->compaction_options_universal.size_ratio;
+  // percentage flexibilty while reducing size amplification
+  uint64_t ratio = options_->compaction_options_universal.
+                     max_size_amplification_percent;
+
+  // The files are sorted from newest first to oldest last.
+  std::vector<int>& file_by_time = current_->files_by_size_[level];
+  assert(file_by_time.size() == current_->files_[level].size());
+
+  unsigned int candidate_count = 0;
+  uint64_t candidate_size = 0;
+  unsigned int start_index = 0;
+  FileMetaData* f = nullptr;
+
+  // Skip files that are already being compacted
+  for (unsigned int loop = 0; loop < file_by_time.size() - 1; loop++) {
+    int index = file_by_time[loop];
+    f = current_->files_[level][index];
+    if (!f->being_compacted) {
+      start_index = loop;         // Consider this as the first candidate.
+      break;
+    }
+    Log(options_->info_log, "Universal: skipping file %ld[%d] compacted %s",
+        f->number, loop, " cannot be a candidate to reduce size amp.\n");
+    f = nullptr;
+  }
+  if (f == nullptr) {
+    return nullptr;             // no candidate files
+  }
+
+  Log(options_->info_log, "Universal: First candidate file %ld[%d] %s",
+      f->number, start_index, " to reduce size amp.\n");
+
+  // keep adding up all the remaining files
+  for (unsigned int loop = start_index; loop < file_by_time.size() - 1;
+       loop++) {
+    int index = file_by_time[loop];
+    f = current_->files_[level][index];
+    if (f->being_compacted) {
+      Log(options_->info_log,
+          "Universal: Possible candidate file %ld[%d] %s.", f->number, loop,
+          " is already being compacted. No size amp reduction possible.\n");
+      return nullptr;
+    }
+    candidate_size += f->file_size;
+    candidate_count++;
+  }
+  if (candidate_count == 0) {
+    return nullptr;
+  }
+
+  // size of earliest file
+  int index = file_by_time[file_by_time.size() - 1];
+  uint64_t earliest_file_size = current_->files_[level][index]->file_size;
+
+  // size amplification = percentage of additional size
+  if (candidate_size * 100 < ratio * earliest_file_size) {
+    Log(options_->info_log,
+        "Universal: size amp not needed. newer-files-total-size %ld "
+        "earliest-file-size %ld",
+        candidate_size, earliest_file_size);
+    return nullptr;
+  } else {
+    Log(options_->info_log,
+        "Universal: size amp needed. newer-files-total-size %ld "
+        "earliest-file-size %ld",
+        candidate_size, earliest_file_size);
+  }
+  assert(start_index >= 0 && start_index < file_by_time.size() - 1);
+
+  // create a compaction request
+  // We always compact all the files, so always compress.
+  Compaction* c = new Compaction(level, level, MaxFileSizeForLevel(level),
+                                 LLONG_MAX, NumberLevels(), false,
+                                 true);
+  c->score_ = score;
+  for (unsigned int loop = start_index; loop < file_by_time.size(); loop++) {
+    int index = file_by_time[loop];
+    f = current_->files_[level][index];
+    c->inputs_[0].push_back(f);
+    Log(options_->info_log,
+        "Universal: size amp picking file %ld[%d] with size %ld",
+        f->number, index, f->file_size);
+  }
+  return c;
+}
+
+//
+// Consider compaction files based on their size differences with
+// the next file in time order.
+//
+Compaction* VersionSet::PickCompactionUniversalReadAmp(
+    int level, double score, unsigned int ratio,
+    unsigned int max_number_of_files_to_compact) {
+
   unsigned int min_merge_width =
     options_->compaction_options_universal.min_merge_width;
   unsigned int max_merge_width =
     options_->compaction_options_universal.max_merge_width;
 
-  if ((current_->files_[level].size() <=
+  // The files are sorted from newest first to oldest last.
+  std::vector<int>& file_by_time = current_->files_by_size_[level];
+  FileMetaData* f = nullptr;
+  bool done = false;
+  int start_index = 0;
+  unsigned int candidate_count;
+  assert(file_by_time.size() == current_->files_[level].size());
+
+  unsigned int max_files_to_compact = std::min(max_merge_width,
+                                       max_number_of_files_to_compact);
+  min_merge_width = std::max(min_merge_width, 2U);
+
+  // Considers a candidate file only if it is smaller than the
+  // total size accumulated so far.
+  for (unsigned int loop = 0; loop < file_by_time.size(); loop++) {
+
+    candidate_count = 0;
+
+    // Skip files that are already being compacted
+    for (f = nullptr; loop < file_by_time.size(); loop++) {
+      int index = file_by_time[loop];
+      f = current_->files_[level][index];
+
+      if (!f->being_compacted) {
+        candidate_count = 1;
+        break;
+      }
+      Log(options_->info_log,
+          "Universal: file %ld[%d] being compacted, skipping",
+          f->number, loop);
+      f = nullptr;
+    }
+
+    // This file is not being compacted. Consider it as the
+    // first candidate to be compacted.
+    uint64_t candidate_size =  f != nullptr? f->file_size : 0;
+    if (f != nullptr) {
+      Log(options_->info_log, "Universal: Possible candidate file %ld[%d].",
+          f->number, loop);
+    }
+
+    // Check if the suceeding files need compaction.
+    for (unsigned int i = loop+1;
+         candidate_count < max_files_to_compact && i < file_by_time.size();
+         i++) {
+      int index = file_by_time[i];
+      FileMetaData* f = current_->files_[level][index];
+      if (f->being_compacted) {
+        break;
+      }
+      // pick files if the total candidate file size (increased by the
+      // specified ratio) is still larger than the next candidate file.
+      uint64_t sz = (candidate_size * (100L + ratio)) /100;
+      if (sz < f->file_size) {
+        break;
+      }
+      candidate_count++;
+      candidate_size += f->file_size;
+    }
+
+    // Found a series of consecutive files that need compaction.
+    if (candidate_count >= (unsigned int)min_merge_width) {
+      start_index = loop;
+      done = true;
+      break;
+    } else {
+      for (unsigned int i = loop;
+           i < loop + candidate_count && i < file_by_time.size(); i++) {
+       int index = file_by_time[i];
+       FileMetaData* f = current_->files_[level][index];
+       Log(options_->info_log,
+           "Universal: Skipping file %ld[%d] with size %ld %d\n",
+           f->number, i, f->file_size, f->being_compacted);
+      }
+    }
+  }
+  if (!done || candidate_count <= 1) {
+    return nullptr;
+  }
+  unsigned int first_index_after = start_index + candidate_count;
+  // Compression is enabled if files compacted earlier already reached
+  // size ratio of compression.
+  bool enable_compression = true;
+  int ratio_to_compress =
+      options_->compaction_options_universal.compression_size_percent;
+  if (ratio_to_compress >= 0) {
+    uint64_t total_size = TotalFileSize(current_->files_[level]);
+    uint64_t older_file_size = 0;
+    for (unsigned int i = file_by_time.size() - 1; i >= first_index_after;
+        i--) {
+      older_file_size += current_->files_[level][file_by_time[i]]->file_size;
+      if (older_file_size * 100L >= total_size * (long) ratio_to_compress) {
+        enable_compression = false;
+        break;
+      }
+    }
+  }
+  Compaction* c = new Compaction(level, level, MaxFileSizeForLevel(level),
+                                 LLONG_MAX, NumberLevels(), false,
+                                 enable_compression);
+  c->score_ = score;
+
+  for (unsigned int i = start_index; i < first_index_after; i++) {
+    int index = file_by_time[i];
+    FileMetaData* f = current_->files_[level][index];
+    c->inputs_[0].push_back(f);
+    Log(options_->info_log, "Universal: Picking file %ld[%d] with size %ld\n",
+        f->number, i, f->file_size);
+  }
+  return c;
+}
+
+//
+// Universal style of compaction. Pick files that are contiguous in
+// time-range to compact.
+//
+Compaction* VersionSet::PickCompactionUniversal(int level, double score) {
+  assert (level == 0);
+
+  if ((current_->files_[level].size() <
       (unsigned int)options_->level0_file_num_compaction_trigger)) {
     Log(options_->info_log, "Universal: nothing to do\n");
     return nullptr;
@@ -2179,127 +2408,29 @@ Compaction* VersionSet::PickCompactionUniversal(int level, double score) {
       current_->files_[level].size(),
       LevelFileSummary(&tmp, 0));
 
-  Compaction* c = nullptr;
-  c = new Compaction(level, level, MaxFileSizeForLevel(level),
-                     LLONG_MAX, NumberLevels());
-  c->score_ = score;
+  // Check for size amplification first.
+  Compaction* c = PickCompactionUniversalSizeAmp(level, score);
+  if (c == nullptr) {
 
-  // The files are sorted from newest first to oldest last.
-  std::vector<int>& file_by_time = current_->files_by_size_[level];
-  FileMetaData* f = nullptr;
-  bool done = false;
-  assert(file_by_time.size() == current_->files_[level].size());
+    // Size amplification is within limits. Try reducing read
+    // amplification while maintaining file size ratios.
+    unsigned int ratio = options_->compaction_options_universal.size_ratio;
+    c = PickCompactionUniversalReadAmp(level, score, ratio, UINT_MAX);
 
-  unsigned int max_files_to_compact = std::min(max_merge_width, UINT_MAX);
-
-  // Make two pass. The first pass considers a candidate file
-  // only if it is smaller than the total size accumulated so far.
-  // The second pass does not look at the slope of the
-  // file-size  curve to decide what to pick for compaction.
-  for (int iter = 0; !done && iter < 2; iter++) {
-
-    for (unsigned int loop = 0; loop < file_by_time.size(); ) {
-
-      // Skip files that are already being compacted
-      for (f = nullptr; loop < file_by_time.size(); loop++) {
-        int index = file_by_time[loop];
-        f = current_->files_[level][index];
-
-        if (!f->being_compacted) {
-          break;
-        }
-        Log(options_->info_log, "Universal: file %ld[%d] being compacted, skipping",
-            f->number, loop);
-        f = nullptr;
-      }
-
-      // This file is not being compacted. Consider it as the
-      // first candidate to be compacted.
-      unsigned int candidate_count = 1;
-      uint64_t candidate_size =  f != nullptr? f->file_size : 0;
-      if (f != nullptr) {
-        Log(options_->info_log, "Universal: Possible candidate file %ld[%d] %s.",
-            f->number, loop, iter == 0? "" : "forced ");
-      }
-
-      // Check if the suceeding files need compaction.
-      for (unsigned int i = loop+1;
-           candidate_count < max_files_to_compact && i < file_by_time.size();
-           i++) {
-        int index = file_by_time[i];
-        FileMetaData* f = current_->files_[level][index];
-        if (f->being_compacted) {
-          break;
-        }
-        // If this is the first iteration, then we pick files if the
-        // total candidate file size (increased by the specified ratio)
-        // is still larger than the next candidate file.
-        if (iter == 0) {
-          uint64_t sz = (candidate_size * (100 + ratio)) /100;
-          if (sz < f->file_size) {
-            break;
-          }
-        }
-        candidate_count++;
-        candidate_size += f->file_size;
-      }
-
-      // Found a series of consecutive files that need compaction.
-      if (candidate_count >= (unsigned int)min_merge_width) {
-        for (unsigned int i = loop; i < loop + candidate_count; i++) {
-          int index = file_by_time[i];
-          FileMetaData* f = current_->files_[level][index];
-          c->inputs_[0].push_back(f);
-          Log(options_->info_log, "Universal: Picking file %ld[%d] with size %ld %s",
-              f->number, i, f->file_size,
-              (iter == 0 ? "" : "forced"));
-        }
-        done = true;
-        break;
-      } else {
-        for (unsigned int i = loop;
-             i < loop + candidate_count && i < file_by_time.size(); i++) {
-         int index = file_by_time[i];
-         FileMetaData* f = current_->files_[level][index];
-         Log(options_->info_log, "Universal: Skipping file %ld[%d] with size %ld %d %s",
-             f->number, i, f->file_size, f->being_compacted,
-              (iter == 0 ? "" : "forced"));
-        }
-      }
-      loop += candidate_count;
-    }
-    assert(done || c->inputs_[0].size() == 0);
-
-    // If we are unable to find a normal compaction run and we are still
-    // above the compaction threshold, iterate again to pick compaction
-    // candidates, this time without considering their size differences.
-    if (!done) {
-      int files_not_in_compaction = 0;
-      for (unsigned int i = 0; i < current_->files_[level].size(); i++) {
-        f = current_->files_[level][i];
-        if (!f->being_compacted) {
-          files_not_in_compaction++;
-        }
-      }
-      int expected_num_files = files_not_in_compaction +
-                compactions_in_progress_[level].size();
-      if (expected_num_files <=
-          options_->level0_file_num_compaction_trigger + 1) {
-        done = true;     // nothing more to do
-      } else {
-        max_files_to_compact = std::min((int)max_merge_width,
-          expected_num_files - options_->level0_file_num_compaction_trigger);
-        Log(options_->info_log, "Universal: second loop with maxfiles %d",
-            max_files_to_compact);
-      }
+    // Size amplification and file size ratios are within configured limits.
+    // If max read amplification is exceeding configured limits, then force
+    // compaction without looking at filesize ratios and try to reduce
+    // the number of files to fewer than level0_file_num_compaction_trigger.
+    if (c == nullptr) {
+      unsigned int num_files = current_->files_[level].size() -
+                               options_->level0_file_num_compaction_trigger;
+      c = PickCompactionUniversalReadAmp(level, score, UINT_MAX, num_files);
     }
   }
-  if (c->inputs_[0].size() <= 1) {
-    Log(options_->info_log, "Universal: only %ld files, nothing to do.\n",
-        c->inputs_[0].size());
-    delete c;
+  if (c == nullptr) {
     return nullptr;
   }
+  assert(c->inputs_[0].size() > 1);
 
   // validate that all the chosen files are non overlapping in time
   FileMetaData* newerfile __attribute__((unused)) = nullptr;
@@ -2311,6 +2442,9 @@ Compaction* VersionSet::PickCompactionUniversal(int level, double score) {
     newerfile = f;
   }
 
+  // The files are sorted from newest first to oldest last.
+  std::vector<int>& file_by_time = current_->files_by_size_[level];
+
   // Is the earliest file part of this compaction?
   int last_index = file_by_time[file_by_time.size()-1];
   FileMetaData* last_file = current_->files_[level][last_index];
@@ -2771,7 +2905,7 @@ Compaction* VersionSet::CompactRange(
 
 Compaction::Compaction(int level, int out_level, uint64_t target_file_size,
   uint64_t max_grandparent_overlap_bytes, int number_levels,
-  bool seek_compaction)
+  bool seek_compaction, bool enable_compression)
     : level_(level),
       out_level_(out_level),
       max_output_file_size_(target_file_size),
@@ -2779,6 +2913,7 @@ Compaction::Compaction(int level, int out_level, uint64_t target_file_size,
       input_version_(nullptr),
       number_levels_(number_levels),
       seek_compaction_(seek_compaction),
+      enable_compression_(enable_compression),
       grandparent_index_(0),
       seen_key_(false),
       overlapped_bytes_(0),
@@ -2952,4 +3087,4 @@ void Compaction::Summary(char* output, int len) {
       level_low_summary, level_up_summary);
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/version_set.h

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -12,9 +17,7 @@
 // Version,VersionSet are thread-compatible, but require external
 // synchronization on all accesses.
 
-#ifndef STORAGE_LEVELDB_DB_VERSION_SET_H_
-#define STORAGE_LEVELDB_DB_VERSION_SET_H_
-
+#pragma once
 #include <map>
 #include <memory>
 #include <set>
@@ -25,18 +28,16 @@
 #include "port/port.h"
 #include "db/table_cache.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 namespace log { class Writer; }
 
 class Compaction;
 class Iterator;
 class MemTable;
-class TableBuilder;
 class TableCache;
 class Version;
 class VersionSet;
-class WritableFile;
 
 // Return the smallest index i such that files[i]->largest >= key.
 // Return files.size() if there is no such file.
@@ -316,7 +317,17 @@ class VersionSet {
   // Returns true iff some level needs a compaction because it has
   // exceeded its target size.
   bool NeedsSizeCompaction() const {
-    for (int i = 0; i < NumberLevels()-1; i++) {
+    // In universal compaction case, this check doesn't really
+    // check the compaction condition, but checks num of files threshold
+    // only. We are not going to miss any compaction opportunity
+    // but it's likely that more compactions are scheduled but
+    // ending up with nothing to do. We can improve it later.
+    // TODO: improve this function to be accurate for universal
+    //       compactions.
+    int num_levels_to_check =
+        (options_->compaction_style != kCompactionStyleUniversal) ?
+            NumberLevels() - 1 : 1;
+    for (int i = 0; i < num_levels_to_check; i++) {
       if (current_->compaction_score_[i] >= 1) {
         return true;
       }
@@ -383,6 +394,13 @@ class VersionSet {
   // Pick files to compact in Universal mode
   Compaction* PickCompactionUniversal(int level, double score);
 
+  // Pick Universal compaction to limit read amplification
+  Compaction* PickCompactionUniversalReadAmp(int level, double score,
+                unsigned int ratio, unsigned int num_files);
+
+  // Pick Universal compaction to limit space amplification.
+  Compaction* PickCompactionUniversalSizeAmp(int level, double score);
+
   // Free up the files that were participated in a compaction
   void ReleaseCompactionFiles(Compaction* c, Status status);
 
@@ -540,6 +558,9 @@ class Compaction {
   // Maximum size of files to build during this compaction.
   uint64_t MaxOutputFileSize() const { return max_output_file_size_; }
 
+  // Whether compression will be enabled for compaction outputs
+  bool enable_compression() const { return enable_compression_; }
+
   // Is this a trivial compaction that can be implemented by just
   // moving a single input file to the next level (no merging or splitting)
   bool IsTrivialMove() const;
@@ -574,7 +595,7 @@ class Compaction {
 
   explicit Compaction(int level, int out_level, uint64_t target_file_size,
     uint64_t max_grandparent_overlap_bytes, int number_levels,
-    bool seek_compaction = false);
+    bool seek_compaction = false, bool enable_compression = true);
 
   int level_;
   int out_level_; // levels to which output files are stored
@@ -585,6 +606,7 @@ class Compaction {
   int number_levels_;
 
   bool seek_compaction_;
+  bool enable_compression_;
 
   // Each compaction reads inputs from "level_" and "level_+1"
   std::vector<FileMetaData*> inputs_[2];      // The two sets of inputs
@@ -620,6 +642,4 @@ class Compaction {
   void ResetNextCompactionIndex();
 };
 
-}  // namespace leveldb
-
-#endif  // STORAGE_LEVELDB_DB_VERSION_SET_H_
+}  // namespace rocksdb

db/version_set_reduce_num_levels.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2012 Facebook. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
@@ -10,7 +15,7 @@
 #include "db/log_writer.h"
 #include "util/logging.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 Status VersionSet::ReduceNumberOfLevels(int new_levels, port::Mutex* mu) {
 

db/version_set_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -7,7 +12,7 @@
 #include "util/testharness.h"
 #include "util/testutil.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class FindFileTest {
  public:
@@ -172,8 +177,8 @@ TEST(FindFileTest, OverlappingFiles) {
   ASSERT_TRUE(Overlaps("600", "700"));
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }

db/write_batch.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -26,7 +31,7 @@
 #include "util/coding.h"
 #include <stdexcept>
 
-namespace leveldb {
+namespace rocksdb {
 
 // WriteBatch header has an 8-byte sequence number followed by a 4-byte count.
 static const size_t kHeader = 12;
@@ -227,4 +232,4 @@ void WriteBatchInternal::Append(WriteBatch* dst, const WriteBatch* src) {
   dst->rep_.append(src->rep_.data() + kHeader, src->rep_.size() - kHeader);
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

db/write_batch_internal.h

@@ -1,16 +1,19 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 
-#ifndef STORAGE_LEVELDB_DB_WRITE_BATCH_INTERNAL_H_
-#define STORAGE_LEVELDB_DB_WRITE_BATCH_INTERNAL_H_
-
+#pragma once
 #include "rocksdb/types.h"
 #include "rocksdb/write_batch.h"
 #include "rocksdb/db.h"
 #include "rocksdb/options.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class MemTable;
 
@@ -51,7 +54,4 @@ class WriteBatchInternal {
   static void Append(WriteBatch* dst, const WriteBatch* src);
 };
 
-}  // namespace leveldb
-
-
-#endif  // STORAGE_LEVELDB_DB_WRITE_BATCH_INTERNAL_H_
+}  // namespace rocksdb

db/write_batch_test.cc

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -12,7 +17,7 @@
 #include "util/logging.h"
 #include "util/testharness.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 static std::string PrintContents(WriteBatch* b) {
   InternalKeyComparator cmp(BytewiseComparator());
@@ -221,8 +226,8 @@ TEST(WriteBatchTest, Continue) {
             handler.seen);
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }

doc/bench/db_bench_sqlite3.cc

@@ -111,7 +111,7 @@ static void WalCheckpoint(sqlite3* db_) {
   }
 }
 
-namespace leveldb {
+namespace rocksdb {
 
 // Helper for quickly generating random data.
 namespace {
@@ -664,7 +664,7 @@ class Benchmark {
 
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
   std::string default_db_path;
@@ -672,7 +672,7 @@ int main(int argc, char** argv) {
     double d;
     int n;
     char junk;
-    if (leveldb::Slice(argv[i]).starts_with("--benchmarks=")) {
+    if (rocksdb::Slice(argv[i]).starts_with("--benchmarks=")) {
       FLAGS_benchmarks = argv[i] + strlen("--benchmarks=");
     } else if (sscanf(argv[i], "--histogram=%d%c", &n, &junk) == 1 &&
                (n == 0 || n == 1)) {
@@ -688,7 +688,7 @@ int main(int argc, char** argv) {
       FLAGS_reads = n;
     } else if (sscanf(argv[i], "--value_size=%d%c", &n, &junk) == 1) {
       FLAGS_value_size = n;
-    } else if (leveldb::Slice(argv[i]) == leveldb::Slice("--no_transaction")) {
+    } else if (rocksdb::Slice(argv[i]) == rocksdb::Slice("--no_transaction")) {
       FLAGS_transaction = false;
     } else if (sscanf(argv[i], "--page_size=%d%c", &n, &junk) == 1) {
       FLAGS_page_size = n;
@@ -707,12 +707,12 @@ int main(int argc, char** argv) {
 
   // Choose a location for the test database if none given with --db=<path>
   if (FLAGS_db == NULL) {
-      leveldb::Env::Default()->GetTestDirectory(&default_db_path);
+      rocksdb::Env::Default()->GetTestDirectory(&default_db_path);
       default_db_path += "/dbbench";
       FLAGS_db = default_db_path.c_str();
   }
 
-  leveldb::Benchmark benchmark;
+  rocksdb::Benchmark benchmark;
   benchmark.Run();
   return 0;
 }

doc/bench/db_bench_tree_db.cc

@@ -80,7 +80,7 @@ static void DBSynchronize(kyotocabinet::TreeDB* db_)
   }
 }
 
-namespace leveldb {
+namespace rocksdb {
 
 // Helper for quickly generating random data.
 namespace {
@@ -479,7 +479,7 @@ class Benchmark {
   }
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
   std::string default_db_path;
@@ -487,7 +487,7 @@ int main(int argc, char** argv) {
     double d;
     int n;
     char junk;
-    if (leveldb::Slice(argv[i]).starts_with("--benchmarks=")) {
+    if (rocksdb::Slice(argv[i]).starts_with("--benchmarks=")) {
       FLAGS_benchmarks = argv[i] + strlen("--benchmarks=");
     } else if (sscanf(argv[i], "--compression_ratio=%lf%c", &d, &junk) == 1) {
       FLAGS_compression_ratio = d;
@@ -517,12 +517,12 @@ int main(int argc, char** argv) {
 
   // Choose a location for the test database if none given with --db=<path>
   if (FLAGS_db == NULL) {
-      leveldb::Env::Default()->GetTestDirectory(&default_db_path);
+      rocksdb::Env::Default()->GetTestDirectory(&default_db_path);
       default_db_path += "/dbbench";
       FLAGS_db = default_db_path.c_str();
   }
 
-  leveldb::Benchmark benchmark;
+  rocksdb::Benchmark benchmark;
   benchmark.Run();
   return 0;
 }

doc/index.html

@@ -25,27 +25,27 @@ creating it if necessary:
   #include <assert>
   #include "leveldb/db.h"
 
-  leveldb::DB* db;
-  leveldb::Options options;
+  rocksdb::DB* db;
+  rocksdb::Options options;
   options.create_if_missing = true;
-  leveldb::Status status = leveldb::DB::Open(options, "/tmp/testdb", &db);
+  rocksdb::Status status = rocksdb::DB::Open(options, "/tmp/testdb", &db);
   assert(status.ok());
   ...
 </pre>
 If you want to raise an error if the database already exists, add
-the following line before the <code>leveldb::DB::Open</code> call:
+the following line before the <code>rocksdb::DB::Open</code> call:
 <pre>
   options.error_if_exists = true;
 </pre>
 <h1>Status</h1>
 <p>
-You may have noticed the <code>leveldb::Status</code> type above.  Values of this
+You may have noticed the <code>rocksdb::Status</code> type above.  Values of this
 type are returned by most functions in <code>leveldb</code> that may encounter an
 error.  You can check if such a result is ok, and also print an
 associated error message:
 <p>
 <pre>
-   leveldb::Status s = ...;
+   rocksdb::Status s = ...;
    if (!s.ok()) cerr &lt;&lt; s.ToString() &lt;&lt; endl;
 </pre>
 <h1>Closing A Database</h1>
@@ -65,9 +65,9 @@ modify/query the database.  For example, the following code
 moves the value stored under key1 to key2.
 <pre>
   std::string value;
-  leveldb::Status s = db-&gt;Get(leveldb::ReadOptions(), key1, &amp;value);
-  if (s.ok()) s = db-&gt;Put(leveldb::WriteOptions(), key2, value);
-  if (s.ok()) s = db-&gt;Delete(leveldb::WriteOptions(), key1);
+  rocksdb::Status s = db-&gt;Get(rocksdb::ReadOptions(), key1, &amp;value);
+  if (s.ok()) s = db-&gt;Put(rocksdb::WriteOptions(), key2, value);
+  if (s.ok()) s = db-&gt;Delete(rocksdb::WriteOptions(), key1);
 </pre>
 
 <h1>Atomic Updates</h1>
@@ -81,12 +81,12 @@ atomically apply a set of updates:
   #include "leveldb/write_batch.h"
   ...
   std::string value;
-  leveldb::Status s = db-&gt;Get(leveldb::ReadOptions(), key1, &amp;value);
+  rocksdb::Status s = db-&gt;Get(rocksdb::ReadOptions(), key1, &amp;value);
   if (s.ok()) {
-    leveldb::WriteBatch batch;
+    rocksdb::WriteBatch batch;
     batch.Delete(key1);
     batch.Put(key2, value);
-    s = db-&gt;Write(leveldb::WriteOptions(), &amp;batch);
+    s = db-&gt;Write(rocksdb::WriteOptions(), &amp;batch);
   }
 </pre>
 The <code>WriteBatch</code> holds a sequence of edits to be made to the database,
@@ -109,7 +109,7 @@ persistent storage.  (On Posix systems, this is implemented by calling
 either <code>fsync(...)</code> or <code>fdatasync(...)</code> or
 <code>msync(..., MS_SYNC)</code> before the write operation returns.)
 <pre>
-  leveldb::WriteOptions write_options;
+  rocksdb::WriteOptions write_options;
   write_options.sync = true;
   db-&gt;Put(write_options, ...);
 </pre>
@@ -144,7 +144,7 @@ the batch.
 A database may only be opened by one process at a time.
 The <code>leveldb</code> implementation acquires a lock from the
 operating system to prevent misuse.  Within a single process, the
-same <code>leveldb::DB</code> object may be safely shared by multiple
+same <code>rocksdb::DB</code> object may be safely shared by multiple
 concurrent threads.  I.e., different threads may write into or fetch
 iterators or call <code>Get</code> on the same database without any
 external synchronization (the leveldb implementation will
@@ -160,7 +160,7 @@ The following example demonstrates how to print all key,value pairs
 in a database.
 <p>
 <pre>
-  leveldb::Iterator* it = db-&gt;NewIterator(leveldb::ReadOptions());
+  rocksdb::Iterator* it = db-&gt;NewIterator(rocksdb::ReadOptions());
   for (it-&gt;SeekToFirst(); it-&gt;Valid(); it-&gt;Next()) {
     cout &lt;&lt; it-&gt;key().ToString() &lt;&lt; ": "  &lt;&lt; it-&gt;value().ToString() &lt;&lt; endl;
   }
@@ -196,10 +196,10 @@ implicit snapshot of the current state.
 Snapshots are created by the DB::GetSnapshot() method:
 <p>
 <pre>
-  leveldb::ReadOptions options;
+  rocksdb::ReadOptions options;
   options.snapshot = db-&gt;GetSnapshot();
   ... apply some updates to db ...
-  leveldb::Iterator* iter = db-&gt;NewIterator(options);
+  rocksdb::Iterator* iter = db-&gt;NewIterator(options);
   ... read using iter to view the state when the snapshot was created ...
   delete iter;
   db-&gt;ReleaseSnapshot(options.snapshot);
@@ -211,7 +211,7 @@ support reading as of that snapshot.
 <h1>Slice</h1>
 <p>
 The return value of the <code>it->key()</code> and <code>it->value()</code> calls above
-are instances of the <code>leveldb::Slice</code> type.  <code>Slice</code> is a simple
+are instances of the <code>rocksdb::Slice</code> type.  <code>Slice</code> is a simple
 structure that contains a length and a pointer to an external byte
 array.  Returning a <code>Slice</code> is a cheaper alternative to returning a
 <code>std::string</code> since we do not need to copy potentially large keys and
@@ -223,10 +223,10 @@ C++ strings and null-terminated C-style strings can be easily converted
 to a Slice:
 <p>
 <pre>
-   leveldb::Slice s1 = "hello";
+   rocksdb::Slice s1 = "hello";
 
    std::string str("world");
-   leveldb::Slice s2 = str;
+   rocksdb::Slice s2 = str;
 </pre>
 A Slice can be easily converted back to a C++ string:
 <pre>
@@ -238,7 +238,7 @@ the external byte array into which the Slice points remains live while
 the Slice is in use.  For example, the following is buggy:
 <p>
 <pre>
-   leveldb::Slice slice;
+   rocksdb::Slice slice;
    if (...) {
      std::string str = ...;
      slice = str;
@@ -255,16 +255,16 @@ which orders bytes lexicographically.  You can however supply a custom
 comparator when opening a database.  For example, suppose each
 database key consists of two numbers and we should sort by the first
 number, breaking ties by the second number.  First, define a proper
-subclass of <code>leveldb::Comparator</code> that expresses these rules:
+subclass of <code>rocksdb::Comparator</code> that expresses these rules:
 <p>
 <pre>
-  class TwoPartComparator : public leveldb::Comparator {
+  class TwoPartComparator : public rocksdb::Comparator {
    public:
     // Three-way comparison function:
     //   if a &lt; b: negative result
     //   if a &gt; b: positive result
     //   else: zero result
-    int Compare(const leveldb::Slice&amp; a, const leveldb::Slice&amp; b) const {
+    int Compare(const rocksdb::Slice&amp; a, const rocksdb::Slice&amp; b) const {
       int a1, a2, b1, b2;
       ParseKey(a, &amp;a1, &amp;a2);
       ParseKey(b, &amp;b1, &amp;b2);
@@ -277,7 +277,7 @@ subclass of <code>leveldb::Comparator</code> that expresses these rules:
 
     // Ignore the following methods for now:
     const char* Name() const { return "TwoPartComparator"; }
-    void FindShortestSeparator(std::string*, const leveldb::Slice&amp;) const { }
+    void FindShortestSeparator(std::string*, const rocksdb::Slice&amp;) const { }
     void FindShortSuccessor(std::string*) const { }
   };
 </pre>
@@ -285,18 +285,18 @@ Now create a database using this custom comparator:
 <p>
 <pre>
   TwoPartComparator cmp;
-  leveldb::DB* db;
-  leveldb::Options options;
+  rocksdb::DB* db;
+  rocksdb::Options options;
   options.create_if_missing = true;
   options.comparator = &amp;cmp;
-  leveldb::Status status = leveldb::DB::Open(options, "/tmp/testdb", &amp;db);
+  rocksdb::Status status = rocksdb::DB::Open(options, "/tmp/testdb", &amp;db);
   ...
 </pre>
 <h2>Backwards compatibility</h2>
 <p>
 The result of the comparator's <code>Name</code> method is attached to the
 database when it is created, and is checked on every subsequent
-database open.  If the name changes, the <code>leveldb::DB::Open</code> call will
+database open.  If the name changes, the <code>rocksdb::DB::Open</code> call will
 fail.  Therefore, change the name if and only if the new key format
 and comparison function are incompatible with existing databases, and
 it is ok to discard the contents of all existing databases.
@@ -339,9 +339,9 @@ compression entirely, but should only do so if benchmarks show a
 performance improvement:
 <p>
 <pre>
-  leveldb::Options options;
-  options.compression = leveldb::kNoCompression;
-  ... leveldb::DB::Open(options, name, ...) ....
+  rocksdb::Options options;
+  options.compression = rocksdb::kNoCompression;
+  ... rocksdb::DB::Open(options, name, ...) ....
 </pre>
 <h2>Cache</h2>
 <p>
@@ -353,10 +353,10 @@ uncompressed block contents.
 <pre>
   #include "leveldb/cache.h"
 
-  leveldb::Options options;
-  options.cache = leveldb::NewLRUCache(100 * 1048576);  // 100MB cache
-  leveldb::DB* db;
-  leveldb::DB::Open(options, name, &db);
+  rocksdb::Options options;
+  options.cache = rocksdb::NewLRUCache(100 * 1048576);  // 100MB cache
+  rocksdb::DB* db;
+  rocksdb::DB::Open(options, name, &db);
   ... use the db ...
   delete db
   delete options.cache;
@@ -373,9 +373,9 @@ displacing most of the cached contents.  A per-iterator option can be
 used to achieve this:
 <p>
 <pre>
-  leveldb::ReadOptions options;
+  rocksdb::ReadOptions options;
   options.fill_cache = false;
-  leveldb::Iterator* it = db-&gt;NewIterator(options);
+  rocksdb::Iterator* it = db-&gt;NewIterator(options);
   for (it-&gt;SeekToFirst(); it-&gt;Valid(); it-&gt;Next()) {
     ...
   }
@@ -407,10 +407,10 @@ a single <code>Get()</code> call may involve multiple reads from disk.
 The optional <code>FilterPolicy</code> mechanism can be used to reduce
 the number of disk reads substantially.
 <pre>
-   leveldb::Options options;
+   rocksdb::Options options;
    options.filter_policy = NewBloomFilter(10);
-   leveldb::DB* db;
-   leveldb::DB::Open(options, "/tmp/testdb", &amp;db);
+   rocksdb::DB* db;
+   rocksdb::DB::Open(options, "/tmp/testdb", &amp;db);
    ... use the database ...
    delete db;
    delete options.filter_policy;
@@ -434,7 +434,7 @@ consider a comparator that ignores trailing spaces when comparing keys.
 Instead, the application should provide a custom filter policy that
 also ignores trailing spaces.  For example:
 <pre>
-  class CustomFilterPolicy : public leveldb::FilterPolicy {
+  class CustomFilterPolicy : public rocksdb::FilterPolicy {
    private:
     FilterPolicy* builtin_policy_;
    public:
@@ -484,7 +484,7 @@ checksums are verified:
   parts of its persistent storage have been corrupted.
 <p>
   If a database is corrupted (perhaps it cannot be opened when
-  paranoid checking is turned on), the <code>leveldb::RepairDB</code> function
+  paranoid checking is turned on), the <code>rocksdb::RepairDB</code> function
   may be used to recover as much of the data as possible
 <p>
 </ul>
@@ -494,11 +494,11 @@ The <code>GetApproximateSizes</code> method can used to get the approximate
 number of bytes of file system space used by one or more key ranges.
 <p>
 <pre>
-   leveldb::Range ranges[2];
-   ranges[0] = leveldb::Range("a", "c");
-   ranges[1] = leveldb::Range("x", "z");
+   rocksdb::Range ranges[2];
+   ranges[0] = rocksdb::Range("a", "c");
+   ranges[1] = rocksdb::Range("x", "z");
    uint64_t sizes[2];
-   leveldb::Status s = db-&gt;GetApproximateSizes(ranges, 2, sizes);
+   rocksdb::Status s = db-&gt;GetApproximateSizes(ranges, 2, sizes);
 </pre>
 The preceding call will set <code>sizes[0]</code> to the approximate number of
 bytes of file system space used by the key range <code>[a..c)</code> and
@@ -508,21 +508,21 @@ bytes of file system space used by the key range <code>[a..c)</code> and
 <h1>Environment</h1>
 <p>
 All file operations (and other operating system calls) issued by the
-<code>leveldb</code> implementation are routed through a <code>leveldb::Env</code> object.
+<code>leveldb</code> implementation are routed through a <code>rocksdb::Env</code> object.
 Sophisticated clients may wish to provide their own <code>Env</code>
 implementation to get better control.  For example, an application may
 introduce artificial delays in the file IO paths to limit the impact
 of <code>leveldb</code> on other activities in the system.
 <p>
 <pre>
-  class SlowEnv : public leveldb::Env {
+  class SlowEnv : public rocksdb::Env {
     .. implementation of the Env interface ...
   };
 
   SlowEnv env;
-  leveldb::Options options;
+  rocksdb::Options options;
   options.env = &amp;env;
-  Status s = leveldb::DB::Open(options, ...);
+  Status s = rocksdb::DB::Open(options, ...);
 </pre>
 <h1>Porting</h1>
 <p>
@@ -531,7 +531,7 @@ specific implementations of the types/methods/functions exported by
 <code>leveldb/port/port.h</code>.  See <code>leveldb/port/port_example.h</code> for more
 details.
 <p>
-In addition, the new platform may need a new default <code>leveldb::Env</code>
+In addition, the new platform may need a new default <code>rocksdb::Env</code>
 implementation.  See <code>leveldb/util/env_posix.h</code> for an example.
 
 <h1>Other Information</h1>

hdfs/README

@@ -1,4 +1,4 @@
-This directory contains the hdfs extensions needed to make leveldb store
+This directory contains the hdfs extensions needed to make rocksdb store
 files in HDFS.
 
 The hdfs.h file is copied from the Apache Hadoop 1.0 source code. 
@@ -7,12 +7,12 @@ It defines the libhdfs library
 data in HDFS.  The libhdfs.a is copied from the Apache Hadoop 1.0 build. 
 It implements the API defined in hdfs.h. If your hadoop cluster is running
 a different hadoop release, then install these two files manually from your
-hadoop distribution and then recompile leveldb.
+hadoop distribution and then recompile rocksdb.
 
-The env_hdfs.h file defines the leveldb objects that are needed to talk to an
+The env_hdfs.h file defines the rocksdb objects that are needed to talk to an
 underlying filesystem. 
 
-If you want to compile leveldb with hdfs support, please set the following
+If you want to compile rocksdb with hdfs support, please set the following
 enviroment variables appropriately:
    USE_HDFS=1
    JAVA_HOME=/usr/local/jdk-6u22-64

hdfs/env_hdfs.h

@@ -1,25 +1,10 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-// Copyright (c) 2012 Facebook. All rights reserved.
-
-#ifndef LEVELDB_HDFS_FILE_H
-#define LEVELDB_HDFS_FILE_H
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 
+#pragma once
 #include <algorithm>
 #include <stdio.h>
 #include <sys/time.h>
@@ -31,7 +16,7 @@
 #ifdef USE_HDFS
 #include "hdfs/hdfs.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 static const std::string kProto = "hdfs://";
 static const std::string pathsep = "/";
@@ -55,7 +40,7 @@ private:
 };
 
 //
-// The HDFS environment for leveldb. This class overrides all the
+// The HDFS environment for rocksdb. This class overrides all the
 // file/dir access methods and delegates the thread-mgmt methods to the
 // default posix environment.
 //
@@ -81,6 +66,10 @@ class HdfsEnv : public Env {
   virtual Status NewWritableFile(const std::string& fname,
                                  WritableFile** result);
 
+  virtual Status NewRandomRWFile(const std::string& fname,
+                                 unique_ptr<RandomRWFile>* result,
+                                 const EnvOptions& options);
+
   virtual bool FileExists(const std::string& fname);
 
   virtual Status GetChildren(const std::string& path,
@@ -107,8 +96,9 @@ class HdfsEnv : public Env {
 
   virtual Status NewLogger(const std::string& fname, Logger** result);
 
-  virtual void Schedule( void (*function)(void* arg), void* arg) {
-    posixEnv->Schedule(function, arg);
+  virtual void Schedule(void (*function)(void* arg), void* arg,
+                        Priority pri = LOW) {
+    posixEnv->Schedule(function, arg, pri);
   }
 
   virtual void StartThread(void (*function)(void* arg), void* arg) {
@@ -140,8 +130,8 @@ class HdfsEnv : public Env {
     return posixEnv->GetAbsolutePath(db_path, output_path);
   }
 
-  virtual void SetBackgroundThreads(int number) {
-    posixEnv->SetBackgroundThreads(number);
+  virtual void SetBackgroundThreads(int number, Priority pri = LOW) {
+    posixEnv->SetBackgroundThreads(number, pri);
   }
 
   virtual std::string TimeToString(uint64_t number) {
@@ -213,12 +203,12 @@ class HdfsEnv : public Env {
   }
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #else // USE_HDFS
 
 
-namespace leveldb {
+namespace rocksdb {
 
 static const Status notsup;
 
@@ -226,7 +216,7 @@ class HdfsEnv : public Env {
 
  public:
   HdfsEnv(const std::string& fsname) {
-    fprintf(stderr, "You have not build leveldb with HDFS support\n");
+    fprintf(stderr, "You have not build rocksdb with HDFS support\n");
     fprintf(stderr, "Please see hdfs/README for details\n");
     throw new std::exception();
   }
@@ -250,6 +240,12 @@ class HdfsEnv : public Env {
     return notsup;
   }
 
+  virtual Status NewRandomRWFile(const std::string& fname,
+                                 unique_ptr<RandomRWFile>* result,
+                                 const EnvOptions& options) {
+    return notsup;
+  }
+
   virtual bool FileExists(const std::string& fname){return false;}
 
   virtual Status GetChildren(const std::string& path,
@@ -279,7 +275,8 @@ class HdfsEnv : public Env {
   virtual Status NewLogger(const std::string& fname,
                            shared_ptr<Logger>* result){return notsup;}
 
-  virtual void Schedule( void (*function)(void* arg), void* arg) {}
+  virtual void Schedule(void (*function)(void* arg), void* arg,
+                        Priority pri = LOW) {}
 
   virtual void StartThread(void (*function)(void* arg), void* arg) {}
 
@@ -296,12 +293,10 @@ class HdfsEnv : public Env {
   virtual Status GetAbsolutePath(const std::string& db_path,
       std::string* outputpath) {return notsup;}
 
-  virtual void SetBackgroundThreads(int number) {}
+  virtual void SetBackgroundThreads(int number, Priority pri = LOW) {}
 
   virtual std::string TimeToString(uint64_t number) { return "";}
 };
 }
 
 #endif // USE_HDFS
-
-#endif // LEVELDB_HDFS_FILE_H

hdfs/hdfs.h

@@ -1,21 +1,8 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #ifndef LIBHDFS_HDFS_H
 #define LIBHDFS_HDFS_H
 

helpers/memenv/memenv.cc

@@ -13,7 +13,7 @@
 #include <string>
 #include <vector>
 
-namespace leveldb {
+namespace rocksdb {
 
 namespace {
 
@@ -383,4 +383,4 @@ Env* NewMemEnv(Env* base_env) {
   return new InMemoryEnv(base_env);
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb

helpers/memenv/memenv.h

@@ -4,7 +4,7 @@
 
 #ifndef STORAGE_ROCKSDB_HELPERS_MEMENV_MEMENV_H_
 #define STORAGE_ROCKSDB_HELPERS_MEMENV_MEMENV_H_
-namespace leveldb {
+namespace rocksdb {
 
 class Env;
 
@@ -14,6 +14,6 @@ class Env;
 // *base_env must remain live while the result is in use.
 Env* NewMemEnv(Env* base_env);
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_HELPERS_MEMENV_MEMENV_H_

helpers/memenv/memenv_test.cc

@@ -12,7 +12,7 @@
 #include <string>
 #include <vector>
 
-namespace leveldb {
+namespace rocksdb {
 
 class MemEnvTest {
  public:
@@ -215,7 +215,7 @@ TEST(MemEnvTest, DBTest) {
   delete iterator;
 
   DBImpl* dbi = reinterpret_cast<DBImpl*>(db);
-  ASSERT_OK(dbi->TEST_CompactMemTable());
+  ASSERT_OK(dbi->TEST_FlushMemTable());
 
   for (size_t i = 0; i < 3; ++i) {
     std::string res;
@@ -226,8 +226,8 @@ TEST(MemEnvTest, DBTest) {
   delete db;
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 int main(int argc, char** argv) {
-  return leveldb::test::RunAllTests();
+  return rocksdb::test::RunAllTests();
 }

include/rocksdb/arena.h

@@ -11,7 +11,7 @@
 #include <limits>
 #include <memory>
 
-namespace leveldb {
+namespace rocksdb {
 
 class Arena {
  public:
@@ -36,6 +36,6 @@ class Arena {
   void operator=(const Arena&);
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_ARENA_H_

include/rocksdb/cache.h

@@ -22,16 +22,28 @@
 #include <stdint.h>
 #include "rocksdb/slice.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 using std::shared_ptr;
 
 class Cache;
 
-// Create a new cache with a fixed size capacity.  This implementation
-// of Cache uses a least-recently-used eviction policy.
+// Create a new cache with a fixed size capacity. The cache is sharded
+// to 2^numShardBits shards, by hash of the key. The total capacity
+// is divided and evenly assigned to each shard. Inside each shard,
+// the eviction is done in two passes: first try to free spaces by
+// evicting entries that are among the most least used removeScanCountLimit
+// entries and do not have reference other than by the cache itself, in
+// the least-used order. If not enough space is freed, further free the
+// entries in least used order.
+//
+// The functions without parameter numShardBits and/or removeScanCountLimit
+// use default values. removeScanCountLimit's default value is 0, which
+// means a strict LRU order inside each shard.
 extern shared_ptr<Cache> NewLRUCache(size_t capacity);
 extern shared_ptr<Cache> NewLRUCache(size_t capacity, int numShardBits);
+extern shared_ptr<Cache> NewLRUCache(size_t capacity, int numShardBits,
+                                     int removeScanCountLimit);
 
 class Cache {
  public:
@@ -101,6 +113,6 @@ class Cache {
   void operator=(const Cache&);
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_UTIL_CACHE_H_

include/rocksdb/compaction_filter.h

@@ -7,7 +7,7 @@
 
 #include <string>
 
-namespace leveldb {
+namespace rocksdb {
 
 class Slice;
 
@@ -66,6 +66,6 @@ class DefaultCompactionFilterFactory : public CompactionFilterFactory {
     }
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_COMPACTION_FILTER_H_

include/rocksdb/comparator.h

@@ -7,13 +7,13 @@
 
 #include <string>
 
-namespace leveldb {
+namespace rocksdb {
 
 class Slice;
 
 // A Comparator object provides a total order across slices that are
 // used as keys in an sstable or a database.  A Comparator implementation
-// must be thread-safe since leveldb may invoke its methods concurrently
+// must be thread-safe since rocksdb may invoke its methods concurrently
 // from multiple threads.
 class Comparator {
  public:
@@ -33,7 +33,7 @@ class Comparator {
   // the comparator implementation changes in a way that will cause
   // the relative ordering of any two keys to change.
   //
-  // Names starting with "leveldb." are reserved and should not be used
+  // Names starting with "rocksdb." are reserved and should not be used
   // by any clients of this package.
   virtual const char* Name() const = 0;
 
@@ -58,6 +58,6 @@ class Comparator {
 // must not be deleted.
 extern const Comparator* BytewiseComparator();
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_COMPARATOR_H_

include/rocksdb/db.h

@@ -14,7 +14,7 @@
 #include "rocksdb/types.h"
 #include "rocksdb/transaction_log.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 using std::unique_ptr;
 
@@ -175,11 +175,11 @@ class DB {
   //
   // Valid property names include:
   //
-  //  "leveldb.num-files-at-level<N>" - return the number of files at level <N>,
+  //  "rocksdb.num-files-at-level<N>" - return the number of files at level <N>,
   //     where <N> is an ASCII representation of a level number (e.g. "0").
-  //  "leveldb.stats" - returns a multi-line string that describes statistics
+  //  "rocksdb.stats" - returns a multi-line string that describes statistics
   //     about the internal operation of the DB.
-  //  "leveldb.sstables" - returns a multi-line string that describes all
+  //  "rocksdb.sstables" - returns a multi-line string that describes all
   //     of the sstables that make up the db contents.
   virtual bool GetProperty(const Slice& property, std::string* value) = 0;
 
@@ -244,8 +244,15 @@ class DB {
   // manifest file is returned in manifest_file_size. The manifest file is an
   // ever growing file, but only the portion specified by manifest_file_size is
   // valid for this snapshot.
+  // Setting flush_memtable to true does Flush before recording the live files.
+  // Setting flush_memtable to false is useful when we don't want to wait for
+  // flush which may have to wait for compaction to complete taking an
+  // indeterminate time. But this will have to use GetSortedWalFiles after
+  // GetLiveFiles to compensate for memtables missed in this snapshot due to the
+  // absence of Flush, by WAL files to recover the database consistently later
   virtual Status GetLiveFiles(std::vector<std::string>&,
-                              uint64_t* manifest_file_size) = 0;
+                              uint64_t* manifest_file_size,
+                              bool flush_memtable = true) = 0;
 
   // Retrieve the sorted list of all wal files with earliest file first
   virtual Status GetSortedWalFiles(VectorLogPtr& files) = 0;
@@ -299,6 +306,6 @@ Status DestroyDB(const std::string& name, const Options& options);
 // on a database that contains important information.
 Status RepairDB(const std::string& dbname, const Options& options);
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_DB_H_

include/rocksdb/env.h

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 //
-// An Env is an interface used by the leveldb implementation to access
+// An Env is an interface used by the rocksdb implementation to access
 // operating system functionality like the filesystem etc.  Callers
 // may wish to provide a custom Env object when opening a database to
 // get fine gain control; e.g., to rate limit file system operations.
@@ -20,7 +20,7 @@
 #include <stdint.h>
 #include "rocksdb/status.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class FileLock;
 class Logger;
@@ -28,6 +28,7 @@ class RandomAccessFile;
 class SequentialFile;
 class Slice;
 class WritableFile;
+class RandomRWFile;
 class Options;
 
 using std::unique_ptr;
@@ -41,7 +42,7 @@ struct EnvOptions {
   EnvOptions();
 
   // construct from Options
-  EnvOptions(const Options& options);
+  explicit EnvOptions(const Options& options);
 
   // If true, then allow caching of data in environment buffers
   bool use_os_buffer;
@@ -72,7 +73,7 @@ class Env {
   // system.  Sophisticated users may wish to provide their own Env
   // implementation instead of relying on this default environment.
   //
-  // The result of Default() belongs to leveldb and must never be deleted.
+  // The result of Default() belongs to rocksdb and must never be deleted.
   static Env* Default();
 
   // Create a brand new sequentially-readable file with the specified name.
@@ -109,6 +110,15 @@ class Env {
                                  unique_ptr<WritableFile>* result,
                                  const EnvOptions& options) = 0;
 
+  // Create an object that both reads and writes to a file on
+  // specified offsets (random access). If file already exists,
+  // does not overwrite it. On success, stores a pointer to the
+  // new file in *result and returns OK. On failure stores nullptr
+  // in *result and returns non-OK.
+  virtual Status NewRandomRWFile(const std::string& fname,
+                                 unique_ptr<RandomRWFile>* result,
+                                 const EnvOptions& options) = 0;
+
   // Returns true iff the named file exists.
   virtual bool FileExists(const std::string& fname) = 0;
 
@@ -162,15 +172,20 @@ class Env {
   // REQUIRES: lock has not already been unlocked.
   virtual Status UnlockFile(FileLock* lock) = 0;
 
-  // Arrange to run "(*function)(arg)" once in a background thread.
-  //
+  enum Priority { LOW, HIGH, TOTAL };
+
+  // Arrange to run "(*function)(arg)" once in a background thread, in
+  // the thread pool specified by pri. By default, jobs go to the 'LOW'
+  // priority thread pool.
+
   // "function" may run in an unspecified thread.  Multiple functions
   // added to the same Env may run concurrently in different threads.
   // I.e., the caller may not assume that background work items are
   // serialized.
   virtual void Schedule(
       void (*function)(void* arg),
-      void* arg) = 0;
+      void* arg,
+      Priority pri = LOW) = 0;
 
   // Start a new thread, invoking "function(arg)" within the new thread.
   // When "function(arg)" returns, the thread will be destroyed.
@@ -210,13 +225,17 @@ class Env {
   virtual Status GetAbsolutePath(const std::string& db_path,
       std::string* output_path) = 0;
 
-  // The number of background worker threads for this environment.
-  // default: 1
-  virtual void SetBackgroundThreads(int number) = 0;
+  // The number of background worker threads of a specific thread pool
+  // for this environment. 'LOW' is the default pool.
+  // default number: 1
+  virtual void SetBackgroundThreads(int number, Priority pri = LOW) = 0;
 
   // Converts seconds-since-Jan-01-1970 to a printable string
   virtual std::string TimeToString(uint64_t time) = 0;
 
+  // Generates a unique id that can be used to identify a db
+  virtual std::string GenerateUniqueId();
+
  private:
   // No copying allowed
   Env(const Env&);
@@ -247,6 +266,13 @@ class SequentialFile {
   //
   // REQUIRES: External synchronization
   virtual Status Skip(uint64_t n) = 0;
+
+  // Remove any kind of caching of data from the offset to offset+length
+  // of this file. If the length is 0, then it refers to the end of file.
+  // If the system is not caching the file contents, then this is a noop.
+  virtual Status InvalidateCache(size_t offset, size_t length) {
+    return Status::NotSupported("InvalidateCache not supported.");
+  }
 };
 
 // A file abstraction for randomly reading the contents of a file.
@@ -292,6 +318,12 @@ class RandomAccessFile {
 
   virtual void Hint(AccessPattern pattern) {}
 
+  // Remove any kind of caching of data from the offset to offset+length
+  // of this file. If the length is 0, then it refers to the end of file.
+  // If the system is not caching the file contents, then this is a noop.
+  virtual Status InvalidateCache(size_t offset, size_t length) {
+    return Status::NotSupported("InvalidateCache not supported.");
+  }
 };
 
 // A file abstraction for sequential writing.  The implementation
@@ -310,7 +342,7 @@ class WritableFile {
 
   /*
    * Sync data and/or metadata as well.
-   * By default, sync only metadata.
+   * By default, sync only data.
    * Override this method for environments where we need to sync
    * metadata as well.
    */
@@ -341,6 +373,14 @@ class WritableFile {
     *block_size = preallocation_block_size_;
   }
 
+  // Remove any kind of caching of data from the offset to offset+length
+  // of this file. If the length is 0, then it refers to the end of file.
+  // If the system is not caching the file contents, then this is a noop.
+  // This call has no effect on dirty pages in the cache.
+  virtual Status InvalidateCache(size_t offset, size_t length) {
+    return Status::NotSupported("InvalidateCache not supported.");
+  }
+
  protected:
   // PrepareWrite performs any necessary preparation for a write
   // before the write actually occurs.  This allows for pre-allocation
@@ -391,6 +431,55 @@ class WritableFile {
   void operator=(const WritableFile&);
 };
 
+// A file abstraction for random reading and writing.
+class RandomRWFile {
+ public:
+  RandomRWFile() {}
+  virtual ~RandomRWFile() {}
+
+  // Write data from Slice data to file starting from offset
+  // Returns IOError on failure, but does not guarantee
+  // atomicity of a write.  Returns OK status on success.
+  //
+  // Safe for concurrent use.
+  virtual Status Write(uint64_t offset, const Slice& data) = 0;
+  // Read up to "n" bytes from the file starting at "offset".
+  // "scratch[0..n-1]" may be written by this routine.  Sets "*result"
+  // to the data that was read (including if fewer than "n" bytes were
+  // successfully read).  May set "*result" to point at data in
+  // "scratch[0..n-1]", so "scratch[0..n-1]" must be live when
+  // "*result" is used.  If an error was encountered, returns a non-OK
+  // status.
+  //
+  // Safe for concurrent use by multiple threads.
+  virtual Status Read(uint64_t offset, size_t n, Slice* result,
+                      char* scratch) const = 0;
+  virtual Status Close() = 0; // closes the file
+  virtual Status Sync() = 0; // sync data
+
+  /*
+   * Sync data and/or metadata as well.
+   * By default, sync only data.
+   * Override this method for environments where we need to sync
+   * metadata as well.
+   */
+  virtual Status Fsync() {
+    return Sync();
+  }
+
+  /*
+   * Pre-allocate space for a file.
+   */
+  virtual Status Allocate(off_t offset, off_t len) {
+    return Status::OK();
+  }
+
+ private:
+  // No copying allowed
+  RandomRWFile(const RandomRWFile&);
+  void operator=(const RandomRWFile&);
+};
+
 // An interface for writing log messages.
 class Logger {
  public:
@@ -470,6 +559,10 @@ class EnvWrapper : public Env {
                          const EnvOptions& options) {
     return target_->NewWritableFile(f, r, options);
   }
+  Status NewRandomRWFile(const std::string& f, unique_ptr<RandomRWFile>* r,
+                         const EnvOptions& options) {
+    return target_->NewRandomRWFile(f, r, options);
+  }
   bool FileExists(const std::string& f) { return target_->FileExists(f); }
   Status GetChildren(const std::string& dir, std::vector<std::string>* r) {
     return target_->GetChildren(dir, r);
@@ -496,8 +589,8 @@ class EnvWrapper : public Env {
     return target_->LockFile(f, l);
   }
   Status UnlockFile(FileLock* l) { return target_->UnlockFile(l); }
-  void Schedule(void (*f)(void*), void* a) {
-    return target_->Schedule(f, a);
+  void Schedule(void (*f)(void*), void* a, Priority pri) {
+    return target_->Schedule(f, a, pri);
   }
   void StartThread(void (*f)(void*), void* a) {
     return target_->StartThread(f, a);
@@ -525,8 +618,8 @@ class EnvWrapper : public Env {
       std::string* output_path) {
     return target_->GetAbsolutePath(db_path, output_path);
   }
-  void SetBackgroundThreads(int num) {
-    return target_->SetBackgroundThreads(num);
+  void SetBackgroundThreads(int num, Priority pri) {
+    return target_->SetBackgroundThreads(num, pri);
   }
   std::string TimeToString(uint64_t time) {
     return target_->TimeToString(time);
@@ -536,6 +629,6 @@ class EnvWrapper : public Env {
   Env* target_;
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_ENV_H_

include/rocksdb/filter_policy.h

@@ -4,8 +4,8 @@
 //
 // A database can be configured with a custom FilterPolicy object.
 // This object is responsible for creating a small filter from a set
-// of keys.  These filters are stored in leveldb and are consulted
-// automatically by leveldb to decide whether or not to read some
+// of keys.  These filters are stored in rocksdb and are consulted
+// automatically by rocksdb to decide whether or not to read some
 // information from disk. In many cases, a filter can cut down the
 // number of disk seeks form a handful to a single disk seek per
 // DB::Get() call.
@@ -18,7 +18,7 @@
 
 #include <string>
 
-namespace leveldb {
+namespace rocksdb {
 
 class Slice;
 

include/rocksdb/iterator.h

@@ -18,7 +18,7 @@
 #include "rocksdb/slice.h"
 #include "rocksdb/status.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class Iterator {
  public:
@@ -97,6 +97,6 @@ extern Iterator* NewEmptyIterator();
 // Return an empty iterator with the specified status.
 extern Iterator* NewErrorIterator(const Status& status);
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_ITERATOR_H_

include/rocksdb/ldb_tool.h

@@ -3,12 +3,13 @@
 #define STORAGE_ROCKSDB_INCLUDE_LDB_TOOL_H
 #include "rocksdb/options.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class LDBTool {
  public:
   void Run(int argc, char** argv, Options = Options());
 };
 
-} // namespace leveldb
+} // namespace rocksdb
+
 #endif // STORAGE_ROCKSDB_INCLUDE_LDB_TOOL_H

include/rocksdb/memtablerep.h

@@ -10,7 +10,7 @@
 // The liberal use of assertions is encouraged to enforce (1).
 //
 // The factory will be passed an Arena object when a new MemTableRep is
-// requested. The API for this object is in leveldb/arena.h.
+// requested. The API for this object is in rocksdb/arena.h.
 //
 // Users can implement their own memtable representations. We include four
 // types built in:
@@ -44,7 +44,7 @@
 #include "rocksdb/slice.h"
 #include "rocksdb/slice_transform.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class MemTableRep {
  public:
@@ -143,6 +143,7 @@ class MemTableRepFactory {
   virtual ~MemTableRepFactory() { };
   virtual std::shared_ptr<MemTableRep> CreateMemTableRep(
     MemTableRep::KeyComparator&, Arena*) = 0;
+  virtual const char* Name() const = 0;
 };
 
 // This creates MemTableReps that are backed by an std::vector. On iteration,
@@ -152,13 +153,16 @@ class MemTableRepFactory {
 // Parameters:
 //   count: Passed to the constructor of the underlying std::vector of each
 //     VectorRep. On initialization, the underlying array will be at least count
-//     size.
+//     bytes reserved for usage.
 class VectorRepFactory : public MemTableRepFactory {
   const size_t count_;
 public:
   explicit VectorRepFactory(size_t count = 0) : count_(count) { }
   virtual std::shared_ptr<MemTableRep> CreateMemTableRep(
     MemTableRep::KeyComparator&, Arena*) override;
+  virtual const char* Name() const override {
+    return "VectorRepFactory";
+  }
 };
 
 // This uses a skip list to store keys. It is the default.
@@ -166,11 +170,14 @@ class SkipListFactory : public MemTableRepFactory {
 public:
   virtual std::shared_ptr<MemTableRep> CreateMemTableRep(
     MemTableRep::KeyComparator&, Arena*) override;
+  virtual const char* Name() const override {
+    return "SkipListFactory";
+  }
 };
 
 // TransformReps are backed by an unordered map of buffers to buckets. When
 // looking up a key, the user key is extracted and a user-supplied transform
-// function (see leveldb/slice_transform.h) is applied to get the key into the
+// function (see rocksdb/slice_transform.h) is applied to get the key into the
 // unordered map. This allows the user to bin user keys based on arbitrary
 // criteria. Two example implementations are UnsortedRepFactory and
 // PrefixHashRepFactory.
@@ -201,6 +208,10 @@ class TransformRepFactory : public MemTableRepFactory {
   virtual std::shared_ptr<MemTableRep> CreateMemTableRep(
     MemTableRep::KeyComparator&, Arena*) override;
 
+  virtual const char* Name() const override {
+    return "TransformRepFactory";
+  }
+
   const SliceTransform* GetTransform() { return transform_; }
 
  protected:
@@ -219,6 +230,9 @@ public:
     : TransformRepFactory(NewNoopTransform(),
                           bucket_count,
                           num_locks) { }
+  virtual const char* Name() const override {
+    return "UnsortedRepFactory";
+  }
 };
 
 // PrefixHashReps bin user keys based on a fixed-size prefix. This optimizes for
@@ -234,6 +248,10 @@ public:
 
   virtual std::shared_ptr<MemTableRep> CreateMemTableRep(
     MemTableRep::KeyComparator&, Arena*) override;
+
+  virtual const char* Name() const override {
+    return "PrefixHashRepFactory";
+  }
 };
 
 }

include/rocksdb/merge_operator.h

@@ -7,9 +7,9 @@
 
 #include <string>
 #include <deque>
-#include "rocksdb/slice.h"  // TODO: Remove this when migration is done;
+#include "rocksdb/slice.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class Slice;
 class Logger;
@@ -142,6 +142,6 @@ class AssociativeMergeOperator : public MergeOperator {
                             Logger* logger) const override;
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_MERGE_OPERATOR_H_

include/rocksdb/options.h

@@ -16,7 +16,7 @@
 #include "rocksdb/memtablerep.h"
 #include "rocksdb/slice_transform.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class Cache;
 class Comparator;
@@ -165,7 +165,7 @@ struct Options {
   // a block is the unit of reading from disk).
 
   // If non-NULL use the specified cache for blocks.
-  // If NULL, leveldb will automatically create and use an 8MB internal cache.
+  // If NULL, rocksdb will automatically create and use an 8MB internal cache.
   // Default: nullptr
   shared_ptr<Cache> block_cache;
 
@@ -351,13 +351,21 @@ struct Options {
   // Default value is 1800 (half an hour).
   int db_stats_log_interval;
 
-  // This specifies the log dir.
+  // This specifies the info LOG dir.
   // If it is empty, the log files will be in the same dir as data.
   // If it is non empty, the log files will be in the specified dir,
   // and the db data dir's absolute path will be used as the log file
   // name's prefix.
   std::string db_log_dir;
 
+  // This specifies the absolute dir path for write-ahead logs (WAL).
+  // If it is empty, the log files will be in the same dir as data,
+  //   dbname is used as the data dir by default
+  // If it is non empty, the log files will be in kept the specified dir.
+  // When destroying the db,
+  //   all log files in wal_dir and the dir itself is deleted
+  std::string wal_dir;
+
   // Disable compaction triggered by seek.
   // With bloomfilter and fast storage, a miss on one level
   // is very cheap if the file handle is cached in table cache
@@ -369,10 +377,23 @@ struct Options {
   // every compaction run.
   uint64_t delete_obsolete_files_period_micros;
 
-  // Maximum number of concurrent background compactions.
+  // Maximum number of concurrent background jobs, submitted to
+  // the default LOW priority thread pool
   // Default: 1
   int max_background_compactions;
 
+  // Maximum number of concurrent background memtable flush jobs, submitted to
+  // the HIGH priority thread pool.
+  // By default, all background jobs (major compaction and memtable flush) go
+  // to the LOW priority pool. If this option is set to a positive number,
+  // memtable flush jobs will be submitted to the HIGH priority pool.
+  // It is important when the same Env is shared by multiple db instances.
+  // Without a separate pool, long running major compaction jobs could
+  // potentially block memtable flush jobs of other db instances, leading to
+  // unnecessary Put stalls.
+  // Default: 0
+  int max_background_flushes;
+
   // Specify the maximal size of the info log file. If the log file
   // is larger than `max_log_file_size`, a new info log file will
   // be created.
@@ -421,6 +442,15 @@ struct Options {
   // Number of shards used for table cache.
   int table_cache_numshardbits;
 
+  // During data eviction of table's LRU cache, it would be inefficient
+  // to strictly follow LRU because this piece of memory will not really
+  // be released unless its refcount falls to zero. Instead, make two
+  // passes: the first pass will release items with refcount = 1,
+  // and if not enough space releases after scanning the number of
+  // elements specified by this parameter, we will remove items in LRU
+  // order.
+  int table_cache_remove_scan_count_limit;
+
   // size of one block in arena memory allocation.
   // If <= 0, a proper value is automatically calculated (usually 1/10 of
   // writer_buffer_size).
@@ -448,7 +478,7 @@ struct Options {
 
   // The number of seconds a WAL(write ahead log) should be kept after it has
   // been marked as Not Live. If the value is set. The WAL files are moved to
-  // the archive direcotory and deleted after the given TTL.
+  // the archive directory and deleted after the given TTL.
   // If set to 0, WAL files are deleted as soon as they are not required by
   // the database.
   // If set to std::numeric_limits<uint64_t>::max() the WAL files will never be
@@ -484,7 +514,7 @@ struct Options {
   // Default: false
   bool skip_log_error_on_recovery;
 
-  // if not zero, dump leveldb.stats to LOG every stats_dump_period_sec
+  // if not zero, dump rocksdb.stats to LOG every stats_dump_period_sec
   // Default: 3600 (1 hour)
   unsigned int stats_dump_period_sec;
 
@@ -548,6 +578,11 @@ struct Options {
   // an application to modify/delete a key-value during background compaction.
   // Default: a factory that doesn't provide any object
   std::shared_ptr<CompactionFilterFactory> compaction_filter_factory;
+
+  // Remove the log file immediately after the corresponding memtable is flushed
+  // to data file.
+  // Default: true
+  bool purge_log_after_memtable_flush;
 };
 
 //
@@ -655,6 +690,6 @@ struct FlushOptions {
   }
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_OPTIONS_H_

include/rocksdb/perf_context.h

@@ -3,7 +3,7 @@
 
 #include <stdint.h>
 
-namespace leveldb {
+namespace rocksdb {
 
 enum PerfLevel {
   kDisable        = 0,  // disable perf stats
@@ -28,11 +28,12 @@ struct PerfContext {
   uint64_t block_read_time;
   uint64_t block_checksum_time;
   uint64_t block_decompress_time;
+  uint64_t internal_key_skipped_count;
+  uint64_t internal_delete_skipped_count;
 };
 
 extern __thread PerfContext perf_context;
 
 }
 
-
 #endif

include/rocksdb/slice.h

@@ -20,7 +20,7 @@
 #include <string.h>
 #include <string>
 
-namespace leveldb {
+namespace rocksdb {
 
 class Slice {
  public:
@@ -91,7 +91,7 @@ class Slice {
             (memcmp(data_, x.data_, x.size_) == 0));
   }
 
- // private: make these public for leveldbjni access
+ // private: make these public for rocksdbjni access
   const char* data_;
   size_t size_;
 
@@ -117,6 +117,6 @@ inline int Slice::compare(const Slice& b) const {
   return r;
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_SLICE_H_

include/rocksdb/slice_transform.h

@@ -13,7 +13,7 @@
 
 #include <string>
 
-namespace leveldb {
+namespace rocksdb {
 
 class Slice;
 

include/rocksdb/statistics.h

@@ -13,7 +13,7 @@
 #include <memory>
 #include <vector>
 
-namespace leveldb {
+namespace rocksdb {
 
 /**
  * Keep adding ticker's here.
@@ -23,58 +23,64 @@ namespace leveldb {
  * And incrementing TICKER_ENUM_MAX.
  */
 enum Tickers {
-  BLOCK_CACHE_MISS = 0,
-  BLOCK_CACHE_HIT = 1,
-  BLOOM_FILTER_USEFUL = 2, // no. of times bloom filter has avoided file reads.
+  BLOCK_CACHE_MISS,
+  BLOCK_CACHE_HIT,
+  BLOOM_FILTER_USEFUL, // no. of times bloom filter has avoided file reads.
+
   /**
    * COMPACTION_KEY_DROP_* count the reasons for key drop during compaction
    * There are 3 reasons currently.
    */
-  COMPACTION_KEY_DROP_NEWER_ENTRY = 3, // key was written with a newer value.
-  COMPACTION_KEY_DROP_OBSOLETE = 4, // The key is obsolete.
-  COMPACTION_KEY_DROP_USER = 5, // user compaction function has dropped the key.
-  // Number of keys written to the database via the Put and Write call's
-  NUMBER_KEYS_WRITTEN = 6,
-  // Number of Keys read,
-  NUMBER_KEYS_READ = 7,
-  // Bytes written / read
-  BYTES_WRITTEN = 8,
-  BYTES_READ = 9,
-  NO_FILE_CLOSES = 10,
-  NO_FILE_OPENS = 11,
-  NO_FILE_ERRORS = 12,
-  // Time system had to wait to do LO-L1 compactions
-  STALL_L0_SLOWDOWN_MICROS = 13,
-  // Time system had to wait to move memtable to L1.
-  STALL_MEMTABLE_COMPACTION_MICROS = 14,
-  // write throttle because of too many files in L0
-  STALL_L0_NUM_FILES_MICROS = 15,
-  RATE_LIMIT_DELAY_MILLIS = 16,
+  COMPACTION_KEY_DROP_NEWER_ENTRY, // key was written with a newer value.
+  COMPACTION_KEY_DROP_OBSOLETE, // The key is obsolete.
+  COMPACTION_KEY_DROP_USER, // user compaction function has dropped the key.
 
-  NO_ITERATORS = 17, // number of iterators currently open
+  // Number of keys written to the database via the Put and Write call's
+  NUMBER_KEYS_WRITTEN,
+  // Number of Keys read,
+  NUMBER_KEYS_READ,
+  // Bytes written / read
+  BYTES_WRITTEN,
+  BYTES_READ,
+  NO_FILE_CLOSES,
+  NO_FILE_OPENS,
+  NO_FILE_ERRORS,
+  // Time system had to wait to do LO-L1 compactions
+  STALL_L0_SLOWDOWN_MICROS,
+  // Time system had to wait to move memtable to L1.
+  STALL_MEMTABLE_COMPACTION_MICROS,
+  // write throttle because of too many files in L0
+  STALL_L0_NUM_FILES_MICROS,
+  RATE_LIMIT_DELAY_MILLIS,
+
+  NO_ITERATORS, // number of iterators currently open
 
   // Number of MultiGet calls, keys read, and bytes read
-  NUMBER_MULTIGET_CALLS = 18,
-  NUMBER_MULTIGET_KEYS_READ = 19,
-  NUMBER_MULTIGET_BYTES_READ = 20,
+  NUMBER_MULTIGET_CALLS,
+  NUMBER_MULTIGET_KEYS_READ,
+  NUMBER_MULTIGET_BYTES_READ,
 
   // Number of deletes records that were not required to be
   // written to storage because key does not exist
-  NUMBER_FILTERED_DELETES = 21,
-  NUMBER_MERGE_FAILURES = 22,
-  SEQUENCE_NUMBER = 23,
+  NUMBER_FILTERED_DELETES,
+  NUMBER_MERGE_FAILURES,
+  SEQUENCE_NUMBER,
 
   // number of times bloom was checked before creating iterator on a
   // file, and the number of times the check was useful in avoiding
   // iterator creation (and thus likely IOPs).
-  BLOOM_FILTER_PREFIX_CHECKED = 24,
-  BLOOM_FILTER_PREFIX_USEFUL = 25,
+  BLOOM_FILTER_PREFIX_CHECKED,
+  BLOOM_FILTER_PREFIX_USEFUL,
 
   // Number of times we had to reseek inside an iteration to skip
   // over large number of keys with same userkey.
-  NUMBER_OF_RESEEKS_IN_ITERATION = 26,
+  NUMBER_OF_RESEEKS_IN_ITERATION,
 
-  TICKER_ENUM_MAX = 27
+  // Record the number of calls to GetUpadtesSince. Useful to keep track of
+  // transaction log iterator refreshes
+  GET_UPDATES_SINCE_CALLS,
+
+  TICKER_ENUM_MAX
 };
 
 // The order of items listed in  Tickers should be the same as
@@ -106,7 +112,8 @@ const std::vector<std::pair<Tickers, std::string>> TickersNameMap = {
   { SEQUENCE_NUMBER, "rocksdb.sequence.number" },
   { BLOOM_FILTER_PREFIX_CHECKED, "rocksdb.bloom.filter.prefix.checked" },
   { BLOOM_FILTER_PREFIX_USEFUL, "rocksdb.bloom.filter.prefix.useful" },
-  { NUMBER_OF_RESEEKS_IN_ITERATION, "rocksdb.number.reseeks.iteration" }
+  { NUMBER_OF_RESEEKS_IN_ITERATION, "rocksdb.number.reseeks.iteration" },
+  { GET_UPDATES_SINCE_CALLS, "rocksdb.getupdatessince.calls" }
 };
 
 /**
@@ -117,27 +124,27 @@ const std::vector<std::pair<Tickers, std::string>> TickersNameMap = {
  * And increment HISTOGRAM_ENUM_MAX
  */
 enum Histograms {
-  DB_GET = 0,
-  DB_WRITE = 1,
-  COMPACTION_TIME = 2,
-  TABLE_SYNC_MICROS = 3,
-  COMPACTION_OUTFILE_SYNC_MICROS = 4,
-  WAL_FILE_SYNC_MICROS = 5,
-  MANIFEST_FILE_SYNC_MICROS = 6,
+  DB_GET,
+  DB_WRITE,
+  COMPACTION_TIME,
+  TABLE_SYNC_MICROS,
+  COMPACTION_OUTFILE_SYNC_MICROS,
+  WAL_FILE_SYNC_MICROS,
+  MANIFEST_FILE_SYNC_MICROS,
   // TIME SPENT IN IO DURING TABLE OPEN
-  TABLE_OPEN_IO_MICROS = 7,
-  DB_MULTIGET = 8,
-  READ_BLOCK_COMPACTION_MICROS = 9,
-  READ_BLOCK_GET_MICROS = 10,
-  WRITE_RAW_BLOCK_MICROS = 11,
+  TABLE_OPEN_IO_MICROS,
+  DB_MULTIGET,
+  READ_BLOCK_COMPACTION_MICROS,
+  READ_BLOCK_GET_MICROS,
+  WRITE_RAW_BLOCK_MICROS,
 
-  STALL_L0_SLOWDOWN_COUNT = 12,
-  STALL_MEMTABLE_COMPACTION_COUNT = 13,
-  STALL_L0_NUM_FILES_COUNT = 14,
-  HARD_RATE_LIMIT_DELAY_COUNT = 15,
-  SOFT_RATE_LIMIT_DELAY_COUNT = 16,
-  NUM_FILES_IN_SINGLE_COMPACTION = 17,
-  HISTOGRAM_ENUM_MAX = 18
+  STALL_L0_SLOWDOWN_COUNT,
+  STALL_MEMTABLE_COMPACTION_COUNT,
+  STALL_L0_NUM_FILES_COUNT,
+  HARD_RATE_LIMIT_DELAY_COUNT,
+  SOFT_RATE_LIMIT_DELAY_COUNT,
+  NUM_FILES_IN_SINGLE_COMPACTION,
+  HISTOGRAM_ENUM_MAX
 };
 
 const std::vector<std::pair<Histograms, std::string>> HistogramsNameMap = {
@@ -191,7 +198,7 @@ class Histogram {
 
 /**
  * A dumb ticker which keeps incrementing through its life time.
- * Not thread safe. Locking is currently managed by external leveldb lock
+ * Thread safe. Locking managed by implementation of this interface.
  */
 class Ticker {
  public:
@@ -251,6 +258,6 @@ inline void SetTickerCount(std::shared_ptr<Statistics> statistics,
   }
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_STATISTICS_H_

include/rocksdb/status.h

@@ -16,7 +16,7 @@
 #include <string>
 #include "rocksdb/slice.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class Status {
  public:
@@ -121,6 +121,6 @@ inline void Status::operator=(const Status& s) {
   }
 }
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_STATUS_H_

include/rocksdb/table_builder.h

@@ -17,7 +17,7 @@
 #include "rocksdb/options.h"
 #include "rocksdb/status.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class BlockBuilder;
 class BlockHandle;
@@ -30,7 +30,12 @@ class TableBuilder {
   // caller to close the file after calling Finish(). The output file
   // will be part of level specified by 'level'.  A value of -1 means
   // that the caller does not know which level the output file will reside.
-  TableBuilder(const Options& options, WritableFile* file, int level=-1);
+  //
+  // If enable_compression=true, this table will follow the compression
+  // setting given in parameter options. If enable_compression=false, the
+  // table will not be compressed.
+  TableBuilder(const Options& options, WritableFile* file, int level=-1,
+               const bool enable_compression=true);
 
   // REQUIRES: Either Finish() or Abandon() has been called.
   ~TableBuilder();
@@ -90,6 +95,6 @@ class TableBuilder {
   void operator=(const TableBuilder&);
 };
 
-}  // namespace leveldb
+}  // namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_TABLE_BUILDER_H_

include/rocksdb/table_stats.h

@@ -0,0 +1,46 @@
+// Copyright (c) 2013 Facebook
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file. See the AUTHORS file for names of contributors.
+#pragma once
+
+#include <string>
+#include <unordered_map>
+
+namespace rocksdb {
+
+// TableStats contains a bunch of read-only stats of its associated
+// table.
+struct TableStats {
+ public:
+  // TODO(kailiu) we do not support user collected stats yet.
+  //
+  // Other than basic table stats, each table may also have the user
+  // collected stats.
+  // The value of the user-collected stats are encoded as raw bytes --
+  // users have to interprete these values by themselves.
+  typedef
+    std::unordered_map<std::string, std::string>
+    UserCollectedStats;
+
+  // the total size of all data blocks.
+  uint64_t data_size = 0;
+  // the total size of all index blocks.
+  uint64_t index_size = 0;
+  // total raw key size
+  uint64_t raw_key_size = 0;
+  // total raw value size
+  uint64_t raw_value_size = 0;
+  // the number of blocks in this table
+  uint64_t num_data_blocks = 0;
+  // the number of entries in this table
+  uint64_t num_entries = 0;
+
+  // The name of the filter policy used in this table.
+  // If no filter policy is used, `filter_policy_name` will be an empty string.
+  std::string filter_policy_name;
+
+  // user collected stats
+  UserCollectedStats user_collected_stats;
+};
+
+}  // namespace rocksdb

include/rocksdb/transaction_log.h

@@ -6,7 +6,7 @@
 #include "rocksdb/types.h"
 #include "rocksdb/write_batch.h"
 
-namespace leveldb {
+namespace rocksdb {
 
 class LogFile;
 typedef std::vector<std::unique_ptr<LogFile>> VectorLogPtr;
@@ -67,8 +67,8 @@ class TransactionLogIterator {
   // REQUIRES: Valid() to be true.
   virtual void Next() = 0;
 
-  // Return's ok if the iterator is valid.
-  // Return the Error when something has gone wrong.
+  // Returns ok if the iterator is valid.
+  // Returns the Error when something has gone wrong.
   virtual Status status() = 0;
 
   // If valid return's the current write_batch and the sequence number of the
@@ -76,6 +76,6 @@ class TransactionLogIterator {
   // ONLY use if Valid() is true and status() is OK.
   virtual BatchResult GetBatch() = 0;
 };
-} //  namespace leveldb
+} //  namespace rocksdb
 
 #endif  // STORAGE_ROCKSDB_INCLUDE_TRANSACTION_LOG_ITERATOR_H_

include/rocksdb/types.h

@@ -3,12 +3,13 @@
 
 #include <stdint.h>
 
-namespace leveldb {
+namespace rocksdb {
 
 // Define all public custom types here.
 
 // Represents a sequence number in a WAL file.
 typedef uint64_t SequenceNumber;
 
-}  //  namespace leveldb
+}  //  namespace rocksdb
+
 #endif //  STORAGE_ROCKSDB_INCLUDE_TYPES_H_