[RocksDB] BackupableDB

Summary:
In this diff I present you BackupableDB v1. You can easily use it to backup your DB and it will do incremental snapshots for you.
Let's first describe how you would use BackupableDB. It's inheriting StackableDB interface so you can easily construct it with your DB object -- it will add a method RollTheSnapshot() to the DB object. When you call RollTheSnapshot(), current snapshot of the DB will be stored in the backup dir. To restore, you can just call RestoreDBFromBackup() on a BackupableDB (which is a static method) and it will restore all files from the backup dir. In the next version, it will even support automatic backuping every X minutes.

There are multiple things you can configure:
1. backup_env and db_env can be different, which is awesome because then you can easily backup to HDFS or wherever you feel like.
2. sync - if true, it *guarantees* backup consistency on machine reboot
3. number of snapshots to keep - this will keep last N snapshots around if you want, for some reason, be able to restore from an earlier snapshot. All the backuping is done in incremental fashion - if we already have 00010.sst, we will not copy it again. *IMPORTANT* -- This is based on assumption that 00010.sst never changes - two files named 00010.sst from the same DB will always be exactly the same. Is this true? I always copy manifest, current and log files.
4. You can decide if you want to flush the memtables before you backup, or you're fine with backing up the log files -- either way, you get a complete and consistent view of the database at a time of backup.
5. More things you can find in BackupableDBOptions

Here is the directory structure I use:

   backup_dir/CURRENT_SNAPSHOT - just 4 bytes holding the latest snapshot
               0, 1, 2, ... - files containing serialized version of each snapshot - containing a list of files
               files/*.sst - sst files shared between snapshots - if one snapshot references 00010.sst and another one needs to backup it from the DB, it will just reference the same file
               files/ 0/, 1/, 2/, ... - snapshot directories containing private snapshot files - current, manifest and log files

All the files are ref counted and deleted immediatelly when they get out of scope.

Some other stuff in this diff:
1. Added GetEnv() method to the DB. Discussed with @haobo and we agreed that it seems right thing to do.
2. Fixed StackableDB interface. The way it was set up before, I was not able to implement BackupableDB.

Test Plan:
I have a unittest, but please don't look at this yet. I just hacked it up to help me with debugging. I will write a lot of good tests and update the diff.

Also, `make asan_check`

Reviewers: dhruba, haobo, emayanke

Reviewed By: dhruba

CC: leveldb, haobo

Differential Revision: https://reviews.facebook.net/D14295

Author: igorcanadi

Makefile

@@ -76,6 +76,7 @@ TESTS = \
 	skiplist_test \
 	stringappend_test \
 	ttl_test \
+	backupable_db_test \
 	version_edit_test \
 	version_set_test \
 	write_batch_test\
@@ -272,6 +273,9 @@ perf_context_test: db/perf_context_test.o $(LIBOBJECTS) $(TESTHARNESS)
 prefix_test: db/prefix_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(CXX) db/prefix_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS)
 
+backupable_db_test: utilities/backupable/backupable_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
+	$(CXX) utilities/backupable/backupable_db_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@  $(LDFLAGS) $(COVERAGEFLAGS)
+
 ttl_test: utilities/ttl/ttl_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(CXX) utilities/ttl/ttl_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@  $(LDFLAGS) $(COVERAGEFLAGS)
 

db/db_impl.cc

@@ -3176,6 +3176,10 @@ Status DBImpl::MakeRoomForWrite(bool force) {
   return s;
 }
 
+const std::string& DBImpl::GetName() const {
+  return dbname_;
+}
+
 Env* DBImpl::GetEnv() const {
   return env_;
 }

db/db_impl.h

@@ -67,6 +67,7 @@ class DBImpl : public DB {
   virtual int NumberLevels();
   virtual int MaxMemCompactionLevel();
   virtual int Level0StopWriteTrigger();
+  virtual const std::string& GetName() const;
   virtual Env* GetEnv() const;
   virtual const Options& GetOptions() const;
   virtual Status Flush(const FlushOptions& options);

db/db_test.cc

@@ -4444,6 +4444,10 @@ class ModelDB: public DB {
   return -1;
   }
 
+  virtual const std::string& GetName() const {
+    return name_;
+  }
+
   virtual Env* GetEnv() const {
     return nullptr;
   }
@@ -4521,6 +4525,7 @@ class ModelDB: public DB {
   };
   const Options options_;
   KVMap map_;
+  std::string name_ = "";
 };
 
 static std::string RandomKey(Random* rnd, int minimum = 0) {

include/rocksdb/db.h

@@ -228,6 +228,10 @@ class DB {
   // Number of files in level-0 that would stop writes.
   virtual int Level0StopWriteTrigger() = 0;
 
+  // Get DB name -- the exact same name that was provided as an argument to
+  // DB::Open()
+  virtual const std::string& GetName() const = 0;
+
   // Get Env object from the DB
   virtual Env* GetEnv() const = 0;
 

include/utilities/backupable_db.h

@@ -0,0 +1,128 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
+// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file. See the AUTHORS file for names of contributors.
+
+#pragma once
+#include "utilities/stackable_db.h"
+#include "rocksdb/env.h"
+#include "rocksdb/status.h"
+
+#include <string>
+#include <map>
+#include <vector>
+
+namespace rocksdb {
+
+struct BackupableDBOptions {
+  // Where to keep the backup files. Has to be different than dbname_
+  // Best to set this to dbname_ + "/backups"
+  // Required
+  std::string backup_dir;
+
+  // Backup Env object. It will be used for backup file I/O. If it's
+  // nullptr, backups will be written out using DBs Env. If it's
+  // non-nullptr, backup's I/O will be performed using this object.
+  // If you want to have backups on HDFS, use HDFS Env here!
+  // Default: nullptr
+  Env* backup_env;
+
+  // Backup info and error messages will be written to info_log
+  // if non-nullptr.
+  // Default: nullptr
+  Logger* info_log;
+
+  // If sync == true, we can guarantee you'll get consistent backup even
+  // on a machine crash/reboot. Backup process is slower with sync enabled.
+  // If sync == false, we don't guarantee anything on machine reboot. However,
+  // chances are some of the backups are consistent.
+  // Default: true
+  bool sync;
+
+  // If true, it will delete whatever backups there are already
+  // Default: false
+  bool destroy_old_data;
+
+  explicit BackupableDBOptions(const std::string& _backup_dir,
+                               Env* _backup_env = nullptr,
+                               Logger* _info_log = nullptr,
+                               bool _sync = true,
+                               bool _destroy_old_data = false) :
+      backup_dir(_backup_dir),
+      backup_env(_backup_env),
+      info_log(_info_log),
+      sync(_sync),
+      destroy_old_data(_destroy_old_data) { }
+};
+
+class BackupEngine;
+
+typedef uint32_t BackupID;
+
+struct BackupInfo {
+  BackupID backup_id;
+  int64_t timestamp;
+  uint64_t size;
+
+  BackupInfo() {}
+  BackupInfo(BackupID _backup_id, int64_t _timestamp, uint64_t _size)
+      : backup_id(_backup_id), timestamp(_timestamp), size(_size) {}
+};
+
+// Stack your DB with BackupableDB to be able to backup the DB
+class BackupableDB : public StackableDB {
+ public:
+  // BackupableDBOptions have to be the same as the ones used in a previous
+  // incarnation of the DB
+  BackupableDB(DB* db, const BackupableDBOptions& options);
+  virtual ~BackupableDB();
+
+  // Captures the state of the database in the latest backup
+  // NOT a thread safe call
+  Status CreateNewBackup(bool flush_before_backup = false);
+  // Returns info about backups in backup_info
+  void GetBackupInfo(std::vector<BackupInfo>* backup_info);
+  // deletes old backups, keeping latest num_backups_to_keep alive
+  Status PurgeOldBackups(uint32_t num_backups_to_keep);
+  // deletes a specific backup
+  Status DeleteBackup(BackupID backup_id);
+
+ private:
+  BackupEngine* backup_engine_;
+};
+
+// Use this class to access information about backups and restore from them
+class RestoreBackupableDB {
+  public:
+   RestoreBackupableDB(Env* db_env, const BackupableDBOptions& options);
+   ~RestoreBackupableDB();
+
+   // Returns info about backups in backup_info
+   void GetBackupInfo(std::vector<BackupInfo>* backup_info);
+
+   // restore from backup with backup_id
+   // IMPORTANT -- if you restore from some backup that is not the latest,
+   // you HAVE to delete all the newer backups immediately, before creating
+   // new backup on the restored database. Otherwise, your new backups
+   // will be corrupted.
+   // TODO should we enforce this somehow?
+   Status RestoreDBFromBackup(BackupID backup_id, const std::string& db_dir,
+                              const std::string& wal_dir);
+
+   // restore from the latest backup
+   Status RestoreDBFromLatestBackup(const std::string& db_dir,
+                                    const std::string& wal_dir);
+   // deletes old backups, keeping latest num_backups_to_keep alive
+   Status PurgeOldBackups(uint32_t num_backups_to_keep);
+   // deletes a specific backup
+   Status DeleteBackup(BackupID backup_id);
+
+ private:
+  BackupEngine* backup_engine_;
+};
+
+} // rocksdb namespace

include/utilities/stackable_db.h

@@ -103,6 +103,10 @@ class StackableDB : public DB {
     return db_->Level0StopWriteTrigger();
   }
 
+  virtual const std::string& GetName() const override {
+    return db_->GetName();
+  }
+
   virtual Env* GetEnv() const override {
     return db_->GetEnv();
   }

util/coding.cc

@@ -217,6 +217,17 @@ Slice GetLengthPrefixedSlice(const char* data) {
   return Slice(p, len);
 }
 
+Slice GetSliceUntil(Slice* slice, char delimiter) {
+  uint32_t len;
+  for (len = 0; len < slice->size() && slice->data()[len] != delimiter; ++len) {
+    // nothing
+  }
+
+  Slice ret(slice->data(), len);
+  slice->remove_prefix(len + ((len < slice->size()) ? 1 : 0));
+  return ret;
+}
+
 void BitStreamPutInt(char* dst, size_t dstlen, size_t offset,
                      uint32_t bits, uint64_t value) {
   assert((offset + bits + 7)/8 <= dstlen);

util/coding.h

@@ -40,6 +40,8 @@ extern bool GetVarint64(Slice* input, uint64_t* value);
 extern bool GetLengthPrefixedSlice(Slice* input, Slice* result);
 extern Slice GetLengthPrefixedSlice(const char* data);
 
+extern Slice GetSliceUntil(Slice* slice, char delimiter);
+
 // Pointer-based variants of GetVarint...  These either store a value
 // in *v and return a pointer just past the parsed value, or return
 // nullptr on error.  These routines only look at bytes in the range