Merge pull request #109 from timvaillancourt/MCB_1.0-bugfix18

MCB_1.0: Use callbacks to check multiprocessing.Pool threads, README updates for release

Author: timvaillancourt

README.rst

@@ -4,27 +4,28 @@ MongoDB Consistent Backup Tool - mongodb-consistent-backup
 About
 ~~~~~
 
-Creates cluster-consistent point-in-time backups of MongoDB via wrapping
-'mongodump'. Backups are remotely-pulled and outputted onto the host
-running the tool.
+Creates cluster-consistent point-in-time backups of MongoDB with optional
+archiving, compression/de-duplication, encryption and upload functionality
 
 Features
 ~~~~~~~~
 
 -  Works on a single replset (2+ members) or a sharded cluster
 -  Auto-discovers healthy members for backup by considering replication
-   lag, replication 'priority' and by preferring 'hidden' members.
+   lag, replication 'priority' and by preferring 'hidden' members
 -  Creates cluster-consistent backups across many separate shards
+-  'mongodump' is the default *(and currently only)* backup method. Other methods coming soon!
 -  Transparent restore process (*just add --oplogReplay flag to your
    mongorestore command*)
--  Archiving and compression of backups
+-  Archiving and compression of backups (*optional*)
 -  Block de-duplication and optional AES encryption at rest via `ZBackup <http://zbackup.org/>`__
-   archiving method
--  AWS S3 Secure/HTTPS Multipart backup uploads (*optional*)
+   archiving method (*optional*)
+-  `AWS S3 <https://aws.amazon.com/s3/>`__ Secure Multipart backup uploads (*optional*)
 -  `Nagios NSCA <https://sourceforge.net/p/nagios/nsca>`__ push
    notification support (*optional*)
 -  Modular backup, archiving, upload and notification components
 -  Multi-threaded, single executable
+-  Auto-scales to number of available CPUs by default
 
 Current Limitations
 ~~~~~~~~~~~~~~~~~~~
@@ -38,47 +39,56 @@ Requirements:
 ~~~~~~~~~~~~~
 
 -  Backup consistency depends on consistent server time across all
-   hosts. Server time **must be synchronized on all nodes** using ntpd
+   hosts! Server time **must be synchronized on all nodes** using ntpd
    and a consistent time source or virtualization guest agent that 
    syncs time
 -  Must have 'mongodump' installed and specified if not at default:
    */usr/bin/mongodump*. Even if you do not run MongoDB 3.2+, it is
-   strongly recommended to use MongoDB 3.2+ binaries due to inline
-   compression, parallelism, etc
+   strongly recommended to use MongoDB 3.2+ mongodump binaries due
+   to inline compression and parallelism features
 -  Must have Python 2.7 installed
 
+Releases
+~~~~~~~~
+
+Pre-built release binaries and packages are available on our `GitHub Releases Page <https://github.com/Percona-Lab/mongodb_consistent_backup/releases>`__. We recommend most users deploy mongodb_consistent_backup using these packages.
+
 Build/Install
 ~~~~~~~~~~~~~
 
-To build on CentOS/RedHat, you wil need the following packages (see
-command):
+To build on CentOS/RedHat, you will need the following packages installed:
 
 ::
 
-    yum install python python-devel python-virtualenv gcc git make libffi-devel openssl-devel
+    $ yum install python python-devel python-virtualenv gcc git make libffi-devel openssl-devel
 
-To install to default '*/usr/local/bin/mongodb-consistent-backup*\ ':
+To build an CentOS/RedHat RPM of the tool *(recommended)*:
 
 ::
 
-    cd path/to/mongo_backup 
-    make
-    make install
+    $ cd /path/to/mongodb_consistent_backup
+    $ make rpm
+
+To build and install from source *(to default '/usr/local/bin/mongodb-consistent-backup')*:
 
-Use the PREFIX= variable to change the installation path (*default:
-/usr/local*), ie: ``make PREFIX=/usr install`` to install to:
-'*/usr/bin/mongodb-consistent-backup*\ '.
+::
+
+    $ cd /path/to/mongodb_consistent_backup
+    $ make
+    $ make install
+
+Use the PREFIX= variable to change the installation path (*default: /usr/local*), ie: ``make PREFIX=/usr install`` to install to: '*/usr/bin/mongodb-consistent-backup*'.
 
 MongoDB Authorization
 ~~~~~~~~~~~~~~~~~~~~~
 
-If your replset/cluster uses `Authentication <https://docs.mongodb.com/manual/core/authentication>`__, you must add a user with the "backup" and "clusterMonitor" built-in auth roles.
+If your replset/cluster uses `Authentication <https://docs.mongodb.com/manual/core/authentication>`__, you must add a user with the `"backup" <https://docs.mongodb.com/manual/reference/built-in-roles/#backup>`__ and `"clusterMonitor" <https://docs.mongodb.com/manual/reference/built-in-roles/#clusterMonitor>`__ built-in auth roles.
 
 To create a user, execute the following **replace the 'pwd' field with a secure password!**:
 
 ::
 
-    db.createUser({
+    db.getSiblingDB("admin").createUser({
             user: "mongodb_consistent_backup",
             pwd: "PASSWORD-HERE",
             roles: [
@@ -94,36 +104,44 @@ Run a Backup
 
 **Using Command-Line Flags**
 
+*Note: username+password is visible in process lists when set using the command-line flags. Use a config file (below) to hide credentials!*
+
 ::
 
-    $ mongodb-consistent-backup -H mongos1.example.com -P 27018 -u mongodb-consistent-backup -p s3cr3t -n prodwebsite -l /opt/mongobackups
+    $ mongodb-consistent-backup -H mongos1.example.com -P 27018 -u mongodb-consistent-backup -p s3cr3t -n prodwebsite -l /var/lib/mongodb-consistent-backup
     ...
     ...
     $ ls /opt/mongobackups
     prodwebsite
 
 **Using a Config File**
 
+The tool supports a YAML-based config file for settings. The config file is loaded first and any additional command-line arguments override the file based config settings.
+
 ::
 
     $ mongodb-consistent-backup --config /etc/mongodb-consistent-backup.yml
     ...
 
+An example *(with comments)* of the YAML-based config file is here: `conf/mongodb-consistent-backup.example.conf <conf/mongodb-consistent-backup.example.conf>`__.
+
+A description of all available config settings can also be listed by passing the '--help' flag to the tool.
+
 Restore a Backup
 ~~~~~~~~~~~~~~~~
 
-The backups are mongorestore compatible. The *--oplogReplay* flag **MUST** be present to replay the oplogs to ensure consistency.
+The backups are mongorestore compatible and stored in a directory per backup. The *--oplogReplay* flag **MUST** be present to replay the oplogs to ensure consistency.
 
 ::
 
     $ tar xfvz <shardname>.tar.gz
     ...
-    $ mongorestore --host mongod12.example.com --port 27017 -u admin -p 123456 --oplogReplay --dir /path/to/backup/dump
+    $ mongorestore --host mongod12.example.com --port 27017 -u admin -p 123456 --oplogReplay --dir /var/lib/mongodb-consistent-backup/default/20170424_0000/rs0/dump
 
 Run as Docker Container (Experimental)
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-*Note: you need to use persistent volumes to store backups long-term on disk when using Docker. Data in Docker containers is destroyed when the container is deleted.*
+*Note: you need to use persistent volumes to store backups and/or config files long-term when using Docker. Data in Docker containers is destroyed when the container is deleted.*
 
 **Via Docker Hub**
 
@@ -143,12 +161,16 @@ Run as Docker Container (Experimental)
 ZBackup Archiving (Optional)
 ~~~~~~~
 
-`ZBackup <http://zbackup.org/>`__ *(with LZMA compression)* is an optional archive method for mongodb_consistent_backup. This archive method significantly reduces disk usage for backups via deduplication and compression. 
+*Note: the ZBackup archive method is not yet compatible with the 'Upload' phase. Disable uploading by setting 'upload.method' to 'none' in the meantime.*
+
+`ZBackup <http://zbackup.org/>`__ *(with LZMA compression)* is an optional archive method for mongodb_consistent_backup. This archive method significantly reduces disk usage for backups via de-duplication and compression. 
 
-ZBackup offers block de-duplication and compression of backups and optionally supports AES-128 encryption at rest. The ZBackup archive method causes backups to be stored via ZBackup at archive time.
+ZBackup offers block de-duplication and compression of backups and optionally supports AES-128 *(CBC mode with PKCS#7 padding)* encryption at rest. The ZBackup archive method causes backups to be stored via ZBackup at archive time.
 
 To enable, ZBackup must be installed on your system and the 'archive.method' config file variable *(or --archive.method flag=)* must be set to 'zbackup'.
 
+ZBackup's compression is most efficient when compression is disabled in the backup phase, to do this set 'backup.<method>.compression' to 'none'.
+
 **Install on CentOS/RHEL**
 
 ::
@@ -161,21 +183,36 @@ To enable, ZBackup must be installed on your system and the 'archive.method' con
 
     $ apt-get install zbackup
 
-ZBackup data is stored in a repository directory named *mongodb_consistent_backup-zbackup* and must be restored using a 'zbackup restore ...' command.
 
-**Get Backup from ZBackup Repo**
+**Get Backup from ZBackup**
+
+ZBackup data is stored in a storage directory named *'mongodb_consistent_backup-zbackup'* and must be restored using a 'zbackup restore ...' command.
+
+::
+
+    $ zbackup restore --password-file /etc/zbackup.passwd /mnt/backup/default/mongodb_consistent_backup-zbackup/backups/20170424_0000.tar | tar -xf
+
+**Delete Backup from ZBackup**
+
+To remove a backup, first delete the .tar file in 'backups' subdir of the ZBackup storage directory. After, run a 'zbackup gc full' garbage collection to remove unused data.
 
 ::
 
-    $ zbackup restore --password-file /etc/zbackup.passwd /mnt/backup/default/mongodb_consistent_backup-zbackup/backups/20170424_0000.tar
+    $ rm -f /mnt/backup/default/mongodb_consistent_backup-zbackup/backups/20170424_0000.tar
+    $ zbackup gc full --password-file /etc/zbackup.passwd /mnt/backup/default/mongodb_consistent_backup-zbackup 
     
 Roadmap
 ~~~~~~~
 
+-  More testing: this project has many flows that probably need more in-depth testing. Please submit and bugs and/or bugfixes!
 -  "Distributed Mode" for running backup on remote hosts *(vs. only on one host)*
--  Support more notification methods *(Prometheus, PagerDuty, etc)* and upload methods *(Google Cloud Storage, Rsync, etc)*
+-  Upload compatibility for ZBackup archive phase *(upload unsupported today)*
+-  Backup retention/rotation *(eg: delete old backups)*
+-  Support more notification methods *(Prometheus, PagerDuty, etc)*
+-  Support more upload methods *(Google Cloud Storage, Rsync, etc)*
 -  Support SSL MongoDB connections
--  Unit tests
+-  Documentation for running under Docker with persistent volumes
+-  Python unit tests
 
 Contact
 ~~~~~~~

mongodb_consistent_backup/Archive/Tar/Tar.py

@@ -2,7 +2,8 @@
 import logging
 
 from copy_reg import pickle
-from multiprocessing import Pool, TimeoutError
+from multiprocessing import Pool
+from time import sleep
 from types import MethodType
 
 from TarThread import TarThread
@@ -30,20 +31,23 @@ def __init__(self, manager, config, timer, base_dir, backup_dir, **kwargs):
         self._pool   = None
         self._pooled = []
 
+    def done(self, done_dir):
+        if done_dir in self._pooled:
+            logging.debug("Archiving completed for: %s" % done_dir)
+            self._pooled.remove(done_dir)
+        else:
+            raise OperationError("Unexpected response from tar thread: %s" % done_dir)
+
     def wait(self):
         if len(self._pooled) > 0:
             self._pool.close()
-            logging.debug("Waiting for tar threads to stop")
-            while len(self._pooled) > 0:
-                try:
-                    item = self._pooled[0]
-                    path, result = item
-                    result.get(1)
-		    logging.debug("Archiving completed for directory: %s" % path)
-                    self._pooled.remove(item)
-                except TimeoutError:
-                    continue
+            while len(self._pooled):
+                logging.debug("Waiting for %i tar thread(s) to stop" % len(self._pooled))
+                sleep(2)
+            self._pool.terminate()
+            logging.debug("Stopped all tar threads")
             self.stopped = True
+            self.running = False
 
     def run(self):
         try:
@@ -64,20 +68,19 @@ def run(self):
                     output_file = "%s.tar" % subdir_name
                     if self.do_gzip():
                         output_file  = "%s.tgz" % subdir_name
-                    result = self._pool.apply_async(TarThread(subdir_name, output_file, self.do_gzip(), self.verbose, self.binary).run)
-                    self._pooled.append((subdir_name, result))
+                    self._pool.apply_async(TarThread(subdir_name, output_file, self.do_gzip(), self.verbose, self.binary).run, callback=self.done)
+                    self._pooled.append(subdir_name)
             except Exception, e:
                 self._pool.terminate()
                 logging.fatal("Could not create tar archiving thread! Error: %s" % e)
                 raise Error(e)
             finally:
                 self.wait()
-            self.completed = True
+                self.completed = True
 
     def close(self, code=None, frame=None):
-        logging.debug("Stopping tar archiving threads")
         if not self.stopped and self._pool is not None:
+            logging.debug("Stopping tar archiving threads")
             self._pool.terminate()
-            self._pool.join()
             logging.info("Stopped all tar archiving threads")
             self.stopped = True

mongodb_consistent_backup/Archive/Tar/TarThread.py

@@ -50,3 +50,4 @@ def run(self):
             else:
                 logging.fatal("Output file: %s already exists!" % self.output_file)
                 sys.exit(1)
+            return self.backup_dir

mongodb_consistent_backup/Common/LocalCommand.py

@@ -58,6 +58,6 @@ def run(self):
         # return exit code from mongodump
         return self._process.returncode
 
-    def close(self):
+    def close(self, frame=None, code=None):
         if self._process:
             self._process.terminate()

mongodb_consistent_backup/Oplog/Resolver/Resolver.py

@@ -5,6 +5,7 @@
 from bson.timestamp import Timestamp
 from copy_reg import pickle
 from multiprocessing import Pool
+from time import sleep
 from types import MethodType
 
 from ResolverThread import ResolverThread
@@ -34,16 +35,23 @@ def __init__(self, manager, config, timer, base_dir, backup_dir, tailed_oplogs,
         self.resolver_summary      = {}
         self.resolver_state        = {}
 
+        self.running   = False
+        self.stopped   = False
+        self.completed = False
+        self._pool     = None
+        self._pooled   = []
         try:
             self._pool = Pool(processes=self.threads(None, 2))
         except Exception, e:
             logging.fatal("Could not start oplog resolver pool! Error: %s" % e)
             raise Error(e)
 
-    def close(self):
-       if self._pool:
+    def close(self, code=None, frame=None):
+       if self._pool and not self.stopped:
+           logging.debug("Stopping all oplog resolver threads")
            self._pool.terminate()
-           self._pool.join()
+           logging.info("Stopped all oplog resolver threads")
+           self.stopped = True
 
     def get_consistent_end_ts(self):
         ts = None
@@ -54,9 +62,28 @@ def get_consistent_end_ts(self):
                     ts = Timestamp(instance['last_ts'].time, 0)
         return ts
 
+    def done(self, done_uri):
+        if done_uri in self._pooled:
+            logging.debug("Resolving completed for: %s" % done_uri)
+            self._pooled.remove(done_uri)
+        else:
+            raise OperationError("Unexpected response from resolver thread: %s" % done_uri)
+
+    def wait(self):
+        if len(self._pooled) > 0:
+            self._pool.close()
+            while len(self._pooled):
+                logging.debug("Waiting for %i oplog resolver thread(s) to stop" % len(self._pooled))
+                sleep(2)
+            self._pool.terminate()
+            logging.debug("Stopped all oplog resolve threads")
+            self.stopped = True
+            self.running = False
+
     def run(self):
-        logging.info("Resolving oplogs (options: threads=%s,compression=%s)" % (self.threads(), self.compression()))
+        logging.info("Resolving oplogs (options: threads=%s, compression=%s)" % (self.threads(), self.compression()))
         self.timer.start(self.timer_name)
+        self.running = True
 
         for shard in self.backup_oplogs:
             backup_oplog = self.backup_oplogs[shard]
@@ -80,14 +107,17 @@ def run(self):
                             backup_oplog.copy(),
                             self.get_consistent_end_ts(),
                             self.do_gzip()
-                        ).run)
+                        ).run, callback=self.done)
+                        self._pooled.append(uri.str())
                     except Exception, e:
                         logging.fatal("Resolve failed for %s! Error: %s" % (uri, e))
                         raise Error(e)
             else:
                 logging.info("No tailed oplog for host %s" % uri)
-        self._pool.close()
-        self._pool.join()
+        self.wait()
+        self.running   = False
+        self.stopped   = True
+        self.completed = True
 
         self.timer.stop(self.timer_name)
         logging.info("Oplog resolving completed in %.2f seconds" % self.timer.duration(self.timer_name))