Review docs, Dockerfile, and tutorials. (#964)

The test image now contains more debugging tools.

Improve our tutorial by not requiring an external docker volume in the
environment, having the volume only available within the docker-compose
context is all we need here.

Small improvement: make the app image respect the SIGTERM that
docker-compose down sends, by rewritting it using Python this time.

Author: DimCitus

Dockerfile

@@ -55,7 +55,11 @@ RUN apt-get update \
     watch \
     lsof \
     psutils \
-	valgrind \
+    psmisc \
+    htop \
+    less \
+	mg \
+    valgrind \
     postgresql-common \
 	&& rm -rf /var/lib/apt/lists/*
 

docs/citus-quickstart.rst

@@ -455,6 +455,15 @@ node too:
    -------
        75
 
+Cleanup
+-------
+
+To dispose of the entire tutorial environment, just use the following command:
+
+::
+
+   $ docker-compose down
+
 Next steps
 ----------
 

docs/citus/Dockerfile.app

@@ -13,6 +13,7 @@ RUN apt-get update \
      watch \
      lsof \
      psutils \
+     python3 \
   && rm -rf /var/lib/apt/lists/*
 
 # we use apt.postgresql.org
@@ -29,5 +30,7 @@ RUN adduser --disabled-password --gecos '' docker
 RUN adduser docker sudo
 RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
 
+COPY ./app.py /usr/src/app/app.py
+
 USER docker
-ENTRYPOINT while true; do date -R; sleep 600; done
+ENTRYPOINT [ "/usr/src/app/app.py" ]

docs/citus/app.py

@@ -0,0 +1,28 @@
+#! /usr/bin/env python3
+
+#
+# Write a Python application that knows how to exit gracefully when
+# receiving SIGTERM (at docker-compose down time), but doesn't know how to
+# do much else.
+#
+
+import sys
+import time
+import signal
+from datetime import datetime
+
+
+def sigterm_handler(_signo, _stack_frame):
+    sys.exit(0)
+
+
+signal.signal(signal.SIGTERM, sigterm_handler)
+
+if __name__ == "__main__":
+    try:
+        while True:
+            print("%s" % datetime.now())
+            time.sleep(600)
+    except BaseException:
+        # Keyboard Interrupt or something
+        pass

docs/install.rst

@@ -41,10 +41,10 @@ To avoid automated creation of a Postgres data directory when installing the
 debian package, follow those steps:
 
 ::
-   
+
   $ curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
   $ echo "deb http://apt.postgresql.org/pub/repos/apt buster-pgdg main" > /etc/apt/sources.list.d/pgdg.list
-  
+
   # bypass initdb of a "main" cluster
   $ echo 'create_main_cluster = false' | sudo tee -a /etc/postgresql-common/createcluster.conf
   $ apt-get update
@@ -54,53 +54,19 @@ That way when it's time to :ref:`pg_autoctl_create_monitor` or
 :ref:`pg_autoctl_create_postgres` there is no confusion about how to handle
 the default Postgres service created by debian: it has not been created at
 all.
-  
+
 Fedora, CentOS, or Red Hat
 --------------------------
 
-Quick install
-~~~~~~~~~~~~~
-
-The following installation method downloads a bash script that automates
-several steps. The full script is available for review at our `package cloud
-installation instructions page`__ url.
-
-__ https://packagecloud.io/citusdata/community/install#bash
-
-.. code-block:: bash
-
-  # add the required packages to your system
-  curl https://install.citusdata.com/community/rpm.sh | sudo bash
-
-  # install pg_auto_failover
-  sudo yum install -y pg-auto-failover14_12
-
-  # confirm installation
-  /usr/pgsql-12/bin/pg_autoctl --version
-
-Manual installation
-~~~~~~~~~~~~~~~~~~~
-
-If you'd prefer to install your repo on your system manually, follow the
-instructions from `package cloud manual installation`__ page. This page will
-guide you with the specific details to achieve the 3 steps:
-
-  1. install the pygpgme yum-utils packages for your distribution
-  2. install a new RPM reposiroty for CitusData packages
-  3. update your local yum cache
-
-Then when that's done, you can proceed with installing pg_auto_failover
-itself as in the previous case:
-
-.. code-block:: bash
-
-  # install pg_auto_failover
-  sudo yum install -y pg-auto-failover14_12
+The Postgres community packaging team for RPM based system has worked on
+supporting pg_auto_failover. Binary packages are available by following the
+documentation at `PostgreSQL Yum Repository`__.
 
-  # confirm installation
-  /usr/pgsql-12/bin/pg_autoctl --version
+__ https://yum.postgresql.org
 
-__ https://packagecloud.io/citusdata/community/install#manual-rpm
+A single package named ``pg_auto_failover`` is available on the RPM based
+systems, containing both the monitor Postgres extension and the pg_autoctl
+command line.
 
 Installing a pgautofailover Systemd unit
 ----------------------------------------
@@ -140,3 +106,34 @@ It is important that PostgreSQL is started by ``pg_autoctl`` rather than by
 systemd itself, as it might be that a failover has been done during a
 reboot, for instance, and that once the reboot complete we want the local
 Postgres to re-join as a secondary node where it used to be a primary node.
+
+
+Building pg_auto_failover from sources
+--------------------------------------
+
+To build the project, make sure you have installed the build-dependencies,
+then just type `make`. You can install the resulting binary using `make
+install`.
+
+For this to work please consider adding both the binary and the source
+repositories to your debian distribution by using the following apt sources,
+as an example targetting the debian bullseye distribution:
+
+::
+
+   deb http://apt.postgresql.org/pub/repos/apt bullseye-pgdg main
+   deb-src http://apt.postgresql.org/pub/repos/apt bullseye-pgdg main
+
+Then we can install the build dependencies for Postgres, knowing that
+pg_auto_failover uses the same build dependencies:
+
+::
+
+   $ sudo apt-get build-dep -y --no-install-recommends postgresql-14
+
+Then build pg_auto_failover from sources with the following instructions:
+
+::
+
+   $ make -s clean && make -s -j12 all
+   $ sudo make -s install

docs/tutorial.rst

@@ -43,8 +43,6 @@ or run the docker build command directly:
 
    $ git clone https://github.com/citusdata/pg_auto_failover
    $ cd pg_auto_failover/docs/tutorial
-
-   $ docker build -t pg_auto_failover:tutorial -f Dockerfile ../..
    $ docker-compose build
 
 Postgres failover with two nodes
@@ -74,7 +72,7 @@ following command:
 
 ::
 
-   $ docker-compose up
+   $ docker-compose up app monitor node1 node2
 
 The command above starts the services up. The first service is the monitor
 and is created with the command ``pg_autoctl create monitor``. The options
@@ -196,6 +194,107 @@ And we can verify that we still have the data available::
        75
    (1 row)
 
+Multiple Standbys Architectures
+-------------------------------
+
+The ``docker-compose.yml`` file comes with a third node that you can bring
+up to obtain the following architecture:
+
+.. figure:: ./tikz/arch-multi-standby.svg
+   :alt: pg_auto_failover Architecture for a standalone PostgreSQL service
+
+   pg_auto_failover architecture with a primary and two standby nodes
+
+Adding a second standby node
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To run a second standby node, or a third Postgres node, simply run the
+following command:
+
+::
+
+   $ docker-compose up -d node3
+
+We can see the resulting replication settings with the following command:
+
+::
+
+   $ docker-compose exec monitor pg_autoctl show settings
+
+     Context |    Name |                   Setting | Value
+   ----------+---------+---------------------------+-------------------------------------------------------------
+   formation | default |      number_sync_standbys | 1
+     primary |   node1 | synchronous_standby_names | 'ANY 1 (pgautofailover_standby_1, pgautofailover_standby_3)'
+        node |   node2 |        candidate priority | 50
+        node |   node1 |        candidate priority | 50
+        node |   node3 |        candidate priority | 50
+        node |   node2 |        replication quorum | true
+        node |   node1 |        replication quorum | true
+        node |   node3 |        replication quorum | true
+
+Editing the replication settings while in production
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It's then possible to change the production architecture obtained with
+playing with the :ref:`architecture_setup` commands. Specifically, try the
+following command to change the candidate_priority of the node3 to zero, in
+order for it to never be a candidate for failover:
+
+::
+
+   $ docker-compose exec node3 pg_autoctl set candidate-priority 0 --name node3
+
+To see the replication settings for all the nodes, the following command can
+be useful, and is described in more details in the :ref:`architecture_setup`
+section.
+
+::
+
+   $ docker-compose exec monitor pg_autoctl show settings
+
+     Context |    Name |                   Setting | Value
+   ----------+---------+---------------------------+-------------------------------------------------------------
+   formation | default |      number_sync_standbys | 1
+     primary |   node1 | synchronous_standby_names | 'ANY 1 (pgautofailover_standby_1, pgautofailover_standby_3)'
+        node |   node2 |        candidate priority | 50
+        node |   node1 |        candidate priority | 50
+        node |   node3 |        candidate priority | 0
+        node |   node2 |        replication quorum | true
+        node |   node1 |        replication quorum | true
+        node |   node3 |        replication quorum | true
+
+Then in a separate terminal (but in the same directory, because of the way
+docker-compose works with projects), you can run the following watch
+command:
+
+::
+
+   $ docker-compose exec monitor pg_autoctl watch
+
+And in the main terminal, while the watch command output is visible, you can
+run a switchover operation:
+
+::
+
+   $ docker-compose exec monitor pg_autoctl perform switchover
+
+Getting familiar with those commands one of you next steps. The manual has
+coverage for them in the following links:
+
+ * :ref:`pg_autoctl_watch`
+ * :ref:`pg_autoctl_show_state`
+ * :ref:`pg_autoctl_show_settings`
+ * :ref:`pg_autoctl_set_node_candidate_priority`
+
+Cleanup
+-------
+
+To dispose of the entire tutorial environment, just use the following command:
+
+::
+
+   $ docker-compose down
+
 Next steps
 ----------
 

docs/tutorial/Dockerfile

@@ -47,5 +47,8 @@ COPY src/ ./src
 RUN make -s clean && make -s install -j8
 RUN cp /usr/lib/postgresql/${PGVERSION}/bin/pg_autoctl /usr/local/bin
 
+RUN install -o docker -m 0755 -d /var/lib/postgres
+VOLUME /var/lib/postgres
+
 USER docker
 ENV PG_AUTOCTL_DEBUG 1

docs/tutorial/Dockerfile.app

@@ -13,6 +13,7 @@ RUN apt-get update \
      watch \
      lsof \
      psutils \
+     python3 \
   && rm -rf /var/lib/apt/lists/*
 
 # we use apt.postgresql.org
@@ -29,5 +30,7 @@ RUN adduser --disabled-password --gecos '' docker
 RUN adduser docker sudo
 RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
 
+COPY ./app.py /usr/src/app/app.py
+
 USER docker
-ENTRYPOINT while true; do date -R; sleep 600; done
+ENTRYPOINT [ "/usr/src/app/app.py" ]

docs/tutorial/Makefile

@@ -3,18 +3,22 @@ CONTAINER_NAME = pg_auto_failover:tutorial
 all: build down up;
 
 build:
-	docker build -t $(CONTAINER_NAME) -f Dockerfile ../..
 	docker-compose build
 
 up:
-	docker-compose up
+	docker-compose up app monitor node1 node2
+
+node3:
+	docker-compose up -d node3
 
 down:
-	docker-compose down --volumes --remove-orphans
-	rm -rf {monitor,node1,node2,node3}/pgaf/
+	docker-compose down
 
 state:
 	docker-compose exec monitor pg_autoctl show state
 
 switchover:
 	docker-compose exec monitor pg_autoctl perform switchover
+
+watch:
+	docker-compose exec monitor pg_autoctl watch

docs/tutorial/app.py

@@ -0,0 +1,28 @@
+#! /usr/bin/env python3
+
+#
+# Write a Python application that knows how to exit gracefully when
+# receiving SIGTERM (at docker-compose down time), but doesn't know how to
+# do much else.
+#
+
+import sys
+import time
+import signal
+from datetime import datetime
+
+
+def sigterm_handler(_signo, _stack_frame):
+    sys.exit(0)
+
+
+signal.signal(signal.SIGTERM, sigterm_handler)
+
+if __name__ == "__main__":
+    try:
+        while True:
+            print("%s" % datetime.now())
+            time.sleep(600)
+    except BaseException:
+        # Keyboard Interrupt or something
+        pass