Support and document persisting the Octavia CA

doc/source/operations/octavia.rst

@@ -65,6 +65,151 @@ The default image path is ``/tmp/amphora-x64-haproxy.qcow2``.
 
   kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/maintenance/octavia-amphora-image-register.yml -e image_path="<path-to-amphora-image>"
 
+Handling TLS certificates
+=========================
+
+Octavia uses mutual TLS to secure communication between the amphorae and
+Octavia services. It uses a private CA to sign both client and server
+certificates. These certificates need to be generated when first deploying
+Octavia, and will later need to be rotated (details below). We use the
+kolla-ansible built-in support for generating these certificates:
+
+.. code-block:: console
+
+   kayobe kolla ansible run octavia-certificates
+
+This command will output certificates and keys in ``${KOLLA_CONFIG_PATH}/octavia-certificates``
+
+Copy the relevant certificates into your kayobe-config:
+
+.. code-block:: console
+
+   cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia
+   cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem .
+   cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem .
+   cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem .
+   cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem .
+
+Encrypt any files containing the keys:
+
+.. code-block:: console
+
+   ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault
+   ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault
+
+Checking certificate expiry
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+   ansible-vault decrypt client.cert-and-key.pem --vault-password-file ~/vault
+   openssl x509 -enddate -noout -in client.cert-and-key.pem
+
+Backing up the octavia-certificates directory
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the root of your kayobe-config checkout:
+
+.. code-block:: console
+
+   tools/backup-octavia-certificates.sh
+
+This will output an encrypted backup to ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar``
+Commit this file to store the backup.
+
+.. _restoring-octavia-certificates-directory:
+
+Restoring octavia-certificates directory when regenerating certificates
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the root of your kayobe-config checkout:
+
+.. code-block:: console
+
+   tools/restore-octavia-certificates.sh
+
+This will use the encrypted backup in ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar``
+to restore ``${KOLLA_CONFIG_PATH}/octavia-certificates``. This will allow you
+to reuse the client CA.
+
+Rotating client.cert-and-key.pem
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This has a lifetime of 1 year.
+
+#. Follow the steps to restore octavia-certificates so you can reuse the client
+   CA. See :ref:`restoring-octavia-certificates-directory`.
+
+#. Make sure your config allows you to regenerate a certificate with the same
+   common name.
+
+   .. code-block:: console
+      :caption: $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/index.txt.attr
+
+      unique_subject = no
+
+#. Remove the old files relating to the client certificate:
+
+   .. code-block:: console
+
+      rm $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/{client.cert-and-key.pem,client.csr.pem,client.cert.pem}
+
+#. Regenerate the certificates
+
+   .. code-block:: console
+
+      kayobe kolla ansible run octavia-certificates
+
+#. Backup your octavia-certificates directory (see previous section).
+
+#. Copy your new certificate to the correct location:
+
+   .. code-block:: console
+
+      cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia
+      cp  $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem .
+      cp  $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem .
+      ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault
+
+#. Reconfigure Octavia
+
+   .. code-block:: console
+
+      kayobe overcloud service reconfigure -kt octavia
+
+#. Run Tempest with the `octavia` test list to check it is working. See
+   :ref:`running_tempest_with_kayobe_automation`.
+
+#. Commit and push any changes.
+
+Rotating the CAs
+~~~~~~~~~~~~~~~~
+
+The CAs have a 10 year lifetime. Simply delete the relevant directory under
+``$KOLLA_CONFIG_PATH/octavia-certificates/`` and regenerate it with:
+
+   .. code-block:: console
+
+      kayobe kolla ansible run octavia-certificates
+
+Copy the relevant certificates into your kayobe-config.
+
+.. code-block:: console
+
+   cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia
+   cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem .
+   cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem .
+   cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem .
+   cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem .
+
+Encrypt any files containing the keys.
+
+.. code-block:: console
+
+   ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault
+   ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault
+
+Follow any instructions in the `upstream docs <https://docs.openstack.org/octavia/latest/admin/guides/operator-maintenance.html>`_.
 
 Manually deleting broken load balancers
 =======================================

doc/source/operations/tempest.rst

@@ -1,3 +1,5 @@
+.. _running_tempest_with_kayobe_automation:
+
 ======================================
 Running Tempest with Kayobe Automation
 ======================================

tools/backup-octavia-certificates.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+set -eu
+
+if [ -z ${KAYOBE_CONFIG_PATH:+x} ]; then
+    1>&2 echo 'Please source kayobe-env'
+    exit 1
+fi
+
+if [ -z ${KAYOBE_VAULT_PASSWORD:+x} ]; then
+    1>&2 echo 'Please set Kayobe vault password'
+    exit 1
+fi
+
+if [ ! -d $KOLLA_CONFIG_PATH/octavia-certificates ]; then
+    1>&2 echo 'Certificates missing'
+    exit 1
+fi
+
+pushd $KOLLA_CONFIG_PATH
+ls octavia-certificates
+tar -c -f - octavia-certificates | ansible-vault encrypt --vault-password-file $KAYOBE_CONFIG_PATH/../../tools/vault-helper > $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar 2>/dev/null
+popd

tools/restore-octavia-certificates.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+set -eu
+
+if [ -z ${KAYOBE_CONFIG_PATH:+x} ]; then
+    1>&2 echo 'Please source kayobe-env'
+    exit -1
+fi
+
+if [ -z ${KAYOBE_VAULT_PASSWORD:+x} ]; then
+    1>&2 echo 'Please set Kayobe vault password'
+    exit -1
+fi
+
+if [ -d $KOLLA_CONFIG_PATH/octavia-certificates ]; then
+    1>&2 echo 'Certificates exists. Please remove if you wish to restore.'
+    exit -1
+fi
+
+cat $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar | ansible-vault decrypt --vault-password-file $KAYOBE_CONFIG_PATH/../../tools/vault-helper 2>/dev/null | tar -xvf - -C $KOLLA_CONFIG_PATH

tools/vault-helper.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+echo "$KAYOBE_VAULT_PASSWORD"