diff --git a/doc/source/operations/octavia.rst b/doc/source/operations/octavia.rst index 52c25f2313..e470bf5047 100644 --- a/doc/source/operations/octavia.rst +++ b/doc/source/operations/octavia.rst @@ -65,6 +65,151 @@ The default image path is ``/tmp/amphora-x64-haproxy.qcow2``. kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/maintenance/octavia-amphora-image-register.yml -e image_path="" +Handling TLS certificates +========================= + +Octavia uses mutual TLS to secure communication between the amphorae and +Octavia services. It uses a private CA to sign both client and server +certificates. These certificates need to be generated when first deploying +Octavia, and will later need to be rotated (details below). We use the +kolla-ansible built-in support for generating these certificates: + +.. code-block:: console + + kayobe kolla ansible run octavia-certificates + +This command will output certificates and keys in ``${KOLLA_CONFIG_PATH}/octavia-certificates`` + +Copy the relevant certificates into your kayobe-config: + +.. code-block:: console + + cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem . + +Encrypt any files containing the keys: + +.. code-block:: console + + ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault + ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault + +Checking certificate expiry +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + ansible-vault decrypt client.cert-and-key.pem --vault-password-file ~/vault + openssl x509 -enddate -noout -in client.cert-and-key.pem + +Backing up the octavia-certificates directory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the root of your kayobe-config checkout: + +.. code-block:: console + + tools/backup-octavia-certificates.sh + +This will output an encrypted backup to ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar`` +Commit this file to store the backup. + +.. _restoring-octavia-certificates-directory: + +Restoring octavia-certificates directory when regenerating certificates +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the root of your kayobe-config checkout: + +.. code-block:: console + + tools/restore-octavia-certificates.sh + +This will use the encrypted backup in ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar`` +to restore ``${KOLLA_CONFIG_PATH}/octavia-certificates``. This will allow you +to reuse the client CA. + +Rotating client.cert-and-key.pem +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This has a lifetime of 1 year. + +#. Follow the steps to restore octavia-certificates so you can reuse the client + CA. See :ref:`restoring-octavia-certificates-directory`. + +#. Make sure your config allows you to regenerate a certificate with the same + common name. + + .. code-block:: console + :caption: $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/index.txt.attr + + unique_subject = no + +#. Remove the old files relating to the client certificate: + + .. code-block:: console + + rm $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/{client.cert-and-key.pem,client.csr.pem,client.cert.pem} + +#. Regenerate the certificates + + .. code-block:: console + + kayobe kolla ansible run octavia-certificates + +#. Backup your octavia-certificates directory (see previous section). + +#. Copy your new certificate to the correct location: + + .. code-block:: console + + cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . + ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault + +#. Reconfigure Octavia + + .. code-block:: console + + kayobe overcloud service reconfigure -kt octavia + +#. Run Tempest with the `octavia` test list to check it is working. See + :ref:`running_tempest_with_kayobe_automation`. + +#. Commit and push any changes. + +Rotating the CAs +~~~~~~~~~~~~~~~~ + +The CAs have a 10 year lifetime. Simply delete the relevant directory under +``$KOLLA_CONFIG_PATH/octavia-certificates/`` and regenerate it with: + + .. code-block:: console + + kayobe kolla ansible run octavia-certificates + +Copy the relevant certificates into your kayobe-config. + +.. code-block:: console + + cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem . + +Encrypt any files containing the keys. + +.. code-block:: console + + ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault + ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault + +Follow any instructions in the `upstream docs `_. Manually deleting broken load balancers ======================================= diff --git a/doc/source/operations/tempest.rst b/doc/source/operations/tempest.rst index 1f6dd43cd9..f63828262c 100644 --- a/doc/source/operations/tempest.rst +++ b/doc/source/operations/tempest.rst @@ -1,3 +1,5 @@ +.. _running_tempest_with_kayobe_automation: + ====================================== Running Tempest with Kayobe Automation ====================================== diff --git a/tools/backup-octavia-certificates.sh b/tools/backup-octavia-certificates.sh new file mode 100755 index 0000000000..89e363f82c --- /dev/null +++ b/tools/backup-octavia-certificates.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +set -eu + +if [ -z ${KAYOBE_CONFIG_PATH:+x} ]; then + 1>&2 echo 'Please source kayobe-env' + exit 1 +fi + +if [ -z ${KAYOBE_VAULT_PASSWORD:+x} ]; then + 1>&2 echo 'Please set Kayobe vault password' + exit 1 +fi + +if [ ! -d $KOLLA_CONFIG_PATH/octavia-certificates ]; then + 1>&2 echo 'Certificates missing' + exit 1 +fi + +pushd $KOLLA_CONFIG_PATH +ls octavia-certificates +tar -c -f - octavia-certificates | ansible-vault encrypt --vault-password-file $KAYOBE_CONFIG_PATH/../../tools/vault-helper > $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar 2>/dev/null +popd diff --git a/tools/restore-octavia-certificates.sh b/tools/restore-octavia-certificates.sh new file mode 100755 index 0000000000..3f0b3f2bf3 --- /dev/null +++ b/tools/restore-octavia-certificates.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +set -eu + +if [ -z ${KAYOBE_CONFIG_PATH:+x} ]; then + 1>&2 echo 'Please source kayobe-env' + exit -1 +fi + +if [ -z ${KAYOBE_VAULT_PASSWORD:+x} ]; then + 1>&2 echo 'Please set Kayobe vault password' + exit -1 +fi + +if [ -d $KOLLA_CONFIG_PATH/octavia-certificates ]; then + 1>&2 echo 'Certificates exists. Please remove if you wish to restore.' + exit -1 +fi + +cat $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar | ansible-vault decrypt --vault-password-file $KAYOBE_CONFIG_PATH/../../tools/vault-helper 2>/dev/null | tar -xvf - -C $KOLLA_CONFIG_PATH diff --git a/tools/vault-helper.sh b/tools/vault-helper.sh new file mode 100755 index 0000000000..453cdebbef --- /dev/null +++ b/tools/vault-helper.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "$KAYOBE_VAULT_PASSWORD"