From 9958a8093391effb515b6cc033f93b1a29bd1a05 Mon Sep 17 00:00:00 2001 From: Matt Crees Date: Tue, 18 Nov 2025 15:16:51 +0000 Subject: [PATCH 1/3] Support and document persisting the Octavia CA --- doc/source/operations/octavia.rst | 141 ++++++++++++++++++++++++++ tools/backup-octavia-certificates.sh | 23 +++++ tools/restore-octavia-certificates.sh | 20 ++++ tools/vault-helper.sh | 3 + 4 files changed, 187 insertions(+) create mode 100755 tools/backup-octavia-certificates.sh create mode 100755 tools/restore-octavia-certificates.sh create mode 100755 tools/vault-helper.sh diff --git a/doc/source/operations/octavia.rst b/doc/source/operations/octavia.rst index 52c25f2313..f8aaeca26f 100644 --- a/doc/source/operations/octavia.rst +++ b/doc/source/operations/octavia.rst @@ -65,6 +65,147 @@ The default image path is ``/tmp/amphora-x64-haproxy.qcow2``. kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/maintenance/octavia-amphora-image-register.yml -e image_path="" +Handling TLS certificates +========================= + +Octavia uses mutual TLS to secure communication between the amphorae and +Octavia services. It uses a private CA to sign both client and server +certificates. We use the kolla-ansible built-in support for generating these +certificates: + +.. code-block:: console + + kayobe kolla ansible run octavia-certificates + +This command will output certificates and keys in ``${KOLLA_CONFIG_PATH}/octavia-certificates`` + +Copy the relevant certificates into your kayobe-config: + +.. code-block:: console + + cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem . + +Encrypt any files containing the keys: + +.. code-block:: console + + ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault + ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault + +Checking certificate expiry +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + ansible-vault decrypt client.cert-and-key.pem --vault-password-file ~/vault + openssl x509 -enddate -noout -in client.cert-and-key.pem + +Backing up the octavia-certificates directory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the root of your kayobe-config checkout: + +.. code-block:: console + + tools/backup-octavia-certificates.sh + +This will output an encrypted backup to ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar`` +Commit this file to store the backup. + +Restoring octavia-certificates directory when regenerating certificates +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the root of your kayobe-config checkout: + +.. code-block:: console + + tools/restore-octavia-certificates.sh + +This will use the encrypted backup in ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar`` +to restore ``${KOLLA_CONFIG_PATH}/octavia-certificates``. This will allow you +to reuse the client CA. + +Rotating client.cert-and-key.pem +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This has a lifetime of 1 year. + +1) Follow the steps to restore octavia-certificates so you can reuse the client + CA. + +2) Make sure your config allows you to regenerate a certificate with the same + common name. + + .. code-block:: console + :caption: $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/index.txt.attr + + unique_subject = no + +3) Remove the old files relating to the client certificate: + + .. code-block:: console + + rm $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/{client.cert-and-key.pem,client.csr.pem,client.cert.pem} + +4) Regenerate the certificates + + .. code-block:: console + + kayobe kolla ansible run octavia-certificates + +5) Backup your octavia-certificates directory (see previous section). + +6) Copy your new certificate to the correct location: + + .. code-block:: console + + cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . + ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault + +7) Reconfigure octavia + + .. code-block:: console + + kayobe overcloud service reconfigure -kt octavia + +8) Run tempest with the `octavia` test list to check it is working. + +9) Commit and push any changes. + +Rotating the CAs +~~~~~~~~~~~~~~~~ + +The CAs have a 10 year lifetime. Simply delete the relevant directory under +``$KOLLA_CONFIG_PATH/octavia-certificates/`` and regenerate it with: + + .. code-block:: console + + kayobe kolla ansible run octavia-certificates + +Copy the relevant certificates into your kayobe-config. + +.. code-block:: console + + cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem . + cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem . + +Encrypt any files containing the keys. + +.. code-block:: console + + ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault + ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault + +Follow any instructions in the `upstream docs `_. Manually deleting broken load balancers ======================================= diff --git a/tools/backup-octavia-certificates.sh b/tools/backup-octavia-certificates.sh new file mode 100755 index 0000000000..89e363f82c --- /dev/null +++ b/tools/backup-octavia-certificates.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +set -eu + +if [ -z ${KAYOBE_CONFIG_PATH:+x} ]; then + 1>&2 echo 'Please source kayobe-env' + exit 1 +fi + +if [ -z ${KAYOBE_VAULT_PASSWORD:+x} ]; then + 1>&2 echo 'Please set Kayobe vault password' + exit 1 +fi + +if [ ! -d $KOLLA_CONFIG_PATH/octavia-certificates ]; then + 1>&2 echo 'Certificates missing' + exit 1 +fi + +pushd $KOLLA_CONFIG_PATH +ls octavia-certificates +tar -c -f - octavia-certificates | ansible-vault encrypt --vault-password-file $KAYOBE_CONFIG_PATH/../../tools/vault-helper > $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar 2>/dev/null +popd diff --git a/tools/restore-octavia-certificates.sh b/tools/restore-octavia-certificates.sh new file mode 100755 index 0000000000..3f0b3f2bf3 --- /dev/null +++ b/tools/restore-octavia-certificates.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +set -eu + +if [ -z ${KAYOBE_CONFIG_PATH:+x} ]; then + 1>&2 echo 'Please source kayobe-env' + exit -1 +fi + +if [ -z ${KAYOBE_VAULT_PASSWORD:+x} ]; then + 1>&2 echo 'Please set Kayobe vault password' + exit -1 +fi + +if [ -d $KOLLA_CONFIG_PATH/octavia-certificates ]; then + 1>&2 echo 'Certificates exists. Please remove if you wish to restore.' + exit -1 +fi + +cat $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar | ansible-vault decrypt --vault-password-file $KAYOBE_CONFIG_PATH/../../tools/vault-helper 2>/dev/null | tar -xvf - -C $KOLLA_CONFIG_PATH diff --git a/tools/vault-helper.sh b/tools/vault-helper.sh new file mode 100755 index 0000000000..453cdebbef --- /dev/null +++ b/tools/vault-helper.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "$KAYOBE_VAULT_PASSWORD" From 4e141c7917f5b5b10fa36206bea4f96a6efb4863 Mon Sep 17 00:00:00 2001 From: Matt Crees Date: Tue, 18 Nov 2025 16:35:45 +0000 Subject: [PATCH 2/3] Reference restoring-octavia-certificates-directory --- doc/source/operations/octavia.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/source/operations/octavia.rst b/doc/source/operations/octavia.rst index f8aaeca26f..b5e6dc242e 100644 --- a/doc/source/operations/octavia.rst +++ b/doc/source/operations/octavia.rst @@ -116,6 +116,8 @@ In the root of your kayobe-config checkout: This will output an encrypted backup to ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar`` Commit this file to store the backup. +.. _restoring-octavia-certificates-directory: + Restoring octavia-certificates directory when regenerating certificates ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -135,7 +137,7 @@ Rotating client.cert-and-key.pem This has a lifetime of 1 year. 1) Follow the steps to restore octavia-certificates so you can reuse the client - CA. + CA. See :ref:`restoring-octavia-certificates-directory`. 2) Make sure your config allows you to regenerate a certificate with the same common name. From 43a0fd009aa87318e25e229492ce9b1cb1a6f2f6 Mon Sep 17 00:00:00 2001 From: Matt Crees Date: Wed, 19 Nov 2025 15:59:09 +0000 Subject: [PATCH 3/3] Updates from review --- doc/source/operations/octavia.rst | 24 +++++++++++++----------- doc/source/operations/tempest.rst | 2 ++ 2 files changed, 15 insertions(+), 11 deletions(-) diff --git a/doc/source/operations/octavia.rst b/doc/source/operations/octavia.rst index b5e6dc242e..e470bf5047 100644 --- a/doc/source/operations/octavia.rst +++ b/doc/source/operations/octavia.rst @@ -70,8 +70,9 @@ Handling TLS certificates Octavia uses mutual TLS to secure communication between the amphorae and Octavia services. It uses a private CA to sign both client and server -certificates. We use the kolla-ansible built-in support for generating these -certificates: +certificates. These certificates need to be generated when first deploying +Octavia, and will later need to be rotated (details below). We use the +kolla-ansible built-in support for generating these certificates: .. code-block:: console @@ -136,10 +137,10 @@ Rotating client.cert-and-key.pem This has a lifetime of 1 year. -1) Follow the steps to restore octavia-certificates so you can reuse the client +#. Follow the steps to restore octavia-certificates so you can reuse the client CA. See :ref:`restoring-octavia-certificates-directory`. -2) Make sure your config allows you to regenerate a certificate with the same +#. Make sure your config allows you to regenerate a certificate with the same common name. .. code-block:: console @@ -147,21 +148,21 @@ This has a lifetime of 1 year. unique_subject = no -3) Remove the old files relating to the client certificate: +#. Remove the old files relating to the client certificate: .. code-block:: console rm $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/{client.cert-and-key.pem,client.csr.pem,client.cert.pem} -4) Regenerate the certificates +#. Regenerate the certificates .. code-block:: console kayobe kolla ansible run octavia-certificates -5) Backup your octavia-certificates directory (see previous section). +#. Backup your octavia-certificates directory (see previous section). -6) Copy your new certificate to the correct location: +#. Copy your new certificate to the correct location: .. code-block:: console @@ -170,15 +171,16 @@ This has a lifetime of 1 year. cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault -7) Reconfigure octavia +#. Reconfigure Octavia .. code-block:: console kayobe overcloud service reconfigure -kt octavia -8) Run tempest with the `octavia` test list to check it is working. +#. Run Tempest with the `octavia` test list to check it is working. See + :ref:`running_tempest_with_kayobe_automation`. -9) Commit and push any changes. +#. Commit and push any changes. Rotating the CAs ~~~~~~~~~~~~~~~~ diff --git a/doc/source/operations/tempest.rst b/doc/source/operations/tempest.rst index 1f6dd43cd9..f63828262c 100644 --- a/doc/source/operations/tempest.rst +++ b/doc/source/operations/tempest.rst @@ -1,3 +1,5 @@ +.. _running_tempest_with_kayobe_automation: + ====================================== Running Tempest with Kayobe Automation ======================================