Fix issue with GRUB defaulting to an old kernel

On some Rocky Linux 9 deployments, we are seeing GRUB defaulting to the
old kernel included in the DIB image, even after newer kernels have been
installed. This appears to be related to the presence of Boot Loader
Specification (BLS) entries with a machine ID lower in alphabetical
order than the current one:

    [stack@host ~]$ sudo cat /etc/machine-id
    cd3361a338fe47348de9937e51a7a4aa
    [stack@host ~]$ sudo ls -l /boot/loader/entries/
    total 20
    -rw-r--r--. 1 root root 449 Mar 31  2023 104a42359fae41b687caac066397aec2-0-rescue.conf
    -rw-r--r--. 1 root root 397 Mar 31  2023 104a42359fae41b687caac066397aec2-5.14.0-162.22.2.el9_1.x86_64.conf
    -rw-r--r--  1 root root 446 Jun  9  2023 cd3361a338fe47348de9937e51a7a4aa-0-rescue.conf
    -rw-r--r--  1 root root 422 Jun  9  2023 cd3361a338fe47348de9937e51a7a4aa-5.14.0-284.11.1.el9_2.x86_64.conf
    -rw-r--r--  1 root root 422 Jan  9 09:40 cd3361a338fe47348de9937e51a7a4aa-5.14.0-284.30.1.el9_2.x86_64.conf

Add a new `reset-bls-entries.yml` playbook which will rename existing
BLS entries using the current machine ID. This should prompt Grub to
pick the most recent kernel on next reboot.

Author: priteau

etc/kayobe/ansible/reset-bls-entries.yml

@@ -0,0 +1,39 @@
+---
+# Custom playbook to reset Boot Loader Specification (BLS) entries to resolve
+# an issue with GRUB defaulting to an old kernel. This is adapted from a Bash
+# script in diskimage-builder:
+# https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/rhel/post-install.d/03-reset-bls-entries
+
+- name: Reset BLS entries
+  hosts: overcloud
+  become: true
+  tags:
+    - reset-bls-entries
+  tasks:
+    - name: Get machine ID
+      command: cat /etc/machine-id
+      register: machine_id
+      check_mode: false
+
+    - name: Find entries with wrong machine ID
+      ansible.builtin.find:
+        paths: /boot/loader/entries
+        patterns: "*.conf"
+      register: bls_entries
+      check_mode: false
+
+    # We set force to false to avoid replacing an existing BLS entry with the
+    # correct machine ID.
+    - name: Rename entries with wrong machine ID
+      copy:
+        src: "/boot/loader/entries/{{ item }}"
+        dest: "/boot/loader/entries/{{ item | ansible.builtin.regex_replace('^[a-f0-9]*', machine_id.stdout) }}"
+        force: false
+        remote_src: true
+      with_items: "{{ bls_entries.files | map(attribute='path') | reject('search', machine_id.stdout) | map('basename') }}"
+
+    - name: Remove entries with wrong machine ID
+      file:
+        path: "/boot/loader/entries/{{ item }}"
+        state: absent
+      with_items: "{{ bls_entries.files | map(attribute='path') | reject('search', machine_id.stdout) | map('basename') }}"

releasenotes/notes/reset-bls-entries-b2bded62c5887937.yaml

@@ -0,0 +1,7 @@
+---
+fixes:
+  - |
+    Add a new ``reset-bls-entries.yml`` custom playbook which will rename
+    existing Boot Loader Specification (BLS) entries using the current machine
+    ID for each host. This should fix an issue with Grub not selecting the most
+    recent kernel during boot.