doc problems with templating out hostvars

Author: sjpb

docs/experimental/compute-init.md

@@ -1,14 +1,17 @@
 # compute-init
 
-TODO: describe current status.
+The following roles are currently functional:
+- resolv_conf
+- etc_hosts
+- stackhpc.openhpc
 
 # Development
 
 To develop/debug this without actually having to build an image:
 
 
 1. Deploy a cluster using tofu and ansible/site.yml as normal. This will
-   additionally configure the control node to export compute hosts over NFS.
+   additionally configure the control node to export compute hostvars over NFS.
    Check the cluster is up.
 
 2. Reimage the compute nodes:
@@ -22,6 +25,10 @@ To develop/debug this without actually having to build an image:
 
         ansible-playbook ansible/fatimage.yml --tags compute_init
 
+    NB: This will also re-export the compute hostvars, as the nodes are not
+    in the builder group, which conveniently means any changes made to that
+    play also get picked up.
+
 5. Fake a reimage of compute to run ansible-init and the compute-init playbook:
 
     On compute node where metadata was added:
@@ -31,8 +38,9 @@ To develop/debug this without actually having to build an image:
 
     Use `systemctl status ansible-init` to view stdout/stderr from Ansible.
 
-Steps 4/5 can be repeated with changes to the compute script. If desirable
-reimage the compute node(s) first as in step 3.
+Steps 4/5 can be repeated with changes to the compute script. If required,
+reimage the compute node(s) first as in step 2 and/or add additional metadata
+as in step 3.
 
 # Results/progress
 
@@ -144,3 +152,40 @@ This commit - shows that hostvars have loaded:
     Dec 13 21:06:20 rl9-compute-0.rl9.invalid ansible-init[27585]: [INFO] ansible-init completed successfully
     Dec 13 21:06:20 rl9-compute-0.rl9.invalid systemd[1]: Finished ansible-init.service.
 
+# Design notes
+
+- In general, we don't want to rely on NFS export. So should e.g. copy files
+  from this mount ASAP in the compute-init script. TODO:
+- There are a few possible approaches:
+
+  1. Control node copies files resulting from role into cluster exports,
+     compute-init copies to local disk. Only works if files are not host-specific
+     Examples: etc_hosts, eessi config?
+  
+  2. Re-implement the role. Works if the role vars are not too complicated,
+     (else they all need to be duplicated in compute-init). Could also only
+     support certain subsets of role functionality or variables
+     Examples: resolv_conf, stackhpc.openhpc
+
+
+# Problems with templated hostvars
+
+Here are all the ones which actually rely on hostvars from other nodes,
+which therefore aren't available:
+
+```
+[root@rl9-compute-0 rocky]# grep hostvars /mnt/cluster/hostvars/rl9-compute-0/hostvars.yml 
+    "grafana_address": "{{ hostvars[groups['grafana'].0].api_address }}",
+    "grafana_api_address": "{{ hostvars[groups['grafana'].0].internal_address }}",
+    "mysql_host": "{{ hostvars[groups['mysql'] | first].api_address }}",
+    "nfs_server_default": "{{ hostvars[groups['control'] | first ].internal_address }}",
+    "openhpc_slurm_control_host": "{{ hostvars[groups['control'].0].api_address }}",
+    "openondemand_address": "{{ hostvars[groups['openondemand'].0].api_address if groups['openondemand'] | count > 0 else '' }}",
+    "openondemand_node_proxy_directives": "{{ _opeonondemand_unset_auth if (openondemand_auth == 'basic_pam' and 'openondemand_host_regex' and groups['grafana'] | length > 0 and hostvars[ groups['grafana']  | first]._grafana_auth_is_anonymous) else '' }}",
+    "openondemand_servername": "{{ hostvars[ groups['openondemand'] | first].ansible_host }}",
+    "prometheus_address": "{{ hostvars[groups['prometheus'].0].api_address }}",
+        "{{ hostvars[groups['freeipa_server'].0].ansible_host }}"
+```
+
+More generally, there is nothing to stop any group var depending on a
+"{{ hostvars[] }}" interpolation ...