Update helm chart to be more easily configurable (#56)

* Update helm chart to be more easily configurable

Signed-off-by: clux <sszynrae@gmail.com>

* do not default generate servicemonitor

Signed-off-by: clux <sszynrae@gmail.com>

* doc update

Signed-off-by: clux <sszynrae@gmail.com>

* can't bump here unless we tag...

Signed-off-by: clux <sszynrae@gmail.com>

---------

Signed-off-by: clux <sszynrae@gmail.com>

Author: clux

README.md

@@ -7,33 +7,44 @@ A rust kubernetes reference controller for a [`Document` resource](https://githu
 
 The `Controller` object reconciles `Document` instances when changes to it are detected, writes to its `.status` object, creates associated events, and uses finalizers for guaranteed delete handling.
 
-## Requirements
-- A Kubernetes cluster / k3d instance
-- The [CRD](yaml/crd.yaml)
-- Opentelemetry collector (**optional**)
+## Installation
 
-### Cluster
-As an example; get `k3d` then:
+### CRD
+Apply the CRD from [cached file](yaml/crd.yaml), or pipe it from `crdgen` to pickup schema changes:
 
 ```sh
-k3d cluster create --registry-create --servers 1 --agents 1 main
-k3d kubeconfig get --all > ~/.kube/k3d
-export KUBECONFIG="$HOME/.kube/k3d"
+cargo run --bin crdgen | kubectl apply -f -
 ```
 
-A default `k3d` setup is fastest for local dev due to its local registry.
+### Controller
 
-### CRD
-Apply the CRD from [cached file](yaml/crd.yaml), or pipe it from `crdgen` (best if changing it):
+Install the controller via `helm` by setting your preferred settings. For defaults:
 
 ```sh
-cargo run --bin crdgen | kubectl apply -f -
+helm template charts/doc-controller | kubectl apply -f -
+kubectl wait --for=condition=available deploy/doc-controller --timeout=30s
+kubectl port-forward service/doc-controller 8080:80
 ```
 
 ### Opentelemetry
-Setup an opentelemetry collector in your cluster. [Tempo](https://github.com/grafana/helm-charts/tree/main/charts/tempo) / [opentelemetry-operator](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-operator) / [grafana agent](https://github.com/grafana/helm-charts/tree/main/charts/agent-operator) should all work out of the box. If your collector does not support grpc otlp you need to change the exporter in [`main.rs`](./src/main.rs).
 
-If you don't have a collector, you can build locally without the `telemetry` feature (`tilt up telemetry`), or pull images [without the `otel` tag](https://hub.docker.com/r/clux/controller/tags/).
+Build and run with `telemetry` feature, or configure it via `helm`:
+
+```sh
+helm template charts/doc-controller --set tracing.enabled=true | kubectl apply -f -
+```
+
+This requires an opentelemetry collector in your cluster. [Tempo](https://github.com/grafana/helm-charts/tree/main/charts/tempo) / [opentelemetry-operator](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-operator) / [grafana agent](https://github.com/grafana/helm-charts/tree/main/charts/agent-operator) should all work out of the box. If your collector does not support grpc otlp you need to change the exporter in [`telemetry.rs`](./src/telemetry.rs).
+
+Note that the [images are pushed either with or without the telemetry feature](https://hub.docker.com/r/clux/controller/tags/) depending on whether the tag includes `otel`.
+
+### Metrics
+
+Metrics is available on `/metrics` and a `ServiceMonitor` is configurable from the chart:
+
+```sh
+helm template charts/doc-controller --set serviceMonitor.enabled=true | kubectl apply -f -
+```
 
 ## Running
 
@@ -43,22 +54,16 @@ If you don't have a collector, you can build locally without the `telemetry` fea
 cargo run
 ```
 
-or, with optional telemetry (change as per requirements):
+or, with optional telemetry:
 
 ```sh
 OPENTELEMETRY_ENDPOINT_URL=https://0.0.0.0:55680 RUST_LOG=info,kube=trace,controller=debug cargo run --features=telemetry
 ```
 
 ### In-cluster
-Use either your locally built image or the one from dockerhub (using opentelemetry features by default). Edit the [deployment](./yaml/deployment.yaml)'s image tag appropriately, and then:
-
-```sh
-kubectl apply -f yaml/deployment.yaml
-kubectl wait --for=condition=available deploy/doc-controller --timeout=20s
-kubectl port-forward service/doc-controller 8080:80
-```
+For prebuilt, edit the [chart values](./charts/doc-controller/values.yaml) or [snapshotted yaml](./yaml/deployment.yaml) and apply as you see fit (like above).
 
-To build and deploy the image quickly, we recommend using [tilt](https://tilt.dev/), via `tilt up` instead.
+To develop by building and deploying the image quickly, we recommend using [tilt](https://tilt.dev/), via `tilt up` instead.
 
 ## Usage
 In either of the run scenarios, your app is listening on port `8080`, and it will observe `Document` events.
@@ -102,7 +107,7 @@ $ curl 0.0.0.0:8080/
 {"last_event":"2019-07-17T22:31:37.591320068Z"}
 ```
 
-The metrics will be scraped by prometheus if you setup a `PodMonitor` or `ServiceMonitor` for it.
+The metrics will be scraped by prometheus if you setup a`ServiceMonitor` for it.
 
 ### Events
 The example `reconciler` only checks the `.spec.hidden` bool. If it does, it updates the `.status` object to reflect whether or not the instance `is_hidden`. It also sends a Kubernetes event associated with the controller. It is visible at the bottom of `kubectl describe doc samuel`.

charts/doc-controller/templates/_helpers.tpl

@@ -17,3 +17,10 @@ app.kubernetes.io/version: {{ .Values.image.tag | default .Chart.AppVersion | qu
 app: {{ include "controller.name" . }}
 {{- end }}
 
+{{- define "controller.tag" -}}
+{{- if .Values.tracing.enabled }}
+{{- "otel-" }}{{ .Values.version | default .Chart.AppVersion }}
+{{- else }}
+{{- .Values.version | default .Chart.AppVersion }}
+{{- end }}
+{{- end }}

charts/doc-controller/templates/deployment.yaml

@@ -30,7 +30,7 @@ spec:
         {{- toYaml .Values.podSecurityContext | nindent 8 }}
       containers:
       - name: {{ .Chart.Name }}
-        image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+        image: {{ .Values.image.repository }}:{{ include "controller.tag" . }}
         imagePullPolicy: {{ .Values.image.pullPolicy }}
         securityContext:
           {{- toYaml .Values.securityContext | nindent 10 }}
@@ -41,11 +41,15 @@ spec:
           containerPort: 8080
           protocol: TCP
         env:
-        # We are pointing to tempo or grafana tracing agent's otlp grpc receiver port
-        - name: OPENTELEMETRY_ENDPOINT_URL
-          value: "https://promstack-tempo.monitoring.svc.cluster.local:4317"
         - name: RUST_LOG
-          value: "info,kube=debug,controller=debug"
+          value: {{ .Values.logging.env_filter }}
+        {{- if .Values.tracing.enabled }}
+        - name: OPENTELEMETRY_ENDPOINT_URL
+          value: {{ .Values.tracing.endpoint }}
+        {{- end }}
+        {{- with .Values.env }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
         readinessProbe:
           httpGet:
             path: /health

charts/doc-controller/templates/servicemonitor.yaml

@@ -0,0 +1,45 @@
+{{- if .Values.serviceMonitor.enabled }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: {{ include "controller.fullname" . }}
+  namespace: {{ .Values.namespace }}
+  labels:
+    {{- include "controller.labels" . | nindent 4 }}
+  {{- with .Values.service.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  endpoints:
+  - port: http
+    {{- with .Values.serviceMonitor.interval }}
+    interval: {{ . }}
+    {{- end }}
+    {{- with .Values.serviceMonitor.scrapeTimeout }}
+    scrapeTimeout: {{ . }}
+    {{- end }}
+    honorLabels: true
+    path: {{ .Values.serviceMonitor.path }}
+    scheme: {{ .Values.serviceMonitor.scheme }}
+    {{- with .Values.serviceMonitor.relabelings }}
+    relabelings:
+      {{- toYaml . | nindent 6 }}
+    {{- end }}
+    {{- with .Values.serviceMonitor.metricRelabelings }}
+    metricRelabelings:
+      {{- toYaml . | nindent 6 }}
+    {{- end }}
+  jobLabel: {{ include "controller.fullname" . }}
+  selector:
+    matchLabels:
+      app: {{ include "controller.fullname" . }}
+  namespaceSelector:
+    matchNames:
+      - {{ .Values.namespace }}
+  {{- with .Values.serviceMonitor.targetLabels }}
+  targetLabels:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+{{- end }}

charts/doc-controller/values.yaml

@@ -1,19 +1,17 @@
 replicaCount: 1
 nameOverride: ""
 namespace: "default"
+version: "" # pin a specific version
 
 image:
   repository: clux/controller
   pullPolicy: IfNotPresent
-  tag: ""
 
 imagePullSecrets: []
 
 serviceAccount:
   annotations: {}
 podAnnotations: {}
-  # prometheus.io/scrape: "true"
-  # prometheus.io/port: "8080"
 
 podSecurityContext: {}
   # fsGroup: 2000
@@ -25,6 +23,16 @@ securityContext: {}
   # runAsNonRoot: true
   # runAsUser: 1000
 
+# Enable the feature-flagged opentelemetry trace layer pushing over grpc
+tracing:
+  enabled: false # prefixes tag with otel
+  endpoint: "https://promstack-tempo.monitoring.svc.cluster.local:4317"
+
+logging:
+  env_filter: info,kube=debug,controller=debug
+
+env: []
+
 service:
   type: ClusterIP
   port: 80
@@ -37,6 +45,7 @@ resources:
     cpu: 50m
     memory: 100Mi
 
-# TODO: evar option for otel
-# TODO: how to select between otel and non otel?
-# TODO: metrics scraping?
+serviceMonitor:
+  enabled: false
+  path: /metrics
+  scheme: http

yaml/deployment.yaml

@@ -89,7 +89,7 @@ spec:
         {}
       containers:
       - name: doc-controller
-        image: "clux/controller:0.12.5"
+        image: clux/controller:0.12.5
         imagePullPolicy: IfNotPresent
         securityContext:
           {}
@@ -105,11 +105,8 @@ spec:
           containerPort: 8080
           protocol: TCP
         env:
-        # We are pointing to tempo or grafana tracing agent's otlp grpc receiver port
-        - name: OPENTELEMETRY_ENDPOINT_URL
-          value: "https://promstack-tempo.monitoring.svc.cluster.local:4317"
         - name: RUST_LOG
-          value: "info,kube=debug,controller=debug"
+          value: info,kube=debug,controller=debug
         readinessProbe:
           httpGet:
             path: /health