Merge pull request #25 from kube-logging/readme

kristofgyuracz · web-flow · commit 5dfb0b0e1cce · 2024-02-05T16:26:41.000+01:00
Add readme with example deployment
diff --git a/.github/release.yml b/.github/release.yml
@@ -0,0 +1,30 @@
+# https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes
+changelog:
+  exclude:
+    labels:
+      - ignore-for-release-note
+  categories:
+    - title: Breaking Changes
+      labels:
+        - breaking-change
+    - title: New Features
+      labels:
+        - feature
+    - title: Enhancements
+      labels:
+        - enhancement
+    - title: Dependency and image updates
+      labels:
+        - dependencies
+    - title: Bug fixes
+      labels:
+        - bugfix
+    - title: Documentation
+      labels:
+        - documentation
+    - title: Testing
+      labels:
+        - testing
+    - title: Other Changes
+      labels:
+        - "*"
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -0,0 +1,21 @@
+name: Release
+
+on:
+  push:
+    tags: ["[0-9]+.[0-9]+.[0-9]+*"]
+
+permissions:
+  contents: read
+
+jobs:
+  artifacts:
+    name: Artifacts
+    uses: ./.github/workflows/artifacts.yaml
+    with:
+      publish: true
+      release: true
+    permissions:
+      contents: read
+      packages: write
+      id-token: write
+      security-events: write
diff --git a/README.md b/README.md
@@ -1,9 +1,7 @@
 # telemetry-controller
-// TODO(user): Add simple overview of use/purpose
-
+The Telemetry Controller is a multi-tenancy focused solution, that facilitates collection of telemetry data from Kubernetes workloads, without any need for changes to the running software.
 ## Description
-// TODO(user): An in-depth paragraph about your project and overview of use
-
+Telemetry-controller can be configured using Custom Resources to set up an opinionated Opentelemetry Collector configuration to route log messages based on rules defined as a Tenant -> Subscription relation map.
 ## Getting Started
 
 ### Prerequisites
@@ -13,51 +11,72 @@
 - Access to a Kubernetes v1.11.3+ cluster.
 
 ### To Deploy on the cluster
-**Build and push your image to the location specified by `IMG`:**
 
+**Install cert-manager, and opentelemtry-operator:**
 ```sh
-make docker-build docker-push IMG=<some-registry>/telemetry-controller:tag
-```
+helm upgrade --install --repo https://charts.jetstack.io cert-manager cert-manager --namespace cert-manager --create-namespace --version v1.13.3 --set installCRDs=true --wait
 
-**NOTE:** This image ought to be published in the personal registry you specified. 
-And it is required to have access to pull the image from the working environment. 
-Make sure you have the proper permission to the registry if the above commands don’t work.
+kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml --wait
+```
 
 **Install the CRDs into the cluster:**
 
 ```sh
 make install
 ```
 
+**Build your image and load it to your KinD nodes**
+```sh
+make docker-build IMG=telemetry-controller:latest
+kind load docker-image telemetry-controller:latest
+```
+
 **Deploy the Manager to the cluster with the image specified by `IMG`:**
 
 ```sh
-make deploy IMG=<some-registry>/telemetry-controller:tag
+make deploy IMG=telemetry-controller:tag
 ```
 
 > **NOTE**: If you encounter RBAC errors, you may need to grant yourself cluster-admin 
 privileges or be logged in as admin.
 
 **Create instances of your solution**
-You can apply the samples (examples) from the config/sample:
+You can deploy the example configuration provided as part of the docs. This will deploy a demo pipeline with one tenant, two subscriptions, and an OpenObserve instance.
+Deploying Openobserve is an optional, but recommended step, logs can be forwarded to any OTLP endpoint. Openobserve provides a UI to visualize the ingested logstream.
 
 ```sh
-kubectl apply -k config/samples/
+# Deploy Openobserve
+kubectl apply -f docs/examples/simple-demo/openobserve.yaml
+
+# Set up portforwarding for Openobserve UI
+kubectl -n openobserve port-forward svc/openobserve 5080:5080 &
 ```
 
->**NOTE**: Ensure that the samples has default values to test it out.
+Open the UI at `localhost:5080`, navigate to the `Ingestion/OTEL Collector` tab, and copy the authorization token as seen on the screenshot.
+![Openobserve auth](docs/assets/openobserve-auth.png)
 
-### To Uninstall
-**Delete the instances (CRs) from the cluster:**
+Paste this token to the example manifests:
+```sh
+sed -i '' -e "s/\<TOKEN\>/INSERT YOUR COPIED TOKEN HERE/" docs/examples/simple-demo/one_tenant_two_subscriptions.yaml
+```
+```sh
+# Deploy the pipeline definition
+kubectl apply -f docs/examples/simple-demo/one_tenant_two_subscriptions.yaml
+```
 
+**Create a workload, which will generate logs for the pipeline:**
 ```sh
-kubectl delete -k config/samples/
+helm install --wait --create-namespace --namespace example-tenant-ns --generate-name oci://ghcr.io/kube-logging/helm-charts/log-generator
 ```
 
-**Delete the APIs(CRDs) from the cluster:**
+**Open the Openobserve UI and inspect the generated log messages**
+![Openobserve logs](docs/assets/openobserve-logs.png)
+
+### To Uninstall
+**Delete the instances (CRs) from the cluster:**
 
 ```sh
-make uninstall
+kubectl delete -f docs/examples/simple-demo/
 ```
 
 **UnDeploy the controller from the cluster:**
@@ -66,16 +85,27 @@ make uninstall
 make undeploy
 ```
 
+**Delete the APIs(CRDs) from the cluster:**
+
+```sh
+make uninstall
+```
+
 ## Contributing
-// TODO(user): Add detailed information on how you would like others to contribute to this project
 
-**NOTE:** Run `make --help` for more information on all potential `make` targets
+If you find this project useful, help us:
+
+- Support the development of this project and star this repo! :star:
+- Help new users with issues they may encounter :muscle:
+- Send a pull request with your new features and bug fixes :rocket: 
+
+Please read the [Organisation's Code of Conduct](https://github.com/kube-logging/.github/blob/main/CODE_OF_CONDUCT.md)!
 
-More information can be found via the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html)
+*For more information, read our organization's [contribution guidelines](https://github.com/kube-logging/.github/blob/main/CONTRIBUTING.md)*.
 
 ## License
 
-Copyright © 2023 Kube logging authors
+Copyright © 2024 Kube logging authors
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
diff --git a/config/manager/kustomization.yaml b/config/manager/kustomization.yaml
@@ -4,5 +4,5 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 images:
 - name: controller
-  newName: controller
+  newName: ghcr.io/kube-logging/telemetry-controller
   newTag: latest
diff --git a/docs/assets/openobserve-auth.png b/docs/assets/openobserve-auth.png
diff --git a/docs/assets/openobserve-logs.png b/docs/assets/openobserve-logs.png
diff --git a/docs/demos/openobserve/start.sh b/docs/demos/openobserve/start.sh
@@ -5,10 +5,13 @@ set -euo pipefail
 KIND_CLUSTER_NAME=${KIND_CLUSTER_NAME:-kind}
 
 # Install OpenObserve
-kubectl create namespace openobserve
-
 kubectl apply -f - <<EOF
 apiVersion: v1
+kind: Namespace
+metadata:
+  name: openobserve
+---
+apiVersion: v1
 kind: Service
 metadata:
   name: openobserve
diff --git a/docs/examples/simple-demo/one_tenant_two_subscriptions.yaml b/docs/examples/simple-demo/one_tenant_two_subscriptions.yaml
@@ -40,9 +40,9 @@ metadata:
   name: subscription-sample-1
   namespace: example-tenant-ns
 spec:
-  ottl: 'route()'
+  ottl: "route()"
   outputs:
-    - name: otlp-test-output
+    - name: otlp-test-output-1
       namespace: collector
 ---
 apiVersion: telemetry.kube-logging.dev/v1alpha1
@@ -51,19 +51,23 @@ metadata:
   name: subscription-sample-2
   namespace: example-tenant-ns
 spec:
-  ottl: 'route()'
+  ottl: "route()"
   outputs:
     - name: otlp-test-output-2
       namespace: collector
 ---
 apiVersion: telemetry.kube-logging.dev/v1alpha1
 kind: OtelOutput
 metadata:
-  name: otlp-test-output
+  name: otlp-test-output-1
   namespace: collector
 spec:
   otlp:
-    endpoint: receiver-collector.example-tenant-ns.svc.cluster.local:4317
+    endpoint: openobserve-otlp-grpc.openobserve.svc.cluster.local:5081
+    headers:
+      Authorization: "Basic <TOKEN>"
+      organization: default
+      stream-name: default
     tls:
       insecure: true
 ---
@@ -74,6 +78,10 @@ metadata:
   namespace: collector
 spec:
   otlp:
-    endpoint: receiver-collector.example-tenant-ns.svc.cluster.local:4317
+    endpoint: openobserve-otlp-grpc.openobserve.svc.cluster.local:5081
+    headers:
+      Authorization: "Basic <TOKEN>"
+      organization: default
+      stream-name: default
     tls:
       insecure: true
diff --git a/docs/examples/simple-demo/openobserve.yaml b/docs/examples/simple-demo/openobserve.yaml
@@ -0,0 +1,98 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: openobserve
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: openobserve
+  namespace: openobserve
+spec:
+  clusterIP: None
+  selector:
+    app: openobserve
+  ports:
+    - name: http
+      port: 5080
+      targetPort: 5080
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: openobserve-otlp-grpc
+  namespace: openobserve
+spec:
+  clusterIP: None
+  selector:
+    app: openobserve
+  ports:
+    - name: otlp-grpc
+      port: 5081
+      targetPort: 5081
+---
+# create statefulset
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: openobserve
+  namespace: openobserve
+  labels:
+    name: openobserve
+spec:
+  serviceName: openobserve
+  replicas: 1
+  selector:
+    matchLabels:
+      name: openobserve
+      app: openobserve
+  template:
+    metadata:
+      labels:
+        name: openobserve
+        app: openobserve
+    spec:
+      securityContext:
+        fsGroup: 2000
+        runAsUser: 10000
+        runAsGroup: 3000
+        runAsNonRoot: true
+      # terminationGracePeriodSeconds: 0
+      containers:
+        - name: openobserve
+          image: public.ecr.aws/zinclabs/openobserve:v0.7.2
+          env:
+            - name: ZO_ROOT_USER_EMAIL
+              value: root@example.com
+            - name: ZO_ROOT_USER_PASSWORD
+              value: Complexpass#123
+            - name: ZO_DATA_DIR
+              value: /data
+          # command: ["/bin/bash", "-c", "while true; do sleep 1; done"]
+          imagePullPolicy: Always
+          resources:
+            limits:
+              cpu: 4096m
+              memory: 2048Mi
+            requests:
+              cpu: 256m
+              memory: 50Mi
+          ports:
+            - containerPort: 5080
+              name: http
+            - containerPort: 50801
+              name: otlp-grpc
+          volumeMounts:
+            - name: data
+              mountPath: /data
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+      spec:
+        accessModes:
+          - ReadWriteOnce
+        # storageClassName: default
+        # NOTE: You can increase the storage size
+        resources:
+          requests:
+            storage: 10Gi
diff --git a/e2e/e2e_test.sh b/e2e/e2e_test.sh
@@ -38,7 +38,7 @@ kubectl wait --namespace opentelemetry-operator-system --for=condition=available
 (cd .. && make manifests generate install)
 
 # Use example
-kubectl apply -f ../docs/examples/simple-demo
+kubectl apply -f ../e2e/testdata/one_tenant_two_subscriptions
 
 if [[ -z "${CI_MODE}" ]]; then
   $(cd .. && timeout 5m make run &)
diff --git a/e2e/testdata/one_tenant_two_subscriptions/one_tenant_two_subscriptions.yaml b/e2e/testdata/one_tenant_two_subscriptions/one_tenant_two_subscriptions.yaml
diff --git a/e2e/testdata/one_tenant_two_subscriptions/receiver.yaml b/e2e/testdata/one_tenant_two_subscriptions/receiver.yaml
