Skip to content

Commit d0c162e

Browse files
fabxcfabxc
authored
Improve Prometheus setup, add injection scripts (#57)
* Improve Prometheus setup, add injection scripts Signed-off-by: Fabian Reinartz <freinartz@google.com> * Improve docs, rename Prometheus Signed-off-by: Fabian Reinartz <freinartz@google.com>
1 parent 20f76ff commit d0c162e

File tree

10 files changed

+363
-253
lines changed

10 files changed

+363
-253
lines changed

‎kube/README.md‎

Lines changed: 32 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,39 @@
1-
# Kubernetes Test Setup
1+
# Kubernetes setup
22

3-
This directory contains files to deploy Prometheus with the sidecar in a Kubernetes
4-
cluster. Additional manifests deploy the Prometheus node exporter and kube-state-metrics,
5-
which provide a further variety of metrics.
3+
This directory contains patch scripts to inject the Prometheus sidecar into
4+
existing Prometheus installations and to deploy a full example setup.
65

7-
To deploy all components:
6+
Required environment variables:
7+
* `KUBE_NAMESPACE`: namespace to run the script against
8+
* `KUBE_CLUSTER`: cluster name parameter for the sidecar
9+
* `GCP_REGION`: GCP region parameter for the sidecar
10+
* `GCP_PROJECT`: GCP project parameter for the sidecar
811

9-
`KUBE_NAMESPACE=sidecar-test GCP_REGION=your_region GCP_PROJECT=your_project_id KUBE_CLUSTER=clustername ./deploy.sh`
12+
## `patch.sh`
1013

11-
Setting `USE_OPERATOR=1` will deploy Prometheus via the [coreos/prometheus-operator](https://github.com/coreos/prometheus-operator).
14+
Inject sidecar into Deployments or StatefulSets:
1215

13-
To tear down everything:
16+
```sh
17+
./patch.sh <deployment|statefulset> <name>
18+
```
1419

15-
`kubectl delete namespace "${KUBE_NAMESPACE}"`
20+
Additional environment variables:
21+
* `DATA_DIR`: data directory for the sidecar
22+
* `DATA_VOLUME`: name of the volume that contains Prometheus's data
1623

24+
## `patch-operated.sh`
25+
26+
Injects sidecar into Prometheus deployments controlled by the [prometheus-operator](https://github.com/coreos/prometheus-operator):
27+
28+
```sh
29+
./patch-operated.sh <prometheus_name>
30+
```
31+
32+
## `full/deploy.sh`
33+
34+
Deploys a basic Prometheus deployment to monitor Kubernetes components and
35+
custom services that are annotated with the well-known `prometheus.io/*` annotations.
36+
37+
```sh
38+
./full/deploy.sh
39+
```

‎kube/deploy.sh‎ renamed to ‎kube/full/deploy.sh‎

Lines changed: 8 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,29 +3,23 @@
33
set -e
44
set -u
55

6+
pushd "$(dirname "$0")"
7+
68
# Override to use a different Docker image version for the sidecar.
79
export SIDECAR_IMAGE_TAG=${SIDECAR_IMAGE_TAG:-'master'}
8-
export USE_OPERATOR=${USE_OPERATOR:-''}
910
export KUBE_NAMESPACE=${KUBE_NAMESPACE:-'default'}
1011

11-
echo "Deploy to namespace ${KUBE_NAMESPACE} for Stackdriver project ${GCP_PROJECT} (location=${GCP_REGION}, cluster=${KUBE_CLUSTER}), operator=${USE_OPERATOR}"
12+
echo "Deploy to namespace ${KUBE_NAMESPACE} for Stackdriver project ${GCP_PROJECT} (location=${GCP_REGION}, cluster=${KUBE_CLUSTER})"
1213

13-
envsubst < prometheus-base.yaml > _prometheus-base.yaml.tmp
14-
envsubst < prometheus-meta-operated.yaml > _prometheus-meta-operated.yaml.tmp
15-
envsubst < prometheus-meta.yaml > _prometheus-meta.yaml.tmp
14+
envsubst < prometheus.yaml > _prometheus.yaml.tmp
1615
envsubst < node-exporter.yaml > _node-exporter.yaml.tmp
1716
envsubst < kube-state-metrics.yaml > _kube-state-metrics.yaml.tmp
1817

19-
kubectl apply -f _prometheus-base.yaml.tmp --as=admin --as-group=system:masters
20-
21-
if [ -n "${USE_OPERATOR}" ]; then
22-
kubectl apply -f _prometheus-meta-operated.yaml.tmp
23-
else
24-
kubectl apply -f _prometheus-meta.yaml.tmp
25-
fi
26-
18+
kubectl apply -f _prometheus.yaml.tmp
2719
kubectl apply -f _node-exporter.yaml.tmp
2820
kubectl apply -f _kube-state-metrics.yaml.tmp --as=admin --as-group=system:masters
2921

30-
rm _*.tmp
22+
DATA_DIR=/data DATA_VOLUME=data-volume ../patch.sh deploy prometheus-meta
3123

24+
rm _*.tmp
25+
popd

‎kube/kube-state-metrics.yaml‎ renamed to ‎kube/full/kube-state-metrics.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,10 @@ kind: Service
124124
metadata:
125125
labels:
126126
k8s-app: kube-state-metrics
127+
annotations:
128+
prometheus.io/scrape: 'true'
129+
prometheus.io/port: '8081'
130+
prometheus.io/port2: '8082'
127131
name: kube-state-metrics
128132
namespace: ${KUBE_NAMESPACE}
129133
spec:

‎kube/node-exporter.yaml‎ renamed to ‎kube/full/node-exporter.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,10 @@ kind: Service
5656
metadata:
5757
labels:
5858
app: node-exporter
59+
annotations:
60+
a: b
61+
prometheus.io/scrape: 'true'
62+
prometheus.io/port: '9101'
5963
name: node-exporter
6064
namespace: ${KUBE_NAMESPACE}
6165
spec:

‎kube/full/prometheus.yaml‎

Lines changed: 251 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,251 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: ${KUBE_NAMESPACE}
5+
---
6+
apiVersion: rbac.authorization.k8s.io/v1beta1
7+
kind: ClusterRole
8+
metadata:
9+
name: prometheus
10+
rules:
11+
- apiGroups: [""]
12+
resources:
13+
- nodes
14+
- nodes/proxy
15+
- services
16+
- endpoints
17+
- pods
18+
verbs: ["get", "list", "watch"]
19+
- apiGroups:
20+
- extensions
21+
resources:
22+
- ingresses
23+
verbs: ["get", "list", "watch"]
24+
- nonResourceURLs: ["/metrics"]
25+
verbs: ["get"]
26+
---
27+
apiVersion: v1
28+
kind: ServiceAccount
29+
metadata:
30+
name: prometheus
31+
namespace: ${KUBE_NAMESPACE}
32+
---
33+
apiVersion: rbac.authorization.k8s.io/v1beta1
34+
kind: ClusterRoleBinding
35+
metadata:
36+
name: prometheus
37+
roleRef:
38+
apiGroup: rbac.authorization.k8s.io
39+
kind: ClusterRole
40+
name: prometheus
41+
subjects:
42+
- kind: ServiceAccount
43+
name: prometheus
44+
namespace: ${KUBE_NAMESPACE}
45+
---
46+
apiVersion: v1
47+
kind: Service
48+
metadata:
49+
labels:
50+
app: prometheus-k8s
51+
annotations:
52+
prometheus.io/scrape: 'true'
53+
prometheus.io/port: '9090'
54+
prometheus.io/port2: '9091'
55+
name: prometheus-k8s
56+
namespace: ${KUBE_NAMESPACE}
57+
spec:
58+
type: LoadBalancer
59+
externalTrafficPolicy: Cluster
60+
ports:
61+
- name: prometheus
62+
nodePort: 32387
63+
port: 9090
64+
protocol: TCP
65+
targetPort: 9090
66+
- name: sidecar
67+
nodePort: 30182
68+
port: 9091
69+
protocol: TCP
70+
targetPort: 9091
71+
selector:
72+
app: prometheus
73+
prometheus: k8s
74+
sessionAffinity: None
75+
---
76+
apiVersion: apps/v1beta2
77+
kind: Deployment
78+
metadata:
79+
name: prometheus-k8s
80+
namespace: ${KUBE_NAMESPACE}
81+
labels:
82+
app: prometheus
83+
prometheus: k8s
84+
spec:
85+
replicas: 1
86+
selector:
87+
matchLabels:
88+
app: prometheus
89+
prometheus: k8s
90+
template:
91+
metadata:
92+
labels:
93+
app: prometheus
94+
prometheus: k8s
95+
spec:
96+
serviceAccount: prometheus
97+
securityContext:
98+
runAsUser: 0
99+
containers:
100+
- name: prometheus
101+
image: quay.io/prometheus/prometheus:v2.4.3
102+
imagePullPolicy: Always
103+
args:
104+
- "--config.file=/etc/prometheus/config/prometheus.yaml"
105+
- "--storage.tsdb.path=/data"
106+
- "--storage.tsdb.min-block-duration=15m"
107+
- "--storage.tsdb.max-block-duration=4h"
108+
- "--storage.tsdb.retention=48h"
109+
ports:
110+
- name: prometheus
111+
containerPort: 9090
112+
volumeMounts:
113+
- name: config-volume
114+
mountPath: /etc/prometheus/config
115+
- name: data-volume
116+
mountPath: /data
117+
volumes:
118+
- name: config-volume
119+
configMap:
120+
name: prometheus-k8s
121+
- name: data-volume
122+
emptyDir: {}
123+
terminationGracePeriodSeconds: 300
124+
---
125+
apiVersion: v1
126+
kind: ConfigMap
127+
metadata:
128+
name: prometheus-k8s
129+
namespace: ${KUBE_NAMESPACE}
130+
data:
131+
prometheus.yaml: |
132+
scrape_configs:
133+
- job_name: kubernetes-apiservers
134+
kubernetes_sd_configs:
135+
- role: endpoints
136+
relabel_configs:
137+
- action: keep
138+
regex: default;kubernetes;https
139+
source_labels:
140+
- __meta_kubernetes_namespace
141+
- __meta_kubernetes_service_name
142+
- __meta_kubernetes_endpoint_port_name
143+
scheme: https
144+
tls_config:
145+
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
146+
insecure_skip_verify: true
147+
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
148+
149+
- job_name: kubernetes-nodes-kubelet
150+
kubernetes_sd_configs:
151+
- role: node
152+
relabel_configs:
153+
- target_label: __address__
154+
regex: "(.+):10250"
155+
source_labels: [__address__]
156+
replacement: "${1}:10255"
157+
158+
- job_name: kubernetes-nodes-cadvisor
159+
kubernetes_sd_configs:
160+
- role: node
161+
relabel_configs:
162+
- target_label: __metrics_path__
163+
replacement: /metrics/cadvisor
164+
- target_label: __address__
165+
regex: "(.+):10250"
166+
source_labels: [__address__]
167+
replacement: "${1}:10255"
168+
169+
# Configuration for the first port (prometheus.io/port) that service
170+
# endpoints are annotated with.
171+
- job_name: kubernetes-service-endpoints1
172+
kubernetes_sd_configs:
173+
- role: endpoints
174+
relabel_configs:
175+
- action: keep
176+
regex: true
177+
source_labels:
178+
- __meta_kubernetes_service_annotation_prometheus_io_scrape
179+
- action: replace
180+
regex: (https?)
181+
source_labels:
182+
- __meta_kubernetes_service_annotation_prometheus_io_scheme
183+
target_label: __scheme__
184+
- action: replace
185+
regex: (.+)
186+
source_labels:
187+
- __meta_kubernetes_service_annotation_prometheus_io_path
188+
target_label: __metrics_path__
189+
- action: replace
190+
regex: ([^:]+)(?::\d+)?;(\d+)
191+
replacement: $1:$2
192+
source_labels:
193+
- __address__
194+
- __meta_kubernetes_service_annotation_prometheus_io_port
195+
target_label: __address__
196+
- action: replace
197+
source_labels:
198+
- __meta_kubernetes_namespace
199+
target_label: k8s_namespace
200+
- action: replace
201+
source_labels:
202+
- __meta_kubernetes_service_name
203+
target_label: k8s_service
204+
- action: replace
205+
source_labels:
206+
- __meta_kubernetes_pod_name
207+
target_label: k8s_pod
208+
209+
# Configuration for the seocnd port (prometheus.io/port2) that service
210+
# endpoints are annotated with.
211+
- job_name: kubernetes-service-endpoints2
212+
kubernetes_sd_configs:
213+
- role: endpoints
214+
relabel_configs:
215+
- action: keep
216+
regex: true
217+
source_labels:
218+
- __meta_kubernetes_service_annotation_prometheus_io_scrape
219+
- action: replace
220+
regex: (https?)
221+
source_labels:
222+
- __meta_kubernetes_service_annotation_prometheus_io_scheme
223+
target_label: __scheme__
224+
- action: replace
225+
regex: (.+)
226+
source_labels:
227+
- __meta_kubernetes_service_annotation_prometheus_io_path
228+
target_label: __metrics_path__
229+
- action: keep
230+
source_labels:
231+
- __meta_kubernetes_service_annotation_prometheus_io_port2
232+
regex: .+
233+
- action: replace
234+
regex: ([^:]+)(?::\d+)?;(\d+)
235+
replacement: $1:$2
236+
source_labels:
237+
- __address__
238+
- __meta_kubernetes_service_annotation_prometheus_io_port2
239+
target_label: __address__
240+
- action: replace
241+
source_labels:
242+
- __meta_kubernetes_namespace
243+
target_label: k8s_namespace
244+
- action: replace
245+
source_labels:
246+
- __meta_kubernetes_service_name
247+
target_label: k8s_service
248+
- action: replace
249+
source_labels:
250+
- __meta_kubernetes_pod_name
251+
target_label: k8s_pod

‎kube/patch-operated.sh‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/sh
2+
3+
set -e
4+
set -u
5+
6+
if [ $# -le 1 ]; then
7+
echo -e "Usage: $0 <prometheus_name>\n"
8+
exit 1
9+
fi
10+
11+
kubectl -n "${KUBE_NAMESPACE}" patch prometheus "$1" --type merge --patch "
12+
spec:
13+
containers:
14+
- name: sidecar
15+
image: gcr.io/prometheus-to-sd/stackdriver-prometheus-sidecar:${SIDECAR_IMAGE_TAG}
16+
imagePullPolicy: Always
17+
args:
18+
- \"--stackdriver.project-id=${GCP_PROJECT}\"
19+
- \"--prometheus.wal-directory=/data/wal\"
20+
- \"--stackdriver.kubernetes.location=${GCP_REGION}\"
21+
- \"--stackdriver.kubernetes.cluster-name=${KUBE_CLUSTER}\"
22+
ports:
23+
- name: sidecar
24+
containerPort: 9091
25+
volumeMounts:
26+
- mountPath: /data
27+
name: prometheus-$1-db
28+
"

0 commit comments

Comments
 (0)