Skip to content

Commit 974b8e9

Browse files
max-cxmax-cx
authored
Merge pull request #101709 from max-cx/OBSDOCS-2519
OBSDOCS-2519: Modularize otel-collector-processors.adoc
2 parents a1d0e34 + 61aeb13 commit 974b8e9

14 files changed

+942
-869
lines changed

‎modules/otel-processors-attributes-processor.adoc‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-collector/otel-collector-processors.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="otel-processors-attributes-processor_{context}"]
7+
= Attributes Processor
8+
9+
[role="_abstract"]
10+
The Attributes Processor can modify attributes of a span, log, or metric. You can configure this processor to filter and match input data and include or exclude such data for specific actions.
11+
12+
This processor operates on a list of actions, executing them in the order specified in the configuration. The following actions are supported:
13+
14+
Insert:: Inserts a new attribute into the input data when the specified key does not already exist.
15+
16+
Update:: Updates an attribute in the input data if the key already exists.
17+
18+
Upsert:: Combines the insert and update actions: Inserts a new attribute if the key does not exist yet. Updates the attribute if the key already exists.
19+
20+
Delete:: Removes an attribute from the input data.
21+
22+
Hash:: Hashes an existing attribute value as SHA1.
23+
24+
Extract:: Extracts values by using a regular expression rule from the input key to the target keys defined in the rule. If a target key already exists, it is overridden similarly to the Span Processor's `to_attributes` setting with the existing attribute as the source.
25+
26+
Convert:: Converts an existing attribute to a specified type.
27+
28+
.OpenTelemetry Collector using the Attributes Processor
29+
[source,yaml]
30+
----
31+
# ...
32+
config:
33+
processors:
34+
attributes/example:
35+
actions:
36+
- key: db.table
37+
action: delete
38+
- key: redacted_span
39+
value: true
40+
action: upsert
41+
- key: copy_key
42+
from_attribute: key_original
43+
action: update
44+
- key: account_id
45+
value: 2245
46+
action: insert
47+
- key: account_password
48+
action: delete
49+
- key: account_email
50+
action: hash
51+
- key: http.status_code
52+
action: convert
53+
converted_type: int
54+
# ...
55+
----

‎modules/otel-processors-batch-processor.adoc‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-collector/otel-collector-processors.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="otel-processors-batch-processor_{context}"]
7+
= Batch Processor
8+
9+
[role="_abstract"]
10+
The Batch Processor batches traces and metrics to reduce the number of outgoing connections needed to transfer the telemetry information.
11+
12+
.Example of the OpenTelemetry Collector custom resource when using the Batch Processor
13+
[source,yaml]
14+
----
15+
# ...
16+
config:
17+
processors:
18+
batch:
19+
timeout: 5s
20+
send_batch_max_size: 10000
21+
service:
22+
pipelines:
23+
traces:
24+
processors: [batch]
25+
metrics:
26+
processors: [batch]
27+
# ...
28+
----
29+
30+
.Parameters used by the Batch Processor
31+
[options="header"]
32+
[cols="a,a,a"]
33+
|===
34+
|Parameter |Description |Default
35+
36+
|`timeout`
37+
|Sends the batch after a specific time duration and irrespective of the batch size.
38+
|`200ms`
39+
40+
|`send_batch_size`
41+
|Sends the batch of telemetry data after the specified number of spans or metrics.
42+
|`8192`
43+
44+
|`send_batch_max_size`
45+
|The maximum allowable size of the batch. Must be equal or greater than the `send_batch_size`.
46+
|`0`
47+
48+
|`metadata_keys`
49+
|When activated, a batcher instance is created for each unique set of values found in the `client.Metadata`.
50+
|`[]`
51+
52+
|`metadata_cardinality_limit`
53+
|When the `metadata_keys` are populated, this configuration restricts the number of distinct metadata key-value combinations processed throughout the duration of the process.
54+
|`1000`
55+
|===

‎modules/otel-processors-cumulative-to-delta-processor.adoc‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-collector/otel-collector-processors.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="otel-processors-cumulative-to-delta-processor_{context}"]
7+
= Cumulative-to-Delta Processor
8+
9+
[role="_abstract"]
10+
The Cumulative-to-Delta Processor converts monotonic, cumulative-sum, and histogram metrics to monotonic delta metrics.
11+
12+
You can filter metrics by using the `include:` or `exclude:` fields and specifying the `strict` or `regexp` metric name matching.
13+
14+
Because this processor calculates delta by storing the previous value of a metric, you must set up the metric source to send the metric data to a single stateful Collector instance rather than a deployment of multiple Collectors.
15+
16+
This processor does not convert non-monotonic sums and exponential histograms.
17+
18+
:FeatureName: The Cumulative-to-Delta Processor
19+
include::snippets/technology-preview.adoc[]
20+
21+
.Example of an OpenTelemetry Collector custom resource with an enabled Cumulative-to-Delta Processor
22+
[source,yaml]
23+
----
24+
# ...
25+
mode: sidecar # <1>
26+
config:
27+
processors:
28+
cumulativetodelta:
29+
include: # <2>
30+
match_type: strict # <3>
31+
metrics: # <4>
32+
- <metric_1_name>
33+
- <metric_2_name>
34+
exclude: # <5>
35+
match_type: regexp
36+
metrics:
37+
- "<regular_expression_for_metric_names>"
38+
# ...
39+
----
40+
<1> To tie the Collector's lifecycle to the metric source, you can run the Collector as a sidecar to the application that emits the cumulative temporality metrics.
41+
<2> Optional: You can limit which metrics the processor converts by explicitly defining which metrics you want converted in this stanza. If you omit this field, the processor converts all metrics, except the metrics that are listed in the `exclude` field.
42+
<3> Defines the value that you provided in the `metrics` field as an exact match by using the `strict` parameter or a regular expression by using the `regex` parameter.
43+
<4> Lists the names of the metrics that you want to convert. The processor converts exact matches or matches for regular expressions. If a metric matches both the `include` and `exclude` filters, the `exclude` filter takes precedence.
44+
<5> Optional: You can exclude certain metrics from conversion by explicitly defining them here.

‎modules/otel-processors-filter-processor.adoc‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-collector/otel-collector-processors.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="otel-processors-filter-processor_{context}"]
7+
= Filter Processor
8+
9+
[role="_abstract"]
10+
The Filter Processor leverages the OpenTelemetry Transformation Language to establish criteria for discarding telemetry data. If any of these conditions are satisfied, the telemetry data are discarded. You can combine the conditions by using the logical OR operator. This processor supports traces, metrics, and logs.
11+
12+
:FeatureName: The Filter Processor
13+
include::snippets/technology-preview.adoc[]
14+
15+
.OpenTelemetry Collector custom resource with an enabled Filter Processor
16+
[source,yaml]
17+
----
18+
# ...
19+
config:
20+
processors:
21+
filter/ottl:
22+
error_mode: ignore # <1>
23+
traces:
24+
span:
25+
- 'attributes["container.name"] == "app_container_1"' # <2>
26+
- 'resource.attributes["host.name"] == "localhost"' # <3>
27+
# ...
28+
----
29+
<1> Defines the error mode. When set to `ignore`, ignores errors returned by conditions. When set to `propagate`, returns the error up the pipeline. An error causes the payload to be dropped from the Collector.
30+
<2> Filters the spans that have the `container.name == app_container_1` attribute.
31+
<3> Filters the spans that have the `host.name == localhost` resource attribute.

‎modules/otel-processors-group-by-attributes-processor.adoc‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-collector/otel-collector-processors.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="otel-processors-group-by-attributes-processor_{context}"]
7+
= Group-by-Attributes Processor
8+
9+
[role="_abstract"]
10+
The Group-by-Attributes Processor groups all spans, log records, and metric datapoints that share the same attributes by reassigning them to a Resource that matches those attributes.
11+
12+
:FeatureName: The Group-by-Attributes Processor
13+
include::snippets/technology-preview.adoc[]
14+
15+
At minimum, configuring this processor involves specifying an array of attribute keys to be used to group spans, log records, or metric datapoints together, as in the following example:
16+
17+
.Example of the OpenTelemetry Collector custom resource when using the Group-by-Attributes Processor
18+
[source,yaml]
19+
----
20+
# ...
21+
config:
22+
processors:
23+
groupbyattrs:
24+
keys: # <1>
25+
- <key1> # <2>
26+
- <key2>
27+
# ...
28+
----
29+
<1> Specifies attribute keys to group by.
30+
<2> If a processed span, log record, or metric datapoint contains at least one of the specified attribute keys, it is reassigned to a Resource that shares the same attribute values; and if no such Resource exists, a new one is created. If none of the specified attribute keys is present in the processed span, log record, or metric datapoint, then it remains associated with its current Resource. Multiple instances of the same Resource are consolidated.

‎modules/otel-processors-kubernetes-attributes-processor.adoc‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-collector/otel-collector-processors.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="otel-processors-kubernetes-attributes-processor_{context}"]
7+
= Kubernetes Attributes Processor
8+
9+
[role="_abstract"]
10+
The Kubernetes Attributes Processor enables automatic configuration of spans, metrics, and log resource attributes by using the Kubernetes metadata.
11+
This processor supports traces, metrics, and logs.
12+
This processor automatically identifies the Kubernetes resources, extracts the metadata from them, and incorporates this extracted metadata as resource attributes into relevant spans, metrics, and logs. It utilizes the Kubernetes API to discover all pods operating within a cluster, maintaining records of their IP addresses, pod UIDs, and other relevant metadata.
13+
14+
.Minimum {product-title} permissions required for the Kubernetes Attributes Processor
15+
[source,yaml]
16+
----
17+
kind: ClusterRole
18+
metadata:
19+
name: otel-collector
20+
rules:
21+
- apiGroups: ['']
22+
resources: ['pods', 'namespaces']
23+
verbs: ['get', 'watch', 'list']
24+
- apiGroups: ['apps']
25+
resources: ['replicasets']
26+
verbs: ['get', 'watch', 'list']
27+
# ...
28+
----
29+
30+
.OpenTelemetry Collector using the Kubernetes Attributes Processor
31+
[source,yaml]
32+
----
33+
# ...
34+
config:
35+
processors:
36+
k8sattributes:
37+
filter:
38+
node_from_env_var: KUBE_NODE_NAME
39+
# ...
40+
----

‎modules/otel-processors-memory-limiter-processor.adoc‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-collector/otel-collector-processors.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="otel-processors-memory-limiter-processor_{context}"]
7+
= Memory Limiter Processor
8+
9+
[role="_abstract"]
10+
The Memory Limiter Processor periodically checks the Collector's memory usage and pauses data processing when the soft memory limit is reached. This processor supports traces, metrics, and logs. The preceding component, which is typically a receiver, is expected to retry sending the same data and may apply a backpressure to the incoming data. When memory usage exceeds the hard limit, the Memory Limiter Processor forces garbage collection to run.
11+
12+
.Example of the OpenTelemetry Collector custom resource when using the Memory Limiter Processor
13+
[source,yaml]
14+
----
15+
# ...
16+
config:
17+
processors:
18+
memory_limiter:
19+
check_interval: 1s
20+
limit_mib: 4000
21+
spike_limit_mib: 800
22+
service:
23+
pipelines:
24+
traces:
25+
processors: [batch]
26+
metrics:
27+
processors: [batch]
28+
# ...
29+
----
30+
31+
.Parameters used by the Memory Limiter Processor
32+
[options="header"]
33+
[cols="a,a,a"]
34+
|===
35+
|Parameter |Description |Default
36+
37+
|`check_interval`
38+
|Time between memory usage measurements. The optimal value is `1s`. For spiky traffic patterns, you can decrease the `check_interval` or increase the `spike_limit_mib`.
39+
|`0s`
40+
41+
|`limit_mib`
42+
|The hard limit, which is the maximum amount of memory in MiB allocated on the heap. Typically, the total memory usage of the OpenTelemetry Collector is about 50 MiB greater than this value.
43+
|`0`
44+
45+
|`spike_limit_mib`
46+
|Spike limit, which is the maximum expected spike of memory usage in MiB. The optimal value is approximately 20% of `limit_mib`. To calculate the soft limit, subtract the `spike_limit_mib` from the `limit_mib`.
47+
|20% of `limit_mib`
48+
49+
|`limit_percentage`
50+
|Same as the `limit_mib` but expressed as a percentage of the total available memory. The `limit_mib` setting takes precedence over this setting.
51+
|`0`
52+
53+
|`spike_limit_percentage`
54+
|Same as the `spike_limit_mib` but expressed as a percentage of the total available memory. Intended to be used with the `limit_percentage` setting.
55+
|`0`
56+
57+
|===

‎modules/otel-processors-probabilistic-sampling-processor.adoc‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
// Module included in the following assemblies:
22
//
3-
// * observability/otel/otel-collector/
3+
// * observability/otel/otel-collector/otel-collector-processors.adoc
44

55
:_mod-docs-content-type: REFERENCE
66
[id="otel-processors-probabilistic-sampling-processor_{context}"]
77
= Probabilistic Sampling Processor
88

9+
[role="_abstract"]
910
If you handle high volumes of telemetry data and seek to reduce costs by reducing processed data volumes, you can use the Probabilistic Sampling Processor as an alternative to the Tail Sampling Processor.
1011

1112
:FeatureName: Probabilistic Sampling Processor

‎modules/otel-processors-resource-detection-processor.adoc‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-collector/otel-collector-processors.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="otel-processors-resource-detection-processor_{context}"]
7+
= Resource Detection Processor
8+
9+
[role="_abstract"]
10+
The Resource Detection Processor identifies host resource details in alignment with OpenTelemetry's resource semantic standards. Using the detected information, this processor can add or replace the resource values in telemetry data. This processor supports traces and metrics. You can use this processor with multiple detectors such as the Docket metadata detector or the `OTEL_RESOURCE_ATTRIBUTES` environment variable detector.
11+
12+
:FeatureName: The Resource Detection Processor
13+
include::snippets/technology-preview.adoc[]
14+
15+
.{product-title} permissions required for the Resource Detection Processor
16+
[source,yaml]
17+
----
18+
kind: ClusterRole
19+
metadata:
20+
name: otel-collector
21+
rules:
22+
- apiGroups: ["config.openshift.io"]
23+
resources: ["infrastructures", "infrastructures/status"]
24+
verbs: ["get", "watch", "list"]
25+
# ...
26+
----
27+
28+
.OpenTelemetry Collector using the Resource Detection Processor
29+
[source,yaml]
30+
----
31+
# ...
32+
config:
33+
processors:
34+
resourcedetection:
35+
detectors: [openshift]
36+
override: true
37+
service:
38+
pipelines:
39+
traces:
40+
processors: [resourcedetection]
41+
metrics:
42+
processors: [resourcedetection]
43+
# ...
44+
----
45+
46+
.OpenTelemetry Collector using the Resource Detection Processor with an environment variable detector
47+
[source,yaml]
48+
----
49+
# ...
50+
config:
51+
processors:
52+
resourcedetection/env:
53+
detectors: [env] # <1>
54+
timeout: 2s
55+
override: false
56+
# ...
57+
----
58+
<1> Specifies which detector to use. In this example, the environment detector is specified.

0 commit comments

Comments
 (0)