Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -114,11 +114,7 @@ Set up the Google Pub/Sub destination and its environment variables when you [se

{{< img src="observability_pipelines/destinations/google_pubsub_env_var.png" alt="The install page showing the Google Pub/Sub environment variable field" style="width:70%;" >}}

By default the Worker sends data to the global endpoint: `https://pubsub.googleapis.com`.

If your Pub/Sub topic is region-specific, configure the Google Pub/Sub alternative endpoint URL with the regional endpoint. See [About Pub/Sub endpoints][5] for more information.

Stored as the environment variable: `DD_OP_DESTINATION_GCP_PUBSUB_ENDPOINT_URL`.
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/google_pubsub %}}

## Troubleshooting

Expand Down Expand Up @@ -163,7 +159,6 @@ A batch of events is flushed when one of these parameters is met. See [event bat
[2]: https://cloud.google.com/docs/authentication#auth-flowchart
[3]: https://cloud.google.com/pubsub/docs/access-control#roles
[4]: https://console.cloud.google.com/iam-admin/serviceaccounts
[5]: https://cloud.google.com/pubsub/docs/reference/service_apis_overview#pubsub_endpoints
[6]: /observability_pipelines/destinations/#event-batching
[7]:https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity
[8]: /observability_pipelines/monitoring_and_maintenance/pipeline_usage_metrics/
17 changes: 1 addition & 16 deletions content/en/observability_pipelines/destinations/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,22 +78,7 @@ Click **Advanced** if you want to set any of the following fields:

### Set environment variables

#### Kafka bootstrap servers
- The host and port of the Kafka bootstrap servers.
- This is the bootstrap server that the client uses to connect to the Kafka cluster and discover all the other hosts in the cluster. The host and port must be entered in the format of `host:port`, such as `10.14.22.123:9092`. If there is more than one server, use commas to separate them.
- Stored as the environment variable: `DD_OP_DESTINATION_KAFKA_BOOTSTRAP_SERVERS`.

#### TLS (when enabled)

- If TLS is enabled, the Kafka TLS passphrase is needed.
- Stored as the environment variable: `DD_OP_DESTINATION_KAFKA_KEY_PASS`.

#### SASL (when enabled)

- Kafka SASL username
- Stored as the environment variable: `DD_OP_DESTINATION_KAFKA_SASL_USERNAME`.
- Kafka SASL password
- Stored as the environment variable: `DD_OP_DESTINATION_KAFKA_SASL_PASSWORD`.
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/kafka %}}

## librdkafka options

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ Some Observability Pipelines components require setting up environment variables
### Logstash
{{% observability_pipelines/configure_existing_pipelines/source_env_vars/logstash %}}

### OpenTelemetry
{{% observability_pipelines/configure_existing_pipelines/source_env_vars/opentelemetry %}}

### Socket

{{% observability_pipelines/configure_existing_pipelines/source_env_vars/socket %}}
Expand Down Expand Up @@ -91,6 +94,12 @@ Some Observability Pipelines components require setting up environment variables
### Elasticsearch
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/elasticsearch %}}

### Kafka
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/kafka %}}

### Google Pub/Sub
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/google_pubsub %}}

### HTTP Client
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/http_client %}}

Expand Down
10 changes: 1 addition & 9 deletions content/en/observability_pipelines/sources/opentelemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,7 @@ Optionally, toggle the switch to enable TLS. If you enable TLS, the following ce

{{< img src="observability_pipelines/sources/otel_env_vars.png" alt="The install page showing the OpenTelemetry environment variable field" style="width:75%;" >}}

You must provide both HTTP and gRPC endpoints. Configure your OTLP exporters to point to one of these endpoints. See [Send logs to the Observability Pipelines Worker](#send-logs-to-the-observability-pipelines-worker) for more information.

- HTTP listener address
- The Observability Pipelines Worker listens to this socket address to receive logs from the OTel collector.
- Stored as the environment variable `DD_OP_SOURCE_OTEL_HTTP_ADDRESS`.

- gRPC listener address
- The Observability Pipelines Worker listens to this socket address to receive logs from the OTel collector.
- Stored as the environment variable `DD_OP_SOURCE_OTEL_GRPC_ADDRESS`.
{{% observability_pipelines/configure_existing_pipelines/source_env_vars/splunk_hec %}}

## Send logs to the Observability Pipelines Worker

Expand Down
4 changes: 4 additions & 0 deletions content/en/observability_pipelines/sources/socket.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,8 @@ Select and set up this source when you [set up a pipeline][1]. The information b

{{% observability_pipelines/source_settings/socket %}}

## Set the environment variables

{{% observability_pipelines/configure_existing_pipelines/source_env_vars/socket %}}

[1]: /observability_pipelines/configuration/set_up_pipelines/
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
By default the Worker sends data to the global endpoint: `https://pubsub.googleapis.com`.

If your Pub/Sub topic is region-specific, configure the Google Pub/Sub alternative endpoint URL with the regional endpoint. See [About Pub/Sub endpoints][10240] for more information.

Stored as the environment variable: `DD_OP_DESTINATION_GCP_PUBSUB_ENDPOINT_URL`.

[10240]: https://cloud.google.com/pubsub/docs/reference/service_apis_overview#pubsub_endpoints
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#### Kafka bootstrap servers
- The host and port of the Kafka bootstrap servers.
- This is the bootstrap server that the client uses to connect to the Kafka cluster and discover all the other hosts in the cluster. The host and port must be entered in the format of `host:port`, such as `10.14.22.123:9092`. If there is more than one server, use commas to separate them.
- Stored as the environment variable: `DD_OP_DESTINATION_KAFKA_BOOTSTRAP_SERVERS`.

#### TLS (when enabled)

- If TLS is enabled, the Kafka TLS passphrase is needed.
- Stored as the environment variable: `DD_OP_DESTINATION_KAFKA_KEY_PASS`.

#### SASL (when enabled)

- Kafka SASL username
- Stored as the environment variable: `DD_OP_DESTINATION_KAFKA_SASL_USERNAME`.
- Kafka SASL password
- Stored as the environment variable: `DD_OP_DESTINATION_KAFKA_SASL_PASSWORD`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
You must provide both HTTP and gRPC endpoints. Configure your OTLP exporters to point to one of these endpoints. See [Send logs to the Observability Pipelines Worker][10230] for more information.

- HTTP listener address
- The Observability Pipelines Worker listens to this socket address to receive logs from the OTel collector.
- Stored as the environment variable `DD_OP_SOURCE_OTEL_HTTP_ADDRESS`.

- gRPC listener address
- The Observability Pipelines Worker listens to this socket address to receive logs from the OTel collector.
- Stored as the environment variable `DD_OP_SOURCE_OTEL_GRPC_ADDRESS`.

[10230]: /observability_pipelines/sources/opentelemetry/#send-logs-to-the-observability-pipelines-worker
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
- Socket address:
- The address and port where the Observability Pipelines Worker listens for incoming logs.
- Stored as the environment variable `DD_OP_SOURCE_SOCKET_ADDRESS`
- Stored as the environment variable `DD_OP_SOURCE_SOCKET_ADDRESS`.

- TLS passphrase:
- Used if TLS is enabled.
- TLS passphrase (when enabled)
- Stored as the environment variable `DD_OP_SOURCE_SOCKET_KEY_PASS`.
Loading