Skip to content

Commit 2e39ec2

Browse files
committed
updating local data guide in example datasets to reflect otel config extension
1 parent ad92b7b commit 2e39ec2

File tree

1 file changed

+50
-34
lines changed
  • docs/use-cases/observability/clickstack/example-datasets

1 file changed

+50
-34
lines changed

docs/use-cases/observability/clickstack/example-datasets/local-data.md

Lines changed: 50 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -45,21 +45,19 @@ Navigate to [`Team Settings`](http://localhost:8080/team) and copy the `Ingestio
4545

4646
<Image img={copy_api_key} alt="Copy API key" size="lg"/>
4747

48-
## Create a local OpenTelemetry configuration {#create-otel-configuration}
48+
## Create a custom OpenTelemetry configuration {#create-otel-configuration}
4949

50-
Create a `otel-local-file-collector.yaml` file with the following content.
51-
52-
**Important**: Populate the value `<YOUR_INGESTION_API_KEY>` with your ingestion API key copied above (not required for HyperDX in ClickHouse Cloud).
50+
Create a `custom-local-config.yaml` file with the following content:
5351

5452
```yaml
5553
receivers:
5654
filelog:
5755
include:
58-
- /var/log/**/*.log # Linux
59-
- /var/log/syslog
60-
- /var/log/messages
61-
- /private/var/log/*.log # macOS
62-
- /tmp/all_events.log # macos - see below
56+
- /host/var/log/**/*.log # Linux logs from host
57+
- /host/var/log/syslog
58+
- /host/var/log/messages
59+
- /host/private/var/log/*.log # macOS logs from host
60+
- /tmp/all_events.log # macOS - see below
6361
start_at: beginning # modify to collect new files only
6462

6563
hostmetrics:
@@ -96,29 +94,25 @@ receivers:
9694
network:
9795
processes:
9896

99-
exporters:
100-
otlp:
101-
endpoint: localhost:4317
102-
headers:
103-
authorization: <YOUR_INGESTION_API_KEY>
104-
tls:
105-
insecure: true
106-
sending_queue:
107-
enabled: true
108-
num_consumers: 10
109-
queue_size: 262144 # 262,144 items × ~8 KB per item ≈ 2 GB
110-
11197
service:
11298
pipelines:
113-
logs:
99+
logs/local:
114100
receivers: [filelog]
115-
exporters: [otlp]
116-
metrics:
101+
processors:
102+
- memory_limiter
103+
- batch
104+
exporters:
105+
- clickhouse
106+
metrics/hostmetrics:
117107
receivers: [hostmetrics]
118-
exporters: [otlp]
108+
processors:
109+
- memory_limiter
110+
- batch
111+
exporters:
112+
- clickhouse
119113
```
120114
121-
This configuration collects system logs and metric for OSX and Linux systems, sending the results to ClickStack via the OTLP endpoint on port 4317.
115+
This configuration collects system logs and metrics for OSX and Linux systems, sending the results to ClickStack. The configuration extends the ClickStack collector by adding new receivers and pipelines—you reference the existing `clickhouse` exporter and processors (`memory_limiter`, `batch`) that are already configured in the base ClickStack collector.
122116

123117
:::note Ingestion timestamps
124118
This configuration adjusts timestamps at ingest, assigning an updated time value to each event. Users should ideally [preprocess or parse timestamps](/use-cases/observability/clickstack/ingesting-data/otel-collector#processing-filtering-transforming-enriching) using OTel processors or operators in their log files to ensure accurate event time is retained.
@@ -136,20 +130,42 @@ Users wanting more detailed logs on OSX can run the command `log stream --debug
136130

137131
## Start the collector {#start-the-collector}
138132

139-
Run the following docker command to start an instance of the OTel collector.
133+
Run the following docker command to start the all-in-one container with your custom configuration:
140134

141135
```shell
142-
docker run --network=host --rm -it \
136+
docker run -d --name clickstack \
137+
-p 8080:8080 -p 4317:4317 -p 4318:4318 \
143138
--user 0:0 \
144-
-v "$(pwd)/otel-local-file-collector.yaml":/etc/otel/config.yaml \
145-
-v /var/log:/var/log:ro \
146-
-v /private/var/log:/private/var/log:ro \
147-
otel/opentelemetry-collector-contrib:latest \
148-
--config /etc/otel/config.yaml
139+
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
140+
-v "$(pwd)/custom-local-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
141+
-v /var/log:/host/var/log:ro \
142+
-v /private/var/log:/host/private/var/log:ro \
143+
docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest
149144
```
150145

151146
:::note Root user
152147
We run the collector as the root user to access all system logs—this is necessary to capture logs from protected paths on Linux-based systems. However, this approach is not recommended for production. In production environments, the OpenTelemetry Collector should be deployed as a local agent with only the minimal permissions required to access the intended log sources.
148+
149+
Note that we mount the host's `/var/log` to `/host/var/log` inside the container to avoid conflicts with the container's own log files.
150+
:::
151+
152+
:::note HyperDX in ClickHouse Cloud
153+
If using HyperDX in ClickHouse Cloud with a standalone collector, use this command instead:
154+
155+
```shell
156+
docker run -d \
157+
-p 4317:4317 -p 4318:4318 \
158+
--user 0:0 \
159+
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
160+
-e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} \
161+
-e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} \
162+
-e CLICKHOUSE_USER=${CLICKHOUSE_USER} \
163+
-e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} \
164+
-v "$(pwd)/custom-local-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
165+
-v /var/log:/host/var/log:ro \
166+
-v /private/var/log:/host/private/var/log:ro \
167+
docker.hyperdx.io/hyperdx/hyperdx-otel-collector
168+
```
153169
:::
154170

155171
The collector will immediately begin collecting local system logs and metrics.
@@ -180,4 +196,4 @@ From the subsequent menu you can select `Percentage` from the `Output format` dr
180196

181197
<Image img={hyperdx_23} alt="Memory % of time" size="lg"/>
182198

183-
</VerticalStepper>
199+
</VerticalStepper>

0 commit comments

Comments
 (0)