filexporter/invalid zstd file

[bmiguel-teixeira]

Component(s)

exporter/file

What happened?

I have messing around with file exporter as a way to store logs in raw format in cold storage. I was looking into the zstd streaming compression as a more efficient way to ingesting logs (writing raw logs uncompressed takes a fair ammount of storage).

But the final generated file seems corrupted or invalid. I am unable to unzip with traditional unzstd tools.

Description

Steps to Reproduce

exporters:
  file:
    path: randomfile
    format: json
    compression: zstd

Run unzstd fileout.zst and it reports out unsupported format

Expected Result

I would expected i would be able to unzstd the created file as if it was a tipical zstd file.

Actual Result

Invalid file

Collector version

0.139.0

Environment information

Environment

Same behavior in Darwin ARM64 and LinuxAMD64

OpenTelemetry Collector configuration

exporters:
  file:
    path: /logs/out/*
    format: json
    compression: zstd

    group_by:
      enabled: true
      max_open_files: 1000
      resource_attribute: hourly_log_batch_name
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
processors:
  batch: {}
  transform/logs:
    log_statements:
    - context: log
      statements:
      - set(resource.attributes["hourly_log_batch_name.name"], "unknown")
receivers:
  filelog:
    include:
    - /logs/logfile.json
    start_at: beginning

service:
  extensions: []
  pipelines:
    logs:
      exporters:
      - file
      processors:
      - transform/logs
      - batch
      receivers:
      - filelog
  telemetry:
    logs:
      level: info

Log output

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/file: @atingchen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bmiguel-teixeira]

Btw,

I was able to uncompress the files by writing a small golang script together with OTEL source code from the collector. It seems to need some type of streaming decompression.

Is this the expected behaviour? To create custom code to decompress the files? Are there any general utilities to bypass this?

The docs are kind unclear about this behaviour.