swap out the existing headers normalization logic with the tag normalizer #5041

wantsui · 2025-11-11T22:18:39Z

What does this PR do?

Uses the new tag normalization logic that was copied from the Trace Agent for headers.

Motivation:

While working on #5033, it was noticed that the current normalization logic in dd-trace-rb is missing some of the logic from https://github.com/DataDog/datadog-agent/blob/45799c842bbd216bcda208737f9f11cade6fdd95/pkg/trace/traceutil/normalize.go#L131 , specifically around:

character limit truncation after 200 characters
consecutive _ are merged
if the first letter is not a letter, look for the next first letter since tags have to start with a letter

Change log entry

feat: swap out the existing headers normalization logic with the tag normalizer

Additional Notes:

I am not sure if we consider this to be a bug or a missing feature, but note that with this new logic, some of the headers results are going to be empty strings ie: "", which can be seen in the tests below.

How to test the change?

I've been using:

bundle exec rspec spec/datadog/tracing/metadata/ext_spec.rb

… This is still missing memoization and additional tests.

…he payload has the process tag only when the feature is enabled.

…versions so this fixes that.

Co-authored-by: Marco Costa <marco.costa@datadoghq.com>

…uby conflict with sqlite and it is not needed for this test

…izer used for spans

github-actions · 2025-11-11T22:18:53Z

👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description.

If changes need to be present in CHANGELOG.md you can state it this way

**Change log entry**

Yes. A brief summary to be placed into the CHANGELOG.md

(possible answers Yes/Yep/Yeah)

Or you can opt out like that

**Change log entry**

None.

(possible answers No/Nope/None)

^{Visited at: 2025-11-12 16:20:11 UTC}

ivoanjo · 2025-11-12T08:42:17Z

spec/datadog/tracing/metadata/ext_spec.rb

      it 'preserves them' do
-        is_expected.to eq(tag)
+        is_expected.to eq("")
      end
    end

    context 'with unsupported characters' do
      let(:tag) { "\\ \t!@?" }

      it 'replaces each with an underscore' do
-        is_expected.to eq('_' * tag.size)
+        # turns into an empty string because in a tag, the first character must be a letter
+        is_expected.to eq('')
      end
    end


When changing the logic -- please update the test description as well! It'd be quite confusing to have a test saying "it replaces each with an underscore" that actually is not true :P

oh! good catch, I'll use a better description for this!

I updated the description but open to feedback on the wording !

Seems reasonable! :)

Thanks! Just a heads up I'm still changing the normalization logic in #5033, so I'll circle back to this PR after the process tags PR to see if I need to update these test assertions.

… environment variable.

Co-authored-by: Sergey Fedorov <oni.strech@gmail.com>

…ormalized

Co-authored-by: Marco Costa <marco.costa@datadoghq.com>

Co-authored-by: Oleg Pudeyev <156273877+p-datadog@users.noreply.github.com>

…xist, small fixes, and add comments to the normalizer.rb explaining the expected usage

Co-authored-by: Oleg Pudeyev <156273877+p-datadog@users.noreply.github.com>

Strech · 2025-11-20T22:10:57Z

spec/datadog/tracing/metadata/ext_spec.rb

+    context 'with a mix of unsupported and supported characters' do
+      let(:tag) { "\\ \t!@?a" }
+
+      it 'the first letter to show up is the supported character' do


We try to make out of this line a real statement, readable by us, now that reads

it the first letter to show up is the supported character

to be honest I don't get what it said. Check out some other tests, you will find something like

it 'returns only supported characters'

or something similar

…xtend self on process.rb

…r and update tests to show some new assertions

Co-authored-by: Sergey Fedorov <oni.strech@gmail.com>

…ts are allowed for tag values

… before setting process tags.

…normalization

pr-commenter · 2025-11-22T00:24:45Z

Benchmarks

Benchmark execution time: 2025-11-22 00:24:16

Comparing candidate commit 2f4d1c6 in PR branch replace-headers-tags-normalization with baseline commit 4e3f8f4 in branch add-process-tags-to-tracing.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 2 unstable metrics.

scenario:tracing - Propagation - Trace Context

🟩 throughput [+3994.855op/s; +4119.039op/s] or [+11.708%; +12.072%]

wantsui and others added 15 commits November 7, 2025 17:26

Add initial attempt at adding process related tags on trace payloads.…

1d8bab2

… This is still missing memoization and additional tests.

Add test for multiple calls to the formatter tags

58592a3

Add tests for trace formatter spec to assert that the first span of t…

7dc9184

…he payload has the process tag only when the feature is enabled.

it turns out you cannot just pin things to rails 7 due to newer ruby …

cad26a6

…versions so this fixes that.

Update lib/datadog/core/environment/process.rb

f31440a

Co-authored-by: Marco Costa <marco.costa@datadoghq.com>

fix string and rename formatted_process_tags_k1_v1 to serialized

cfec602

remove unneeded line

8dae705

remove server type for now until more research is done

055586f

Add new tag normalizer logic following the trace agent.

cacb500

lint fix

7661a3f

add missing files from prototype command

7825940

Add missing constants to ext rbs file

5de6efd

jruby fix for the process spec

f5ca84a

remove the active record during rails creation because it caused a jr…

9ad5be5

…uby conflict with sqlite and it is not needed for this test

Swap out the existing headers normalization logic with the tag normal…

ceaf7d4

…izer used for spans

github-actions bot added the tracing label Nov 11, 2025

wantsui requested a review from marcotc November 11, 2025 22:18

ivoanjo reviewed Nov 12, 2025

View reviewed changes

wantsui and others added 11 commits November 12, 2025 13:57

update the comments for the test

0848833

fix another comment

9fc4009

Bring tag normalization to 1:1 parity with the Trace Agent

a66e635

Add changes from code review around comments and add test for the new…

ec1e930

… environment variable.

Merge branch 'master' into add-process-tags-to-tracing

4073ab5

Remove the rails gem install from process_spec

22a3680

Remove 1 sec delay.

5784833

Update sig/datadog/core/environment/ext.rbs

2b705e3

Co-authored-by: Sergey Fedorov <oni.strech@gmail.com>

Update lib/datadog/tracing/transport/trace_formatter.rb

e3deb4c

Co-authored-by: Sergey Fedorov <oni.strech@gmail.com>

Add improvements for long strings.

4747259

small improvement to the whitespace removal.

41bc6c0

wantsui and others added 15 commits November 19, 2025 11:07

Add a special character into the test app name to show that it gets n…

4210d74

…ormalized

Update lib/datadog/core/normalizer.rb

f9af946

Co-authored-by: Marco Costa <marco.costa@datadoghq.com>

Update lib/datadog/core/normalizer.rb

381fbe2

Co-authored-by: Marco Costa <marco.costa@datadoghq.com>

Fixes for new constant names

138dff8

Change to byteslice

2449153

fix lint.

5252259

remove process_spec from main rake task

0eaf302

Update spec/datadog/core/normalizer_spec.rb

fbfecfe

Co-authored-by: Oleg Pudeyev <156273877+p-datadog@users.noreply.github.com>

Update spec/datadog/tracing/transport/trace_formatter_spec.rb

cc2225f

Co-authored-by: Oleg Pudeyev <156273877+p-datadog@users.noreply.github.com>

Merge branch 'master' into add-process-tags-to-tracing

62822ab

Remove the unless check and replace with an assertion that the file e…

a77e63b

…xist, small fixes, and add comments to the normalizer.rb explaining the expected usage

Update spec/datadog/core/environment/process_spec.rb

439e81a

Co-authored-by: Oleg Pudeyev <156273877+p-datadog@users.noreply.github.com>

fix lint

9e45ade

Rename Normalizer to TagNormalizer.

0ab4fef

Update lib/datadog/core/environment/process.rb

6577d3f

Co-authored-by: Oleg Pudeyev <156273877+p-datadog@users.noreply.github.com>

Strech reviewed Nov 20, 2025

View reviewed changes

wantsui and others added 12 commits November 20, 2025 17:33

Add api private comment to the tag normalizer and refactor away the e…

a336c66

…xtend self on process.rb

Fix steep errors on the process rbs file

0d229de

Refactor the utils encode call so it can be used in the tag normalize…

e83bc4a

…r and update tests to show some new assertions

Update Rakefile

ce1759f

Co-authored-by: Sergey Fedorov <oni.strech@gmail.com>

Update lib/datadog/core/tag_normalizer.rb

8b978c6

Co-authored-by: Sergey Fedorov <oni.strech@gmail.com>

Add lint fixes and remove unneeded regex at the end.

f4c9d49

fix rbs file for deleted variable

eae4eb9

remove unneeded conditional

f3f1480

Add a log if the process tags cannot be obtained

b48d20d

Fix regex and reuse the same test cases to show that the leading digi…

3d33291

…ts are allowed for tag values

Attempt to retrieve as many non empty string process tags as possible…

4e3f8f4

… before setting process tags.

Merge branch 'add-process-tags-to-tracing' into replace-headers-tags-…

2f4d1c6

…normalization

Base automatically changed from add-process-tags-to-tracing to master December 4, 2025 17:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

swap out the existing headers normalization logic with the tag normalizer #5041

swap out the existing headers normalization logic with the tag normalizer #5041

wantsui commented Nov 11, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 11, 2025 •

edited

Loading

Uh oh!

ivoanjo Nov 12, 2025

Uh oh!

wantsui Nov 12, 2025

Uh oh!

wantsui Nov 12, 2025

Uh oh!

ivoanjo Nov 19, 2025

Uh oh!

wantsui Nov 19, 2025

Uh oh!

Strech Nov 20, 2025

Uh oh!

pr-commenter bot commented Nov 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

swap out the existing headers normalization logic with the tag normalizer #5041

Are you sure you want to change the base?

swap out the existing headers normalization logic with the tag normalizer #5041

Conversation

wantsui commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ivoanjo Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

wantsui Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

wantsui Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

ivoanjo Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

wantsui Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Strech Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

pr-commenter bot commented Nov 22, 2025

Benchmarks

scenario:tracing - Propagation - Trace Context

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wantsui commented Nov 11, 2025 •

edited

Loading

github-actions bot commented Nov 11, 2025 •

edited

Loading