Support for presets, enabled, and disabled for detectors, propagators, and instrumentations

[pellared]

I'd like to propose a new configuration approach for OpenTelemetry configuration, specifically for the following components like detectors, propagators, and instrumentation.

Why

From experience with other ecosystems (notably OTel .NET Automatic Instrumentation and popular linters like golangci-lint), many users want to quickly enable a sane and practical default set of configuration values, or easily enable all, disable all, or fine-tune a subset. This pattern is especially useful for onboarding and simplifies configuration for both beginners and advanced users.

What

Add support for the following configuration values:

• presets: Predefined, recommended sets of options (e.g., “default”, “all”, “stable”, "none", "fast", "common").
• enabled: Explicit list of items to enable.
• disabled: Explicit list of items to disable.

For example, a configuration could look like (simplified):

instrumentation:
  preset: all         # Enable all available instrumentation
  disabled: [redis]   # Exclude one specific instrumentation that causes issues

propagators:
  preset: default     # The OpenTelemetry recommended defaults

detectors:
  preset: none        # A fine-grained list of detectors
  enabled: [gcp]

Benefits:

• Simplifies configuration for most use cases, especially for new users.
• Allows fine-grained configuration.
• Aligns with user expectations from .NET auto instrumentation and other similar projects.

Considerations:

• Preset names and their actual contents should be documented.
• Define behavior if an item exists in both enabled and disabled lists.
  • Option A: error
  • Option B: disabled has higher priority (precedence)
• Define how to configure specific item (e.g. redis instrumentation)
  • Option A: following configuration patterns used by golangci-lint for items, add separate settings node e.g. instrumentation-settings

instrumentation:
  preset: all         # Enable all available instrumentation

instrumentation-settings:
  redis:
    metrics-enabled: false

• Option B: the items in enabled can be (or are) objects (not just names)

instrumentation:
  preset: all         # Enable all available instrumentation
  enabled:
    - redis:
        metrics-enabled: false

or maybe

instrumentation:
  preset: all         # Enable all available instrumentation
  enabled:
    redis:
        metrics-enabled: false

but then we would have different syntax for enabled and disabled which feels akward

Additional (optional) features:

• Allow users to easily extend or override presets for their specific needs

Side note: I think it may be worth checking configuration patterns used by golangci-lint. Similar patterns could be used for configuring instrumentation, propagators, detectors.

Drawbacks:

• Presets are "magic" that can cause different behavior between implementation versions. E.g. stable instrumentation preset may have more entries for OTel .NET Auto in v1.45.0 than v1.23.0. Moreover, people reading the config content would need to understand what is behind the preset. Without the presets everything is explicitly specified.
• The config structure is more complex.
• Implementation is harder.

PS. Initially I was even against the environment variable substitution behavior.

[pellared]

CC @open-telemetry/dotnet-instrumentation-maintainers

[Kielek]

I will support this. Only comment (but it might be language specific).
We have intrumentation per signal in OTel .NET AutoInstrumentation:

instruemntation:
   traces:
     redis:
        some_strange_setting: true
   metrics:
     redis:
         some_other_settings: "no"

[jaydeluca]

I'm not sure how consistent this needs to be across languages, but just for context, we are doing things a little differently in java for our instrumentation configs, example.

for our "global" enable/disable, we used "instrumentation_mode", which has two options as of now, "default" and "none". "default" means all instrumentations are enabled, except for the ones that we disable by default due to various reasons, and "none" means everything is off. We then have individual "enabled" per instrumentation.

So to turn everything off and selectively enable it looks like:

instrumentation/development:
  java:
    agent:
      instrumentation_mode: none
    tomcat:
      enabled: true

cc @zeitlinger @trask

[pellared]

We have settings per signal in OTel .NET AutoInstrumentation:

I think you wanted to say "instrumentation [library] per signal".

In most (all?) other languages the pattern is that the instrumentation library instruments with many signal types. Therefore, I propose that the "signal configuration" is part of the configuration of given instrumentation; even if for .NET it means that one instrumentation in config maps to multiple instrumentation libraries (e.g. for trace+metrics).

[ysolomchenko]

I like the idea of adding presets. We can keep the existing structure but add presets that take priority.
That is, there would be two configuration options: using presets (e.g., all enabled, only stable, etc.)
and the ability to configure manually with the current structure.

[pellared]

I added some "drawbacks" to the description.

[robsunday]

This idea of presets is interesting, but it involves some risks. On one hand, it's easy to turn on some defaults, but on the other hand, these defaults are hidden in the code. Moreover, they may silently change from version to version.

I think the intention of declarative config was to be very explicit, so perhaps we should provide some YAML templates with default settings instead of adding another option to turn on defaults. Templates could be versioned and published, so users would always know what the defaults are.

There is one issue with this approach: the YAML specification does not support file inclusion (however, some parsers have such an extension). Therefore, a template should most probably be a complete YAML file with some predefined elements. A template could be used as is, or it could serve as a base for additional customization.

We could also consider having multiple small templates of YAML sections that could be composed into a final config YAML by some additional tool (there are plenty of such tools).

[jack-berg]

I acknowledge there is a need for these types of common configuration, but prefer to accomplish this by maintaining / publishing starter template config files. I imagine us expanding the set in /examples to accommodate common configs in different ecosystems and for different use cases.

Declarative configuration has a "what you see is what you get" philosophy, and presets would be moving away this.

This idea has come up in the past in a slightly different form. Arguments have been made along the lines of "this config is too verbose, I want to write something simpler with defaults that just work for most people". We have talked about helm playing a role here. For example, opentelemetry's helm chart for the collector has a simplified config interface with a variety of "preset" options (they even use the same vocabulary suggested here). Helm templates interpret those presets and output the more verbose / detailed kubernetes / collector configuration YAML. This allows:

• The k8s / collector configuration data models to very closely mirror the underlying concepts, with minimal magic
• Simplified configuration interface for those who want it
• Folks using these helm charts can still ultimately see the fully rendered configuration YAML and modify it very precisely if they desire

Similarly, I think the declarative configuration data model should very closely mirror the underlying concepts, with simplified terse configuration stories and associated magic being reserved for higher order abstractions.

[pellared]

Declarative configuration has a "what you see is what you get" philosophy, and presets would be moving away this.

I totally agree with this and I support this philosophy.

On the other hand, note that one could say that the environment variable substitution feature is also against this philosophy (sic!).

I think that we lack high-level design documentation for declarative config that captures the design principles, values, scope, constrains, rejected proposals. It would help

1. understanding what at are the best use cases and strategies involving declarative config
2. contributions and maintenance

If we have a prioritized list of design value then it would be easier to validate if things like env vars or presets support are following the design principles or not.

PS. I have not proposed the preset feature because I have some strong opinion. I created this issue because I think this is something that should be transparently discussed. Especially that the declarative configuration philosophy is different from the current env var based configuration philosophy in Java and .NET Automatic Instrumentation. I think that this shift is good, but it should be clearly documented to both users and contributors.

[nrcventura]

I have seen problems on both sides of this discussion, and it comes down to people generally leaving their file or environment variable based configuration alone between upgrades of their auto instrumentation solution. Once they have a configuration working they tend to leave it alone, and upgrade paths for auto instrumentation generally leave these configurations alone because they are defined and maintained by the person who defined them. This leads to two possible situations with auto-instrumentation as it supports more instrumentation out of the box with newer releases.

1. Someone expects the newly added instrumentation to be enabled by default.
2. Someone expects the newly added instrumentation to be disabled by default.

Whether declarative configuration supports presets or not seems to be related to which of these expectations OpenTelemetry wants to define as the expected behavior. Is there a strong opinion on which stance we should take? Or do we need some way to declaratively opt-in to one behavior or another?

On the dotnet side, I think there has been a strong precedent from multiple auto instrumentation solutions from different vendors, including OpenTelemetry, where new instrumentation (as it is added to auto instrumentation) is enabled by default. The current version of declarative configuration, seems to make that precedent harder to follow, so it would be good to know if this is intentional, and have a place where we can see the documented rationale for it.

[jack-berg]

I think that we lack high-level design documentation for declarative config that captures the design principles, values, scope, constrains, rejected proposals.

The schema modeling rules section is supposed to cover some of what you're describing here. I don't think "what you see is what you get" philosophy is listed anywhere tho. I'll make a note to document that.

@nrcventura a couple of weeks ago we were discussing this in the spec SIG. I'll try to recap key parts of the discussions here.

Currently the spec has boolean naming guidance which we follow in declarative config which states that properties should be named enabled / disabled such that a default value of false aligns with the target default. This has been the point of a lot of debate, but the point I really care about is this guidance blows up with instrumentation, which evolves from experimental to stable:

# initially when instrumentation foo is experimental and is disabled by default
foo:
  enabled: false # property is named enabled such that a default of false aligns with the target behavior
---
# later when instrumentation foo is stable and is enabled by default
foo:
  disabled: false # property is named disabled such that a default value of false aligns with the target behavior

Each and every instrumentation which evolves from experimental to stable ends up with both enabled and disabled. Yikes.

Now let's take the other side and align on naming the property enabled all the time:

# initially when instrumentation foo is experimental and is disabled by default
foo:
  enabled: false # property is named enabled and defaults to false
---
# later when instrumentation foo is stable and is enabled by default
foo:
  enabled: true # property is named enabled and defaults to false

Now we have default values which switch from false to true when an instrumentation stabilizes, which is a breaking change. Note we're allowed to make breaking changes until a type is stable, but still we'd rather not. So this doesn't fare much better.

A solution was discussed where we introduce a new higher order property, which dictates which instrumentations are enabled based on the value of the property and the stability of each instrumentation. It would work something like:

instrumentation/development:
  general:
    # dictates the stability threshold controlling which instrumentations are enabled by default
    default_instrumentation_stability_threshold: stable # or beta, alpha, etc
  java:
    # a stable instrumentation which is enabled by default
    foo:
    # an experimental instrumentation that we enable by flipping enabled
    bar:
      enabled: true

This solves the problems I mentioned above: We don't have to change from enabled to disabled as instrumentations stabilize. We don't have to change a property default value as instrumentation stabilizes.

It solves part of the problem you express. A user upgrading to a new version of an auto instrumentation agent will get new instrumentations enabled without needing to touch their config file. However, as you note, the user who expects all new instrumentations to be disabled until opted in will be mad. But I think this second user is wrong: If you set default_instrumentation_stability_threshold: stable, its unreasonable to be mad about stable instrumentations to be enabled by default. If you set default_instrumentation_stability_threshold: experimental, then you like to live life on the edge and should read the release notes, but you shouldn't be mad if new experimental instrumentations are added and enabled by default.

Vendors with auto instrumentation distributions can tip the scales by doing pre-processing on configuration files to do things like always enable some set of experimental instrumentations regardless of default_instrumentation_stability_threshold. Of course this violates the WYSIWYG principle, but vendors are going to vendor.

[nrcventura]

@jack-berg Your explanation is really helpful in understanding even more of the upgrading scenarios that should be considered.

One part that I don't quite understand yet is.

A user upgrading to a new version of an auto instrumentation agent will get new instrumentations enabled without needing to touch their config file.

In that case if someone starts with the following configuration.

instrumentation/development:
  general:
    # dictates the stability threshold controlling which instrumentations are enabled by default
    default_instrumentation_stability_threshold: stable # or beta, alpha, etc
  java:
    # a stable instrumentation which is enabled by default
    foo:
    # an experimental instrumentation that we enable by flipping enabled
    bar:
      enabled: true

And now they upgrade straight to a version that supports a stable buzz instrumentation without changing this file.

It's my current understanding that if buzz is not added to the configuration file, it should not be enabled. But perhaps that's my misunderstanding of how this is supposed to work. The part that I think needs clarification for me is, what should the behavior of not including buzz in the configuration file be, and is that dependent on the "default" value of the enabled setting for that instrumentation. In this example, if buzz is not defined in the file and the "default" enabled value is true, should the buzz instrumentation be enabled? Is the expected behavior different if the "default" enabled value is false?

[jack-berg]

It's my current understanding that if buzz is not added to the configuration file, it should not be enabled. But perhaps that's my misunderstanding of how this is supposed to work.

No in this idea, you wouldn't need to add buzz to the file. I see how the fact that I added the stable foo to the file is confusing. I added it for demonstration purposes not because I think it should be required.