The http.route in metric#http_server_request_duration_seconds_bucket shows the {jobName} placeholder instead of the actual job name.

[vijaybtech13]

Describe the bug

The SpringBoot application has been instrumented with the OpenTelemetry Java agent. A custom actuator endpoint, /actuator/jobs/DAILY_JOB, has been implemented. However, the generated metric lacks the actual job name in the http.route field.

Metric:
http_server_request_duration_seconds_bucket{
http_route="/actuator/jobs/{jobName}",
http_request_method="POST",
…
}

Steps to reproduce

Access the custom actuator endpoint /actuator/jobs/DAILY_JOB.
Review the generated metric.

Expected behavior

There are multiple actuator jobs in the application, making it essential to include the job name in the metric to differentiate between them. How can I obtain the job name in the metric?

Metric:
http_server_request_duration_seconds_bucket{
http_route="/actuator/jobs/DAILY_JOB",
http_request_method="POST",
job_name="DAILY_JOB"
…
}

Actual behavior

The generated metric lacks the actual job name in the http.route field.

Metric:
http_server_request_duration_seconds_bucket{
http_route="/actuator/jobs/{jobName}",
http_request_method="POST",
…
}

Javaagent or library instrumentation version

Paketo Buildpack for OpenTelemetry 2.17.0 , OpenTelemetry Java Agent 2.21.0

Environment

SpringBoot - 3.4.10
JDK - 21
Server Information - OS: Ubuntu focal (20.04), Kernel: 5.15.0-1077-aws, Platform: Amazon EC2

Additional context

Tried approach:
I attempted the following steps to display the actual job name in the metric, but did not achieve the expected result.

Step 1 - Set span attribute in the controller
@component
@endpoint(id = "jobs")
public class JobsEndpoint {

@ReadOperation
public String runJob(@Selector String jobName) {

    Span span = Span.current();
    span.setAttribute("job.name", jobName);
span.setAttribute("http.route", "/actuator/jobs/" + jobName);

    return "Job " + jobName + " executed";
}

}

Step 2 - Added the following otel env variables
OTEL_JAVAAGENT_ENABLED: "true"
OTEL_METRICS_EXPORTER: otlp
OTEL_ATTRIBUTE_FILTER: metrics:job.name
OTEL_SEMCONV_STABILITY_OPT_IN: http
OTEL_JAVAAGENT_EXPERIMENTAL_SPAN_METRICS_ENABLED: true

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[vijaybtech13]

Hi Team - Any update on this case?

[laurit]

Route is determined based on the request path mapping that matched the request. If you want the route to be /actuator/jobs/DAILY_JOB then one way would be to add a mapping that uses this path. Another options would be to add dependency to https://central.sonatype.com/artifact/io.opentelemetry.instrumentation/opentelemetry-instrumentation-api and call

opentelemetry-java-instrumentation/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/semconv/http/HttpServerRoute.java

Line 60 in 6d6915e

public static void update(Context context, HttpServerRouteSource source, String httpRoute) {

with the desired route. One more option we could consider would be to introduces declarative config based customization rules similar to what was added for http client url.template in #15217

[vijaybtech13]

Hi @laurit - Thanks for your suggestion.
If I understood correctly, below are your 3 different suggestions.

1. Change the API implementation to a REST Controller instead of a Custom Actuator Endpoint.
2. Call HttpServerRoute.update with the desired route.
3. Introduce declarative config-based customization rules in a future OpenTelemetry agent version by the core team.

[vijaybtech13]

I have tried the second suggestion. Despite adding the dependency and calling the update method, the issue is still not resolved. Did I miss anything?
Here are the changes made:

Endpoint class:
@component
@endpoint(id = "jobs")
public class JobsEndpoint {
private final Logger logger;
List jobs;
List jobNames;

@Autowired
public JobsEndpoint(Logger logger, List<RunnableJob> jobs) {
    this.logger = logger;
    this.jobs = jobs;
    this.jobNames = jobs.stream().map(RunnableJob::getName).collect(Collectors.toList());
}

@WriteOperation(produces = MediaType.APPLICATION_JSON)
public void executeJob(@Selector String jobName) {
    logger.info("Executing job: {}", jobName);
    //Overriding http.route with the actual jobName
    HttpServerRoute.update(Context.current(), HttpServerRouteSource.CONTROLLER, "/actuator/jobs/" + jobName);

    //Business logic
}

}

Gradle change:
implementation "io.opentelemetry.instrumentation:opentelemetry-instrumentation-api:2.21.0"

Generated Metric:
http_server_request_duration_seconds_bucket
{
http_request_method="POST",
http_response_status_code="204",
http_route="/actuator/jobs/{jobName}"
}

[laurit]

Probably the issue is that route value can only be overwritten by a more specific route. If spring instrumentation has already set the route at CONTROLLER level then you can't overwrite it with your own route at the same level. Try

opentelemetry-java-instrumentation/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/semconv/http/HttpServerRouteSource.java

Line 32 in e5d150d

NESTED_CONTROLLER(4, /* useFirst= */ false);

that one always overwrites.