feat: Add OpenTelemetry instrumentation #525

soapun · 2025-10-26T13:35:27Z

No description provided.

…hecks from pre-commit, fix taskiq-python#417

danfimov · 2025-11-04T08:40:26Z

docs/README.md

+      content: "97DC185FE0A2F5B123861F0790FDFB26" # pragma: allowlist secret
  - - meta
    - name: "yandex-verification"
-      content: "9b105f7c58cbc920"


Why pragma: allowlist secret is here?

As I see, we already exclude this file from detect-secrets pre-commit hook check. Is some other linter checks fails on this lines?

even with exclusion in .pre-commit-config.yaml, still seeing this without pragmas

Detect secrets...........................................................Failed - hook id: detect-secrets - exit code: 1 ERROR: Potential secrets about to be committed to git repo! Secret Type: Base64 High Entropy String Location: docs\README.md:17 Secret Type: Hex High Entropy String Location: docs\README.md:20 Secret Type: Hex High Entropy String Location: docs\README.md:23 Possible mitigations: - For information about putting your secrets in a safer place, please ask in #security - Mark false positives with an inline `pragma: allowlist secret` comment If a secret has already been committed, visit https://help.github.com/articles/removing-sensitive-data-from-a-repository

Thats strange...
I checked on my fork and anything seems to be fine:

(taskiq)  git pull Already up to date. (taskiq)  git st On branch master Your branch is up to date with 'origin/master'. nothing to commit, working tree clean (taskiq)  pre-commit install pre-commit installed at .git/hooks/pre-commit (taskiq)  pre-commit run detect-secrets --all-files Detect secrets...........................................................Passed

Are you sure that you run poetry install --all-extras && pre-commit install after rebase? I don't really know that else can be a problem

In any case - it's not a blocker to merge this MR. I will deal with it later if we don't find the root cause of this strange pre-commit hook behaviour.

I guess the root cause is me commiting from windows :)
On mac the exclusions are working as expected

rollbacked the file

danfimov · 2025-11-04T08:43:20Z

taskiq/package.py

@@ -0,0 +1,2 @@
+# for compatibility with opentelemetry-instrumentation
+_instruments = ("taskiq >= 0.11.19",)


Just wondering: do we need to update version in this file on every release?

No. It's a version from which we start supporting opentelemetry instrumentation

some examples
asyncpg
fastapi
celery

danfimov · 2025-11-04T08:51:16Z

tests/opentelemetry/test_tasks.py

+
+        self.assertEqual(result.return_value, {"key": "value"})
+
+    @pytest.mark.anyio


I enabled anyio_mode = "auto" flag for pytest. So tests doesn't need to explicitly say that they are async with pytest.mark.anyio marker.

You can just remove it and tests should work as they were)

danfimov · 2025-11-09T00:21:59Z

taskiq/instrumentation.py

+---
+"""
+
+from __future__ import annotations


Seems like everything works fine without this import. Maybe we can remove it?

As far as I can see, there are no other places in the taskiq repo where __future__ is used.

removed this

taskiq/instrumentation.py

s3rius · 2025-11-10T08:32:41Z

taskiq/instrumentation.py

+    broker = InMemoryBroker()
+
+    @broker.on_event(TaskiqEvents.WORKER_STARTUP)
+    async def startup(state: TaskiqState) -> None:


Can you do it in main? So you won't need to create to on_event.

s3rius · 2025-11-10T08:35:46Z

taskiq/middlewares/opentelemetry_middleware.py

+    if not span.is_recording():
+        return
+
+    for key in TASKIQ_CONTEXT_ATTRIBUTES:


Why do we track only particular labels? Let's just iterate over all available labels.

Also, I think that we also want to extract args and kwargs to set it on the span. For better tracing. Instead of passing labels, you can pass TaskiqMessage itself and get everthing from it.

s3rius · 2025-11-10T08:41:14Z

taskiq/middlewares/opentelemetry_middleware.py

+        ctx = retrieve_context(message, is_publish=True)
+
+        if ctx is None:
+            logger.warning("no existing span found for task_id=%s", message.task_id)


Suggested change

logger.warning("no existing span found for task_id=%s", message.task_id)

logger.debug("no existing span found for task_id=%s", message.task_id)

s3rius · 2025-11-10T08:41:52Z

taskiq/middlewares/opentelemetry_middleware.py

+            span.set_attribute(_TASK_NAME_KEY, message.task_name)
+            set_attributes_from_context(span, message.labels)
+
+        activation = trace.use_span(span, end_on_exit=True)


Here's a small quetsion about spans. It seems like you use spans as events.

From my understanding it makes sense to only have 1 single span for task execution. And inside of that span we can use events to show what was going on during the exeuction.

I'm not sure if it's the best way. I need to try it out.

This PR is heavily inspired by celery and remoulade instrumentations. Both of them create separate span for task sending. I guess to fit producer-consumer semantics

add opentelemetry instrumentation, suppress false-positive security c…

fbe4b78

…hecks from pre-commit, fix taskiq-python#417

danfimov reviewed Nov 4, 2025

View reviewed changes

soapun added 3 commits November 8, 2025 19:01

Merge branch 'master' into feat/add-otel-instrumentation

b09c4f1

delete pytest.mark.anyio

41eef7f

upd version from which otel is supported

78aa5ee

danfimov reviewed Nov 9, 2025

View reviewed changes

taskiq/instrumentation.py Outdated Show resolved Hide resolved

soapun added 2 commits November 9, 2025 13:05

delete pragmas

2d856ee

delete unused statements

e7614c9

s3rius requested changes Nov 10, 2025

View reviewed changes

		@@ -0,0 +1,2 @@
		# for compatibility with opentelemetry-instrumentation
		_instruments = ("taskiq >= 0.11.19",)


		self.assertEqual(result.return_value, {"key": "value"})

		@pytest.mark.anyio

	logger.warning("no existing span found for task_id=%s", message.task_id)
	logger.debug("no existing span found for task_id=%s", message.task_id)

Uh oh!

feat: Add OpenTelemetry instrumentation #525

Are you sure you want to change the base?

feat: Add OpenTelemetry instrumentation #525

Uh oh!

Conversation

soapun commented Oct 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danfimov Nov 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

danfimov Nov 9, 2025 •

edited

Loading