Skip to content

Commit 15b6948

Browse files
lrafeeiTimPansinodependabot[bot]mergify[bot]umaannamalai
authored
Merge main into develop-11.0.0 (#1503)
* Enable coverage for motor instrumentation (#1487) * Drop CI Image Build Caching (#1493) * Add deprecation warning for modules (#1490) * Add deprecation warning for modules * Megalinter fixes * Bump github/codeql-action in the github_actions group (#1492) Bumps the github_actions group with 1 update: [github/codeql-action](https://github.com/github/codeql-action). Updates `github/codeql-action` from 3.30.1 to 3.30.3 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@f1f6e5f...192325c) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 3.30.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github_actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add Additional Delay to Test Startup Fixture (#1494) * Add additional delay to test startup fixture * Fix assertion error message --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Temporarily remove pypy310 test from pymysql and aiomysql (#1501) * Remove pymysql pypy310 test * Remove aiomysql pypy310 test * Falcon sanic testing fix (#1495) * Remove unsupported Sanic from tests * Remove unsupported Falcon from tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Camunda Pyzeebe Instrumentation (#1385) * add initial pyzeebe instrumentation/tests * second attempt at tests (failing) * updated tests - 1/2 passing locally * combine function_trace client tests * fix: py agent team feedback #1 * fix: pyzeebe tests * chore: ruff formatting * fix: review updates #2 * fix: next round of updates * fix: more updates based on feedback * fix: resource parameter capture * chore: ruff lint fixes * fix: no txn tests;add resourceCount attr back --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> * Format with ruff (#1505) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Keagan Peet <nobrac34@gmail.com>
1 parent bdcf2e8 commit 15b6948

File tree

14 files changed

+566
-17
lines changed

14 files changed

+566
-17
lines changed

.github/workflows/build-ci-image.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,8 +89,6 @@ jobs:
8989
platforms: ${{ matrix.platform }}
9090
labels: ${{ steps.meta.outputs.labels }}
9191
outputs: type=image,name=ghcr.io/${{ steps.image-name.outputs.IMAGE_NAME }},push-by-digest=true,name-canonical=true,push=true
92-
cache-from: type=gha,scope=build-${{ matrix.cache_tag }}
93-
cache-to: type=gha,scope=build-${{ matrix.cache_tag }}
9492

9593
- name: Export Digest
9694
run: |

.github/workflows/trivy.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,6 @@ jobs:
6161

6262
- name: Upload Trivy scan results to GitHub Security tab
6363
if: ${{ github.event_name == 'schedule' }}
64-
uses: github/codeql-action/upload-sarif@f1f6e5f6af878fb37288ce1c627459e94dbf7d01 # 3.30.1
64+
uses: github/codeql-action/upload-sarif@192325c86100d080feab897ff886c34abd4c83a3 # 3.30.3
6565
with:
6666
sarif_file: "trivy-results.sarif"

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@ share/python-wheels/
3434
MANIFEST
3535
_version.py
3636
version.txt
37+
version.py
38+
_version.py
3739

3840
# PyInstaller
3941
# Usually these files are written by a python script from a template

codecov.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@ ignore:
2323
- "newrelic/hooks/database_psycopg2ct.py"
2424
- "newrelic/hooks/datastore_aioredis.py"
2525
- "newrelic/hooks/datastore_aredis.py"
26-
- "newrelic/hooks/datastore_motor.py"
2726
- "newrelic/hooks/datastore_pyelasticsearch.py"
2827
- "newrelic/hooks/external_dropbox.py"
2928
- "newrelic/hooks/external_facepy.py"

newrelic/config.py

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import threading
2121
import time
2222
import traceback
23+
from datetime import datetime, timezone
2324
from pathlib import Path
2425

2526
import newrelic.api.application
@@ -54,6 +55,8 @@
5455

5556
_logger = logging.getLogger(__name__)
5657

58+
DEPRECATED_MODULES = {"aioredis": datetime(2022, 2, 22, 0, 0, tzinfo=timezone.utc)}
59+
5760

5861
def _map_aws_account_id(s):
5962
return newrelic.core.config._map_aws_account_id(s, _logger)
@@ -1048,6 +1051,18 @@ def _module_import_hook(target, module, function):
10481051
def _instrument(target):
10491052
_logger.debug("instrument module %s", ((target, module, function),))
10501053

1054+
# Deprecation warning for archived/unsupported modules
1055+
library_name = target.__package__.split(".")[0]
1056+
1057+
if library_name in DEPRECATED_MODULES:
1058+
_logger.warning(
1059+
"%(module)s has been archived by the developers "
1060+
"and has not been supported since %(date)s. %(module)s "
1061+
"support will be removed from New Relic in a future "
1062+
"release.",
1063+
{"module": library_name, "date": DEPRECATED_MODULES[library_name].strftime("%B %d, %Y")},
1064+
)
1065+
10511066
try:
10521067
instrumented = target._nr_instrumented
10531068
except AttributeError:
@@ -4054,6 +4069,12 @@ def _process_module_builtin_defaults():
40544069
"newrelic.hooks.framework_azurefunctions",
40554070
"instrument_azure_functions_worker_dispatcher",
40564071
)
4072+
_process_module_definition(
4073+
"pyzeebe.client.client", "newrelic.hooks.external_pyzeebe", "instrument_pyzeebe_client_client"
4074+
)
4075+
_process_module_definition(
4076+
"pyzeebe.worker.job_executor", "newrelic.hooks.external_pyzeebe", "instrument_pyzeebe_worker_job_executor"
4077+
)
40574078

40584079

40594080
def _process_module_entry_points():

newrelic/core/attribute.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,12 @@
100100
"response.headers.contentType",
101101
"response.status",
102102
"server.address",
103+
"zeebe.client.bpmnProcessId",
104+
"zeebe.client.messageName",
105+
"zeebe.client.correlationKey",
106+
"zeebe.client.messageId",
107+
"zeebe.client.resourceCount",
108+
"zeebe.client.resourceFile",
103109
}
104110

105111
MAX_NUM_USER_ATTRIBUTES = 128

newrelic/hooks/external_pyzeebe.py

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# Copyright 2010 New Relic, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
16+
import logging
17+
18+
from newrelic.api.application import application_instance
19+
from newrelic.api.function_trace import FunctionTrace
20+
from newrelic.api.transaction import current_transaction
21+
from newrelic.api.web_transaction import WebTransaction
22+
from newrelic.common.object_wrapper import wrap_function_wrapper
23+
24+
_logger = logging.getLogger(__name__)
25+
26+
CLIENT_ATTRIBUTES_DEPLOY_RESOURCE_LOG_MSG = "Exception occurred in PyZeebe instrumentation: Failed to extract resource count/file for method `deploy_resource`. Report this issue to New Relic support."
27+
28+
29+
# Adds client method params as txn or span attributes
30+
def _add_client_input_attributes(method_name, trace, args, kwargs):
31+
bpmn_id = extract_agent_attribute_from_methods(
32+
args, kwargs, method_name, ("run_process", "run_process_with_result"), "bpmn_process_id", 0
33+
)
34+
if bpmn_id:
35+
trace._add_agent_attribute("zeebe.client.bpmnProcessId", bpmn_id)
36+
37+
msg_name = extract_agent_attribute_from_methods(args, kwargs, method_name, ("publish_message"), "name", 0)
38+
if msg_name:
39+
trace._add_agent_attribute("zeebe.client.messageName", msg_name)
40+
41+
correlation_key = extract_agent_attribute_from_methods(
42+
args, kwargs, method_name, ("publish_message"), "correlation_key", 1
43+
)
44+
if correlation_key:
45+
trace._add_agent_attribute("zeebe.client.correlationKey", correlation_key)
46+
47+
message_id = extract_agent_attribute_from_methods(args, kwargs, method_name, ("publish_message"), "message_id", 4)
48+
if message_id:
49+
trace._add_agent_attribute("zeebe.client.messageId", message_id)
50+
51+
resource = extract_agent_attribute_from_methods(args, {}, method_name, ("deploy_resource"), None, 0)
52+
if resource:
53+
try:
54+
trace._add_agent_attribute("zeebe.client.resourceFile", resource)
55+
trace._add_agent_attribute("zeebe.client.resourceCount", len(list(args)))
56+
except Exception:
57+
_logger.warning(CLIENT_ATTRIBUTES_DEPLOY_RESOURCE_LOG_MSG, exc_info=True)
58+
59+
60+
def extract_agent_attribute_from_methods(args, kwargs, method_name, methods, param, index):
61+
try:
62+
if method_name in methods:
63+
value = kwargs.get(param)
64+
if not value and args and len(args) > index:
65+
value = args[index]
66+
return value
67+
except Exception:
68+
_logger.warning(
69+
"Exception occurred in PyZeebe instrumentation: failed to extract %s from %s. Report this issue to New Relic support.",
70+
param,
71+
method_name,
72+
exc_info=True,
73+
)
74+
75+
76+
# Async wrapper that instruments router/worker annotations`
77+
async def _nr_wrapper_execute_one_job(wrapped, instance, args, kwargs):
78+
job = args[0] if args else kwargs.get("job")
79+
process_id = getattr(job, "bpmn_process_id", None) or "UnknownProcess"
80+
task_type = getattr(job, "type", None) or "UnknownType"
81+
txn_name = f"{process_id}/{task_type}"
82+
83+
with WebTransaction(application_instance(), txn_name, group="ZeebeTask") as txn:
84+
if job is not None:
85+
if hasattr(job, "key"):
86+
txn.add_custom_attribute("zeebe.job.key", job.key)
87+
if hasattr(job, "type"):
88+
txn.add_custom_attribute("zeebe.job.type", job.type)
89+
if hasattr(job, "bpmn_process_id"):
90+
txn.add_custom_attribute("zeebe.job.bpmnProcessId", job.bpmn_process_id)
91+
if hasattr(job, "process_instance_key"):
92+
txn.add_custom_attribute("zeebe.job.processInstanceKey", job.process_instance_key)
93+
if hasattr(job, "element_id"):
94+
txn.add_custom_attribute("zeebe.job.elementId", job.element_id)
95+
96+
return await wrapped(*args, **kwargs)
97+
98+
99+
# Async wrapper that instruments a ZeebeClient method.
100+
def _nr_client_wrapper(method_name):
101+
async def _client_wrapper(wrapped, instance, args, kwargs):
102+
txn = current_transaction()
103+
if not txn:
104+
return await wrapped(*args, **kwargs)
105+
106+
with FunctionTrace(name=method_name, group="ZeebeClient") as trace:
107+
_add_client_input_attributes(method_name, trace, args, kwargs)
108+
return await wrapped(*args, **kwargs)
109+
110+
return _client_wrapper
111+
112+
113+
# Instrument JobExecutor.execute_one_job to create a background transaction per job (invoked from @router.task or @worker.task annotations)
114+
def instrument_pyzeebe_worker_job_executor(module):
115+
if hasattr(module, "JobExecutor"):
116+
wrap_function_wrapper(module, "JobExecutor.execute_one_job", _nr_wrapper_execute_one_job)
117+
118+
119+
# Instrument ZeebeClient methods to trace client calls.
120+
def instrument_pyzeebe_client_client(module):
121+
target_methods = ("run_process", "run_process_with_result", "deploy_resource", "publish_message")
122+
123+
for method_name in target_methods:
124+
if hasattr(module, "ZeebeClient"):
125+
if hasattr(module.ZeebeClient, method_name):
126+
wrap_function_wrapper(module, f"ZeebeClient.{method_name}", _nr_client_wrapper(method_name))

tests/external_pyzeebe/_mocks.py

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Copyright 2010 New Relic, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
16+
from types import SimpleNamespace
17+
18+
from pyzeebe.grpc_internals.zeebe_adapter import ZeebeAdapter
19+
20+
# Dummy response objects with only required fields
21+
DummyCreateProcessInstanceResponse = SimpleNamespace(process_instance_key=12345)
22+
23+
DummyCreateProcessInstanceWithResultResponse = SimpleNamespace(
24+
process_instance_key=45678, variables={"result": "success"}
25+
)
26+
27+
DummyDeployResourceResponse = SimpleNamespace(key=67890, deployments=[], tenant_id=None)
28+
29+
DummyPublishMessageResponse = SimpleNamespace(key=99999, tenant_id=None)
30+
31+
32+
# Dummy RPC stub coroutines
33+
async def dummy_create_process_instance(
34+
self,
35+
bpmn_process_id: str,
36+
variables: dict = None, # noqa: RUF013
37+
version: int = -1,
38+
tenant_id: str = None, # noqa: RUF013
39+
):
40+
"""Simulate ZeebeAdapter.create_process_instance"""
41+
return DummyCreateProcessInstanceResponse
42+
43+
44+
async def dummy_create_process_instance_with_result(
45+
self,
46+
bpmn_process_id: str,
47+
variables: dict = None, # noqa: RUF013
48+
version: int = -1,
49+
timeout: int = 0,
50+
variables_to_fetch=None,
51+
tenant_id: str = None, # noqa: RUF013
52+
):
53+
"""Simulate ZeebeAdapter.create_process_instance_with_result"""
54+
return DummyCreateProcessInstanceWithResultResponse
55+
56+
57+
async def dummy_deploy_resource(*resource_file_path: str, tenant_id: str = None): # noqa: RUF013
58+
"""Simulate ZeebeAdapter.deploy_resource"""
59+
# Create dummy deployment metadata for each provided resource path
60+
deployments = [
61+
SimpleNamespace(
62+
resource_name=str(path),
63+
bpmn_process_id="dummy_process",
64+
process_definition_key=123,
65+
version=1,
66+
tenant_id=tenant_id if tenant_id is not None else None,
67+
)
68+
for path in resource_file_path
69+
]
70+
# Create a dummy response with a list of deployments
71+
return SimpleNamespace(
72+
deployment_key=333333, deployments=deployments, tenant_id=tenant_id if tenant_id is not None else None
73+
)
74+
75+
76+
async def dummy_publish_message(
77+
self,
78+
name: str,
79+
correlation_key: str,
80+
variables: dict = None, # noqa: RUF013
81+
time_to_live_in_milliseconds: int = 60000,
82+
message_id: str = None, # noqa: RUF013
83+
tenant_id: str = None, # noqa: RUF013
84+
):
85+
"""Simulate ZeebeAdapter.publish_message"""
86+
# Return the dummy response (contains message key)
87+
return SimpleNamespace(key=999999, tenant_id=tenant_id if tenant_id is not None else None)
88+
89+
90+
async def dummy_complete_job(self, job_key: int, variables: dict):
91+
"""Simulate JobExecutor.complete_job"""
92+
self._last_complete = {"job_key": job_key, "variables": variables}
93+
return None
94+
95+
96+
class DummyZeebeAdapter(ZeebeAdapter):
97+
"""Simulate a ZeebeAdapter so JobExecutor can be instatiated w/o gRPC channel"""
98+
99+
def __init__(self):
100+
self.completed_job_key = None
101+
self.completed_job_vars = None
102+
103+
async def complete_job(self, job_key: int, variables: dict):
104+
self.completed_job_key = job_key
105+
self.completed_job_vars = variables
106+
return None

tests/external_pyzeebe/conftest.py

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Copyright 2010 New Relic, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
from testing_support.fixture.event_loop import event_loop as loop
16+
from testing_support.fixtures import collector_agent_registration_fixture, collector_available_fixture
17+
18+
_default_settings = {
19+
"package_reporting.enabled": False, # Turn off package reporting for testing as it causes slow downs.
20+
"transaction_tracer.explain_threshold": 0.0,
21+
"transaction_tracer.transaction_threshold": 0.0,
22+
"transaction_tracer.stack_trace_threshold": 0.0,
23+
"debug.log_data_collector_payloads": True,
24+
"debug.record_transaction_failure": True,
25+
}
26+
27+
collector_agent_registration = collector_agent_registration_fixture(
28+
app_name="Python Agent Test (external_pyzeebe)", default_settings=_default_settings
29+
)

tests/external_pyzeebe/test.bpmn

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<definitions
3+
xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
4+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
5+
xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI"
6+
xmlns:omgdc="http://www.omg.org/spec/DD/20100524/DC"
7+
xmlns:omgdi="http://www.omg.org/spec/DD/20100524/DI"
8+
targetNamespace="http://example.com/bpmn"
9+
xsi:schemaLocation="http://www.omg.org/spec/BPMN/20100524/MODEL BPMN20.xsd">
10+
11+
<!-- Define the process with a unique id and name -->
12+
<process id="dummyProcess" name="Dummy Process" isExecutable="true">
13+
<!-- Start Event -->
14+
<startEvent id="StartEvent_1" name="Start"/>
15+
16+
<!-- A simple Service Task representing work -->
17+
<serviceTask id="ServiceTask_1" name="Perform Work"/>
18+
19+
<!-- End Event -->
20+
<endEvent id="EndEvent_1" name="End"/>
21+
22+
<!-- Sequence Flows connecting Start → Service Task → End -->
23+
<sequenceFlow id="Flow_1" sourceRef="StartEvent_1" targetRef="ServiceTask_1"/>
24+
<sequenceFlow id="Flow_2" sourceRef="ServiceTask_1" targetRef="EndEvent_1"/>
25+
</process>
26+
27+
<!-- (Optional) BPMNDiagram section can be added for graphical layout, but omitted here -->
28+
</definitions>

0 commit comments

Comments
 (0)