[beta] Python Profiling support (#244)

* feat: Python Profiler WIP

* feat: start on cold start, stop never

* breaking: bump dd trace to 1.2.1

* feat: Add major version notes

* fix: readme header size

* fix: Fix test using old method. Describe change in changes list

* fix: use tracer instead of correlation ids

* feat: Add the profiler

* fix: remove debug

* feat: Remove profiling comment line

* feat: Go to ddtrace 1.4. Conditionally import profiler.

* fix: remove duplicate declaration of profiling_env_var

* feat: Black, with the right version

* feat: Readme on profiler beta period

* fix: Remove dev changes used to package dd-trace locally

* feat: Remove == from if True

* feat: Bump layer size maximums

* feat: Remove python36 integration tests, as we can no longer create stacks with this version of python

* feat: Update snapshots

* feat: More python 3.6 deprecation

* feat: Readme update - dropping python36

Author: astuyve

Dockerfile

@@ -17,7 +17,3 @@ RUN find ./python/lib/$runtime/site-packages -name \*.pyc -delete
 # Remove botocore (40MB) to reduce package size. aws-xray-sdk
 # installs it, while it's already provided by the Lambda Runtime.
 RUN rm -rf ./python/lib/$runtime/site-packages/botocore*
-
-# Remove profiling (7MB) to reduce package size.
-# Continous profiling is not yet supported anyway.
-RUN rm -rf ./python/lib/$runtime/site-packages/ddtrace/profiling

README.md

@@ -24,11 +24,16 @@ When opening an issue, include the Datadog Lambda Library version, Python versio
 
 You can also open an issue for a feature request.
 
+## Lambda Profiling Beta
+
+Datadog's Continuous Profiler is now available in beta for Python. This optional feature is enabled by setting the `DD_PROFILING_ENABLED` environment variable to `True`. During the beta period, profiling is available at no cost.
+
 ## Major Version Notes
 
-### 4.x
+### 4.x / Layer version 61+
 
-- `dd-trace` upgraded from 0.61 to 1.2, full release notes are available [here](https://ddtrace.readthedocs.io/en/stable/release_notes.html#v1-0-0)
+- Python3.6 support has been [deprecated](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html) by AWS, and support removed from this library.
+- `dd-trace` upgraded from 0.61 to 1.4, full release notes are available [here](https://ddtrace.readthedocs.io/en/stable/release_notes.html#v1-0-0)
   - `get_correlation_ids()` has been changed to `get_log_correlation_context()`, which now returns a dictionary containing the active `span_id`, `trace_id`, as well as `service` and `env`.
 
 ## Contributing

datadog_lambda/wrapper.py

@@ -36,11 +36,17 @@
 from datadog_lambda.trigger import extract_trigger_tags, extract_http_status_code_tag
 from datadog_lambda.tag_object import tag_object
 
+profiling_env_var = os.environ.get("DD_PROFILING_ENABLED", "false").lower() == "true"
+if profiling_env_var:
+    from ddtrace.profiling import profiler
+
 logger = logging.getLogger(__name__)
 
 dd_capture_lambda_payload_enabled = (
     os.environ.get("DD_CAPTURE_LAMBDA_PAYLOAD", "false").lower() == "true"
 )
+service_env_var = os.environ.get("DD_SERVICE", "DefaultServiceName")
+env_env_var = os.environ.get("DD_ENV", None)
 
 """
 Usage:
@@ -111,7 +117,8 @@ def __init__(self, func):
                 os.environ.get("DD_TRACE_MANAGED_SERVICES", "true").lower() == "true"
             )
             self.response = None
-
+            if profiling_env_var:
+                self.prof = profiler.Profiler(env=env_env_var, service=service_env_var)
             if self.extractor_env:
                 extractor_parts = self.extractor_env.rsplit(".", 1)
                 if len(extractor_parts) == 2:
@@ -181,7 +188,8 @@ def _before(self, event, context):
                 )
             else:
                 set_correlation_ids()
-
+            if profiling_env_var and is_cold_start():
+                self.prof.start(stop_on_exit=False, profile_children=True)
             logger.debug("datadog_lambda_wrapper _before() done")
         except Exception:
             traceback.print_exc()

poetry.lock

@@ -7,9 +7,9 @@
 
 # Compares layer size to threshold, and fails if below that threshold
 
-# 5 mb size limit
-MAX_LAYER_COMPRESSED_SIZE_KB=$(expr 5 \* 1024) 
-MAX_LAYER_UNCOMPRESSED_SIZE_KB=$(expr 16 \* 1024) 
+# 6 mb size limit
+MAX_LAYER_COMPRESSED_SIZE_KB=$(expr 6 \* 1024)
+MAX_LAYER_UNCOMPRESSED_SIZE_KB=$(expr 24 \* 1024)
 
 
 LAYER_FILES_PREFIX="datadog_lambda_py"