fix: metrics not being flushed on every invocation (#45)

heitorlessa · Tom McCarthy · web-flow · commit afe74e5458c0 · 2020-05-29T17:57:46.000+01:00
* [Sync Master] 0.7.0 release (#22) * docs: add pypi badge * fix: add missing single_metric example; test var name * chore: pypi monthly download badge * chore: fix github badge typo * feat: add docs to CI * fix: CI attempt 2 * fix: CI attempt 3 * fix: CI attempt 3 * fix: CI attempt 4 * chore: clean up CI workflows * Decorator factory Feat: Create your own middleware (#17) * feat(utils): add decorator factory * improv: use partial to reduce complexity * improv: add error handling * chore: type hint * docs: include pypi downloads badge * feat: opt in to trace each middleware that runs * improv: add initial util tests * improv: test explicit and implicit trace_execution * improv: test decorator with params * chore: linting * docs: include utilities * improv: correct tests, dec_factory only for func * improv: make util name more explicit * improv: doc trace_execution, fix casting * docs: add limitations, improve syntax * docs: use new docs syntax * fix: remove middleware decorator from libs * feat: build docs in CI * chore: linting * fix: CI python-version type * chore: remove docs CI * chore: kick CI * chore: include build badge master branch * chore: refactor naming * fix: rearrange tracing tests * improv(tracer): toggle default auto patching * feat(tracer): retrieve registered class instance * fix(Makefile): make cov target more explicit * improv(Register): support multiple classes reg. * improv(Register): inject class methods correctly * docs: add how to reutilize Tracer * improv(tracer): test auto patch method * improv: address nicolas feedback * improv: update example to reflect middleware feat * fix: metric dimension in root blob * chore: version bump Co-authored-by: heitorlessa <lessa@amazon.co.uk> Co-authored-by: heitorlessa <lessa@amazon.co.uk> * feat: add algolia search for docs and api ref (#39) (#40) Signed-off-by: heitorlessa <lessa@amazon.co.uk> * fix: revert makefile build-docs-api Signed-off-by: heitorlessa <lessa@amazon.co.uk> * fix: metric_set reuse #43 Signed-off-by: heitorlessa <lessa@amazon.co.uk> * fix: clear metrics Signed-off-by: heitorlessa <lessa@amazon.co.uk> * fix: update serialize_metrics helper function to use MetricManager instead of Metrics * fix: clear metrics after lambda invocation #43 Signed-off-by: heitorlessa <lessa@amazon.co.uk> * improv: document metrics tests, remove redundants Signed-off-by: heitorlessa <lessa@amazon.co.uk> #43 * chore: linting Signed-off-by: heitorlessa <lessa@amazon.co.uk> * docs: add section to flush metrics manually Signed-off-by: heitorlessa <lessa@amazon.co.uk> * docs: include EMF Json object Signed-off-by: heitorlessa <lessa@amazon.co.uk> * chore: bump version 0.9.4 Signed-off-by: heitorlessa <lessa@amazon.co.uk> Co-authored-by: Tom McCarthy <thommcc@amazon.com>
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,11 @@
 # HISTORY 
 
+## May 29th
+
+**0.9.4**
+
+* **Metrics**: Bugfix - Metrics were not correctly flushed, and cleared on every invocation
+
 ## May 16th
 
 **0.9.3**
diff --git a/Makefile b/Makefile
@@ -33,9 +33,8 @@ build-docs:
 	@$(MAKE) build-docs-website
 	@$(MAKE) build-docs-api
 
-build-docs-api:
-	pip install pdoc3~=0.7.5
-	pdoc3 --html --output-dir dist/api/ ./aws_lambda_powertools --force
+build-docs-api: dev
+	poetry run pdoc --html --output-dir dist/api/ ./aws_lambda_powertools --force
 	mv dist/api/aws_lambda_powertools/* dist/api/
 	rm -rf dist/api/aws_lambda_powertools
 
diff --git a/aws_lambda_powertools/metrics/base.py b/aws_lambda_powertools/metrics/base.py
@@ -50,8 +50,8 @@ class MetricManager:
     """
 
     def __init__(self, metric_set: Dict[str, str] = None, dimension_set: Dict = None, namespace: str = None):
-        self.metric_set = metric_set or {}
-        self.dimension_set = dimension_set or {}
+        self.metric_set = metric_set if metric_set is not None else {}
+        self.dimension_set = dimension_set if dimension_set is not None else {}
         self.namespace = os.getenv("POWERTOOLS_METRICS_NAMESPACE") or namespace
         self._metric_units = [unit.value for unit in MetricUnit]
         self._metric_unit_options = list(MetricUnit.__members__)
@@ -116,7 +116,10 @@ def add_metric(self, name: str, unit: MetricUnit, value: Union[float, int]):
             logger.debug(f"Exceeded maximum of {MAX_METRICS} metrics - Publishing existing metric set")
             metrics = self.serialize_metric_set()
             print(json.dumps(metrics))
-            self.metric_set = {}
+
+            # clear metric set only as opposed to metrics and dimensions set
+            # since we could have more than 100 metrics
+            self.metric_set.clear()
 
     def serialize_metric_set(self, metrics: Dict = None, dimensions: Dict = None) -> Dict:
         """Serializes metric and dimensions set
diff --git a/aws_lambda_powertools/metrics/metrics.py b/aws_lambda_powertools/metrics/metrics.py
@@ -65,8 +65,15 @@ def do_something():
     _metrics = {}
     _dimensions = {}
 
-    def __init__(self, metric_set=None, dimension_set=None, namespace=None):
-        super().__init__(metric_set=self._metrics, dimension_set=self._dimensions, namespace=namespace)
+    def __init__(self):
+        self.metric_set = self._metrics
+        self.dimension_set = self._dimensions
+        super().__init__(metric_set=self.metric_set, dimension_set=self.dimension_set)
+
+    def clear_metrics(self):
+        logger.debug("Clearing out existing metric set from memory")
+        self.metric_set.clear()
+        self.dimension_set.clear()
 
     def log_metrics(self, lambda_handler: Callable[[Any, Any], Any] = None):
         """Decorator to serialize and publish metrics at the end of a function execution.
@@ -101,6 +108,7 @@ def decorate(*args, **kwargs):
                 response = lambda_handler(*args, **kwargs)
             finally:
                 metrics = self.serialize_metric_set()
+                self.clear_metrics()
                 logger.debug("Publishing metrics", {"metrics": metrics})
                 print(json.dumps(metrics))
 
diff --git a/docs/content/core/metrics.mdx b/docs/content/core/metrics.mdx
@@ -122,6 +122,25 @@ def lambda_handler(evt, ctx):
     ...
 ```
 
+## Flushing metrics manually
+
+If you prefer not to use `log_metrics` because you might want to encapsulate additional logic when doing so, you can manually flush and clear metrics as follows:
+
+```python:title=manual_metric_serialization.py
+import json
+from aws_lambda_powertools.metrics import Metrics, MetricUnit
+
+metrics = Metrics()
+metrics.add_metric(name="ColdStart", unit="Count", value=1)
+metrics.add_dimension(name="service", value="booking")
+
+# highlight-start
+your_metrics_object = metrics.serialize_metric_set() 
+metrics.clear_metrics()
+print(json.dumps(your_metrics_object))
+# highlight-end
+```
+
 ## Testing your code
 
 Use `POWERTOOLS_METRICS_NAMESPACE` env var when unit testing your code to ensure a metric namespace object is created, and your code doesn't fail validation.
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "aws_lambda_powertools"
-version = "0.9.3"
+version = "0.9.4"
 description = "Python utilities for AWS Lambda functions including but not limited to tracing, logging and custom metric"
 authors = ["Amazon Web Services"]
 classifiers=[
diff --git a/tests/functional/test_metrics.py b/tests/functional/test_metrics.py
diff --git a/tests/unit/test_tracing.py b/tests/unit/test_tracing.py
