Add test coverage to CI workflow (#2690)

* Add test coverage to workflow

* Add diff_cover package

* Update worklow

* Update CONTRIBUTING.md

Author: pamelafox

.github/workflows/python-test.yaml

@@ -29,6 +29,9 @@ jobs:
         node_version: ["20.14", "22"]
     steps:
         - uses: actions/checkout@v5
+          with:
+            # Fetch full history so diff-cover can compute a merge base with origin/main
+            fetch-depth: 0
         - name: Install uv
           uses: astral-sh/setup-uv@v6
           with:
@@ -61,7 +64,14 @@ jobs:
           run: black . --check --verbose
         - name: Run Python tests
           if: runner.os != 'Windows'
-          run: pytest -s -vv --cov --cov-fail-under=89
+          run: pytest -s -vv --cov --cov-report=xml --cov-fail-under=89
+        - name: Check diff coverage
+          if: runner.os != 'Windows'
+          run: |
+            BASE_REF="${{ github.base_ref }}"
+            if [ -z "$BASE_REF" ]; then BASE_REF="main"; fi
+            git fetch origin "$BASE_REF:refs/remotes/origin/$BASE_REF"
+            diff-cover coverage.xml --compare-branch="origin/$BASE_REF" --fail-under=90
         - name: Run E2E tests with Playwright
           id: e2e
           if: runner.os != 'Windows'

.gitignore

@@ -54,6 +54,7 @@ coverage.xml
 .hypothesis/
 .pytest_cache/
 cover/
+coverage_report.html
 
 # Translations
 *.mo

CONTRIBUTING.md

@@ -17,8 +17,9 @@ contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additio
 - [Running unit tests](#running-unit-tests)
 - [Running E2E tests](#running-e2e-tests)
 - [Code style](#code-style)
-- [Adding new azd environment variables](#adding-new-azd-environment-variables)
-- [Adding new UI strings](#adding-new-ui-strings)
+- [Adding new features](#adding-new-features)
+  - [Adding new azd environment variables](#adding-new-azd-environment-variables)
+  - [Adding new UI strings](#adding-new-ui-strings)
 
 ## Submitting a Pull Request (PR)
 
@@ -62,10 +63,18 @@ Run the tests:
 python -m pytest
 ```
 
-Check the coverage report to make sure your changes are covered.
+If test snapshots need updating (and the changes are expected), you can update them by running:
 
 ```shell
-python -m pytest --cov
+python -m pytest --snapshot-update
+```
+
+Once tests are passing, generate a coverage report to make sure your changes are covered:
+
+```shell
+pytest --cov --cov-report=xml && \
+diff-cover coverage.xml --format html:coverage_report.html && \
+open coverage_report.html
 ```
 
 ## Running E2E tests
@@ -118,7 +127,15 @@ python -m black <path-to-file>
 
 If you followed the steps above to install the pre-commit hooks, then you can just wait for those hooks to run `ruff` and `black` for you.
 
-## Adding new azd environment variables
+## Adding new features
+
+We recommend using GitHub Copilot Agent mode when adding new features,
+as this project includes [.github/copilot-instructions.md](.github/copilot-instructions.md) file
+that instructs Copilot on how to generate code for common code changes.
+
+If you are not using Copilot Agent mode, consult both that file and suggestions below.
+
+### Adding new azd environment variables
 
 When adding new azd environment variables, please remember to update:
 
@@ -128,7 +145,7 @@ When adding new azd environment variables, please remember to update:
 1. [ADO pipeline](.azdo/pipelines/azure-dev.yml).
 1. [Github workflows](.github/workflows/azure-dev.yml)
 
-## Adding new UI strings
+### Adding new UI strings
 
 When adding new UI strings, please remember to update all translations.
 For any translations that you generate with an AI tool,

requirements-dev.txt

@@ -12,4 +12,5 @@ pytest-snapshot
 pre-commit
 pip-tools
 mypy==1.14.1
+diff_cover
 axe-playwright-python