Collect Anonymous Analytics Data on Opt-In Basis

- Adds optional analytics data collection using Posthog
- Adds documentation on what data is collected and why

Resolves #63

Author: swissmanu

.github/ISSUE_TEMPLATE/request_analytics_data_access.md

@@ -0,0 +1,10 @@
+---
+name: Request Access To Analytics Data
+about:
+title:
+labels: ''
+assignees: ''
+---
+
+<!-- Please state why you request access to our analytics data and what you plan to do with it. -->
+<!-- Refer to https://github.com/swissmanu/rxjs-debugging-for-vscode/blob/main/ANALYTICS.md for further information. -->

.github/workflows/build.yml

@@ -90,6 +90,9 @@ jobs:
 
       - name: Production Build
         run: yarn nx run-many --target=build --configuration=prod --projects=extension,runtime-webpack
+        env:
+          POSTHOG_HOST: snapshot
+          POSTHOG_PROJECT_API_KEY: snapshot
 
       - name: 'extension: Create VSIX'
         run: yarn nx package extension
@@ -111,16 +114,54 @@ jobs:
           path: 'packages/runtime-webpack/*.tgz'
           retention-days: 10
 
-  publish:
+  publish-marketplace:
     if: github.event == 'release' && github.event.action == 'published'
     needs:
       - unit-test
+      - lint
       - integration-test
+    runs-on: ubuntu-latest
+    environment:
+      name: production
+      url: https://marketplace.visualstudio.com/items?itemName=manuelalabor.rxjs-debugging-for-vs-code
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Install Node.js
+        uses: actions/setup-node@v2
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          registry-url: 'https://registry.npmjs.org'
+
+      - name: Install Dependencies
+        run: yarn --immutable
+
+      - name: Create Production Build
+        run: yarn nx run-many --target=build --configuration=prod --projects=extension
+        env:
+          POSTHOG_HOST: ${{ secrets.POSTHOG_HOST }}
+          POSTHOG_PROJECT_API_KEY: ${{ secrets.POSTHOG_PROJECT_API_KEY }}
+
+      - name: 'Publish to Marketplace'
+        run: yarn nx run-many --target=publish --projects=extension
+        env:
+          VSCE_PAT: ${{ secrets.AZURE_ACCESS_TOKEN }}
+
+  publish-npm:
+    if: github.event == 'release' && github.event.action == 'published'
+    needs:
+      - unit-test
       - lint
+      - integration-test
     runs-on: ubuntu-latest
+    environment:
+      name: production
+      url: https://www.npmjs.com/package/@rxjs-debugging/runtime-webpack
     steps:
       - name: Checkout
         uses: actions/checkout@v2
+
       - name: Install Node.js
         uses: actions/setup-node@v2
         with:
@@ -129,11 +170,11 @@ jobs:
 
       - name: Install Dependencies
         run: yarn --immutable
-      - name: Production Build
-        run: yarn nx run-many --target=build --configuration=prod --projects=extension,runtime-webpack
 
-      - name: 'Publish'
+      - name: Create Production Build
+        run: yarn nx run-many --target=build --configuration=prod --projects=runtime-webpack
+
+      - name: 'Publish to npm'
         run: yarn nx run-many --target=publish --projects=runtime-webpack
         env:
-          VSCE_PAT: ${{ secrets.AZURE_ACCESS_TOKEN }}
           NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

ANALYTICS.md

@@ -0,0 +1,51 @@
+# Extension Usage Analytics
+
+> 🥽 For Science!
+
+The initial version of "RxJS Debugging for Visual Studio Code" resulted from a [research project](./RESEARCH.md). Doing (serious) research relies on empirical data. Thus, "RxJS Debugging for Visual Studio Code" asks its users to opt-in to collecting user behavior analytics data on its first activation.
+
+It is essential for us that our users understand what data we collect and why we do it. This document gives full disclosure on [every event and data point](#tracked-events) we collect. We reveal further [where and how information is stored](#data-transmission-and-storage) and how you can [access it for your research work or contribution to the extension](#open-source-open-research-and-open-data) itself.
+
+## Tracked Events
+
+If analytics is enabled, the extension tracks user behavior at events detailed below. Each event consists of an [anonymized machine identifier provided by Visual Studio Code](https://code.visualstudio.com/api/references/vscode-api#env) and a set of event-specific data points.
+
+All data points are carefully crafted to protect the users' privacy while providing empirical evidence for future research work. The machine identifier **DOES NOT** reveal the identity of users accordingly. Its sole purpose is the consolidation of events that belong together over time.
+
+We discard IP addresses before storing any tracking event. This makes it impossible to reconstruct or estimate your geographical location later.
+
+The following list documents all tracked analytic events. Feel free to review their implementation directly in the source code: [packages/extension/src/analytics](./packages/extension/src/analytics).
+
+### Extension Started
+
+| Data Point                                                   | Reason                                                       | Example Values              |
+| ------------------------------------------------------------ | ------------------------------------------------------------ | --------------------------- |
+| [**Visual Studio Code Version**](https://code.visualstudio.com/api/references/vscode-api#env) | The version of your Visual Studio Code installation.<br />This data point helps us to understand, which versions of Visual Studio Code are relevant for our users. It allows us to decide on if we can stop supporting outdated versions of Visual Studio Code, or not. | `1.61.0`                    |
+| [**Visual Studio Code Language**](https://code.visualstudio.com/api/references/vscode-api#env) | The preferred language of your Visual Studio Code installation.<br />This data point allows us to prioritze the languages for which we might translate "RxJS Debugging for Visual Studio Code" next. | `en-US`, `de-CH`, `fr`, ... |
+| **Extension Version**                                        | Identifies the version of "RxJS Debugging for Visual Studio Code" currently installed on your machine.<br />This data point helps us understand how our users install updates of our extension after release. | `1.0.0`                     |
+
+### Debug Session Started
+
+| Data Point       | Reason                                                       | Example Values      |
+| ---------------- | ------------------------------------------------------------ | ------------------- |
+| **Runtime Type** | The runtime type declares how the RxJS debugger connects to your application.<br />This data point helps us to understand what kind of RxJS applications (e.g. backend or frontend) our users debug most. | `nodejs`, `webpack` |
+
+### Debug Session Stopped
+
+*This event does not include any additional data points.*
+
+### Operator Log Point Enabled/Disabled
+
+| Data Point        | Reason                                                       | Example Values        |
+| ----------------- | ------------------------------------------------------------ | --------------------- |
+| **Operator Name** | Identifies **[built-in operators](https://rxjs.dev/api?type=function)** for which you enable/disable a log point. We will **NOT** track the name of a custom operator nor anything else related to your source code (line numbers, structure, etc.)<br />This data point helps us to understand which operators are the most problematic ones for our users. Thus, it helps us to build better debugging tools in the future accordingly. | `map`, `flatMap`, ... |
+
+## Data Transmission and Storage
+
+All analytic events are securely transmitted over an HTTPS connection.
+
+Usage analytics data is collected using [Posthog](https://posthog.com/). It runs on the premises of the [Eastern Switzerland University of Applied Sciences (OST)](https://ost.ch) where all data ist stored as well.
+
+## Open Source, Open Research and Open Data
+
+[Posthog](https://posthog.com/) does not allow the creation of read-only users at the time of writing this document. 🙏 Please [create an issue](https://github.com/swissmanu/rxjs-debugging-for-vscode/issues/new?template=request_analytics_data_access.md) using the appropriate template if you want to access to the collected analytics data for your own research project or contribution to the extension. We happily assist you either with an export of a data set or grant you access to Posthog itself if required.

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Next
 
 - Feature: Support Debugging of Browser-based Applications [#43](https://github.com/swissmanu/rxjs-debugging-for-vscode/issues/43)
+- Feature: Collect Analytics Data on Opt-In [#63](https://github.com/swissmanu/rxjs-debugging-for-vscode/issues/63)
 - Improvement: Add Integration Test for Operator Log Points [#49](https://github.com/swissmanu/rxjs-debugging-for-vscode/issues/49)
 - Bugfix: Enabled Log Point stays where it was enabled once [#102](https://github.com/swissmanu/rxjs-debugging-for-vscode/issues/102)
 

README.md

@@ -61,6 +61,10 @@ Jump right in and explore, how "RxJS Debugging for Visual Studio Code" can impro
 
 https://github.com/swissmanu/playground-rxjs-debugging-for-vscode
 
+## Analytics Data
+
+The "RxJS Debugger for Visual Studio Code" extension collects usage analytics data from users who opted-in. See [ANALYTICS.md](./ANALYTICS.md) for more information on what data is collected and why.
+
 ## Research
 
 This extension is based on research by Manuel Alabor. See [RESEARCH.md](./RESEARCH.md) for more information.

RESEARCH.md

@@ -3,7 +3,9 @@
 "RxJS Debugging for Visual Studio Code" is the result the master studies research by Manuel Alabor on the topic how developers debug RxJS-based code.
 
 - Manuel Alabor and Markus Stolze. 2020. Debugging of  RxJS-based applications. In Proceedings of the 7th ACM SIGPLAN          International Workshop on Reactive and Event-Based Languages and Systems (REBLS 2020). Association for Computing Machinery, New York, NY, USA, 15–24. DOI: https://doi.org/10.1145/3427763.3428313
-
 - Manuel Alabor. 2021. User Journey: Debugging of RxJS-Based Applications. https://alabor.me/research/user-journey-debugging-of-rxjs-based-applications/
 
-  
+
+## Open Research
+
+"RxJS Debugging for Visual Studio Code" collects usage analytics data on an opt-in basis. We provide open access to this data for research projects. Please refer to [ANALYTICS.md](./ANALYTICS.md) for further information.

packages/extension/.gitignore

@@ -1,3 +1,3 @@
 docs/
-README.md
+*.md
 LICENSE

packages/extension/package.json

@@ -32,17 +32,22 @@
     "configuration": {
       "title": "%rxjs-debugging.config.title%",
       "properties": {
-        "debug.rxjs.recommendOperatorLogPointsWithAnIcon": {
+        "rxjsDebugging.recommendOperatorLogPointsWithAnIcon": {
           "description": "%rxjs-debugging.config.recommendOperatorLogPointsWithAnIcon%",
           "type": "boolean",
           "default": true
         },
-        "debug.rxjs.hideLiveLogWhenStoppingDebugger": {
+        "rxjsDebugging.hideLiveLogWhenStoppingDebugger": {
           "description": "%rxjs-debugging.config.hideLiveLogWhenStoppingDebugger%",
           "type": "boolean",
           "default": true
         },
-        "debug.rxjs.logLevel": {
+        "rxjsDebugging.enableUsageAnalytics": {
+          "markdownDescription": "%rxjs-debugging.config.enableUsageAnalytics%",
+          "type": "boolean",
+          "default": false
+        },
+        "rxjsDebugging.logLevel": {
           "description": "%rxjs-debugging.config.logLevel%",
           "type": "string",
           "default": "Never",
@@ -88,6 +93,7 @@
   "devDependencies": {
     "@rollup/plugin-commonjs": "21.0.1",
     "@rollup/plugin-inject": "4.0.3",
+    "@rollup/plugin-json": "4.1.0",
     "@rollup/plugin-node-resolve": "13.0.6",
     "@rollup/plugin-typescript": "8.3.0",
     "@types/jest": "26.0.24",
@@ -110,6 +116,7 @@
     "@rxjs-debugging/runtime": "0.0.0",
     "@rxjs-debugging/telemetry": "0.0.0",
     "inversify": "5.1.1",
+    "posthog-node": "1.2.0",
     "reflect-metadata": "0.1.13",
     "spicery": "2.1.2",
     "typescript": "4.4.4",

packages/extension/package.nls.json

@@ -3,6 +3,7 @@
   "rxjs-debugging.config.hideLiveLogWhenStoppingDebugger": "When enabled, live log values are hidden automatically after stopping the debugger.",
   "rxjs-debugging.config.recommendOperatorLogPointsWithAnIcon": "When enabled, recommended operator log points will be indicated using an icon at the beginning of the line. Hint: You can toggle this setting using the related command in the command palette as well.",
   "rxjs-debugging.config.logLevel": "Internal log level for the extension. (reload required)",
+  "rxjs-debugging.config.enableUsageAnalytics": "When enabled, anonymized behavior data is collected to further improve RxJS Debugging and fuel future research projects looking into RxJS Debugging.\n\n\nYour individual identity will neither be revealed nor can be reconstructed at any time.\n\nSee [ANALYTICS.md](https://github.com/swissmanu/rxjs-debugging-for-vscode/blob/main/ANALYTICS.md) for a full disclosure on what data for what reason gets collected, where it is stored and how you can get access to it.\n\n\nThank you for considering your contribution 🙏",
 
   "rxjs-debugging.command.toggleOperatorLogPointGutterIcon": "Toggle gutter icon for recommended operator log points"
 }

packages/extension/rollup.config.js

@@ -1,12 +1,16 @@
 import commonjs from '@rollup/plugin-commonjs';
 import inject from '@rollup/plugin-inject';
+import json from '@rollup/plugin-json';
 import nodeResolve from '@rollup/plugin-node-resolve';
 import typescript from '@rollup/plugin-typescript';
 import * as path from 'path';
 import copy from 'rollup-plugin-copy';
 import { terser } from 'rollup-plugin-terser';
+import packageJson from './package.json';
 
 const terserOptions = { format: { comments: () => false } };
+const POSTHOG_PROJECT_API_KEY = process.env['POSTHOG_PROJECT_API_KEY'] ?? null;
+const POSTHOG_HOST = process.env['POSTHOG_HOST'] ?? null;
 
 /**
  * @type {import('rollup').RollupOptions}
@@ -15,6 +19,20 @@ export default ({ configMode }) => {
   const doProductionBuild = configMode === 'production';
   const doTestBuild = configMode === 'test';
 
+  if (doProductionBuild && (POSTHOG_PROJECT_API_KEY === null || POSTHOG_HOST === null)) {
+    console.error(
+      'POSTHOG_PROJECT_API_KEY and/or POSTHOG_HOST environment variables are missing. Cannot create production build.'
+    );
+    process.exit(1);
+    return;
+  }
+
+  const intro = `
+const EXTENSION_VERSION = "${packageJson.version}";
+const POSTHOG_PROJECT_API_KEY = "${POSTHOG_PROJECT_API_KEY}";
+const POSTHOG_HOST = "${POSTHOG_HOST}"
+`;
+
   return {
     input: {
       extension: 'src/extension.ts',
@@ -24,12 +42,14 @@ export default ({ configMode }) => {
       dir: 'out',
       format: 'commonjs',
       sourcemap: !doProductionBuild,
+      intro,
     },
     external: [
       'vscode',
       ...(doProductionBuild ? [] : ['typescript']), // Faster dev builds without including TypeScript
     ],
     plugins: [
+      json(),
       commonjs({
         ignore: ['bufferutil', 'utf-8-validate'], // Ignore optional peer dependencies of ws
       }),
@@ -46,7 +66,7 @@ export default ({ configMode }) => {
         targets: [
           { src: '../runtime-nodejs/out/**/*', dest: './out/runtime-nodejs' },
           { src: '../../docs/*', dest: './docs' },
-          { src: '../../README.md', dest: './' },
+          { src: '../../*.md', dest: './' },
           { src: '../../LICENSE', dest: './' },
         ],
       }),