Skip to content

Commit 8852b88

Browse files
jumskijumski
committed
chore: update Supabase dependencies and improve e2e test infrastructure (#292)
# Update Supabase JS SDK and Improve Edge Worker E2E Testing This PR updates the Supabase JS SDK from version 2.55.0 to 2.74.0 and significantly improves the edge worker E2E testing infrastructure. Key changes: - Added a new script `sync-e2e-deps.sh` that copies built packages to a vendor directory for E2E tests - Created a new import map in `supabase/functions/deno.json` to properly resolve dependencies during tests - Added a new `serve:functions:e2e` task that uses the vendor directory for testing - Removed SQL views and functions that are no longer needed (`active_workers`, `inactive_workers`, `spawn_worker`) - Updated worker detection logic to use direct queries instead of views - Improved worker startup logic with retry mechanism for better test reliability - Added E2E test environment variables to `.env` file - Added `--no-verify-jwt` flag to function serving to simplify testing These changes make the E2E tests more reliable and easier to run, while also keeping the codebase up-to-date with the latest Supabase SDK.
1 parent b8abac8 commit 8852b88

File tree

21 files changed

+402
-200
lines changed

21 files changed

+402
-200
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 57 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -54,14 +54,62 @@ jobs:
5454
run: pnpm nx affected -t build --configuration=production --parallel --exclude=playground --base="$NX_BASE" --head="$NX_HEAD"
5555

5656

57-
# ────────────────────────────────── 2. DEPLOY PLAYGROUND ───────────────────────────
57+
# ─────────────────────────────────────── 2. EDGE-WORKER E2E ──────────────────────────────────────
58+
edge-worker-e2e:
59+
runs-on: ubuntu-latest
60+
env:
61+
NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}
62+
steps:
63+
- uses: actions/checkout@v4
64+
with:
65+
fetch-depth: 0
66+
67+
- uses: ./.github/actions/setup
68+
69+
- name: Setup Deno
70+
uses: denoland/setup-deno@v2
71+
with:
72+
deno-version: '1.45.2'
73+
74+
- name: Install sqruff
75+
uses: quarylabs/install-sqruff-cli-action@main
76+
77+
- name: Setup Atlas
78+
uses: ariga/setup-atlas@master
79+
with:
80+
cloud-token: ${{ secrets.ATLAS_CLOUD_TOKEN }}
81+
82+
- name: Set Nx SHAs for affected commands
83+
uses: nrwl/nx-set-shas@v4
84+
85+
- name: Verify NX_BASE and NX_HEAD are set
86+
run: echo "BASE=$NX_BASE HEAD=$NX_HEAD"
87+
88+
- name: Check if edge-worker e2e tests are affected
89+
id: check-affected
90+
run: |
91+
if pnpm nx show projects --affected -t test:e2e --base="$NX_BASE" --head="$NX_HEAD" | grep -q "^edge-worker$"; then
92+
echo "affected=true" >> $GITHUB_OUTPUT
93+
echo "Edge-worker e2e tests are affected by changes"
94+
else
95+
echo "affected=false" >> $GITHUB_OUTPUT
96+
echo "Edge-worker e2e tests are not affected by changes - skipping"
97+
fi
98+
99+
- name: Run edge-worker e2e tests
100+
if: steps.check-affected.outputs.affected == 'true'
101+
run: pnpm nx affected -t test:e2e --parallel --base="$NX_BASE" --head="$NX_HEAD"
102+
103+
104+
# ────────────────────────────────── 3. DEPLOY PLAYGROUND ───────────────────────────
58105
deploy-playground:
59-
needs: build-and-test
60-
if: >-
61-
${{
62-
(github.event_name == 'pull_request') ||
63-
(github.ref == 'refs/heads/main' && github.event_name == 'push')
64-
}}
106+
needs: [build-and-test, edge-worker-e2e]
107+
if: false # Disabled
108+
# if: >-
109+
# ${{
110+
# (github.event_name == 'pull_request') ||
111+
# (github.ref == 'refs/heads/main' && github.event_name == 'push')
112+
# }}
65113
runs-on: ubuntu-latest
66114
environment: ${{ github.event_name == 'pull_request' && 'preview' || 'production' }}
67115
env:
@@ -95,9 +143,9 @@ jobs:
95143
preview-url: https://pr-${{ github.event.pull_request.number }}--pgflow-demo.netlify.app
96144
production-url: https://playground.pgflow.dev
97145

98-
# ────────────────────────────────── 3. DEPLOY WEBSITE ───────────────────────────
146+
# ────────────────────────────────── 4. DEPLOY WEBSITE ───────────────────────────
99147
deploy-website:
100-
needs: build-and-test
148+
needs: [build-and-test, edge-worker-e2e]
101149
runs-on: ubuntu-latest
102150
environment: ${{ github.event_name == 'pull_request' && 'preview' || 'production' }}
103151
env:

‎pkgs/edge-worker/CLAUDE.md‎

Lines changed: 28 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -53,13 +53,34 @@ The Edge Worker:
5353

5454
## Testing
5555

56-
The package has different test types:
57-
58-
- **Unit tests** (`tests/unit/`) - For testing isolated components
59-
- **Integration tests** (`tests/integration/`) - For testing integration with PostgreSQL
60-
- **E2E tests** (`tests/e2e/`) - For testing end-to-end workflows (not automated for now)
61-
62-
Tests use Deno's built-in testing framework. Database tests require Docker to run a PostgreSQL instance.
56+
The package has three test types with different infrastructure requirements:
57+
58+
- **Unit tests** (`tests/unit/`) - Testing isolated components
59+
- No external dependencies required
60+
61+
- **Integration tests** (`tests/integration/`) - Testing integration with PostgreSQL
62+
- Requires: Supabase database (via `db:ensure`)
63+
- Runs in CI
64+
65+
- **E2E tests** (`tests/e2e/`) - Testing end-to-end workflows with Edge Functions
66+
- Requires: Supabase database + Edge Functions server (via `serve:functions:e2e`)
67+
- Uses `continuous: true` in Nx to keep server running during tests
68+
- Tests worker scaling behavior, CPU limits, and concurrency (20k+ messages)
69+
- Takes 2+ minutes to complete
70+
- Runs in CI as separate parallel job (only when edge-worker is affected)
71+
- Run locally with: `pnpm nx test:e2e edge-worker`
72+
73+
Tests use Deno's built-in testing framework. Database tests require Docker to run a PostgreSQL instance via Supabase CLI.
74+
75+
### E2E Test Architecture
76+
77+
E2E tests verify real Edge Function execution:
78+
1. `sync-e2e-deps` copies built packages (core, dsl, edge-worker) to `supabase/functions/_vendor`
79+
2. `serve:functions:e2e` starts Supabase Functions server in background (continuous task)
80+
3. Tests make HTTP requests to spawn workers and verify behavior
81+
4. Tests in `tests/e2e/`:
82+
- `restarts.test.ts` - Worker auto-restart when CPU clock limit hits
83+
- `performance.test.ts` - Processing thousands of queued messages concurrently
6384

6485
## Usage Examples
6586

‎pkgs/edge-worker/deno.lock‎

Lines changed: 29 additions & 19 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkgs/edge-worker/project.json‎

Lines changed: 28 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,6 @@
100100
"cache": false,
101101
"inputs": [
102102
"{workspaceRoot}/pkgs/core/supabase/migrations/**/*.sql",
103-
"{projectRoot}/sql/**/*.sql",
104103
"^production"
105104
],
106105
"options": {
@@ -109,7 +108,6 @@
109108
"mkdir -p supabase/migrations/",
110109
"rm -f supabase/migrations/*.sql",
111110
"cp ../core/supabase/migrations/*.sql supabase/migrations/",
112-
"cp sql/*_*.sql supabase/migrations/",
113111
"supabase db reset"
114112
],
115113
"parallel": false
@@ -123,7 +121,7 @@
123121
"options": {
124122
"cwd": "pkgs/edge-worker",
125123
"commands": [
126-
"supabase functions serve --env-file supabase/functions/.env"
124+
"supabase functions serve --env-file supabase/functions/.env --no-verify-jwt"
127125
],
128126
"parallel": false
129127
}
@@ -132,11 +130,7 @@
132130
"executor": "nx:run-commands",
133131
"local": true,
134132
"dependsOn": ["^verify-migrations", "^dump-realtime-schema"],
135-
"inputs": [
136-
"default",
137-
"^production",
138-
"{projectRoot}/scripts/ensure-db"
139-
],
133+
"inputs": ["default", "^production", "{projectRoot}/scripts/ensure-db"],
140134
"options": {
141135
"cwd": "pkgs/edge-worker",
142136
"commands": ["./scripts/ensure-db"],
@@ -169,9 +163,34 @@
169163
"parallel": false
170164
}
171165
},
166+
"sync-e2e-deps": {
167+
"executor": "nx:run-commands",
168+
"dependsOn": ["^build"],
169+
"local": true,
170+
"inputs": ["^production"],
171+
"options": {
172+
"cwd": "pkgs/edge-worker",
173+
"commands": ["./scripts/sync-e2e-deps.sh"],
174+
"parallel": false
175+
}
176+
},
177+
"serve:functions:e2e": {
178+
"executor": "nx:run-commands",
179+
"continuous": true,
180+
"dependsOn": ["sync-e2e-deps", "supabase:start"],
181+
"local": true,
182+
"cache": false,
183+
"options": {
184+
"cwd": "pkgs/edge-worker",
185+
"commands": [
186+
"supabase functions serve --env-file supabase/functions/.env --import-map supabase/functions/deno.json --no-verify-jwt"
187+
],
188+
"parallel": false
189+
}
190+
},
172191
"test:e2e": {
173192
"executor": "nx:run-commands",
174-
"dependsOn": ["db:ensure", "^build"],
193+
"dependsOn": ["supabase:reset", "serve:functions:e2e"],
175194
"local": true,
176195
"inputs": ["default", "^production"],
177196
"options": {

‎pkgs/edge-worker/scripts/sync-e2e-deps.sh‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
5+
EDGE_WORKER_DIR="$(dirname "$SCRIPT_DIR")"
6+
MONOREPO_ROOT="$(cd "$EDGE_WORKER_DIR/../.." && pwd)"
7+
VENDOR_DIR="$EDGE_WORKER_DIR/supabase/functions/_vendor"
8+
9+
echo "🔄 Syncing edge function dependencies for e2e tests..."
10+
11+
# Clean and create vendor directory
12+
rm -rf "$VENDOR_DIR"
13+
mkdir -p "$VENDOR_DIR/@pgflow"
14+
15+
# Verify builds succeeded
16+
if [ ! -d "$MONOREPO_ROOT/pkgs/core/dist" ]; then
17+
echo "❌ Error: core package build failed - dist directory not found"
18+
exit 1
19+
fi
20+
21+
if [ ! -d "$MONOREPO_ROOT/pkgs/dsl/dist" ]; then
22+
echo "❌ Error: dsl package build failed - dist directory not found"
23+
exit 1
24+
fi
25+
26+
# Copy core package
27+
echo "📋 Copying @pgflow/core..."
28+
mkdir -p "$VENDOR_DIR/@pgflow/core"
29+
cp -r "$MONOREPO_ROOT/pkgs/core/dist/"* "$VENDOR_DIR/@pgflow/core/"
30+
cp "$MONOREPO_ROOT/pkgs/core/package.json" "$VENDOR_DIR/@pgflow/core/"
31+
32+
# Copy dsl package
33+
echo "📋 Copying @pgflow/dsl..."
34+
mkdir -p "$VENDOR_DIR/@pgflow/dsl"
35+
cp -r "$MONOREPO_ROOT/pkgs/dsl/dist/"* "$VENDOR_DIR/@pgflow/dsl/"
36+
cp "$MONOREPO_ROOT/pkgs/dsl/package.json" "$VENDOR_DIR/@pgflow/dsl/"
37+
38+
# Copy edge-worker source (not built) - preserving directory structure
39+
echo "📋 Copying @pgflow/edge-worker..."
40+
mkdir -p "$VENDOR_DIR/@pgflow/edge-worker"
41+
# Copy the entire src directory to maintain relative imports
42+
cp -r "$MONOREPO_ROOT/pkgs/edge-worker/src" "$VENDOR_DIR/@pgflow/edge-worker/"
43+
44+
# Simple fix: replace .js with .ts in imports
45+
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i 's/\.js"/\.ts"/g' {} +
46+
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i "s/\.js'/\.ts'/g" {} +
47+
48+
# Create a redirect index.ts at the root that points to src/index.ts
49+
cat > "$VENDOR_DIR/@pgflow/edge-worker/index.ts" << 'EOF'
50+
// Re-export from the src directory to maintain compatibility
51+
export * from './src/index.ts';
52+
EOF
53+
54+
# Create _internal.ts redirect as well since edge-worker exports this path
55+
cat > "$VENDOR_DIR/@pgflow/edge-worker/_internal.ts" << 'EOF'
56+
// Re-export from the src directory to maintain compatibility
57+
export * from './src/_internal.ts';
58+
EOF
59+
60+
# Verify key files exist
61+
if [ ! -f "$VENDOR_DIR/@pgflow/core/index.js" ]; then
62+
echo "⚠️ Warning: @pgflow/core/index.js not found after copy"
63+
fi
64+
65+
if [ ! -f "$VENDOR_DIR/@pgflow/dsl/index.js" ]; then
66+
echo "⚠️ Warning: @pgflow/dsl/index.js not found after copy"
67+
fi
68+
69+
if [ ! -f "$VENDOR_DIR/@pgflow/edge-worker/src/index.ts" ]; then
70+
echo "⚠️ Warning: @pgflow/edge-worker/src/index.ts not found after copy"
71+
fi
72+
73+
echo "✅ Dependencies synced to $VENDOR_DIR"

‎pkgs/edge-worker/sql/990_active_workers.sql‎

Lines changed: 0 additions & 11 deletions
This file was deleted.

0 commit comments

Comments
 (0)