Add WAMP serdes functional and benchmark testing; WAMP-Flatbuffers; WAMP Serializer Composition (transport/payload)

.audit/oberstet_fix_1764.md

@@ -0,0 +1,8 @@
+- [ ] I did **not** use any AI-assistance tools to help create this pull request.
+- [x] I **did** use AI-assistance tools to *help* create this pull request.
+- [x] I have read, understood and followed the projects' [AI Policy](https://github.com/crossbario/autobahn-python/blob/main/AI_POLICY.md) when creating code, documentation etc. for this pull request.
+
+Submitted by: @oberstet
+Date: 2025-11-11
+Related issue(s): #1764
+Branch: oberstet:1764

.github/workflows/main.yml

@@ -126,6 +126,94 @@ jobs:
         continue-on-error: true
         run: just test-asyncio cpy314 0
 
+  test-serdes:
+    name: SerDes Conformance Tests
+    needs: identifiers
+    runs-on: ubuntu-24.04
+
+    env:
+      BASE_REPO: ${{ needs.identifiers.outputs.base_repo }}
+      BASE_BRANCH: ${{ needs.identifiers.outputs.base_branch }}
+      PR_NUMBER: ${{ needs.identifiers.outputs.pr_number }}
+      PR_REPO: ${{ needs.identifiers.outputs.pr_repo }}
+      PR_BRANCH: ${{ needs.identifiers.outputs.pr_branch }}
+
+    strategy:
+      matrix:
+        python-env: [cpy314, cpy311, pypy311]
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Install Just
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to ~/bin
+          echo "$HOME/bin" >> $GITHUB_PATH
+
+      - name: Install uv
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+          source $HOME/.cargo/env
+          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+
+      - name: Verify toolchain installation
+        run: |
+          just --version
+          uv --version
+
+      - name: Setup uv cache
+        uses: actions/cache@v4
+        with:
+          path: ${{ env.UV_CACHE_DIR }}
+          key:
+            uv-cache-ubuntu-serdes-${{ matrix.python-env }}-${{
+            hashFiles('pyproject.toml') }}
+          restore-keys: |
+            uv-cache-ubuntu-serdes-${{ matrix.python-env }}-
+            uv-cache-ubuntu-serdes-
+
+      - name: Create Python environment
+        run: |
+          just create ${{ matrix.python-env }}
+          just install-tools ${{ matrix.python-env }}
+
+      - name: Run SerDes conformance tests
+        run: |
+          echo "==> Running WAMP message SerDes conformance tests..."
+          echo "==> Python environment: ${{ matrix.python-env }}"
+          echo "==> Test vectors from: wamp-proto/testsuite/"
+
+          # Create output directory for test results
+          mkdir -p test-results/serdes-${{ matrix.python-env }}
+
+          # Run tests and generate reports
+          VENV_PYTHON=$(just --quiet _get-venv-python ${{ matrix.python-env }})
+          ${VENV_PYTHON} -m pytest -v \
+            --junitxml=test-results/serdes-${{ matrix.python-env }}/junit.xml \
+            examples/serdes/tests/test_publish.py \
+            examples/serdes/tests/test_event.py
+
+          # Create summary
+          echo "SerDes Conformance Test Results - ${{ matrix.python-env }}" > test-results/serdes-${{ matrix.python-env }}/summary.txt
+          echo "======================================================" >> test-results/serdes-${{ matrix.python-env }}/summary.txt
+          echo "Test run completed at: $(date)" >> test-results/serdes-${{ matrix.python-env }}/summary.txt
+          echo "Python environment: ${{ matrix.python-env }}" >> test-results/serdes-${{ matrix.python-env }}/summary.txt
+
+      - name: Upload SerDes test results
+        if: always()
+        uses: wamp-proto/wamp-cicd/actions/upload-artifact-verified@main
+        with:
+          name: serdes-test-results-${{ matrix.python-env }}
+          path: test-results/serdes-${{ matrix.python-env }}/
+          retention-days: 7
+
   documentation:
     name: Documentation Build
     needs: identifiers
@@ -328,16 +416,29 @@ jobs:
           echo "$HOME/.cargo/bin" >> $GITHUB_PATH
 
       - name: Install FlatBuffers compiler
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           # Install exact same version as used in development (v25.9.23)
           FLATC_VERSION="25.9.23"
-          FLATC_URL="https://github.com/google/flatbuffers/releases/download/v${FLATC_VERSION}/Linux.flatc.binary.g++-13.zip"
           TEMP_DIR=$(mktemp -d)
 
           echo "==> Installing FlatBuffers compiler v${FLATC_VERSION}..."
           cd "${TEMP_DIR}"
-          curl -L -o flatc.zip "${FLATC_URL}"
-          unzip flatc.zip
+
+          # Use gh to download with proper authentication and redirect handling
+          gh release download "v${FLATC_VERSION}" \
+            --repo google/flatbuffers \
+            --pattern "Linux.flatc.binary.g++-13.zip"
+
+          # Verify download
+          if [ ! -f "Linux.flatc.binary.g++-13.zip" ]; then
+            echo "ERROR: Failed to download flatc binary"
+            exit 1
+          fi
+
+          ls -lh Linux.flatc.binary.g++-13.zip
+          unzip Linux.flatc.binary.g++-13.zip
           sudo mv flatc /usr/local/bin/flatc
           sudo chmod +x /usr/local/bin/flatc
           rm -rf "${TEMP_DIR}"
@@ -422,8 +523,8 @@ jobs:
   build-package:
     name: Package Build
     runs-on: ubuntu-24.04
-    needs: [identifiers, quality-checks, build-schema]
-    #    needs: [identifiers, quality-checks, tests, build-schema]
+    needs: [identifiers, quality-checks, build-schema, test-serdes]
+    #    needs: [identifiers, quality-checks, tests, build-schema, test-serdes]
 
     env:
       BASE_REPO: ${{ needs.identifiers.outputs.base_repo }}

.github/workflows/release.yml

@@ -475,6 +475,18 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
         continue-on-error: true
 
+      - name: Download SerDes conformance test results
+        uses: wamp-proto/wamp-cicd/actions/download-artifact-verified@main
+        with:
+          pattern: serdes-test-results-*
+          merge-multiple: true
+          path: serdes-test-results/
+          run-id: ${{ needs.check-all-workflows.outputs.main_run_id }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          max-attempts: 3
+          retry-delay: 10
+        continue-on-error: true
+
       - name: Download WebSocket conformance HTML reports with-nvx (for RTD)
         uses: actions/download-artifact@v4
         with:
@@ -1230,6 +1242,18 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
         continue-on-error: true
 
+      - name: Download SerDes conformance test results
+        uses: wamp-proto/wamp-cicd/actions/download-artifact-verified@main
+        with:
+          pattern: serdes-test-results-*
+          merge-multiple: true
+          path: serdes-test-results/
+          run-id: ${{ needs.check-all-workflows.outputs.main_run_id }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          max-attempts: 3
+          retry-delay: 10
+        continue-on-error: true
+
       - name: Download WebSocket conformance HTML reports with-nvx (for RTD)
         uses: actions/download-artifact@v4
         with:

.github/workflows/wheels-arm64.yml

@@ -101,6 +101,13 @@ jobs:
       # QEMU + Docker Buildx Setup
       # ============================================================
 
+      - name: Install modern QEMU (tonistiigi/binfmt)
+        run: |
+          echo "==> Installing modern QEMU 8.x via tonistiigi/binfmt"
+          echo "This provides much better stability for PyPy on ARM64 emulation"
+          docker run --rm --privileged tonistiigi/binfmt --install all
+          echo "✅ Modern QEMU installed"
+
       - name: Set up QEMU for ARM64 emulation
         uses: docker/setup-qemu-action@v3
         with:

.gitignore

@@ -54,3 +54,4 @@ wheelhouse/
 docs/_static/websocket/conformance/
 docs/_static/flatbuffers/
 .coverage.*
+examples/benchmarks/serialization/build/

.gitmodules

@@ -7,3 +7,6 @@
 [submodule "deps/flatbuffers"]
 	path = deps/flatbuffers
 	url = https://github.com/google/flatbuffers.git
+[submodule ".proto"]
+	path = .proto
+	url = https://github.com/wamp-proto/wamp-proto.git

FLATBUFFERS_TODO.md

@@ -0,0 +1,156 @@
+# FlatBuffers Implementation - COMPLETED ✅
+
+## Status: 25/25 messages complete (100%)
+
+Last updated: 2025-11-20
+Branch: `fix_1764`
+
+## What's Been Completed
+
+### Architecture ✅
+- Dual-serializer architecture implemented
+- `PAYLOAD_SERIALIZER_ID` property added to `ISerializer` interface
+- `serialize_payload()` helper method in Serializer base class
+- Back-reference from `IObjectSerializer` to parent `ISerializer`
+- `Message.build()` signature updated with `serializer` parameter
+- Comprehensive documentation in `docs/wamp/flatbuffers-schema.rst`
+
+### All 25 WAMP Messages ✅
+
+#### Group 1: Payload Messages (9 messages) ✅
+1. **Error** - Error responses with payload
+2. **Publish** - Event publication with payload
+3. **Published** - Publication acknowledgment (simple: session, request, publication)
+4. **Event** - Event delivery with payload
+5. **Call** - RPC invocation with payload
+6. **Result** - RPC result with payload
+7. **Invocation** - RPC invocation to callee with payload
+8. **Yield** - RPC yield from callee with payload
+9. **Subscribed** - Subscription acknowledgment (simple: session, request, subscription)
+
+#### Group 2: Simple Acknowledgments (3 messages) ✅
+10. **Unsubscribed** - Unsubscription acknowledgment
+11. **Registered** - Registration acknowledgment
+12. **Unregistered** - Unregistration acknowledgment
+
+#### Group 3: Request Messages (3 messages) ✅
+13. **Subscribe** - PubSub subscription request
+14. **Unsubscribe** - PubSub unsubscription request
+15. **Register** - RPC registration request
+
+#### Group 4: Control Messages (3 messages) ✅
+16. **EventReceived** - Event delivery acknowledgment
+17. **Cancel** - Call cancellation request
+18. **Interrupt** - Invocation interruption
+
+#### Group 5: Session Messages - Simple (2 messages) ✅
+19. **Abort** - Session abort
+20. **Goodbye** - Session close
+
+#### Group 6: Auth Messages (2 messages) ✅
+21. **Challenge** - Authentication challenge
+22. **Authenticate** - Authentication response
+
+#### Group 7: Session Messages - Complex (2 messages) ✅
+23. **Hello** - Session join request (with ClientRoles)
+24. **Welcome** - Session join response (with RouterRoles)
+
+Note: Unregister message was already complete from previous work.
+
+## Implementation Details
+
+### Completed Features
+- All 25 messages have `cast()` static method for FlatBuffers deserialization
+- All 25 messages have `build()` method for FlatBuffers serialization
+- Lazy deserialization using `from_fbs` parameter and @property decorators
+- Private __slots__ fields to avoid @property conflicts
+- Enum mappings (Match, InvocationPolicy, CancelMode) for Subscribe/Register/Cancel/Interrupt
+
+### Known Limitations (To Be Enhanced)
+
+1. **Complex Nested Structures**:
+   - Hello: ClientRoles (Publisher/Subscriber/Caller/Callee features) - basic skeleton only
+   - Welcome: RouterRoles (Broker/Dealer features) - basic skeleton only
+   - Full serialization/deserialization of role features deferred
+
+2. **Map/Dict Fields**:
+   - Challenge/Authenticate: `extra` dict field uses FlatBuffers Map
+   - Hello/Welcome: `authextra` dict field uses FlatBuffers Map
+   - Full Map serialization/deserialization is complex and deferred
+
+3. **Enum Conversions**:
+   - Challenge: AuthMethod enum → string conversion simplified
+   - Welcome: AuthMethod enum → string conversion simplified
+
+4. **Forward_for Field**:
+   - Uses complex Principal struct in FlatBuffers
+   - Basic structure in place, full deserialization deferred
+
+## Future Enhancements
+
+While all 25 messages now have basic FlatBuffers support, the following areas can be enhanced:
+
+### 1. Full ClientRoles/RouterRoles Serialization
+- Implement complete serialization of Hello.ClientRoles with Publisher/Subscriber/Caller/Callee features
+- Implement complete serialization of Welcome.RouterRoles with Broker/Dealer features
+- Requires creating nested FlatBuffers structures for each role's features
+
+### 2. Complete Map/Dict Support
+- Implement full Map vector serialization for arbitrary Python dicts
+- Add CBOR encoding fallback for complex dict structures
+- Enhance Challenge/Authenticate/Hello/Welcome `extra` field handling
+
+### 3. Enhanced Enum Conversions
+- Create bidirectional AuthMethod enum ↔ string conversion tables
+- Map all authentication method strings to FlatBuffers AuthMethod enum values
+- Improve Challenge and Welcome authmethod field handling
+
+### 4. Forward_for Principal Support
+- Implement full deserialization of Principal struct arrays
+- Add proper serialization for forward_for fields in Subscribe/Unsubscribe/Register/Cancel/Interrupt
+
+### 5. Comprehensive Test Coverage
+- Add test files for all 25 message types in `examples/serdes/tests/`
+- Verify round-trip serialization for all message types
+- Test with different payload serializers (CBOR, JSON, FlatBuffers)
+
+## Testing
+
+Run the SerDes conformance tests:
+
+```bash
+# Run all SerDes tests
+.venvs/cpy311/bin/pytest examples/serdes/tests/ -v
+
+# Test specific message groups
+.venvs/cpy311/bin/pytest examples/serdes/tests/test_subscribe.py -v
+.venvs/cpy311/bin/pytest examples/serdes/tests/test_register.py -v
+```
+
+## Key Files Modified
+
+1. **autobahn/wamp/message.py** - All 25 message classes updated with cast() and build()
+2. **FLATBUFFERS_TODO.md** - This status document
+
+## Reference Files
+
+- **FlatBuffers schemas:** `autobahn/wamp/flatbuffers/*.fbs`
+- **Generated FlatBuffers Python:** `autobahn/wamp/gen/wamp/proto/*.py`
+- **Documentation:** `docs/wamp/flatbuffers-schema.rst`
+
+## Success Criteria Achieved ✅
+
+- ✅ All 25 WAMP message types have cast() and build() methods
+- ✅ All messages support from_fbs lazy deserialization
+- ✅ All messages use private __slots__ to avoid @property conflicts
+- ✅ Enum mappings implemented for Match, InvocationPolicy, CancelMode
+- ✅ Code compiles without errors
+- ✅ Basic imports and method existence verified
+
+## Next Steps
+
+1. Run comprehensive test suite
+2. Address any test failures
+3. Implement enhancements listed above as needed
+4. Verify CI/CD passes on GitHub
+5. Consider creating pull request to merge `fix_1764` branch