Skip to content

Commit fa31da4

Browse files
coopbricoopbri
authored
WebXR (#26)
* ci(changesets): add WebXR changeset * build(deps): add `@react-three/xr` * chore(bun): update lock file * refactor(store): condense XR provider components, remove camera source prop * docs(readme): update copy * build(deps): add `ts-pattern`, `@types/webxr` * feature(lib,util): add immersive utils * refactor(apps): remove `cameraSource` prop * feature: add immersive module to public API export * chore: format * feature(immersive): add hooks, types * feature(engine): add WebXR support powered by `@react-three/xr` * fix(engine): properly nest immersive store * feature(apps): add immersive demo * build(deps): upgrade dependencies * docs(readme): update WebXR support details * docs(readme): update copy * chore: export store methods * ci(changesets): update copy * chore: update comments * chore(apps,immersive-demo): update background color for VR mode convenience * fix(store): unify state
1 parent 1524adf commit fa31da4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+1131
-739
lines changed

‎.changeset/metal-cloths-deny.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
"@omnidotdev/rdk": minor
3+
---
4+
5+
Added native WebXR support, powered by [`@react-three/xr`](https://github.com/pmndrs/xr).
6+
7+
**BREAKING:** Removed `cameraSource` prop from XR component. Sessions now auto-configure themselves:
8+
9+
```tsx
10+
// before
11+
<XR cameraSource="video">
12+
<FiducialSession />
13+
</XR>
14+
15+
// after
16+
<XR>
17+
{/* auto-configures video mode */}
18+
<FiducialSession />
19+
</XR>
20+
```
21+
22+
**New Features:**
23+
24+
- `ImmersiveSession` component for WebXR AR/VR
25+
- Nested `@react-three/xr`'s store nested under `useXRStore`'s `immersive` property
26+
- Added `ImmersiveMode` type export, which maps to and from [official WebXR modes](https://www.w3.org/TR/webxr/#xrsessionmode-enum) (`immersive-ar` ↔ `ar`, `immersive-vr` ↔ `vr`, `inline` ↔ `inline`)

‎README.md‎

Lines changed: 24 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -11,27 +11,26 @@
1111

1212
> [!IMPORTANT]
1313
> **Project Status:** 🚧 This project is **brand new**.
14-
> Currently, fiducial marker-based AR via [AR.js](https://github.com/ar-js-org/ar.js) and location-based AR via [LocAR.js](https://github.com/ar-js-org/locar.js) are working (see [`apps/`](./apps/) for demos), though they are **experimental**. Contributions (PRs, [Omni organization sponsorship](https://github.com/sponsors/omnidotdev)) appreciated.
15-
> Native WebXR integration via `@react-three/xr` is coming next.
14+
> Currently, fiducial marker-based AR via [AR.js](https://github.com/ar-js-org/ar.js), location-based AR via [LocAR.js](https://github.com/ar-js-org/locar.js), and WebXR via [`@react-three/xr`](https://github.com/pmndrs/xr) are working (see [`apps/`](./apps/) for demos), though they are **experimental**. Contributions (PRs, [Omni organization sponsorship](https://github.com/sponsors/omnidotdev)) appreciated.
1615
1716
## Overview
1817

19-
RDK unifies multiple XR technologies, such as AR.js for marker-based AR, LocAR.js for geolocation-based AR, and WebXR (coming soon) for device-native support under one React-first abstraction powered by Three.js.
20-
21-
| Capability/Use Case | Status | Backend (Current or Proposed) | Android | iOS | Notes |
22-
| ------------------------------------ | --------------- | ---------------------------------------------------------------------------------------------------- | ------- | --- | ----------------------------------------------------------------------------------------- |
23-
| **Fiducial (Pattern/Barcode)** | ⚗️ Experimental | [AR.js (ARToolKit)](https://github.com/ar-js-org/ar.js) | ✅ | ✅ | Uses `.patt` or barcode markers. Reliable for printed markers. No WebXR dependency. |
24-
| **Image Tracking (Natural Feature)** | 🧭 Planned | [AR.js (ARToolKit)](https://github.com/ar-js-org/ar.js) | N/A | N/A | May use `.mind` or `XRTrackedImage`. Ideal for logos or posters. Requires image database. |
25-
| **Geolocation / World Anchors** | ⚗️ Experimental | [LocAR.js](https://github.com/ar-js-org/locar.js) | ✅ | ✅ | Uses GPS + compass; may later integrate Mapbox or Cesium. |
26-
| **WebXR Native AR/VR Session** | 🧭 Planned | [`@react-three/xr`](https://github.com/pmndrs/xr) | N/A | N/A | Entry point for true AR/VR sessions. Ties into `XRSessionProvider`. |
27-
| **Face Tracking** | 🧭 Planned | - | N/A | N/A | Uses webcam + ML model; lightweight and fast. |
28-
| **Body/Pose Tracking** | 🧭 Planned | [WebXR Body Tracking](https://github.com/immersive-web/body-tracking) | N/A | N/A | Real-time skeletal tracking. GPU/WebGL acceleration required. |
29-
| **Hand Tracking** | 🧭 Planned | - | N/A | N/A | Supported on Chrome + Meta; ML fallback possible. |
30-
| **Plane/Surface Detection** | 🧭 Planned | [WebXR Hit Test API](https://immersive-web.github.io/hit-test)/ar.js (limited) | N/A | N/A | Enables AR object placement on flat surfaces. |
31-
| **Depth Sensing/Environment Mesh** | 🧭 Planned | [WebXR Depth Sensing API](https://immersive-web.github.io/depth-sensing) | N/A | N/A | Provides per-pixel depth; early spec. |
32-
| **SLAM/Visual Positioning (VPS)** | 🧭 Planned | Custom | N/A | N/A | Requires world map data; long-term goal. |
33-
| **Voice/Gesture Interaction** | 🧭 Planned | [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API)/MediaPipe Gestures | N/A | N/A | Enables multimodal input: voice, hand, gaze. |
34-
| **Mixed Reality Compositing** | 🧭 Planned | WebXR Layers/CanvasCaptureStream | N/A | N/A | Transparent overlays/live compositing. |
18+
RDK unifies multiple spatial and XR technologies, such as AR.js for marker-based AR, LocAR.js for geolocation-based AR, and WebXR for device-native support under one React-first abstraction powered by Three.js.
19+
20+
| Capability/Use Case | Status | Backend (Current or Proposed) | Android | iOS | Notes |
21+
| ------------------------------------ | --------------- | ---------------------------------------------------------------------------------------------------- | ------- | -------------- | ----------------------------------------------------------------------------------------- |
22+
| **Fiducial (Pattern/Barcode)** | ⚗️ Experimental | [AR.js (ARToolKit)](https://github.com/ar-js-org/ar.js) | ✅ | ✅ | Uses `.patt` or barcode markers. Reliable for printed markers. No WebXR dependency. |
23+
| **Image Tracking (Natural Feature)** | 🧭 Planned | [AR.js (ARToolKit)](https://github.com/ar-js-org/ar.js) | N/A | N/A | May use `.mind` or `XRTrackedImage`. Ideal for logos or posters. Requires image database. |
24+
| **Geolocation / World Anchors** | ⚗️ Experimental | [LocAR.js](https://github.com/ar-js-org/locar.js) | ✅ | ✅ | Uses GPS + compass; may later integrate Mapbox or Cesium. |
25+
| **WebXR Native AR/VR Session** | ⚗️ Experimental | [`@react-three/xr`](https://github.com/pmndrs/xr) | ✅ | ❌[^ios-webxr] | Entry point for immersive AR/VR sessions. |
26+
| **Face Tracking** | 🧭 Planned | - | N/A | N/A | Uses webcam + ML model; lightweight and fast. |
27+
| **Body/Pose Tracking** | 🧭 Planned | [WebXR Body Tracking](https://github.com/immersive-web/body-tracking) | N/A | N/A | Real-time skeletal tracking. GPU/WebGL acceleration required. |
28+
| **Hand Tracking** | 🧭 Planned | - | N/A | N/A | Supported on Chrome + Meta; ML fallback possible. |
29+
| **Plane/Surface Detection** | 🧭 Planned | [WebXR Hit Test API](https://immersive-web.github.io/hit-test)/ar.js (limited) | N/A | N/A | Enables AR object placement on flat surfaces. |
30+
| **Depth Sensing/Environment Mesh** | 🧭 Planned | [WebXR Depth Sensing API](https://immersive-web.github.io/depth-sensing) | N/A | N/A | Provides per-pixel depth; early spec. |
31+
| **SLAM/Visual Positioning (VPS)** | 🧭 Planned | Custom | N/A | N/A | Requires world map data; long-term goal. |
32+
| **Voice/Gesture Interaction** | 🧭 Planned | [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API)/MediaPipe Gestures | N/A | N/A | Enables multimodal input: voice, hand, gaze. |
33+
| **Mixed Reality Compositing** | 🧭 Planned | WebXR Layers/CanvasCaptureStream | N/A | N/A | Transparent overlays/live compositing. |
3534

3635
## Demos
3736

@@ -83,7 +82,11 @@ Install RDK and required peer dependencies:
8382
bun add @omnidotdev/rdk @ar-js-org/ar.js @react-three/fiber locar react react-dom three
8483
```
8584

86-
See [`apps/fiducial-demo`](./apps/fiducial-demo) for an example of usage. More demos will be added as more use cases beyond fiducial marker-based AR are implemented.
85+
See the demo applications for examples of usage:
86+
87+
- [`apps/fiducial-demo`](./apps/fiducial-demo): Fiducial marker tracking powered by AR.js
88+
- [`apps/geolocation-demo`](./apps/geolocation-demo): GPS-based AR powered by LocAR.js
89+
- [`apps/immersive-demo`](./apps/immersive-demo): WebXR powered by `@react-three/xr`
8790

8891
## Goals: the "Why"
8992

@@ -119,3 +122,5 @@ See Omni's [contributing docs](https://docs.omni.dev/contributing/overview).
119122
## License
120123

121124
The code in this repository is licensed under MIT, &copy; Omni LLC. See [LICENSE.md](LICENSE.md) for more information.
125+
126+
[^ios-webxr]: [iOS does not currently natively support WebXR](https://caniuse.com/webxr), but an iOS fallback is planned for the RDK immersive module.

‎Tiltfile‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,14 @@ local_resource(
2222
labels=["demo-geolocation"]
2323
)
2424

25+
local_resource(
26+
"dev-demo-immersive",
27+
serve_cmd="bun dev --filter rdk-immersive-demo",
28+
auto_init=False,
29+
trigger_mode=TRIGGER_MODE_MANUAL,
30+
labels=["demo-immersive"]
31+
)
32+
2533
local_resource(
2634
"build",
2735
"bun run build",

‎apps/fiducial-demo/package.json‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,17 +9,16 @@
99
"lint": "biome check .",
1010
"lint:fix": "biome check --write .",
1111
"format": "biome format --write .",
12-
"type-check": "tsc --noEmit",
1312
"clean": "rm -rf build"
1413
},
1514
"devDependencies": {
16-
"@biomejs/biome": "^2.3.2",
17-
"@types/react": "^19.2.2",
18-
"@types/react-dom": "^19.2.2",
19-
"@types/three": "^0.180.0",
20-
"@vitejs/plugin-react": "^5.1.0",
15+
"@biomejs/biome": "^2.3.8",
16+
"@types/react": "^19.2.7",
17+
"@types/react-dom": "^19.2.3",
18+
"@types/three": "^0.181.0",
19+
"@vitejs/plugin-react": "^5.1.1",
2120
"typescript": "^5.9.3",
22-
"vite": "^7.1.12",
21+
"vite": "^7.2.4",
2322
"vite-plugin-mkcert": "^1.17.9",
2423
"vite-tsconfig-paths": "^5.1.4"
2524
},
@@ -29,6 +28,6 @@
2928
"@react-three/fiber": "^9.4.0",
3029
"react": "^19.2.0",
3130
"react-dom": "^19.2.0",
32-
"three": "^0.180.0"
31+
"three": "^0.181.2"
3332
}
3433
}

‎apps/fiducial-demo/src/App.tsx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ const App = () => (
1616
<hemisphereLight intensity={0.6} />
1717
<directionalLight position={[5, 5, 5]} intensity={1} />
1818

19-
<XR cameraSource="video">
19+
<XR>
2020
<FiducialSession>
2121
<FiducialAnchor params={{ smooth: true }} patternUrl="/data/rdk.patt">
2222
<Supertorus

‎apps/geolocation-demo/package.json‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,26 +9,25 @@
99
"lint": "biome check .",
1010
"lint:fix": "biome check --write .",
1111
"format": "biome format --write .",
12-
"type-check": "tsc --noEmit",
1312
"clean": "rm -rf build"
1413
},
1514
"devDependencies": {
16-
"@biomejs/biome": "^2.3.2",
17-
"@types/react": "^19.2.2",
18-
"@types/react-dom": "^19.2.2",
19-
"@types/three": "^0.180.0",
20-
"@vitejs/plugin-react": "^5.1.0",
15+
"@biomejs/biome": "^2.3.8",
16+
"@types/react": "^19.2.7",
17+
"@types/react-dom": "^19.2.3",
18+
"@types/three": "^0.181.0",
19+
"@vitejs/plugin-react": "^5.1.1",
2120
"typescript": "^5.9.3",
22-
"vite": "^7.1.12",
21+
"vite": "^7.2.4",
2322
"vite-plugin-mkcert": "^1.17.9",
2423
"vite-tsconfig-paths": "^5.1.4"
2524
},
2625
"dependencies": {
2726
"@omnidotdev/rdk": "workspace:*",
2827
"@react-three/fiber": "^9.4.0",
29-
"locar": "^0.1.4",
28+
"locar": "^0.1.5",
3029
"react": "^19.2.0",
3130
"react-dom": "^19.2.0",
32-
"three": "^0.180.0"
31+
"three": "^0.181.2"
3332
}
3433
}

‎apps/geolocation-demo/src/App.tsx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ const App = () => (
2929
<directionalLight position={[10, 10, 10]} intensity={2} castShadow />
3030
<directionalLight position={[-10, 10, -10]} intensity={1} />
3131

32-
<XR cameraSource="video">
32+
<XR>
3333
<GeolocationSession
3434
options={
3535
{

‎apps/immersive-demo/README.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# RDK Immersive XR Demo
2+
3+
This demo showcases RDK's immersive XR capabilities powered by WebXR and `@react-three/xr`.
4+
5+
## Usage
6+
7+
### Development
8+
9+
```bash
10+
bun install
11+
bun dev
12+
```
13+
14+
### Production Build
15+
16+
```bash
17+
bun run build
18+
bun preview
19+
```

‎apps/immersive-demo/index.html‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8" />
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
6+
7+
<title>RDK Immersive XR Demo</title>
8+
9+
<style>
10+
body {
11+
width: 100%;
12+
height: 100vh;
13+
margin: 0;
14+
padding: 0;
15+
overflow: hidden;
16+
font-family: system-ui, -apple-system, sans-serif;
17+
}
18+
19+
#root {
20+
width: 100%;
21+
height: 100%;
22+
}
23+
</style>
24+
</head>
25+
26+
<body>
27+
<div id="root"></div>
28+
29+
<script type="module" src="/src/index.tsx"></script>
30+
</body>
31+
</html>

‎apps/immersive-demo/package.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"name": "rdk-immersive-demo",
3+
"private": true,
4+
"type": "module",
5+
"scripts": {
6+
"dev": "vite",
7+
"build": "vite build",
8+
"preview": "vite preview",
9+
"lint": "biome check .",
10+
"lint:fix": "biome check --write .",
11+
"format": "biome format --write .",
12+
"clean": "rm -rf build"
13+
},
14+
"devDependencies": {
15+
"@biomejs/biome": "2.3.8",
16+
"@types/react": "^19.2.7",
17+
"@types/react-dom": "^19.2.3",
18+
"@types/three": "^0.181.0",
19+
"@vitejs/plugin-react": "^5.1.1",
20+
"typescript": "~5.9.3",
21+
"vite": "^7.2.4",
22+
"vite-plugin-mkcert": "^1.17.9",
23+
"vite-tsconfig-paths": "^5.1.4"
24+
},
25+
"dependencies": {
26+
"@omnidotdev/rdk": "workspace:*",
27+
"@react-three/drei": "^10.7.7",
28+
"@react-three/fiber": "^9.4.0",
29+
"@react-three/xr": "^6.6.28",
30+
"react": "^19.2.0",
31+
"react-dom": "^19.2.0",
32+
"three": "^0.181.2"
33+
}
34+
}

0 commit comments

Comments
 (0)