Skip to content

Commit e54122c

Browse files
nasif-conasif-co
committed
Adding the depthEstimation documentation
1 parent 14f6c5c commit e54122c

File tree

6 files changed

+257
-0
lines changed

6 files changed

+257
-0
lines changed

‎docs/reference/depthestimation.md‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# DepthEstimation
2+
## Description
3+
4+
The ml5.js DepthEstimation module offers a pretrained model for inferring depth maps **from images of people**, estimating the distance between each pixel and the camera that captured the image. The model used is TensorFlow's [AR Portrait Depth](https://blog.tensorflow.org/2022/05/portrait-depth-api-turning-single-image.html) which is designed specifically for portrait images and does not perform very well with images of other types of subjects.
5+
6+
## Quick Start
7+
8+
Get up and running with the [webcam example](https://editor.p5js.org/nasif-co/sketches/Pep6DjEtD), which shows a realtime depth map estimated from the webcam video.
9+
10+
</br>
11+
12+
[DEMO](iframes/depthestimation ":include :type=iframe width=100% height=550px")
13+
14+
## Initialization and options
15+
To get started, create an instance of `ml5.depthEstimation` in your preload function, to allow the model to load
16+
```js
17+
function preload() {
18+
depthEstimator = ml5.depthEstimation({
19+
// Use options here to configure how the model behaves, like:
20+
flipHorizontal: true,
21+
// See a list of the available options below
22+
});
23+
}
24+
```
25+
26+
The main available options are:
27+
28+
- `flipHorizontal`: Used to mirror the depth map horizontally
29+
- Default: `false`
30+
- Accepted values: `true`, `false` (boolean).
31+
- `dilationFactor`: Sets how many pixels around the detected edges of a person should be ignored. This is useful because depth values are inaccurate and noisy around the contours.
32+
- Default: `4`
33+
- Accepted values: `0` to `10` (integer).
34+
- `colormap`: Defines how the depth map is drawn; either Grayscale, mapping depth from black (far) to white (close), or Color, mapping depth using the whole range of color hues.
35+
- Default: `'GRAYSCALE'`
36+
- Accepted values: `'GRAYSCALE'` or `'COLOR'` (string).
37+
- `minDepth`: Sets the depth value that will map to the 'close' color.
38+
- Default: `0.2`
39+
- Accepted values: `0` to `1` (float). Must be less than `maxDepth`.
40+
- `maxDepth`: Sets the depth value that will map to the 'far' color.
41+
- Default: `0.75`
42+
- Accepted values: `0` to `1` (float). Must be greater than `minDepth`.
43+
- `normalizeDynamically`: Whether to do a manual mapping (using maxDepth and minDepth) or do it dynamically; recording the lowest and highest values detected in the depth map on every frame and using them as the mapping limits. This means that any particular color will not always represent the same absolute distance from the screen.
44+
- Default: `false`
45+
- Accepted values: `true`, `false` (boolean). Setting to `true` will ignore `minDepth` and `maxDepth` options
46+
- `normalizationSmoothingFactor`: Only used if normalizing dynamically. Sets how much to smooth the varying maximum and minimum depth values detected during normalization. Higher values result in faster reaction to changes. Lower values result in smoother changes.
47+
- Default: `0.5`
48+
- Accepted values: `0` to `1` (float).
49+
50+
51+
### p5.js 2.0
52+
You can also use this module with p5.js 2.0! Instead of creating `ml5.depthEstimation` in preload, do it using your async `setup()` and `await`:
53+
```js
54+
async function setup() {
55+
// Load the depth estimation model
56+
depthEstimator = await ml5.depthEstimation({
57+
// Options go here
58+
});
59+
//the rest of your setup goes here
60+
}
61+
```
62+
63+
## Estimating depth
64+
As with many other ml5 models, you have two options to run depth estimation on the image, video or webcam of your choice: _Continuous Estimation_ and _Single Shot Estimation_ .
65+
66+
### Continuous estimation
67+
This method is used to continuously estimate depth on every frame of a video or webcam feed.
68+
```js
69+
// Make sure to load the model in preload or async in p5 2.0!
70+
function setup() {
71+
// Create the video capture element
72+
webcam = createCapture(VIDEO);
73+
74+
// Start continuous depth estimation on the webcam feed
75+
depthEstimator.estimateStart(webcam, gotResults);
76+
}
77+
78+
function gotResults(result) {
79+
// The most recent depth map is in the result object!
80+
}
81+
```
82+
Using this method, the depth estimator will take care of doing estimation of a frame and waiting for it to finish before working on the next frame. Any time a depth map is ready, it will fire the callback function to provide it.
83+
84+
### Single shot estimation
85+
This method is used to estimate depth once, for a single image:
86+
```js
87+
// Make sure to load the image and the model in preload or asyn in p5 2.0!
88+
function setup() {
89+
// Estimate depth from the loaded image
90+
depthEstimator.estimate(img, gotResults);
91+
}
92+
93+
function gotResults(result) {
94+
// The depth map is in the result object!
95+
}
96+
```
97+
Because the estimation takes time, we pass in a callback function that will fire when estimation is ready. The `estimate` method is called in setup because it **will only run once**. If calling it multiple times, it is prudent to wait for each operation to finish before starting the next one.
98+
99+
## Using the depth result
100+
Whenever the callback function fires, we have acces to the depth result that contains all the depth information.
101+
```js
102+
let depthMap;
103+
104+
function gotResults(result) {
105+
// Save the depth result in a variable
106+
depthMap = result;
107+
}
108+
```
109+
This result is an object, and it contains the following properties:
110+
- `image`: A p5 image of the depth map in the chosen colormap.
111+
- Type: `p5.Image` object
112+
- `getDepthAt(x, y)`: Function that returns the depth value of the pixel in the location passed in to the x and y parameters.
113+
- Type: function.
114+
- `data`: The raw depth values for each pixel in a two dimensional array format.
115+
- Type: 2D array
116+
- `mask`: The mask of the people detected in the image and for whom depth values were estimated. It can be used directly with the `mask()` function in p5.
117+
- Type: `p5.Image` object
118+
- `sourceFrame`: The exact frame that was analyzed to create the depth map. Because depth estimation is not immediate, the result can fall out of sync from the source video. By using this value instead of the video, the depth data is guaranteed to be in sync. See it in action in the 'Keeping data in sync' section of [this article](https://ml5js.org/blog/bringing-depth-estimation/).
119+
- Type: `p5.Image`
120+
- `width`: Width of the depth map
121+
- Type: number
122+
- `height`: Height of the depth map
123+
- Type: number
124+
125+
## Examples
126+
- [Webcam](https://editor.p5js.org/nasif-co/sketches/Pep6DjEtD): Show the live depth map of the video captured by the webcam.
127+
- [Video](https://editor.p5js.org/nasif-co/sketches/vifmzXg6o): Generate the depth map of a video file as it plays.
128+
- [Single Image](https://editor.p5js.org/nasif-co/sketches/_TcZofgrt): Depth map of an image using single-shot estimation.
129+
- [Mask Background](https://editor.p5js.org/nasif-co/sketches/Z_1xMhUPl): Showcases how to mask out the background from the depth result.
130+
- [Point Cloud](https://editor.p5js.org/nasif-co/sketches/VbT8hEoDz): Creates a live 3D point cloud visualization of the webcam video.
131+
- [Mesh](https://editor.p5js.org/nasif-co/sketches/X-e1DEZr4): Creates a live 3D mesh geometry of the webcam video.
132+
133+
## Learn more
134+
Check out the community article [Finding the Z-axis](https://ml5js.org/blog/bringing-depth-estimation/) to learn more about the way depth estimation was implemented into ml5.

‎docs/reference/iframes/depthestimation/index.html‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
4+
<head>
5+
<meta charset="UTF-8">
6+
<meta http-equiv="X-UA-Compatible" content="IE=edge">
7+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
8+
9+
<!-- iframe CSS & JS -->
10+
<link rel="stylesheet" type="text/css" href="/css/global.css">
11+
<link rel="stylesheet" type="text/css" href="../style-iframe.css">
12+
<script src="../script-navbar.js"></script>
13+
</head>
14+
15+
<body>
16+
<script>
17+
// Navbar will be added on the top of the page.
18+
// You can provide a link to the script file on the p5 web editor.
19+
navbar("https://editor.p5js.org/nasif-co/sketches/Pep6DjEtD");
20+
</script>
21+
22+
<iframe src="ready.html" name="script-iframe"></iframe>
23+
</body>
24+
25+
</html>

‎docs/reference/iframes/depthestimation/ready.html‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
4+
<head>
5+
<meta charset="UTF-8">
6+
<meta http-equiv="X-UA-Compatible" content="IE=edge">
7+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
8+
9+
<!-- iframe CSS -->
10+
<link rel="stylesheet" type="text/css" href="../../../css/global.css">
11+
<link rel="stylesheet" type="text/css" href="../style-iframe.css">
12+
</head>
13+
14+
<body>
15+
<div class="content-container">
16+
<h1>DepthEstimation Webcam Demo</h1>
17+
<p>
18+
Press the <b>▶︎</b> button to try it out.
19+
Make sure to allow access to the webcam.
20+
</br></br>
21+
Click <b>Open in p5.js Web Editor</b> to view the complete code.
22+
</p>
23+
</div>
24+
</body>
25+
26+
</html>

‎docs/reference/iframes/depthestimation/run.html‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
4+
<head>
5+
<meta charset="UTF-8">
6+
<meta http-equiv="X-UA-Compatible" content="IE=edge">
7+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
8+
9+
<!-- iframe Libraries -->
10+
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.11.8/p5.min.js"></script>
11+
<script src="https://assets.editor.p5js.org/670c5b6fda1724bfcf2e1772/ccf1c3e5-f3d2-4459-b0aa-9347d950c747.js?v=1759711481857"></script>
12+
13+
<!-- iframe CSS & JS -->
14+
<link rel="stylesheet" type="text/css" href="../../../css/global.css">
15+
<link rel="stylesheet" type="text/css" href="../style-iframe.css">
16+
</head>
17+
18+
<body>
19+
<script src="script.js"></script>
20+
</body>
21+
22+
</html>

‎docs/reference/iframes/depthestimation/script.js‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
/*
2+
* 👋 Hello! This is an ml5.js example made and shared with ❤️.
3+
* Learn more about the ml5.js project: https://ml5js.org/
4+
* ml5.js license and Code of Conduct: https://github.com/ml5js/ml5-next-gen/blob/main/LICENSE.md
5+
*
6+
* This example demonstrates running depth estimation real-time on your webcam.
7+
*/
8+
9+
let depthEstimator;
10+
let webcam;
11+
let depthMap;
12+
13+
// Video dimensions
14+
let videoWidth = 640;
15+
let videoHeight = 480;
16+
17+
function preload() {
18+
// Load and start the depth estimation model
19+
depthEstimator = ml5.depthEstimation();
20+
}
21+
22+
function setup() {
23+
// Create a canvas the size of the webcam video
24+
createCanvas(videoWidth, videoHeight);
25+
26+
// Create the video capture element
27+
webcam = createCapture(VIDEO);
28+
webcam.size(videoWidth, videoHeight); // Set video size
29+
webcam.hide(); // Hide the default HTML video element
30+
31+
// Start continuous depth estimation on the webcam feed and make "gotResults" the callback function
32+
depthEstimator.estimateStart(webcam, gotResults);
33+
}
34+
35+
function draw() {
36+
background(0);
37+
38+
// If depth estimation results are available
39+
if (depthMap) {
40+
// Draw the depth map
41+
image(depthMap.image, 0, 0);
42+
}
43+
}
44+
45+
// Callback function that receives the depth estimation results
46+
function gotResults(result) {
47+
// Store the latest result in the global variable depthMap
48+
depthMap = result;
49+
}

‎docs/sidebar.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
- [ImageClassifier](/reference/image-classifier.md)
2626
- [SoundClassifier](/reference/sound-classifier.md)
2727
- [Sentiment](/reference/sentiment.md)
28+
- [DepthEstimation](/reference/depthestimation.md)
2829

2930
<div class="sidebar-spacer">&nbsp;</div>
3031

0 commit comments

Comments
 (0)