You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
?> For more information on importing the ml5 library, check out the [Getting Started](/?id=set-up-ml5js) page.
31
+
32
+
Create an instance of `ml5.depthEstimation` in your preload function, to allow the model to load
16
33
```js
17
34
functionpreload() {
18
35
depthEstimator =ml5.depthEstimation({
19
-
// Use options here to configure how the model behaves, like:
20
-
flipHorizontal:true,
21
-
// See a list of the available options below
36
+
// Use options here to configure how the model behaves.
37
+
// See a full list of options below, in the 'Methods' section of this reference page
22
38
});
23
39
}
24
40
```
41
+
For the full list of options, check out the [methods section](#ml5depthestimation) below!
25
42
26
-
The main available options are:
27
-
28
-
-`flipHorizontal`: Used to mirror the depth map horizontally
29
-
- Default: `false`
30
-
- Accepted values: `true`, `false` (boolean).
31
-
-`dilationFactor`: Sets how many pixels around the detected edges of a person should be ignored. This is useful because depth values are inaccurate and noisy around the contours.
32
-
- Default: `4`
33
-
- Accepted values: `0` to `10` (integer).
34
-
-`colormap`: Defines how the depth map is drawn; either Grayscale, mapping depth from black (far) to white (close), or Color, mapping depth using the whole range of color hues.
35
-
- Default: `'GRAYSCALE'`
36
-
- Accepted values: `'GRAYSCALE'` or `'COLOR'` (string).
37
-
-`minDepth`: Sets the depth value that will map to the 'close' color.
38
-
- Default: `0.2`
39
-
- Accepted values: `0` to `1` (float). Must be less than `maxDepth`.
40
-
-`maxDepth`: Sets the depth value that will map to the 'far' color.
41
-
- Default: `0.75`
42
-
- Accepted values: `0` to `1` (float). Must be greater than `minDepth`.
43
-
-`normalizeDynamically`: Whether to do a manual mapping (using maxDepth and minDepth) or do it dynamically; recording the lowest and highest values detected in the depth map on every frame and using them as the mapping limits. This means that any particular color will not always represent the same absolute distance from the screen.
44
-
- Default: `false`
45
-
- Accepted values: `true`, `false` (boolean). Setting to `true` will ignore `minDepth` and `maxDepth` options
46
-
-`normalizationSmoothingFactor`: Only used if normalizing dynamically. Sets how much to smooth the varying maximum and minimum depth values detected during normalization. Higher values result in faster reaction to changes. Lower values result in smoother changes.
47
-
- Default: `0.5`
48
-
- Accepted values: `0` to `1` (float).
49
-
50
-
51
-
### p5.js 2.0
43
+
#### p5.js 2.0
52
44
You can also use this module with p5.js 2.0! Instead of creating `ml5.depthEstimation` in preload, do it using your async `setup()` and `await`:
53
45
```js
54
46
asyncfunctionsetup() {
@@ -60,10 +52,12 @@ async function setup() {
60
52
}
61
53
```
62
54
63
-
## Estimating depth
55
+
###Estimating depth
64
56
As with many other ml5 models, you have two options to run depth estimation on the image, video or webcam of your choice: _Continuous Estimation_ and _Single Shot Estimation_ .
65
57
66
-
### Continuous estimation
58
+
For any of these, make sure you first load the image, video or start the webcam capture. This is the media we will pass to the model.
59
+
60
+
#### Continuous estimation
67
61
This method is used to continuously estimate depth on every frame of a video or webcam feed.
68
62
```js
69
63
// Make sure to load the model in preload or async in p5 2.0!
@@ -81,7 +75,7 @@ function gotResults(result) {
81
75
```
82
76
Using this method, the depth estimator will take care of doing estimation of a frame and waiting for it to finish before working on the next frame. Any time a depth map is ready, it will fire the callback function to provide it.
83
77
84
-
### Single shot estimation
78
+
####Single shot estimation
85
79
This method is used to estimate depth once, for a single image:
86
80
```js
87
81
// Make sure to load the image and the model in preload or asyn in p5 2.0!
@@ -96,7 +90,7 @@ function gotResults(result) {
96
90
```
97
91
Because the estimation takes time, we pass in a callback function that will fire when estimation is ready. The `estimate` method is called in setup because it **will only run once**. If calling it multiple times, it is prudent to wait for each operation to finish before starting the next one.
98
92
99
-
## Using the depth result
93
+
###Using the depth result
100
94
Whenever the callback function fires, we have acces to the depth result that contains all the depth information.
101
95
```js
102
96
let depthMap;
@@ -106,29 +100,109 @@ function gotResults(result) {
106
100
depthMap = result;
107
101
}
108
102
```
109
-
This result is an object, and it contains the following properties:
103
+
The `result` is a `DepthEstimationResult` object that contains the depth map and other relevant data. Save it to variable so you can use it inside the p5 `draw()` loop!
104
+
105
+
For more information, on the structure and data contained in the result, check out [DepthEstimationResult Structure](#depthestimationresult) below.
106
+
107
+
## Methods
108
+
109
+
### ml5.depthEstimation()
110
+
111
+
This method is used to initialize the depth estimation object.
112
+
113
+
In p5.js 1.x.x, use it inside the `preload()` function:
- `flipHorizontal`: Used to mirror the depth map horizontally
129
+
- Default: `false`
130
+
- Accepted values: `true`, `false` (boolean).
131
+
- `dilationFactor`: Sets how many pixels around the detected edges of a person should be ignored. This is useful because depth values are inaccurate and noisy around the contours.
132
+
- Default: `4`
133
+
- Accepted values: `0` to `10` (integer).
134
+
- `colormap`: Defines how the depth map is drawn; either Grayscale, mapping depth from black (far) to white (close), or Color, mapping depth using the whole range of color hues.
135
+
- Default: `'GRAYSCALE'`
136
+
- Accepted values: `'GRAYSCALE'` or `'COLOR'` (string).
137
+
- `minDepth`: Sets the depth value that will map to the 'close' color.
138
+
- Default: `0.2`
139
+
- Accepted values: `0` to `1` (float). Must be less than `maxDepth`.
140
+
- `maxDepth`: Sets the depth value that will map to the 'far' color.
141
+
- Default: `0.75`
142
+
- Accepted values: `0` to `1` (float). Must be greater than `minDepth`.
143
+
- `normalizeDynamically`: Whether to do a manual mapping (using maxDepth and minDepth) or do it dynamically; recording the lowest and highest values detected in the depth map on every frame and using them as the mapping limits. This means that any particular color will not always represent the same absolute distance from the screen.
144
+
- Default: `false`
145
+
- Accepted values: `true`, `false` (boolean). Setting to `true` will ignore `minDepth` and `maxDepth` options
146
+
- `normalizationSmoothingFactor`: Only used if normalizing dynamically. Sets how much to smooth the varying maximum and minimum depth values detected during normalization. Higher values result in faster reaction to changes. Lower values result in smoother changes.
147
+
- Default: `0.5`
148
+
- Accepted values: `0` to `1` (float).
149
+
150
+
**Returns:**
151
+
152
+
- **Object**: `depthEstimation` object that contains the methods to run estimation.
153
+
154
+
### depthEstimator.estimateStart()
155
+
This method is used for _Continuous Estimation_: estimating depth on a video/webcam continuously, for each frame. Calling it will initiate an estimation loop, running until `depthEstimator.estimateStop()` is called.
156
+
157
+
```js
158
+
depthEstimator.estimateStart(media, callback)
159
+
```
160
+
161
+
**Parameters:**
162
+
163
+
- **media**: An HTML or p5.js image, video, or canvas element to estimate a depth map for continuously.
164
+
- **callback(result)**: A callback function that will be called *every time* an estimation result is available. The `result` is a `DepthEstimationResult` object. Check the section on it below for details on its structure.
165
+
166
+
### depthEstimator.estimateStop()
167
+
This method is used to stop an estimation loop that was previously started by a call to `depthEstimator.estimateStart()`.
168
+
169
+
```js
170
+
depthEstimator.estimateStop()
171
+
```
172
+
173
+
### depthEstimator.estimate()
174
+
This method is used for _Single Shot Estimation_: estimating depth one time on a single image or video/webcam frame.
175
+
176
+
```js
177
+
depthEstimator.estimate(media, callback)
178
+
```
179
+
180
+
**Parameters:**
181
+
182
+
- **media**: An HTML or p5.js image, video, or canvas element to estimate a depth map for.
183
+
- **callback(result)**: A callback function that will be called when estimation is ready. The `result` is a `DepthEstimationResult` object. Check the section on it below for details on its structure.
184
+
185
+
186
+
### DepthEstimationResult
187
+
This is the object that is passed as an argument to the callback functions of `depthEstimator.estimateStart()` or `depthEstimator.estimate()`. It contains the result of the depth estimation process and other useful relevant data
188
+
189
+
These are its properties:
190
+
110
191
- `image`: A p5 image of the depth map in the chosen colormap.
111
192
- Type: `p5.Image` object
112
-
-`getDepthAt(x, y)`: Function that returns the depth value of the pixel in the location passed in to the x and y parameters.
113
-
- Type: function.
193
+
- `getDepthAt(x, y)`: Function that returns the depth value of the pixel at `x, y`.
194
+
- Type: Function.
195
+
- Returns: Floating point number in the 0 - 1 range.
114
196
- `data`: The raw depth values for each pixel in a two dimensional array format.
115
-
- Type: 2D array
197
+
- Type: 2D array of floating point numbers in the 0 - 1 range.
116
198
- `mask`: The mask of the people detected in the image and for whom depth values were estimated. It can be used directly with the `mask()` function in p5.
117
199
- Type: `p5.Image` object
118
-
-`sourceFrame`: The exact frame that was analyzed to create the depth map. Because depth estimation is not immediate, the result can fall out of sync from the source video. By using this value instead of the video, the depth data is guaranteed to be in sync. See it in action in the 'Keeping data in sync' section of [this article](https://ml5js.org/blog/bringing-depth-estimation/).
200
+
- `sourceFrame`: The exact frame that was analyzed to create the depth map. Because depth estimation is not immediate, the result can fall out of sync from the source video. By using this value instead of the video, the depth data is guaranteed to be in sync. See a [demo sketch](https://editor.p5js.org/nasif-co/sketches/Z_1xMhUPl) showcasing the difference.
119
201
- Type: `p5.Image`
120
202
- `width`: Width of the depth map
121
-
- Type: number
203
+
- Type: number (integer)
122
204
- `height`: Height of the depth map
123
-
- Type: number
124
-
125
-
## Examples
126
-
-[Webcam](https://editor.p5js.org/nasif-co/sketches/Pep6DjEtD): Show the live depth map of the video captured by the webcam.
127
-
-[Video](https://editor.p5js.org/nasif-co/sketches/vifmzXg6o): Generate the depth map of a video file as it plays.
128
-
-[Single Image](https://editor.p5js.org/nasif-co/sketches/_TcZofgrt): Depth map of an image using single-shot estimation.
129
-
-[Mask Background](https://editor.p5js.org/nasif-co/sketches/Z_1xMhUPl): Showcases how to mask out the background from the depth result.
130
-
-[Point Cloud](https://editor.p5js.org/nasif-co/sketches/VbT8hEoDz): Creates a live 3D point cloud visualization of the webcam video.
131
-
-[Mesh](https://editor.p5js.org/nasif-co/sketches/X-e1DEZr4): Creates a live 3D mesh geometry of the webcam video.
205
+
- Type: number (integer)
132
206
133
207
## Learn more
134
208
Check out the community article [Finding the Z-axis](https://ml5js.org/blog/bringing-depth-estimation/) to learn more about the way depth estimation was implemented in ml5.
0 commit comments