Skip to content

Commit cf79c41

Browse files
authored
[SN-92] create a foundry + send to annotate workflow example (#1339)
1 parent 88d8f61 commit cf79c41

File tree

1 file changed

+376
-0
lines changed

1 file changed

+376
-0
lines changed
Lines changed: 376 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,376 @@
1+
{
2+
"nbformat": 4,
3+
"nbformat_minor": 2,
4+
"metadata": {},
5+
"cells": [
6+
{
7+
"metadata": {},
8+
"source": [
9+
"<td>\n",
10+
" <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=190/></a>\n",
11+
"</td>"
12+
],
13+
"cell_type": "markdown"
14+
},
15+
{
16+
"metadata": {},
17+
"source": [
18+
"<td>\n",
19+
"<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/master/examples/foundry/object_detection.ipynb\" target=\"_blank\"><img\n",
20+
"src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
21+
"</td>\n",
22+
"\n",
23+
"<td>\n",
24+
"<a href=\"https://github.com/Labelbox/labelbox-python/tree/master/examples/foundry/object_detection.ipynb\" target=\"_blank\"><img\n",
25+
"src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
26+
"</td>"
27+
],
28+
"cell_type": "markdown"
29+
},
30+
{
31+
"metadata": {},
32+
"source": [
33+
"# Foundry overview\n",
34+
"\n",
35+
"This notebook is used to go over the basic of foundry through the Python SDK"
36+
],
37+
"cell_type": "markdown"
38+
},
39+
{
40+
"metadata": {},
41+
"source": [
42+
"Foundry incorporates foundational models into your Labelbox workflow. You can use Foundry to:\n",
43+
"\n",
44+
"* Predict (infer) labels from your data\n",
45+
"* Compare the performance of different foundational models with your data and ontologies.\n",
46+
"* Prototype, diagnose, and refine a machine learning app to solve specific business needs.\n",
47+
"\n",
48+
"Foundry creates model runs that predict data row annotations based on your input."
49+
],
50+
"cell_type": "markdown"
51+
},
52+
{
53+
"metadata": {},
54+
"source": [
55+
"!pip install -q labelbox"
56+
],
57+
"cell_type": "code",
58+
"outputs": [],
59+
"execution_count": null
60+
},
61+
{
62+
"metadata": {},
63+
"source": [
64+
"import labelbox as lb\n",
65+
"from labelbox.schema.conflict_resolution_strategy import ConflictResolutionStrategy\n",
66+
"import uuid"
67+
],
68+
"cell_type": "code",
69+
"outputs": [],
70+
"execution_count": null
71+
},
72+
{
73+
"metadata": {},
74+
"source": [
75+
"# API Key and Client\n",
76+
"\n",
77+
"Provide a valid API key below in order to properly connect to the Labelbox Client."
78+
],
79+
"cell_type": "markdown"
80+
},
81+
{
82+
"metadata": {},
83+
"source": [
84+
"# Add your API key\n",
85+
"API_KEY = \"\"\n",
86+
"# To get your API key go to: Workspace settings -> API -> Create API Key\n",
87+
"client = lb.Client(api_key=API_KEY)"
88+
],
89+
"cell_type": "code",
90+
"outputs": [],
91+
"execution_count": null
92+
},
93+
{
94+
"metadata": {},
95+
"source": [
96+
"# End-to-end example: Run foundry and send to annotate from catalog"
97+
],
98+
"cell_type": "markdown"
99+
},
100+
{
101+
"metadata": {},
102+
"source": [
103+
"## Step 1: Import data rows into catelog"
104+
],
105+
"cell_type": "markdown"
106+
},
107+
{
108+
"metadata": {},
109+
"source": [
110+
"# send a sample image as data row for a dataset\n",
111+
"global_key = str(uuid.uuid4())\n",
112+
"\n",
113+
"test_img_url = {\n",
114+
" \"row_data\":\n",
115+
" \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg\",\n",
116+
" \"global_key\":\n",
117+
" global_key\n",
118+
"}\n",
119+
"\n",
120+
"dataset = client.create_dataset(name=\"foundry-demo-dataset\")\n",
121+
"task = dataset.create_data_rows([test_img_url])\n",
122+
"task.wait_till_done()\n",
123+
"\n",
124+
"print(f\"Errors: {task.errors}\")\n",
125+
"print(f\"Failed data rows: {task.failed_data_rows}\")"
126+
],
127+
"cell_type": "code",
128+
"outputs": [
129+
{
130+
"name": "stdout",
131+
"output_type": "stream",
132+
"text": [
133+
"Errors: None\n",
134+
"Failed data rows: None\n"
135+
]
136+
}
137+
],
138+
"execution_count": null
139+
},
140+
{
141+
"metadata": {},
142+
"source": [
143+
"## Step 2: Create/select an ontology that matches model\n",
144+
"\n",
145+
"Your project should have the correct ontology setup with all the tools and classifications supported for your model and data type.\n",
146+
"\n",
147+
"For example, when using Amazon Rekognition you would need to create a bounding box annotation for your ontology since it only supports object detection. Likewise when using YOLOv8 you would need to create a classification annotation for your ontology since it only supports image classification. \n",
148+
"\n",
149+
"In this tutorial, we will use Amazon Rekognition to detect objects in an image dataset. "
150+
],
151+
"cell_type": "markdown"
152+
},
153+
{
154+
"metadata": {},
155+
"source": [
156+
"# Create ontology with two bounding boxes that is included with Amazon Rekognition: Car and Person \n",
157+
"ontology_builder = lb.OntologyBuilder(\n",
158+
" classifications=[],\n",
159+
" tools=[\n",
160+
" lb.Tool(tool=lb.Tool.Type.BBOX, name=\"Car\"),\n",
161+
" lb.Tool(tool=lb.Tool.Type.BBOX, name=\"Person\")\n",
162+
" ]\n",
163+
")\n",
164+
"\n",
165+
"ontology = client.create_ontology(\"Image Bounding Box Annotation Demo Foundry\",\n",
166+
" ontology_builder.asdict(),\n",
167+
" media_type=lb.MediaType.Image)"
168+
],
169+
"cell_type": "code",
170+
"outputs": [],
171+
"execution_count": null
172+
},
173+
{
174+
"metadata": {},
175+
"source": [
176+
"## Step 3: Create a labeling project\n",
177+
"\n",
178+
"Connect the ontology to the labeling project"
179+
],
180+
"cell_type": "markdown"
181+
},
182+
{
183+
"metadata": {},
184+
"source": [
185+
"project = client.create_project(name=\"Foundry Image Demo\",\n",
186+
" media_type=lb.MediaType.Image)\n",
187+
"\n",
188+
"project.setup_editor(ontology)"
189+
],
190+
"cell_type": "code",
191+
"outputs": [],
192+
"execution_count": null
193+
},
194+
{
195+
"metadata": {},
196+
"source": [
197+
"## Step 4: Create foundry application in UI\n",
198+
"\n",
199+
"Currently we do not support this workflow through the SDK\n",
200+
"#### Workflow:\n",
201+
"\n",
202+
"1. Navigate to model and select ***Create*** > ***App***\n",
203+
"\n",
204+
"2. Select ***Amazon Rekognition*** and name your foundry application\n",
205+
"\n",
206+
"3. Customize your perimeters and then select ***Save & Create***"
207+
],
208+
"cell_type": "markdown"
209+
},
210+
{
211+
"metadata": {},
212+
"source": [
213+
"#Select your foundry application inside the UI and copy the APP ID from the top right corner\n",
214+
"AMAZON_REKOGNITION_APP_ID = \"\""
215+
],
216+
"cell_type": "code",
217+
"outputs": [],
218+
"execution_count": null
219+
},
220+
{
221+
"metadata": {},
222+
"source": [
223+
"## Step 5: Run foundry app on data rows\n",
224+
"\n",
225+
"This step is meant to generate annotations that can later be reused as pre-labels in a project. You must provide your app ID from the previous step for this method to run, please see the [Foundry Apps Guide](https://docs.labelbox.com/docs/foundry-apps#run-app-using-sdk) for more information.\n"
226+
],
227+
"cell_type": "markdown"
228+
},
229+
{
230+
"metadata": {},
231+
"source": [
232+
"task = client.run_foundry_app(model_run_name=f\"Amazon-{str(uuid.uuid4())}\",\n",
233+
" data_rows=lb.GlobalKeys(\n",
234+
" [global_key] # Provide a list of global keys \n",
235+
" ), \n",
236+
" app_id=AMAZON_REKOGNITION_APP_ID)\n",
237+
"\n",
238+
"task.wait_till_done()\n",
239+
"\n",
240+
"print(f\"Errors: {task.errors}\") \n",
241+
"\n",
242+
"#Obtain model run ID from task\n",
243+
"MODEL_RUN_ID = task.metadata[\"modelRunId\"]"
244+
],
245+
"cell_type": "code",
246+
"outputs": [
247+
{
248+
"name": "stdout",
249+
"output_type": "stream",
250+
"text": [
251+
"Errors: None\n"
252+
]
253+
}
254+
],
255+
"execution_count": null
256+
},
257+
{
258+
"metadata": {},
259+
"source": [
260+
"## Step 6: Map ontology through the UI\n",
261+
"\n",
262+
"Mapping a model's ontology to a project's ontology is currently not supported through the SDK, however, to showcase how to send foundry predictions to a project, we are going to generate the mapping of the foundry app ontology to the project ontology through the UI.\n",
263+
"\n",
264+
"#### Workflow\n",
265+
"\n",
266+
"1. Navigate to your dataset you created for your model run\n",
267+
"2. Select **Select all** in the top right corner\n",
268+
"3. Select **Manage selection** > **Send to Annotate**\n",
269+
"4. Specify the project we created from the project dropdown menu\n",
270+
"5. Selecting a workflow step is not required since we are not sending annotations from the UI to a project using this notebook \n",
271+
"6. Mark **Include model predictions** then scroll down and select **Map**\n",
272+
"7. Select the incoming ontology and matching ontology feature for both Car and Person\n",
273+
"8. Once both features are mapped press the **Copy ontology mapping as JSON** in the top right corner\n",
274+
"9. Do not save this configuration, since we are not sending predictions to a project using this UI modal. We will be sending predictions in the following steps using the SDK"
275+
],
276+
"cell_type": "markdown"
277+
},
278+
{
279+
"metadata": {},
280+
"source": [
281+
"# Copy map ontology through the UI then paste JSON file here\n",
282+
"PREDICTIONS_ONTOLOGY_MAPPING = {}"
283+
],
284+
"cell_type": "code",
285+
"outputs": [],
286+
"execution_count": null
287+
},
288+
{
289+
"metadata": {},
290+
"source": [
291+
"## Step 7: Send model generated annotations from catalog to annotate\n",
292+
"\n",
293+
"### Parameters\n",
294+
"\n",
295+
"When you send predicted data rows to annotate from catalog, you may choose to include or exclude certain parameters, at a minimum a predictions_ontology_mapping will need to be provided:\n",
296+
"\n",
297+
"* `predictions_ontology_mapping`\n",
298+
" - A dictionary containing the mapping of the model's ontology feature schema ids to the project's ontology feature schema ids\n",
299+
"* `exclude_data_rows_in_project`\n",
300+
" - Excludes data rows that are already in the project. \n",
301+
"* `override_existing_annotations_rule` \n",
302+
" - The strategy defining how to handle conflicts in classifications between the data rows that already exist in the project and incoming predictions from the source model run or annotations from the source project. \n",
303+
" * Defaults to ConflictResolutionStrategy.KeepExisting\n",
304+
" * Options include:\n",
305+
" * ConflictResolutionStrategy.KeepExisting\n",
306+
" * ConflictResolutionStrategy.OverrideWithPredictions\n",
307+
" * ConflictResolutionStrategy.OverrideWithAnnotations\n",
308+
"* `param batch_priority`\n",
309+
" - The priority of the batch.\n"
310+
],
311+
"cell_type": "markdown"
312+
},
313+
{
314+
"metadata": {},
315+
"source": [
316+
"model_run = client.get_model_run(MODEL_RUN_ID)\n",
317+
"\n",
318+
"send_to_annotations_params = {\n",
319+
" \"predictions_ontology_mapping\": PREDICTIONS_ONTOLOGY_MAPPING,\n",
320+
" \"exclude_data_rows_in_project\": False,\n",
321+
" \"override_existing_annotations_rule\": ConflictResolutionStrategy.OverrideWithPredictions,\n",
322+
" \"batch_priority\": 5,\n",
323+
"}\n",
324+
"\n",
325+
"\n",
326+
"task = model_run.send_to_annotate_from_model(\n",
327+
" destination_project_id=project.uid,\n",
328+
" task_queue_id=None, #ID of workflow task, set ID to None if you want to convert pre-labels to ground truths or obtain task queue id through project.task_queues().\n",
329+
" batch_name=\"Foundry Demo Batch\",\n",
330+
" data_rows=lb.GlobalKeys(\n",
331+
" [global_key] # Provide a list of global keys from foundry app task\n",
332+
" ),\n",
333+
" params=send_to_annotations_params\n",
334+
" )\n",
335+
"\n",
336+
"task.wait_till_done()\n",
337+
"\n",
338+
"print(f\"Errors: {task.errors}\")"
339+
],
340+
"cell_type": "code",
341+
"outputs": [
342+
{
343+
"name": "stdout",
344+
"output_type": "stream",
345+
"text": [
346+
"Errors: None\n"
347+
]
348+
}
349+
],
350+
"execution_count": null
351+
},
352+
{
353+
"metadata": {},
354+
"source": [
355+
"## Clean up"
356+
],
357+
"cell_type": "markdown"
358+
},
359+
{
360+
"metadata": {},
361+
"source": [
362+
"# project.delete()\n",
363+
"# dataset.delete()\n",
364+
"# model_run.delete()"
365+
],
366+
"cell_type": "code",
367+
"outputs": [],
368+
"execution_count": null
369+
},
370+
{
371+
"metadata": {},
372+
"source": [],
373+
"cell_type": "markdown"
374+
}
375+
]
376+
}

0 commit comments

Comments
 (0)