Skip to content

Commit 6e70103

Browse files
gustavocidornelaswhoseoyster
gustavocidornelas
authored and
whoseoyster
committed
Improves organization of monitoring quickstart notebook
1 parent 7cbd63a commit 6e70103

File tree

1 file changed

+111
-135
lines changed

1 file changed

+111
-135
lines changed

‎examples/monitoring/quickstart/monitoring-quickstart.ipynb‎

Lines changed: 111 additions & 135 deletions
Original file line numberDiff line numberDiff line change
@@ -10,213 +10,131 @@
1010
"\n",
1111
"# <a id=\"top\">Monitoring quickstart</a>\n",
1212
"\n",
13-
"This notebook illustrates a typical monitoring flow using Openlayer.\n",
13+
"This notebook illustrates a typical monitoring flow using Openlayer. For more details, refer to the [How to set up monitoring guide](https://docs.openlayer.com/docs/set-up-monitoring) from the documentation.\n",
1414
"\n",
1515
"\n",
1616
"## <a id=\"toc\">Table of contents</a>\n",
1717
"\n",
18-
"1. [**Creating a project and an inference pipeline**](#inference-pipeline) \n",
18+
"1. [**Creating a project and an inference pipeline**](#inference-pipeline) \n",
1919
"\n",
20-
"2. [**Uploading a reference dataset**](#reference-dataset)\n",
20+
"2. [**Publishing batches of production data**](#publish-batches)\n",
2121
"\n",
22-
"3. [**Publishing batches of production data**](#publish-batches)\n",
22+
"3. [(Optional) **Uploading a reference dataset**](#reference-dataset)\n",
2323
"\n",
24+
"4. [(Optional) **Publishing ground truths**](#ground-truths)\n",
2425
"\n",
25-
"4. [**Publishing ground truths**](#ground-truths)"
26-
]
27-
},
28-
{
29-
"cell_type": "markdown",
30-
"id": "c4ea849d",
31-
"metadata": {},
32-
"source": [
33-
"## <a id=\"inference-pipeline\"> 1. Creating a project and an inference pipeline </a>\n",
34-
"\n",
35-
"[Back to top](#top)"
26+
"Before we start, let's download the sample data and import pandas."
3627
]
3728
},
3829
{
3930
"cell_type": "code",
4031
"execution_count": null,
41-
"id": "05f27b6c",
32+
"id": "3d193436",
4233
"metadata": {},
4334
"outputs": [],
4435
"source": [
45-
"!pip install openlayer"
36+
"%%bash\n",
37+
"\n",
38+
"if [ ! -e \"churn_train.csv\" ]; then\n",
39+
" curl \"https://openlayer-static-assets.s3.us-west-2.amazonaws.com/examples-datasets/monitoring/churn_train.csv\" --output \"churn_train.csv\"\n",
40+
"fi\n",
41+
"\n",
42+
"if [ ! -e \"prod_data_no_ground_truths.csv\" ]; then\n",
43+
" curl \"https://openlayer-static-assets.s3.us-west-2.amazonaws.com/examples-datasets/monitoring/prod_data_no_ground_truths.csv\" --output \"prod_data_no_ground_truths.csv\"\n",
44+
"fi\n",
45+
"\n",
46+
"if [ ! -e \"prod_ground_truths.csv\" ]; then\n",
47+
" curl \"https://openlayer-static-assets.s3.us-west-2.amazonaws.com/examples-datasets/monitoring/prod_ground_truths.csv\" --output \"prod_ground_truths.csv\"\n",
48+
"fi"
4649
]
4750
},
4851
{
4952
"cell_type": "code",
5053
"execution_count": null,
51-
"id": "8504e063",
54+
"id": "9dce8f60",
5255
"metadata": {},
5356
"outputs": [],
5457
"source": [
55-
"from openlayer.tasks import TaskType\n",
56-
"import openlayer\n",
57-
"\n",
58-
"client = openlayer.OpenlayerClient(\"YOUR_API_KEY_HERE\")\n",
59-
"project = client.create_or_load_project(\n",
60-
" name=\"Churn Prediction \",\n",
61-
" task_type=TaskType.TabularClassification,\n",
62-
")"
58+
"import pandas as pd"
6359
]
6460
},
6561
{
6662
"cell_type": "markdown",
67-
"id": "ed0c9bf6",
68-
"metadata": {},
69-
"source": [
70-
"Now that you are authenticated and have a project on the platform, it's time to create an inference pipeline. Creating an inference pipeline is what enables the monitoring capabilities in a project."
71-
]
72-
},
73-
{
74-
"cell_type": "code",
75-
"execution_count": null,
76-
"id": "147b5294",
63+
"id": "c4ea849d",
7764
"metadata": {},
78-
"outputs": [],
7965
"source": [
80-
"inference_pipeline = project.create_inference_pipeline()\n",
66+
"## <a id=\"inference-pipeline\"> 1. Creating a project and an inference pipeline </a>\n",
8167
"\n",
82-
"# Or \n",
83-
"# inference_pipeline = project.load_inference_pipeline(name=\"Production\")"
68+
"[Back to top](#top)"
8469
]
8570
},
8671
{
8772
"cell_type": "code",
8873
"execution_count": null,
89-
"id": "61e916c2",
74+
"id": "05f27b6c",
9075
"metadata": {},
9176
"outputs": [],
9277
"source": [
93-
"inference_pipeline"
94-
]
95-
},
96-
{
97-
"cell_type": "markdown",
98-
"id": "39592b32",
99-
"metadata": {},
100-
"source": [
101-
"## <a id=\"reference-dataset\"> 2. Uploading a reference dataset </a>\n",
102-
"\n",
103-
"[Back to top](#top)\n",
104-
"\n",
105-
"A reference dataset is optional, but it enables drift monitoring. Ideally, the reference dataset is a representative sample of the training set used to train the deployed model. In this section, we first load the dataset and then we upload it to Openlayer using the `upload_reference_dataframe` method.\n",
106-
"\n",
107-
"### <a id=\"download-reference\"> Downloading the data </a>"
78+
"!pip install openlayer"
10879
]
10980
},
11081
{
11182
"cell_type": "code",
11283
"execution_count": null,
113-
"id": "4b5be714",
84+
"id": "8504e063",
11485
"metadata": {},
11586
"outputs": [],
11687
"source": [
117-
"%%bash\n",
88+
"import openlayer\n",
11889
"\n",
119-
"if [ ! -e \"churn_train.csv\" ]; then\n",
120-
" curl \"https://openlayer-static-assets.s3.us-west-2.amazonaws.com/examples-datasets/monitoring/churn_train.csv\" --output \"churn_train.csv\"\n",
121-
"fi"
90+
"client = openlayer.OpenlayerClient(\"YOUR_API_KEY_HERE\")"
12291
]
12392
},
12493
{
12594
"cell_type": "code",
12695
"execution_count": null,
127-
"id": "31809ca9",
96+
"id": "5377494b",
12897
"metadata": {},
12998
"outputs": [],
13099
"source": [
131-
"import pandas as pd\n",
100+
"from openlayer.tasks import TaskType\n",
132101
"\n",
133-
"training_set = pd.read_csv(\"./churn_train.csv\")"
102+
"project = client.create_project(\n",
103+
" name=\"Churn Prediction\",\n",
104+
" task_type=TaskType.TabularClassification,\n",
105+
")"
134106
]
135107
},
136108
{
137109
"cell_type": "markdown",
138-
"id": "a6336802",
139-
"metadata": {},
140-
"source": [
141-
"### <a id=\"upload-reference\"> Uploading the dataset to Openlayer </a>"
142-
]
143-
},
144-
{
145-
"cell_type": "code",
146-
"execution_count": null,
147-
"id": "0f8e23e3",
110+
"id": "ed0c9bf6",
148111
"metadata": {},
149-
"outputs": [],
150112
"source": [
151-
"dataset_config = {\n",
152-
" \"categoricalFeatureNames\": [\"Gender\", \"Geography\"],\n",
153-
" \"classNames\": [\"Retained\", \"Exited\"],\n",
154-
" \"featureNames\": [\n",
155-
" \"CreditScore\", \n",
156-
" \"Geography\",\n",
157-
" \"Gender\",\n",
158-
" \"Age\", \n",
159-
" \"Tenure\",\n",
160-
" \"Balance\",\n",
161-
" \"NumOfProducts\",\n",
162-
" \"HasCrCard\",\n",
163-
" \"IsActiveMember\",\n",
164-
" \"EstimatedSalary\",\n",
165-
" \"AggregateRate\",\n",
166-
" \"Year\"\n",
167-
" ],\n",
168-
" \"labelColumnName\": \"Exited\",\n",
169-
" \"label\": \"training\"\n",
170-
"}"
113+
"Now that you are authenticated and have a project on the platform, it's time to create an inference pipeline. Creating an inference pipeline is what enables the monitoring capabilities in a project."
171114
]
172115
},
173116
{
174117
"cell_type": "code",
175118
"execution_count": null,
176-
"id": "f6cf719f",
119+
"id": "147b5294",
177120
"metadata": {},
178121
"outputs": [],
179122
"source": [
180-
"inference_pipeline.upload_reference_dataframe(\n",
181-
" dataset_df=training_set,\n",
182-
" dataset_config=dataset_config\n",
183-
")"
123+
"inference_pipeline = project.create_inference_pipeline()"
184124
]
185125
},
186126
{
187127
"cell_type": "markdown",
188128
"id": "3c8608ea",
189129
"metadata": {},
190130
"source": [
191-
"## <a id=\"publish-batches\"> 3. Publishing batches of data </a>\n",
131+
"## <a id=\"publish-batches\"> 2. Publishing production data </a>\n",
192132
"\n",
193133
"[Back to top](#top)\n",
194134
"\n",
195135
"In production, as the model makes predictions, the data can be published to Openlayer. This is done with the `publish_batch_data` method. \n",
196136
"\n",
197-
"The data published to Openlayer can have a column with **inference ids** and another with **timestamps** (UNIX ms format). These are both optional and, if not provided, will receive default values. The inference id is particularly important if you wish to publish ground truths at a later time. "
198-
]
199-
},
200-
{
201-
"cell_type": "markdown",
202-
"id": "e83afd5b",
203-
"metadata": {},
204-
"source": [
205-
"### <a id=\"download-batches\"> Download the data </a>"
206-
]
207-
},
208-
{
209-
"cell_type": "code",
210-
"execution_count": null,
211-
"id": "80a5462e",
212-
"metadata": {},
213-
"outputs": [],
214-
"source": [
215-
"%%bash\n",
216-
"\n",
217-
"if [ ! -e \"prod_data_no_ground_truths.csv\" ]; then\n",
218-
" curl \"https://openlayer-static-assets.s3.us-west-2.amazonaws.com/examples-datasets/monitoring/prod_data_no_ground_truths.csv\" --output \"prod_data_no_ground_truths.csv\"\n",
219-
"fi"
137+
"The data published to Openlayer can have a column with **inference ids** and another with **timestamps** (UNIX sec format). These are both optional and, if not provided, will receive default values. The inference id is particularly important if you wish to publish ground truths at a later time. "
220138
]
221139
},
222140
{
@@ -244,7 +162,7 @@
244162
{
245163
"cell_type": "code",
246164
"execution_count": null,
247-
"id": "cc126446",
165+
"id": "25b66229",
248166
"metadata": {},
249167
"outputs": [],
250168
"source": [
@@ -332,36 +250,94 @@
332250
},
333251
{
334252
"cell_type": "markdown",
335-
"id": "fbc1fca3",
253+
"id": "d00f6e8e",
336254
"metadata": {},
337255
"source": [
338-
"## <a id=\"ground-truths\"> 4. Publishing ground truths for past batches </a>\n",
256+
"**That's it!** You're now able to set up goals and alerts for your production data. The next sections are optional and enable some features on the platform."
257+
]
258+
},
259+
{
260+
"cell_type": "markdown",
261+
"id": "39592b32",
262+
"metadata": {},
263+
"source": [
264+
"## <a id=\"reference-dataset\"> 3. Uploading a reference dataset </a>\n",
339265
"\n",
340266
"[Back to top](#top)\n",
341267
"\n",
342-
"The `publish_ground_truths` method can be used to update the ground truths for batches of data already published to the Openlayer platform. The inference id is what gets used to merge the ground truths with the corresponding rows."
268+
"A reference dataset is optional, but it enables drift monitoring. Ideally, the reference dataset is a representative sample of the training set used to train the deployed model. In this section, we first load the dataset and then we upload it to Openlayer using the `upload_reference_dataframe` method."
269+
]
270+
},
271+
{
272+
"cell_type": "code",
273+
"execution_count": null,
274+
"id": "31809ca9",
275+
"metadata": {},
276+
"outputs": [],
277+
"source": [
278+
"training_set = pd.read_csv(\"./churn_train.csv\")"
343279
]
344280
},
345281
{
346282
"cell_type": "markdown",
347-
"id": "95bc9594",
283+
"id": "a6336802",
348284
"metadata": {},
349285
"source": [
350-
"### <a id=\"download-truth\"> Download the data </a>"
286+
"### <a id=\"upload-reference\"> Uploading the dataset to Openlayer </a>"
351287
]
352288
},
353289
{
354290
"cell_type": "code",
355291
"execution_count": null,
356-
"id": "9ab9790c",
292+
"id": "0f8e23e3",
357293
"metadata": {},
358294
"outputs": [],
359295
"source": [
360-
"%%bash\n",
296+
"dataset_config = {\n",
297+
" \"categoricalFeatureNames\": [\"Gender\", \"Geography\"],\n",
298+
" \"classNames\": [\"Retained\", \"Exited\"],\n",
299+
" \"featureNames\": [\n",
300+
" \"CreditScore\", \n",
301+
" \"Geography\",\n",
302+
" \"Gender\",\n",
303+
" \"Age\", \n",
304+
" \"Tenure\",\n",
305+
" \"Balance\",\n",
306+
" \"NumOfProducts\",\n",
307+
" \"HasCrCard\",\n",
308+
" \"IsActiveMember\",\n",
309+
" \"EstimatedSalary\",\n",
310+
" \"AggregateRate\",\n",
311+
" \"Year\"\n",
312+
" ],\n",
313+
" \"labelColumnName\": \"Exited\",\n",
314+
" \"label\": \"training\"\n",
315+
"}"
316+
]
317+
},
318+
{
319+
"cell_type": "code",
320+
"execution_count": null,
321+
"id": "f6cf719f",
322+
"metadata": {},
323+
"outputs": [],
324+
"source": [
325+
"inference_pipeline.upload_reference_dataframe(\n",
326+
" dataset_df=training_set,\n",
327+
" dataset_config=dataset_config\n",
328+
")"
329+
]
330+
},
331+
{
332+
"cell_type": "markdown",
333+
"id": "fbc1fca3",
334+
"metadata": {},
335+
"source": [
336+
"## <a id=\"ground-truths\"> 4. Publishing ground truths for past batches </a>\n",
361337
"\n",
362-
"if [ ! -e \"prod_ground_truths.csv\" ]; then\n",
363-
" curl \"https://openlayer-static-assets.s3.us-west-2.amazonaws.com/examples-datasets/monitoring/prod_ground_truths.csv\" --output \"prod_ground_truths.csv\"\n",
364-
"fi"
338+
"[Back to top](#top)\n",
339+
"\n",
340+
"The ground truths are needed to create Performance goals. The `publish_ground_truths` method can be used to update the ground truths for batches of data already published to the Openlayer platform. The inference id is what gets used to merge the ground truths with the corresponding rows."
365341
]
366342
},
367343
{

0 commit comments

Comments
 (0)