Skip to content

Commit 711350b

Browse files
fercerfercer
authored
Merge pull request #9 from fercer/dev-content
Updated content to add dask distributed to the material
2 parents eaadeac + f6fa53b commit 711350b

7 files changed

+902
-487
lines changed

‎notebooks/Solved_Zarr_and_Dask_for_large-scale_imaging-Part-1.ipynb‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2313,6 +2313,15 @@
23132313
"import tifffile"
23142314
]
23152315
},
2316+
{
2317+
"cell_type": "code",
2318+
"execution_count": null,
2319+
"metadata": {},
2320+
"outputs": [],
2321+
"source": [
2322+
"input_path = r\"C:\\Users\\Public\\Documents\\WSI_example\\CMU-1.svs\""
2323+
]
2324+
},
23162325
{
23172326
"cell_type": "code",
23182327
"execution_count": 48,
@@ -2332,7 +2341,7 @@
23322341
}
23332342
],
23342343
"source": [
2335-
"tif_store = tifffile.imread(r\"C:\\Users\\Public\\Documents\\WSI_example\\CMU-1.svs\", aszarr=True)\n",
2344+
"tif_store = tifffile.imread(input_path, aszarr=True)\n",
23362345
"tif_store"
23372346
]
23382347
},

‎notebooks/Solved_Zarr_and_Dask_for_large-scale_imaging-Part-2.ipynb‎

Lines changed: 755 additions & 415 deletions
Large diffs are not rendered by default.

‎notebooks/Zarr_and_Dask_for_large-scale_imaging-Part-1.ipynb‎

Lines changed: 10 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -256,23 +256,6 @@
256256
"- [ ] Create a delayed function (decorated with `@dask.delayed`) that can be applied lazily"
257257
]
258258
},
259-
{
260-
"cell_type": "code",
261-
"execution_count": null,
262-
"metadata": {},
263-
"outputs": [],
264-
"source": [
265-
"\n",
266-
"def grid_x(height, width, offset = 0):\n",
267-
" x = np.arange(offset, offset + width)\n",
268-
" return np.tile(x, (height, 1))\n",
269-
"\n",
270-
"\n",
271-
"def grid_y(height, width, offset = 0):\n",
272-
" y = np.arange(offset, offset + height)\n",
273-
" return np.tile(y[:, None], (1, width))"
274-
]
275-
},
276259
{
277260
"cell_type": "code",
278261
"execution_count": null,
@@ -288,13 +271,6 @@
288271
"## 1.4 Stack, Concatenate, and Block"
289272
]
290273
},
291-
{
292-
"cell_type": "markdown",
293-
"metadata": {},
294-
"source": [
295-
"- [ ] Find a way to create a $1000\\times1000$ pixels image by joining multiple $500\\times500$ pixels blocks "
296-
]
297-
},
298274
{
299275
"cell_type": "code",
300276
"execution_count": null,
@@ -331,6 +307,15 @@
331307
"import tifffile"
332308
]
333309
},
310+
{
311+
"cell_type": "code",
312+
"execution_count": null,
313+
"metadata": {},
314+
"outputs": [],
315+
"source": [
316+
"input_path = r\"C:\\Users\\Public\\Documents\\WSI_example\\CMU-1.svs\""
317+
]
318+
},
334319
{
335320
"cell_type": "code",
336321
"execution_count": null,
@@ -463,31 +448,6 @@
463448
"ℹ Dask arrays already work with `skimage` functions without calling `.compute()`"
464449
]
465450
},
466-
{
467-
"cell_type": "code",
468-
"execution_count": null,
469-
"metadata": {},
470-
"outputs": [],
471-
"source": [
472-
"from dask_image import ndfilters"
473-
]
474-
},
475-
{
476-
"cell_type": "code",
477-
"execution_count": null,
478-
"metadata": {},
479-
"outputs": [],
480-
"source": []
481-
},
482-
{
483-
"cell_type": "code",
484-
"execution_count": 1,
485-
"metadata": {},
486-
"outputs": [],
487-
"source": [
488-
"from skimage.filters import threshold_multiotsu"
489-
]
490-
},
491451
{
492452
"cell_type": "code",
493453
"execution_count": null,
@@ -502,13 +462,6 @@
502462
"- [ ] Visualize the results using `Matplotlib`"
503463
]
504464
},
505-
{
506-
"cell_type": "code",
507-
"execution_count": null,
508-
"metadata": {},
509-
"outputs": [],
510-
"source": []
511-
},
512465
{
513466
"cell_type": "code",
514467
"execution_count": null,
@@ -523,7 +476,7 @@
523476
"provenance": []
524477
},
525478
"kernelspec": {
526-
"display_name": "scale-up",
479+
"display_name": "Python 3 (ipykernel)",
527480
"language": "python",
528481
"name": "python3"
529482
},

‎notebooks/Zarr_and_Dask_for_large-scale_imaging-Part-2.ipynb‎

Lines changed: 121 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,42 @@
5252
"# 2. Compute on Zarr arrays"
5353
]
5454
},
55+
{
56+
"cell_type": "markdown",
57+
"metadata": {},
58+
"source": [
59+
"## 2.1 Set up a Dask cluster"
60+
]
61+
},
62+
{
63+
"cell_type": "markdown",
64+
"metadata": {},
65+
"source": [
66+
"- [ ] Use the Jupyter's Dask extension to start a distributed cluster\n",
67+
"- [ ] Click the \"+ New\" button at the bottom of the plugin\n",
68+
"\n",
69+
"![image](dask_extension.png)"
70+
]
71+
},
72+
{
73+
"cell_type": "markdown",
74+
"metadata": {},
75+
"source": [
76+
"- [ ] Click the \"<>\" button to inject the code needed to connect with this cluster\n",
77+
"\n",
78+
"![image](dask_extension_ready.png)"
79+
]
80+
},
5581
{
5682
"cell_type": "code",
57-
"execution_count": 1,
83+
"execution_count": null,
84+
"metadata": {},
85+
"outputs": [],
86+
"source": []
87+
},
88+
{
89+
"cell_type": "code",
90+
"execution_count": null,
5891
"metadata": {
5992
"executionInfo": {
6093
"elapsed": 976,
@@ -78,6 +111,22 @@
78111
"import skimage"
79112
]
80113
},
114+
{
115+
"cell_type": "markdown",
116+
"metadata": {},
117+
"source": [
118+
"Change this to the actual path where the image is stored"
119+
]
120+
},
121+
{
122+
"cell_type": "code",
123+
"execution_count": null,
124+
"metadata": {},
125+
"outputs": [],
126+
"source": [
127+
"input_path = r\"C:\\Users\\Public\\Documents\\WSI_example\\CMU-1_Crop.ome.zarr\""
128+
]
129+
},
81130
{
82131
"cell_type": "code",
83132
"execution_count": null,
@@ -124,7 +173,12 @@
124173
"outputs": [],
125174
"source": [
126175
"from cellpose import models, transforms\n",
127-
"import torch"
176+
"import torch\n",
177+
"\n",
178+
"gpu = torch.cuda.is_available()\n",
179+
"model_type = \"cyto3\"\n",
180+
"\n",
181+
"cellpose_model = models.CellposeModel(gpu=gpu, model_type=model_type)"
128182
]
129183
},
130184
{
@@ -147,7 +201,24 @@
147201
"execution_count": null,
148202
"metadata": {},
149203
"outputs": [],
150-
"source": []
204+
"source": [
205+
"img_t = transforms.convert_image(da_sel.compute(), channel_axis=2, channels=[0, 0])\n",
206+
"img_t = transforms.normalize_img(img_t, invert=False, axis=2)\n",
207+
"\n",
208+
"labels, _, _ = cellpose_model.eval(img_t[None, ...], diameter=None, flow_threshold=None, channels=[0, 0])"
209+
]
210+
},
211+
{
212+
"cell_type": "code",
213+
"execution_count": null,
214+
"metadata": {
215+
"id": "iRTdgTzi_KfU"
216+
},
217+
"outputs": [],
218+
"source": [
219+
"plt.imshow(da_sel)\n",
220+
"plt.imshow(skimage.color.label2rgb(labels), alpha=0.5)"
221+
]
151222
},
152223
{
153224
"cell_type": "markdown",
@@ -249,7 +320,7 @@
249320
},
250321
{
251322
"cell_type": "code",
252-
"execution_count": 78,
323+
"execution_count": null,
253324
"metadata": {},
254325
"outputs": [],
255326
"source": [
@@ -377,7 +448,7 @@
377448
},
378449
{
379450
"cell_type": "code",
380-
"execution_count": 267,
451+
"execution_count": null,
381452
"metadata": {
382453
"id": "2yzM2YNRcAFI"
383454
},
@@ -443,9 +514,39 @@
443514
{
444515
"cell_type": "code",
445516
"execution_count": null,
446-
"metadata": {},
517+
"metadata": {
518+
"cellView": "form",
519+
"id": "Bh6c9EVy9hST"
520+
},
447521
"outputs": [],
448-
"source": []
522+
"source": [
523+
"with ProgressBar():\n",
524+
" for s in range(1, 6):\n",
525+
" da_labels = da.from_zarr(\"CMU-1_Crop_labels_cellpose_cyto3.zarr\", component=str(s - 1))\n",
526+
"\n",
527+
" da_labels_ds = da_labels[::2, ::2]\n",
528+
" da_labels_ds = da_labels_ds.rechunk()\n",
529+
"\n",
530+
" da_labels_ds.to_zarr(\n",
531+
" \"CMU-1_Crop_labels_cellpose_cyto3.zarr\",\n",
532+
" component=str(s),\n",
533+
" write_empty_chunks=False,\n",
534+
" compressor=zarr.Blosc(clevel=9),\n",
535+
" overwrite=True\n",
536+
" )"
537+
]
538+
},
539+
{
540+
"cell_type": "code",
541+
"execution_count": null,
542+
"metadata": {
543+
"cellView": "form",
544+
"id": "Bh6c9EVy9hST"
545+
},
546+
"outputs": [],
547+
"source": [
548+
"z_labels = zarr.open(\"CMU-1_Crop_labels_cellpose_cyto3.zarr\", mode=\"a\")"
549+
]
449550
},
450551
{
451552
"cell_type": "markdown",
@@ -456,7 +557,7 @@
456557
},
457558
{
458559
"cell_type": "code",
459-
"execution_count": 282,
560+
"execution_count": null,
460561
"metadata": {
461562
"cellView": "form",
462563
"id": "Bh6c9EVy9hST"
@@ -483,7 +584,7 @@
483584
" \"path\" : str(s),\n",
484585
" \"coordinateTransformations\": [\n",
485586
" {\n",
486-
" \"scale\" : [ 1.0, 1.0, 1.0, 4.942E-4 / (2**s), 4.942E-4 / (2**s)],\n",
587+
" \"scale\" : [4.942E-4 / (2**s), 4.942E-4 / (2**s)],\n",
487588
" \"type\" : \"scale\"\n",
488589
" }\n",
489590
" ]\n",
@@ -495,6 +596,17 @@
495596
"]"
496597
]
497598
},
599+
{
600+
"cell_type": "code",
601+
"execution_count": null,
602+
"metadata": {
603+
"id": "giJcwW_F3vrF"
604+
},
605+
"outputs": [],
606+
"source": [
607+
"z_labels = zarr.open(\"CMU-1_Crop_labels_cellpose_cyto3.zarr\", mode=\"r\")"
608+
]
609+
},
498610
{
499611
"cell_type": "markdown",
500612
"metadata": {},

‎notebooks/dask_extension.png‎

31.3 KB
Loading

‎notebooks/dask_extension_ready.png‎

14.5 KB
Loading

‎requirements.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,12 @@ Additionally, install Jupyterlab to follow the notebooks, and its Dask extension
3535
conda install -c conda-forge jupyterlab=4.4.3 dask-labextension=7.0.0
3636
```
3737

38-
## A test image
38+
## Test images
3939

40-
We'll work with big microscopy images. An example of that kind of image can be found [here](https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/CMU-1.svs).
41-
So it is a good idea to have it downloaded as well.
40+
We'll work with big microscopy images: whole slide images. These are images obtained by scanning slides on which thin slices of tissue have been mounted. A number of examples of these kind of image can be found in the [OpenSlide](https://openslide.org) test data, for example in the [Aperio SVS format](https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/). These are TIFF variants that can be read with the [`Tifffile` Python library](https://github.com/cgohlke/tifffile).
4241

43-
A crop of that image can be found [here](https://drive.google.com/file/d/17owNcq_Or6aBAyUVE33fyHSS0VvKuHSw/view?usp=sharing), along with its `Zarr` version [here](https://drive.google.com/file/d/1BmNxOrO3vOFPR-PCnV00DYgFsD1sDu47/view?usp=sharing).
42+
For this workshop, we recommend downloading a smaller crop of the CMU-1 image from [our google drive](https://drive.google.com/file/d/17owNcq_Or6aBAyUVE33fyHSS0VvKuHSw/view?usp=sharing), along with [its `Zarr` version](https://drive.google.com/file/d/1BmNxOrO3vOFPR-PCnV00DYgFsD1sDu47/view?usp=sharing).
4443

45-
A smaller example image can be found [here](https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/CMU-1-Small-Region.svs), and its corresponding `Zarr` version [here](https://drive.google.com/file/d/1MifgafB5mhVAvqjzEAAR_zQajAU5Dcya/view?usp=drive_link).
44+
A alternative even smaller example image can be found [in the OpenSlide test data](https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/CMU-1-Small-Region.svs), and its corresponding `Zarr` version can also be downloaded from [our google drive](https://drive.google.com/file/d/1MifgafB5mhVAvqjzEAAR_zQajAU5Dcya/view?usp=drive_link).
45+
46+
**Please pre-download either pair of images before the tutorial!**

0 commit comments

Comments
 (0)