You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is the repo for **AI for Oceans** from Code.org.
7
7
8
-
Like the Dance Party repo, it is a standalone repo that is published as an NPM package, and consumed by the main repo.
8
+
Like the Dance Party repo, it is a standalone repo that is published as an [NPM package](https://www.npmjs.com/package/@code-dot-org/ml-activities), and consumed by the [main repo](https://github.com/code-dot-org/code-dot-org).
9
9
10
-
AI for Oceans was produced for the Hour of Code in 2019. It provides the student experience for the 5 interactive levels in the AI for Oceans script at https://studio.code.org/s/oceans.
10
+
**AI for Oceans** was produced for the Hour of Code in 2019. This module provides the student experience for the 5 interactive levels in the **AI for Oceans** script at https://studio.code.org/s/oceans.
11
11
12
12
# Design notes
13
13
14
14
## Modes
15
15
16
-
These 5 levels are invoked with a "mode" parameter:
16
+
These 5 levels are invoked with a "mode" (stored internally as `appMode`) parameter:
17
17
18
-
### fishvtrash
18
+
### `fishvtrash`
19
19
20
20
The user trains the AI to differentiate between fish versus trash, and then examine the results.
21
21
22
-
### creaturesvtrashdemo
22
+
### `creaturesvtrashdemo`
23
23
24
24
Next, the concept of non-fish sea creatures is introduced to show that AI is only as good as its training. In this mode, the experience is abbreviated: the user doesn't do training, but rather the mode demonstrates what happens when fish-specific training encounters non-fish.
25
25
26
-
### creaturesvtrash
26
+
### `creaturesvtrash`
27
27
28
28
In this mode, the user trains the AI again, but this time encountering fish, non-fish creatures, and trash.
29
29
30
-
### short
30
+
### `short`
31
31
32
32
In this mode, the user chooses from one of six adjectives and then categorizes fish based on that. The AI is trained on which fish fit into this arbitrary category or not, and then demonstrates this training.
33
33
34
-
### long
34
+
### `long`
35
35
36
36
In this mode, the user chooses from one of fifteen adjectives. With more subjectivity in this list, the user can explore more subtle implications of training and recognition.
37
37
38
+
## ML technology
38
39
39
40
Adapted from content at https://code.org/oceans:
40
41
41
-
Levels 2-4 use a pretrained model provided by the [TensorFlow](https://www.tensorflow.org/) [MobileNet](https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md) project. A MobileNet model is a [convolutional neural network](https://developers.google.com/machine-learning/practica/image-classification/convolutional-neural-networks) that has been trained on [ImageNet](http://www.image-net.org/), a dataset of over 14 million images hand-annotated with words such as "balloon" or "strawberry". In order to customize this model with the labeled training data the student generates in this activity, we use a technique called [Transfer Learning](https://en.wikipedia.org/wiki/Transfer_learning). Each image in the training dataset is fed to MobileNet, as pixels, to obtain a list of annotations that are most likely to apply to it. Then, for a new image, we feed it to MobileNet and compare its resulting list of annotations to those from the training dataset. We classify the new image with the same label (such as "fish" or "not fish") as the images from the training set with the most similar results.
42
+
> Levels 2-4 (`fishvtrash`, `creaturesvtrashdemo`, `creaturesvtrash`) use a pretrained model provided by the [TensorFlow](https://www.tensorflow.org/) [MobileNet](https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md) project. A MobileNet model is a [convolutional neural network](https://developers.google.com/machine-learning/practica/image-classification/convolutional-neural-networks) that has been trained on [ImageNet](http://www.image-net.org/), a dataset of over 14 million images hand-annotated with words such as "balloon" or "strawberry". In order to customize this model with the labeled training data the student generates in this activity, we use a technique called [Transfer Learning](https://en.wikipedia.org/wiki/Transfer_learning). Each image in the training dataset is fed to MobileNet, as pixels, to obtain a list of annotations that are most likely to apply to it. Then, for a new image, we feed it to MobileNet and compare its resulting list of annotations to those from the training dataset. We classify the new image with the same label (such as "fish" or "not fish") as the images from the training set with the most similar results.
42
43
43
-
Levels 6-8 use a [Support-Vector Machine](https://en.wikipedia.org/wiki/Support-vector_machine) (SVM). We look at each component of the fish (such as eyes, mouth, body) and assemble all of the metadata for the components (such as number of teeth, body shape) into a vector of numbers for each fish. We use these vectors to train the SVM. Based on the training data, the SVM separates the "space" of all possible fish into two parts, which correspond to the classes we are trying to learn (such as "blue" or "not blue").
44
+
> Levels 6-8 (`short`, `long`) use a [Support-Vector Machine](https://en.wikipedia.org/wiki/Support-vector_machine) (SVM). We look at each component of the fish (such as eyes, mouth, body) and assemble all of the metadata for the components (such as number of teeth, body shape) into a vector of numbers for each fish. We use these vectors to train the SVM. Based on the training data, the SVM separates the "space" of all possible fish into two parts, which correspond to the classes we are trying to learn (such as "blue" or "not blue").
44
45
45
-
## Screens
46
+
## Scenes
46
47
47
-
The AI for Oceans script forms a linear narrative structure. The app is designed to deliver the interactive levels for this script, one mode at a time, with no need to persist data to the browser or server between each level.
48
+
The **AI for Oceans** script presents a linear narrative structure. The app is designed to deliver the interactive levels for this script, one mode at a time, with no need to persist data to the browser or server between each level.
48
49
49
-
The app itself contains a variety of "scenes", with each mode using a different subset. The scenes are as follows:
50
+
The app itself contains a variety of "scenes", with each mode using a different subset. The scenes (known as `currentMode` internally) are as follows:
The user shows the results of the predictions. The user can toggle between the matching & non-matching sets. In short & long, the user can click each item to view additional information about the AI's recognition.
70
86
87
+
In the `short` and `long` modes, the pond also has a metapanel which can show general information about the ML processing, or, when a fish is selected, specific information about that fish's categorization:
The app uses two layers in the DOM. Underneath, a canvas provides the background and all the sprites. On top, a regular DOM uses HTML elements to provide the user interface.
95
+
The app uses two layers in the DOM. Underneath, a canvas provides the background and all the sprites. On top, a regular DOM uses HTML elements to provide the user interface. The HTML interface is implemented in React.
74
96
75
97
The app is fully responsive by scaling the canvas and also scaling the size of the HTML elements correspondingly. The UI simply shrinks to match the underlying canvas.
76
98
@@ -94,12 +116,13 @@ See the implementation at https://github.com/code-dot-org/ml-activities/blob/mai
94
116
95
117
This simple system enabled the team to add a detailed narrative voice to the script, and allowed a variety of team members to contribute text.
96
118
119
+
# Additional information
120
+
97
121
## Common operations
98
122
99
123
The documentation for common operations for AI Lab is comprehensive and should apply to this project too: https://github.com/code-dot-org/ml-playground#common-operations
0 commit comments