Update tutorial to use pytorch/text-generator (#1278)

Author: deliahu

docs/cluster-management/install.md

@@ -18,8 +18,8 @@ You must have [Docker](https://docs.docker.com/install) installed to run Cortex
 # clone the Cortex repository
 git clone -b master https://github.com/cortexlabs/cortex.git
 
-# navigate to the TensorFlow iris classification example
-cd cortex/examples/tensorflow/iris-classifier
+# navigate to the Pytorch text generator example
+cd cortex/examples/pytorch/text-generator
 
 # deploy the model as a realtime api
 cortex deploy
@@ -28,18 +28,18 @@ cortex deploy
 cortex get --watch
 
 # stream logs from the api
-cortex logs iris-classifier
+cortex logs text-generator
 
 # get the api's endpoint
-cortex get iris-classifier
+cortex get text-generator
 
 # classify a sample
-curl -X POST -H "Content-Type: application/json" \
-  -d '{ "sepal_length": 5.2, "sepal_width": 3.6, "petal_length": 1.4, "petal_width": 0.3 }' \
-  <API endpoint>
+curl <API endpoint> \
+  -X POST -H "Content-Type: application/json" \
+  -d '{"text": "machine learning is"}' \
 
 # delete the api
-cortex delete iris-classifier
+cortex delete text-generator
 ```
 
 ## Running at scale on AWS
@@ -56,12 +56,12 @@ cortex cluster up
 cortex env default aws
 ```
 
-You can now run the same commands shown above to deploy the iris classifier to AWS (if you didn't set the default CLI environment, add `--env aws` to the `cortex` commands).
+You can now run the same commands shown above to deploy the text generator to AWS (if you didn't set the default CLI environment, add `--env aws` to the `cortex` commands).
 
 ## Next steps
 
 <!-- CORTEX_VERSION_MINOR -->
-* Try the [tutorial](../../examples/sklearn/iris-classifier/README.md) to learn more about how to use Cortex.
+* Try the [tutorial](../../examples/pytorch/text-generator/README.md) to learn more about how to use Cortex.
 * Deploy one of our [examples](https://github.com/cortexlabs/cortex/tree/master/examples).
 * See our [exporting guide](../guides/exporting.md) for how to export your model to use in an API.
 * See [uninstall](uninstall.md) if you'd like to spin down your cluster.

docs/deployments/batch-api/api-configuration.md

@@ -41,7 +41,7 @@ See additional documentation for [compute](../compute.md), [networking](../netwo
     model_path: <string>  # S3 path to an exported model (e.g. s3://my-bucket/exported_model) (either this or 'models' must be provided)
     signature_key: <string>  # name of the signature def to use for prediction (required if your model has more than one signature def)
     models:  # use this when multiple models per API are desired (either this or 'model_path' must be provided)
-      - name: <string> # unique name for the model (e.g. iris-classifier) (required)
+      - name: <string> # unique name for the model (e.g. text-generator) (required)
         model_path: <string>  # S3 path to an exported model (e.g. s3://my-bucket/exported_model) (required)
         signature_key: <string>  # name of the signature def to use for prediction (required if your model has more than one signature def)
       ...
@@ -75,7 +75,7 @@ See additional documentation for [compute](../compute.md), [networking](../netwo
     path: <string>  # path to a python file with an ONNXPredictor class definition, relative to the Cortex root (required)
     model_path: <string>  # S3 path to an exported model (e.g. s3://my-bucket/exported_model.onnx) (either this or 'models' must be provided)
     models:  # use this when multiple models per API are desired (either this or 'model_path' must be provided)
-      - name: <string> # unique name for the model (e.g. iris-classifier) (required)
+      - name: <string> # unique name for the model (e.g. text-generator) (required)
         model_path: <string>  # S3 path to an exported model (e.g. s3://my-bucket/exported_model.onnx) (required)
         signature_key: <string>  # name of the signature def to use for prediction (required if your model has more than one signature def)
       ...

docs/deployments/batch-api/deployment.md

@@ -122,6 +122,6 @@ deleting my-api
 ## Additional resources
 
 <!-- CORTEX_VERSION_MINOR -->
-* [Tutorial](../../../examples/batch/image-classifier/README.md) provides a step-by-step walkthrough of deploying an iris classifier API
+* [Tutorial](../../../examples/batch/image-classifier/README.md) provides a step-by-step walkthrough of deploying an image classification batch API
 * [CLI documentation](../../miscellaneous/cli.md) lists all CLI commands
 * [Examples](https://github.com/cortexlabs/cortex/tree/master/examples/batch) demonstrate how to deploy models from common ML libraries

docs/deployments/batch-api/predictors.md

@@ -22,7 +22,7 @@ The following files can also be added at the root of the project's directory:
 For example, if your directory looks like this:
 
 ```text
-./iris-classifier/
+./my-classifier/
 ├── cortex.yaml
 ├── values.json
 ├── predictor.py
@@ -191,7 +191,7 @@ class TensorFlowPredictor:
 <!-- CORTEX_VERSION_MINOR -->
 Cortex provides a `tensorflow_client` to your Predictor's constructor. `tensorflow_client` is an instance of [TensorFlowClient](https://github.com/cortexlabs/cortex/tree/master/pkg/workloads/cortex/lib/client/tensorflow.py) that manages a connection to a TensorFlow Serving container to make predictions using your model. It should be saved as an instance variable in your Predictor, and your `predict()` function should call `tensorflow_client.predict()` to make an inference with your exported TensorFlow model. Preprocessing of the JSON payload and postprocessing of predictions can be implemented in your `predict()` function as well.
 
-When multiple models are defined using the Predictor's `models` field, the `tensorflow_client.predict()` method expects a second argument `model_name` which must hold the name of the model that you want to use for inference (for example: `self.client.predict(payload, "iris-classifier")`). See the [multi model guide](../../guides/multi-model.md#tensorflow-predictor) for more information.
+When multiple models are defined using the Predictor's `models` field, the `tensorflow_client.predict()` method expects a second argument `model_name` which must hold the name of the model that you want to use for inference (for example: `self.client.predict(payload, "text-generator")`). See the [multi model guide](../../guides/multi-model.md#tensorflow-predictor) for more information.
 
 For proper separation of concerns, it is recommended to use the constructor's `config` parameter for information such as from where to download the model and initialization files, or any configurable model parameters. You define `config` in your [API configuration](api-configuration.md), and it is passed through to your Predictor's constructor. The `config` parameters in the `API configuration` can be overridden by providing `config` in the job submission requests.
 
@@ -260,7 +260,7 @@ class ONNXPredictor:
 <!-- CORTEX_VERSION_MINOR -->
 Cortex provides an `onnx_client` to your Predictor's constructor. `onnx_client` is an instance of [ONNXClient](https://github.com/cortexlabs/cortex/tree/master/pkg/workloads/cortex/lib/client/onnx.py) that manages an ONNX Runtime session to make predictions using your model. It should be saved as an instance variable in your Predictor, and your `predict()` function should call `onnx_client.predict()` to make an inference with your exported ONNX model. Preprocessing of the JSON payload and postprocessing of predictions can be implemented in your `predict()` function as well.
 
-When multiple models are defined using the Predictor's `models` field, the `onnx_client.predict()` method expects a second argument `model_name` which must hold the name of the model that you want to use for inference (for example: `self.client.predict(model_input, "iris-classifier")`). See the [multi model guide](../../guides/multi-model.md#onnx-predictor) for more information.
+When multiple models are defined using the Predictor's `models` field, the `onnx_client.predict()` method expects a second argument `model_name` which must hold the name of the model that you want to use for inference (for example: `self.client.predict(model_input, "text-generator")`). See the [multi model guide](../../guides/multi-model.md#onnx-predictor) for more information.
 
 For proper separation of concerns, it is recommended to use the constructor's `config` parameter for information such as from where to download the model and initialization files, or any configurable model parameters. You define `config` in your [API configuration](api-configuration.md), and it is passed through to your Predictor's constructor. The `config` parameters in the `API configuration` can be overridden by providing `config` in the job submission requests.
 

docs/deployments/python-packages.md

@@ -7,7 +7,7 @@ _WARNING: you are on the master branch, please refer to the docs on the branch t
 You can install your required PyPI packages and import them in your Python files using pip. Cortex looks for a `requirements.txt` file in the top level Cortex project directory (i.e. the directory which contains `cortex.yaml`):
 
 ```text
-./iris-classifier/
+./my-classifier/
 ├── cortex.yaml
 ├── predictor.py
 ├── ...
@@ -56,7 +56,7 @@ On GitHub, you can generate a personal access token by following [these steps](h
 Python packages can also be installed by providing a `setup.py` that describes your project's modules. Here's an example directory structure:
 
 ```text
-./iris-classifier/
+./my-classifier/
 ├── cortex.yaml
 ├── predictor.py
 ├── ...
@@ -78,7 +78,7 @@ In this case, `requirements.txt` will have this form:
 Cortex supports installing Conda packages. We recommend only using Conda when your required packages are not available in PyPI. Cortex looks for a `conda-packages.txt` file in the top level Cortex project directory (i.e. the directory which contains `cortex.yaml`):
 
 ```text
-./iris-classifier/
+./my-classifier/
 ├── cortex.yaml
 ├── predictor.py
 ├── ...

docs/deployments/realtime-api.md

@@ -40,7 +40,7 @@ The Cortex Cluster will automatically scale based on the incoming traffic and th
 
 ## Next steps
 
-* Try the [tutorial](../../examples/sklearn/iris-classifier/README.md) to deploy a Realtime API locally or on AWS.
+* Try the [tutorial](../../examples/pytorch/text-generator/README.md) to deploy a Realtime API locally or on AWS.
 * See our [exporting guide](../guides/exporting.md) for how to export your model to use in a Realtime API.
 * See the [Predictor docs](realtime-api/predictors.md) for how to implement a Predictor class.
 * See the [API configuration docs](realtime-api/api-configuration.md) for a full list of features that can be used to deploy your Realtime API.

docs/deployments/realtime-api/api-configuration.md

@@ -63,7 +63,7 @@ See additional documentation for [parallelism](parallelism.md), [autoscaling](au
     model_path: <string>  # S3 path to an exported model (e.g. s3://my-bucket/exported_model) (either this or 'models' must be provided)
     signature_key: <string>  # name of the signature def to use for prediction (required if your model has more than one signature def)
     models:  # use this when multiple models per API are desired (either this or 'model_path' must be provided)
-      - name: <string> # unique name for the model (e.g. iris-classifier) (required)
+      - name: <string> # unique name for the model (e.g. text-generator) (required)
         model_path: <string>  # S3 path to an exported model (e.g. s3://my-bucket/exported_model) (required)
         signature_key: <string>  # name of the signature def to use for prediction (required if your model has more than one signature def)
       ...
@@ -119,7 +119,7 @@ See additional documentation for [parallelism](parallelism.md), [autoscaling](au
     path: <string>  # path to a python file with an ONNXPredictor class definition, relative to the Cortex root (required)
     model_path: <string>  # S3 path to an exported model (e.g. s3://my-bucket/exported_model.onnx) (either this or 'models' must be provided)
     models:  # use this when multiple models per API are desired (either this or 'model_path' must be provided)
-      - name: <string> # unique name for the model (e.g. iris-classifier) (required)
+      - name: <string> # unique name for the model (e.g. text-generator) (required)
         model_path: <string>  # S3 path to an exported model (e.g. s3://my-bucket/exported_model.onnx) (required)
         signature_key: <string>  # name of the signature def to use for prediction (required if your model has more than one signature def)
       ...

docs/deployments/realtime-api/deployment.md

@@ -26,7 +26,7 @@ $ cortex get my-api
 status   up-to-date   requested   last update   avg request   2XX
 live     1            1           1m            -             -
 
-endpoint: http://***.amazonaws.com/iris-classifier
+endpoint: http://***.amazonaws.com/text-generator
 ...
 ```
 
@@ -63,6 +63,6 @@ deleting my-api
 ## Additional resources
 
 <!-- CORTEX_VERSION_MINOR -->
-* [Tutorial](../../../examples/sklearn/iris-classifier/README.md) provides a step-by-step walkthrough of deploying an iris classifier API
+* [Tutorial](../../../examples/pytorch/text-generator/README.md) provides a step-by-step walkthrough of deploying a text generation API
 * [CLI documentation](../../miscellaneous/cli.md) lists all CLI commands
 * [Examples](https://github.com/cortexlabs/cortex/tree/master/examples) demonstrate how to deploy models from common ML libraries

docs/deployments/realtime-api/predictors.md

@@ -24,7 +24,7 @@ The following files can also be added at the root of the project's directory:
 For example, if your directory looks like this:
 
 ```text
-./iris-classifier/
+./my-classifier/
 ├── cortex.yaml
 ├── values.json
 ├── predictor.py
@@ -97,48 +97,25 @@ Your `predictor` method can return different types of objects such as `JSON`-par
 Many of the [examples](https://github.com/cortexlabs/cortex/tree/master/examples) use the Python Predictor, including all of the PyTorch examples.
 
 <!-- CORTEX_VERSION_MINOR -->
-Here is the Predictor for [examples/pytorch/iris-classifier](https://github.com/cortexlabs/cortex/tree/master/examples/pytorch/iris-classifier):
+Here is the Predictor for [examples/pytorch/text-generator](https://github.com/cortexlabs/cortex/tree/master/examples/pytorch/text-generator):
 
 ```python
-import re
 import torch
-import boto3
-from model import IrisNet
+from transformers import GPT2Tokenizer, GPT2LMHeadModel
 
-labels = ["setosa", "versicolor", "virginica"]
 
 class PythonPredictor:
     def __init__(self, config):
-        # download the model
-        bucket, key = re.match("s3://(.+?)/(.+)", config["model"]).groups()
-        s3 = boto3.client("s3")
-        s3.download_file(bucket, key, "model.pth")
-
-        # initialize the model
-        model = IrisNet()
-        model.load_state_dict(torch.load("model.pth"))
-        model.eval()
-
-        self.model = model
+        self.device = "cuda" if torch.cuda.is_available() else "cpu"
+        print(f"using device: {self.device}")
+        self.tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
+        self.model = GPT2LMHeadModel.from_pretrained("gpt2").to(self.device)
 
     def predict(self, payload):
-        # Convert the request to a tensor and pass it into the model
-        input_tensor = torch.FloatTensor(
-            [
-                [
-                    payload["sepal_length"],
-                    payload["sepal_width"],
-                    payload["petal_length"],
-                    payload["petal_width"],
-                ]
-            ]
-        )
-
-        # Run the prediction
-        output = self.model(input_tensor)
-
-        # Translate the model output to the corresponding label string
-        return labels[torch.argmax(output[0])]
+        input_length = len(payload["text"].split())
+        tokens = self.tokenizer.encode(payload["text"], return_tensors="pt").to(self.device)
+        prediction = self.model.generate(tokens, max_length=input_length + 20, do_sample=True)
+        return self.tokenizer.decode(prediction[0])
 ```
 
 ### Pre-installed packages
@@ -256,7 +233,7 @@ class TensorFlowPredictor:
 <!-- CORTEX_VERSION_MINOR -->
 Cortex provides a `tensorflow_client` to your Predictor's constructor. `tensorflow_client` is an instance of [TensorFlowClient](https://github.com/cortexlabs/cortex/tree/master/pkg/workloads/cortex/lib/client/tensorflow.py) that manages a connection to a TensorFlow Serving container to make predictions using your model. It should be saved as an instance variable in your Predictor, and your `predict()` function should call `tensorflow_client.predict()` to make an inference with your exported TensorFlow model. Preprocessing of the JSON payload and postprocessing of predictions can be implemented in your `predict()` function as well.
 
-When multiple models are defined using the Predictor's `models` field, the `tensorflow_client.predict()` method expects a second argument `model_name` which must hold the name of the model that you want to use for inference (for example: `self.client.predict(payload, "iris-classifier")`). See the [multi model guide](../../guides/multi-model.md#tensorflow-predictor) for more information.
+When multiple models are defined using the Predictor's `models` field, the `tensorflow_client.predict()` method expects a second argument `model_name` which must hold the name of the model that you want to use for inference (for example: `self.client.predict(payload, "text-generator")`). See the [multi model guide](../../guides/multi-model.md#tensorflow-predictor) for more information.
 
 For proper separation of concerns, it is recommended to use the constructor's `config` parameter for information such as configurable model parameters or download links for initialization files. You define `config` in your [API configuration](api-configuration.md), and it is passed through to your Predictor's constructor.
 
@@ -352,7 +329,7 @@ class ONNXPredictor:
 <!-- CORTEX_VERSION_MINOR -->
 Cortex provides an `onnx_client` to your Predictor's constructor. `onnx_client` is an instance of [ONNXClient](https://github.com/cortexlabs/cortex/tree/master/pkg/workloads/cortex/lib/client/onnx.py) that manages an ONNX Runtime session to make predictions using your model. It should be saved as an instance variable in your Predictor, and your `predict()` function should call `onnx_client.predict()` to make an inference with your exported ONNX model. Preprocessing of the JSON payload and postprocessing of predictions can be implemented in your `predict()` function as well.
 
-When multiple models are defined using the Predictor's `models` field, the `onnx_client.predict()` method expects a second argument `model_name` which must hold the name of the model that you want to use for inference (for example: `self.client.predict(model_input, "iris-classifier")`). See the [multi model guide](../../guides/multi-model.md#onnx-predictor) for more information.
+When multiple models are defined using the Predictor's `models` field, the `onnx_client.predict()` method expects a second argument `model_name` which must hold the name of the model that you want to use for inference (for example: `self.client.predict(model_input, "text-generator")`). See the [multi model guide](../../guides/multi-model.md#onnx-predictor) for more information.
 
 For proper separation of concerns, it is recommended to use the constructor's `config` parameter for information such as configurable model parameters or download links for initialization files. You define `config` in your [API configuration](api-configuration.md), and it is passed through to your Predictor's constructor.
 

docs/deployments/realtime-api/traffic-splitter.md

@@ -78,5 +78,5 @@ Note that this will not delete the Realtime APIs targeted by the Traffic Splitte
 
 <!-- CORTEX_VERSION_MINOR -->
 * [Traffic Splitter Tutorial](../../../examples/traffic-splitter/README.md) provides a step-by-step walkthrough for deploying an Traffic Splitter
-* [Realtime API Tutorial](../../../examples/sklearn/iris-classifier/README.md) provides a step-by-step walkthrough of deploying an iris classifier Realtime API
+* [Realtime API Tutorial](../../../examples/pytorch/text-generator/README.md) provides a step-by-step walkthrough of deploying a realtime API for text generation
 * [CLI documentation](../../miscellaneous/cli.md) lists all CLI commands