Skip to content

Commit 5f4b6e4

Browse files
RobertLucianRobertLucian
authored
Add guide to avoid Docker Hub rate limit (#1576)
1 parent ca44cee commit 5f4b6e4

File tree

2 files changed

+151
-0
lines changed

2 files changed

+151
-0
lines changed

‎docs/guides/docker-hub-rate-limiting.md‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
# Docker Hub rate limiting
2+
3+
_WARNING: you are on the master branch, please refer to the docs on the branch that matches your `cortex version`_
4+
5+
Docker Hub's [newly enforced rate-limiting policy](https://www.docker.com/increase-rate-limits) can negatively impact your cluster. This is much likelier to be an issue if you've set `subnet_visibility: private` in your cluster configuration file, since with private subnets, all requests from all nodes are routed through the NAT Gateway, and will therefore have the same IP address (docker imposes the rate limit per IP address). If you haven't specified `subnet_visibility` or have set `subnet_visibility: public`, this is less likely to be an issue for you, since each instance will have its own IP address.
6+
7+
We are actively working on a long term resolution to this problem. In the meantime, there are two ways to avoid this issue:
8+
9+
## Paid Docker Hub subscription
10+
11+
One option is to pay for the Docker Hub subscription to remove the limit on the number of image pulls. Docker Hub's updated pricing model allows unlimited pulls on a _Pro_ subscription for individuals as described [here](https://www.docker.com/pricing).
12+
13+
By default, the Cortex cluster pulls the images as an anonymous user. Follow [this guide](private-docker.md) to configure your Cortex cluster to pull the images as an authenticated user.
14+
15+
## Push to AWS ECR (Elastic Container Registry)
16+
17+
You can configure the Cortex cluster to use images from a different registry. A good choice is ECR on AWS. When an ECR repository resides in the same region as your Cortex cluster, there are no costs incurred when pulling images.
18+
19+
### Step 1
20+
21+
Make sure you have the [aws](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html) and [docker](https://docs.docker.com/get-docker/) CLIs installed.
22+
23+
### Step 2
24+
25+
Export the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables in your current shell, or run `aws configure`. These credentials must have access to push to ECR.
26+
27+
### Step 3
28+
29+
Choose a region for your cluster and ECR repositories. In this guide, we'll assume the region is `us-west-2`.
30+
31+
Also, take note of your AWS account ID. The account ID can be found in the _My Account_ section of your AWS console.
32+
33+
### Step 4
34+
35+
You can use the script below to push the images from Docker Hub to your ECR registry. Make sure to update the `ecr_region`, `aws_account_id`, and `cortex_version` variables at the top of the file. Copy-paste the contents into a new file (e.g. `ecr.sh`), and then run `chmod +x ecr.sh`, followed by `./ecr.sh`. It is recommended to run this from an EC2 instance in the same region as your ECR repository, since it will be much faster.
36+
37+
```bash
38+
#!/bin/bash
39+
set -euo pipefail
40+
41+
# user set variables
42+
ecr_region="us-west-2"
43+
aws_account_id="620970939130" # example account ID
44+
cortex_version="0.22.1"
45+
46+
source_registry="docker.io/cortexlabs"
47+
destination_registry="${aws_account_id}.dkr.ecr.${ecr_region}.amazonaws.com/cortexlabs"
48+
49+
aws ecr get-login-password --region $ecr_region | docker login --username AWS --password-stdin $destination_registry
50+
51+
# images for the cluster
52+
cluster_images=(
53+
"manager"
54+
"request-monitor"
55+
"downloader"
56+
"operator"
57+
"cluster-autoscaler"
58+
"metrics-server"
59+
"inferentia"
60+
"neuron-rtd"
61+
"nvidia"
62+
"fluentd"
63+
"statsd"
64+
"istio-proxy"
65+
"istio-pilot"
66+
)
67+
68+
# images for the APIs (you may delete any images that your APIs don't use)
69+
api_images=(
70+
"python-predictor-cpu"
71+
"python-predictor-gpu"
72+
"python-predictor-inf"
73+
"tensorflow-serving-cpu"
74+
"tensorflow-serving-gpu"
75+
"tensorflow-serving-inf"
76+
"tensorflow-predictor"
77+
"onnx-predictor-cpu"
78+
"onnx-predictor-gpu"
79+
"python-predictor-cpu-slim"
80+
"python-predictor-gpu-slim"
81+
"python-predictor-inf-slim"
82+
"tensorflow-predictor-slim"
83+
"onnx-predictor-cpu-slim"
84+
"onnx-predictor-gpu-slim"
85+
)
86+
images=( "${cluster_images[@]}" "${api_images[@]}" )
87+
88+
extra_tags_for_slim_python_predictor=(
89+
"cuda10.0"
90+
"cuda10.1"
91+
"cuda10.2"
92+
"cuda11.0"
93+
)
94+
95+
# create the image repositories
96+
for image in "${images[@]}"; do
97+
aws ecr create-repository --repository-name=cortexlabs/$image --region=$ecr_region || true
98+
done
99+
100+
# pull the images from Docker Hub and push them to ECR
101+
for image in "${images[@]}"; do
102+
if [ "$image" = "python-predictor-gpu-slim" ]; then
103+
for extra_tag in "${extra_tags_for_slim_python_predictor[@]}"; do
104+
docker image pull "$source_registry/$image:$cortex_version-$extra_tag"
105+
docker image tag "$source_registry/$image:$cortex_version-$extra_tag" "$destination_registry/$image:$cortex_version-$extra_tag"
106+
docker image push "$destination_registry/$image:$cortex_version-$extra_tag"
107+
echo
108+
done
109+
else
110+
docker image pull "$source_registry/$image:$cortex_version"
111+
docker image tag "$source_registry/$image:$cortex_version" "$destination_registry/$image:$cortex_version"
112+
docker image push "$destination_registry/$image:$cortex_version"
113+
echo
114+
fi
115+
done
116+
117+
echo "###############################################"
118+
echo
119+
echo "add the following images to your cortex cluster configuration file (e.g. cluster.yaml):"
120+
echo "-----------------------------------------------"
121+
for cluster_image in "${cluster_images[@]}"; do
122+
echo "image_$cluster_image: $destination_registry/$cluster_image:$cortex_version"
123+
done
124+
echo -e "-----------------------------------------------\n"
125+
126+
echo "use the following images in your API configuration files (e.g. cortex.yaml):"
127+
echo "-----------------------------------------------"
128+
for api_image in "${api_images[@]}"; do
129+
if [ "$api_image" = "python-predictor-gpu-slim" ]; then
130+
for extra_tag in "${extra_tags_for_slim_python_predictor[@]}"; do
131+
echo "$destination_registry/$api_image:$cortex_version-$extra_tag"
132+
done
133+
else
134+
echo "$destination_registry/$api_image:$cortex_version"
135+
fi
136+
done
137+
echo "-----------------------------------------------"
138+
```
139+
140+
The first list of images that were printed (the cluster images) can be directly copy-pasted in your [cluster configuration file](../cluster-management/config.md) before spinning up your cluster.
141+
142+
The second list of images that were printed (the API images) can be used in your [API configuration files](../deployments/realtime-api/api-configuration.md). The images are specified in `predictor.image` (and `predictor.tensorflow_serving_image` for APIs with `kind: tensorflow`). Be advised that by default, the Docker Hub images are used for your predictors, so you will need to specify your ECR image paths for all of your APIs.
143+
144+
## Step 6
145+
146+
Spin up your Cortex cluster using your updated cluster configuration file (e.g. `cortex cluster up --config cluster.yaml`).
147+
148+
## Cleanup
149+
150+
You can delete your ECR images from the [AWS ECR dashboard](https://console.aws.amazon.com/ecr/repositories) (set your region in the upper right corner). Make sure all of your Cortex clusters have been deleted before deleting any ECR images.

‎docs/summary.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@
7676
* [SSH into worker instance](guides/ssh-instance.md)
7777
* [Single node deployment](guides/single-node-deployment.md)
7878
* [Set up kubectl](guides/kubectl-setup.md)
79+
* [Docker Hub rate limiting](guides/docker-hub-rate-limiting.md)
7980
* [Private docker registry](guides/private-docker.md)
8081
* [Set up REST API Gateway](guides/rest-api-gateway.md)
8182
* [Install CLI on Windows](guides/windows-cli.md)

0 commit comments

Comments
 (0)