chore: add pytorch base image dockerfiles

Author: starpit

deploy/bases/README.md

@@ -0,0 +1,52 @@
+# Guide for building multi-arch docker images
+
+The Dockerfiles contained in the subdirectories build small-ish (~1Gi)
+base images that helps with some common PyTorch scenarios.
+
+- `lightning`: 827Mi
+- `pyarrow`: 1.01Gi (183Mi delta over `lightning`)
+- `transformers`: 1.09Gi (8Mi delta over `pyarrow`)
+
+## Creating a `docker buildx` build config
+
+To build for both amd64 and aarch4 (e.g. for running on Apple
+Silicon), you can use `docker buildx`.
+
+First, you will need to create a buildx config, because the default
+one does not seem to work well for cross-platform builds. Here we name
+that config `multiarch`, but it can be named anything.
+
+We have found that using the `kubernetes` driver works better than the
+default `docker-container` driver.
+
+```shell
+docker buildx create --name multiarch --use --driver kubernetes --driver-opt=qemu.install=true
+```
+
+The `--use` part tells buildx to set this as the default builder.
+
+_Note_: We tell docker buildx to use qemu, so that the amd64 images
+can be built on an aarch64 host.
+
+_Warning_: If your KUBECONFIG env var is colon-separated, this is
+something that `docker buildx` apparently does not support. You will
+need to run any `docker buildx` commands with
+e.g. `KUBECONFIG=~/.kube/config`, i.e. pointing it to a single
+filepath.
+
+## Starting a build
+
+Now you should be able to build and push:
+
+```shell
+# whatever you need here, but note that the Dockerfiles
+# may need to change, as you update these image names and tags
+TAG=ghcr.io/project-codeflare/lightning:0.0.5
+
+# build for aarch64 (arm64/v8) and amd64
+PLATFORMS=linux/arm64/v8,linux/amd64
+
+# note here that we build and push in one step; for some reason,
+# buildx fails when only building
+docker buildx build --push --platforms $PLATFORMS --tag $TAG .
+```

deploy/bases/lightning/Dockerfile

@@ -0,0 +1,53 @@
+FROM ubuntu:20.04
+
+ARG PYTHON_VERSION=39
+ARG CONDA_VERSION=4.9.2
+ARG TARGETARCH
+
+ENV \
+    DEBIAN_FRONTEND="noninteractive" \
+    CONDA_ENV="lightning" \
+    PATH="/root/miniconda3/bin:$PATH" \
+    LD_LIBRARY_PATH="/root/miniconda3/lib:$LD_LIBRARY_PATH"
+
+COPY requirements.txt requirements.txt
+
+RUN echo "TARGETARCH=$TARGETARCH" && \
+    if [ "$TARGETARCH" = "arm64" ]; then TARGETARCH="aarch64"; else TARGETARCH="x86_64"; fi && \
+    echo "TARGETARCH=$TARGETARCH" && \
+    apt-get update -qq --fix-missing && \
+    apt-get install -y --no-install-recommends \
+        build-essential \
+        ca-certificates \
+        cmake \
+        curl \
+        git \
+        jq \
+        libopenmpi-dev \
+        unzip \
+        wget \
+    && \
+    # Install conda and python.
+    # NOTE new Conda does not forward the exit status... https://github.com/conda/conda/issues/8385
+    curl -o ~/miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-py${PYTHON_VERSION}_${CONDA_VERSION}-Linux-${TARGETARCH}.sh && \
+    chmod +x ~/miniconda.sh && \
+    ~/miniconda.sh -b && \
+    rm ~/miniconda.sh && \
+    pip install -r requirements.txt && \
+    # Cleaning
+    apt-get remove -y \
+        build-essential \
+        ca-certificates \
+        cmake \
+        curl \
+        git \
+        jq \
+        libopenmpi-dev \
+        unzip \
+        wget && \
+    apt-get autoremove -y && \
+    apt-get clean && \
+    rm -rf /root/.cache && \
+    rm -rf /var/lib/apt/lists/* && \
+    chgrp -R 0 /root && \
+    chmod -R g+rwX /root

deploy/bases/lightning/requirements.txt

@@ -0,0 +1,10 @@
+numpy>=1.17.2
+torch>=1.11.0
+pytorch-lightning>=2.0.0
+future>=0.17.1
+PyYAML>=5.1
+tqdm>=4.57.0
+fsspec[http]>=2021.06.1
+
+# Examples
+torchvision>=0.12.0

deploy/bases/pyarrow/Dockerfile

@@ -0,0 +1,28 @@
+FROM ghcr.io/project-codeflare/lightning:0.0.5
+
+ENV \
+    DEBIAN_FRONTEND="noninteractive" \
+    CONDA_ENV="lightning" \
+    PATH="/root/miniconda3/bin:$PATH" \
+    LD_LIBRARY_PATH="/root/miniconda3/lib:$LD_LIBRARY_PATH"
+
+COPY requirements.txt requirements.txt
+
+RUN unset LD_LIBRARY_PATH && \
+    apt-get update -qq --fix-missing && \
+    apt-get install -y --no-install-recommends \
+        build-essential \
+        ca-certificates \
+        cmake && \
+    pip install -r requirements.txt && \
+    # Cleaning
+    apt-get remove -y \
+        build-essential \
+        ca-certificates \
+        cmake && \
+    apt-get autoremove -y && \
+    apt-get clean && \
+    rm -rf /root/.cache && \
+    rm -rf /var/lib/apt/lists/* && \
+    chgrp -R 0 /root && \
+    chmod -R g+rwX /root

deploy/bases/pyarrow/requirements.txt

@@ -0,0 +1,6 @@
+pyarrow
+pandas
+ibm-cos-sdk
+
+# helpful and small utilities
+psutil

deploy/bases/transformers/Dockerfile

@@ -0,0 +1,28 @@
+FROM ghcr.io/project-codeflare/pyarrow:0.0.4
+
+ENV \
+    DEBIAN_FRONTEND="noninteractive" \
+    CONDA_ENV="lightning" \
+    PATH="/root/miniconda3/bin:$PATH" \
+    LD_LIBRARY_PATH="/root/miniconda3/lib:$LD_LIBRARY_PATH"
+
+COPY requirements.txt requirements.txt
+
+RUN unset LD_LIBRARY_PATH && \
+    apt-get update -qq --fix-missing && \
+    apt-get install -y --no-install-recommends \
+        build-essential \
+        ca-certificates \
+        cmake && \
+    pip install -r requirements.txt && \
+    # Cleaning
+    apt-get remove -y \
+        build-essential \
+        ca-certificates \
+        cmake && \
+    apt-get autoremove -y && \
+    apt-get clean && \
+    rm -rf /root/.cache && \
+    rm -rf /var/lib/apt/lists/* && \
+    chgrp -R 0 /root && \
+    chmod -R g+rwX /root

deploy/bases/transformers/requirements.txt

@@ -0,0 +1 @@
+transformers