A couple of years ago, I prepared a lecture on deploying ML applications for the Faculty of Computational Mathematics and Cybernetics at Moscow State University. We started with a single machine and a GPU, then added Docker, Compose, and finally Kubernetes as the deployment problems changed.
Tools and practices for deploying ML applications evolve, but the core engineering challenges remain: reproducing the environment, giving the application access to a GPU, and running it across multiple servers.
Start with one machine and a GPU
The simplest setup is to run the application directly on the host. Install the GPU driver and the runtime the model needs, put the weights on disk, and start the Python process. If you’re working on one machine, that’s enough to try the model and check that inference runs on the GPU.
The first problem comes when you need to repeat the setup on another machine or run two applications with different dependencies. You now have to keep track of which versions of Python and its libraries each application expects.
Docker packages the environment
On the server, the application needs the same version of Python, libraries, and system dependencies it had on the developer’s machine. In the lecture, I used a Dockerfile to choose a base image, install dependencies, copy the code, and set the startup command. Docker builds an image from that file and starts a container from the image.
The slide uses Python 3.8 to illustrate the structure of a Dockerfile.
FROM python:3.8-slim-buster
RUN pip install pandas scikit-learn
COPY app.py /
ENV PORT=8080
CMD ["python", "app.py"]
A GPU adds a dependency on the host. For NVIDIA, the host needs a driver and a configured NVIDIA Container Toolkit. Docker’s --gpus option gives the container access to the device. Running nvidia-smi inside the container checks that it can see the GPU. The next test is to send the model a request and verify that inference actually runs on the GPU. NVIDIA Container Toolkit setup.
This is the GPU check from the lecture. Use a CUDA image tag available for your host.
docker run --gpus all nvidia/cuda:11.0-base nvidia-smi
Compose runs the application on one server
Once the application has several components, you also need to describe how they communicate and where their data lives. In the lecture, I used StyleGAN as an example model. The Python service calls it at http://model:8000/check, where model is the Compose service name. Only the Python service’s port needs to be exposed externally. The weights are mounted into the model container, and PostgreSQL stores its data in a named volume. You can recreate the containers and keep the database data and the same weights file.
On a single server, docker compose up brings the stack up together. If only the model uses a GPU, its container can be the only one with access to it. Compose networking · GPU access in Compose.
This excerpt assumes you have both service directories and the weights file. Set POSTGRES_PASSWORD before starting Compose.
services:
server:
build: ./server
environment:
PORT: 8888
MODEL_ENDPOINT: "http://model:8000/check"
ports:
- "8888:8888"
model:
build: ./model
volumes:
- type: bind
source: ./model/stylegan-ffhq-1024x1024.pkl
target: /app/model.pkl
read_only: true
postgres:
image: postgres:16
environment:
POSTGRES_USER: user
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
volumes:
- postgres-data:/var/lib/postgresql/data
volumes:
postgres-data:
What changes when one server is no longer enough
On a single server, the machine’s address and its GPUs are known upfront. Once you add more servers, you need to decide where each replica runs, where to send requests as replicas move, and what happens if a host goes down. A node can have plenty of CPU capacity and still be unable to run a model replica if it has no available GPU.
Kubernetes places replicas and routes requests
A Deployment defines the Pod template and desired replica count. Kubernetes schedules Pods on nodes with the requested resources. If a node drops out of the cluster, a Deployment can bring up replacements on another node with capacity. A Service gives clients a stable address and sends requests to ready replicas. An Ingress controller can route external HTTP traffic to that Service, for example at /check.
In the example, each model replica requests one GPU by setting nvidia.com/gpu: 1 under resources.limits. The NVIDIA Device Plugin advertises that resource. If the cluster has two matching GPUs available and the Deployment calls for three replicas, the third waits for a GPU to become available. The model’s replica count is tied to the cluster’s GPU capacity. Scheduling GPUs in Kubernetes.
The manifests below use the same app: mlapp label in the Deployment and the Service selector. They assume the image is available to the cluster, the application listens on port 1234, and an Ingress controller serves the nginx class.
apiVersion: apps/v1
kind: Deployment
metadata:
name: mlapp-deployment
labels:
app: mlapp
spec:
replicas: 3
selector:
matchLabels:
app: mlapp
template:
metadata:
labels:
app: mlapp
spec:
containers:
- name: mlapp-container
image: myregistry.com/mlapp:v0.2.3-fix3-gpu
resources:
limits:
nvidia.com/gpu: 1
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
---
apiVersion: v1
kind: Service
metadata:
name: mlapp-service
spec:
selector:
app: mlapp
ports:
- protocol: TCP
port: 80
targetPort: 1234
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: mlapp-ingress
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /check
pathType: Prefix
backend:
service:
name: mlapp-service
port:
number: 80
Helm makes deployment settings reusable
When the same application needs to be installed with different images, replica counts, or storage settings, those parameters can go in values.yaml and the manifests can be packaged as a Helm chart. I showed both ways of working with charts in the lecture. The examples covered installing an existing JupyterLab chart with TensorFlow and GPU access and creating a chart from a template. Anatomy of a Helm chart.
The slides start the custom chart with helm create mychart. The chart’s Deployment template reads the image from values.yaml using this expression.
replicaCount: 1
image:
repository: nginx
tag: stable
pullPolicy: IfNotPresent
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
If the model fits on one machine, that’s where I’d start. Docker helps once the environment becomes hard to reproduce. Kubernetes earns its place when you need to schedule replicas across hosts, route requests to them, and recover from host failures. Each step makes more sense once you’ve run into the problem it solves.