6  Containerization and orchestration

Software portability is a nonfunctional requirement related to the ability of running the same software in different environments. The heavy solution are virtual machines, the balanced solution are containers.

VM vs Containers

A single host struggles to run more than one VM, while it can run tens to hundreds of containers. Moreover, we can deploy one service per container, so it’s easier to scale up services independently.

Containers support microservice architectures, that are, software system as a collection of services in which each service provides a small, independent amount of functionality, packaged in a separate container. Each service communicates with other services through service interfaces, usually, RESTful APIs. The total functionality of the system is derived from composing multiple services.

6.1 Docker

Docker Architecture

Docker Compose is a tool for running multi-container applications on Docker. The pipeline is:

1. Define your app’s environment with a Dockerfile
2. Define the services that make up your app in docker-compose.yml
3. Run docker compose up and Compose will start and run your entire app

Issues to be addressed when deploying models as services:

• Which Docker host(s) should receive the container?
• When a model is deployed in several copies, how can the workload be balanced?
• What happens if the model becomes unresponsive? How can that be detected and a container reprovisioned?
• How can a model running on multiple machines be upgraded?