Difference between revisions of "Containerization"
Jump to navigation
Jump to search
Line 78: | Line 78: | ||
* then, on server: <code>docker login nvcr.io</code> - user $oauthtoken, password use NGC API key | * then, on server: <code>docker login nvcr.io</code> - user $oauthtoken, password use NGC API key | ||
===Pull/Run software=== | ===Pull/Run software=== | ||
− | * <code> | + | * new way: |
− | * <code>nvidia-docker run -it --name | + | ** <code>docker run --gpus all -it --name tensorflow_1910_py3 -d --net=host -v /home/colettace/projects:/projects --ipc=host --ulimit memlock=-1 nvcr.io/nvidia/tensorflow:19.10-py3</code> |
− | * <code>nvidia-docker run -it --name | + | * old way: |
− | * <code>nvidia-docker run -it --name | + | ** <code>nvidia-docker run -it --name digits_1903 -d --net=host -v /home/colettace/projects:/projects -v /home/colettace/nvidia_digits_jobs:/workspace/jobs --ipc=host --ulimit memlock=-1 nvcr.io/nvidia/digits:19.03-tensorflow</code> |
+ | ** <code>nvidia-docker run -it --name coletta_mxnet1903 -d --net=host -v /home/colettace/projects:/projects --ipc=host --ulimit memlock=-1 nvcr.io/nvidia/mxnet:19.03-py3</code> | ||
+ | ** <code>nvidia-docker run -it --name tensorflow_1903_py3 -d --net=host -v /home/colettace/projects:/projects -v /home/colettace/nvidia_digits_jobs:/workspace/jobs --ipc=host --ulimit memlock=-1 nvcr.io/nvidia/tensorflow:19.03-py3</code> | ||
** inside containter: <code>jupyter lab --no-browser --allow-root</code> | ** inside containter: <code>jupyter lab --no-browser --allow-root</code> | ||
* others (optional) | * others (optional) |
Revision as of 21:06, 30 October 2019
Contents
Concepts
- Docker Glossary
- Get Started - End-to-end tutorial
- How to create-react-app with docker
- react-base Base docker image for using React on top of npm
- Another
Image
- The basis of containers
- An ordered collection of root filesystem changes and the corresponding execution parameters for use within a container runtime.
- An image typically contains a union of layered filesystems stacked on top of each other.
- READ-ONLY: image does not have state and never changes.
Container
- A "stateful" instance of a docker image
- Contains 1. A Docker image 2. An execution environment 3. A standard set of instructions
- The concept is borrowed from Shipping Containers, which define a standard to ship goods globally. Docker defines a standard to ship software.
Layer
- A modification to the image, represented by an instruction in the Dockerfile
- Layers are applied in sequence to the base image to create the final image.
- When an image is updated or rebuilt, only layers that change need to be updated, and unchanged layers are cached locally. This is part of why Docker images are so fast and lightweight.
Dockerfile
- A text document that contains all the commands you would normally execute manually in order to build a Docker image.
Docker Hub
- The Docker Hub is a centralized resource for working with Docker and its components
- Docker image hosting, etc.
Composer
- Compose is a tool for defining and running multi-container Docker applications
- Use a YAML file to configure your application’s services
- Compose file provides a way to document and configure all of the application's service dependencies (databases, queues, caches, web service APIs, etc).
- Using the Compose command line tool you can create and start one or more containers for each dependency with a single command (docker-compose up).
- how to use:
- Define your app’s environment with a Dockerfile
- Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.
- Run
docker-compose up
- Latest is v3.x compose file format (reference)
- Use cases:
- Development environment that's isolated
- Automated testing environments
- Single host deployment - deploy to a remote docker engine. Could be a single server (single instance provisioned with docker machine), or scale up to run compose apps on a swarm cluster
Composer in production
- Define an additional Compose file, say production.yml, which specifies production-appropriate configuration
docker-compose -f docker-compose.yml -f production.yml up -d
- Removing any volume bindings for application code, so that code stays inside the container and can't be changed from outside
- Binding to different ports on the host
- Adding extra services such as a log aggregator
- etc
Docker Machine
- Docker Machine is a tool that lets you install Docker Engine on virtual hosts, and manage the hosts with docker-machine commands.
- You can use Machine to create Docker hosts on your local Mac or Windows box, on your company network, in your data center, or on cloud providers like Azure, AWS, or Digital Ocean.
- Using docker-machine commands, you can start, inspect, stop, and restart a managed host, upgrade the Docker client and daemon, and configure a Docker client to talk to your host.
Sysadmin
sudo systemctl start docker
sudo systemctl enable docker
- for CentOS7
journalctl -u docker
- daemon log for CentOS7
General
docker version
- docker server (dockerd) & docker client (docker) version infodocker info
- docker server informationdocker images -a
- list the installed images, -a includes the exited onesdocker ps -a
- List of containers (running process image, age, status, name, and port information)docker stats
- Resource utilizationdocker logs [OPTIONS] CONTAINER
- --details, -f, -tail n, --timestamps, --untildocker exec -it HEXADECIMAL bash
- ssh into container, just do this to begin with and keep a tmux open with it.docker pull
- pull from dockerhubdocker rm Container_ID_or_Name
- remove containerdocker rmi $(docker images -a -q)
- remove all images
NVIDIA Docker/NVIDIA GPU Cloud
nvidia-docker
NVIDIA GPU Cloud
- NGC website - NGC registry page
- On website registry page, generate NGC API key then store it somewhere safe
- I put it in a file in the home directory
- not a big deal if you lose it, just generate a new one, old key become invalid
- then, on server:
docker login nvcr.io
- user $oauthtoken, password use NGC API key
Pull/Run software
- new way:
docker run --gpus all -it --name tensorflow_1910_py3 -d --net=host -v /home/colettace/projects:/projects --ipc=host --ulimit memlock=-1 nvcr.io/nvidia/tensorflow:19.10-py3
- old way:
nvidia-docker run -it --name digits_1903 -d --net=host -v /home/colettace/projects:/projects -v /home/colettace/nvidia_digits_jobs:/workspace/jobs --ipc=host --ulimit memlock=-1 nvcr.io/nvidia/digits:19.03-tensorflow
nvidia-docker run -it --name coletta_mxnet1903 -d --net=host -v /home/colettace/projects:/projects --ipc=host --ulimit memlock=-1 nvcr.io/nvidia/mxnet:19.03-py3
nvidia-docker run -it --name tensorflow_1903_py3 -d --net=host -v /home/colettace/projects:/projects -v /home/colettace/nvidia_digits_jobs:/workspace/jobs --ipc=host --ulimit memlock=-1 nvcr.io/nvidia/tensorflow:19.03-py3
- inside containter:
jupyter lab --no-browser --allow-root
- others (optional)
docker pull nvcr.io/nvidia/digits:18.06
nvidia-docker run --name digits -d -p 5000:5000 -v /jobs:/jobs -v /datasets:/datasets nvcr.io/nvidia/digits:18.06
docker pull nvidia/caffe
docker run --runtime=nvidia --name digits -d -p 5000:5000 -v /jobs:/jobs -v /datasets:/datasets -it nvidia/digits