Docker (software): Difference between revisions

(38 intermediate revisions by the same user not shown)

Line 8:

===Ubuntu===

[https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-engine---community-1 Reference]

{{hidden | Install Script |

# Uninstall old docker

Line 36:

Line 37:

sudo apt-get install docker-ce docker-ce-cli containerd.io

</syntaxhighlight>

}}

===Windows===

Line 44:

Line 46:

==Guides==

[https://docs.docker.com/get-started/ Get Started]

==Dockerfile==

How to write a dockerfile

==CLI Usage==

===Images===

For the most part, you don't need to worry about images as <code>docker run</code> and <code>docker-compose</code> will download and build images for you as needed.

# List images.

docker image ls

# Prune unused images.

docker image prune -a

# Copy image.

docker tag $SOURCE $TARGET

docker push $TARGET

</syntaxhighlight>

;Notes

* Pruning with <code>docker system prune</code> will also delete images.

* Omitting <code>-a</code> will only prune dangling (untagged) images.

===Containers===

Line 64:

Line 83:

** To restrict listing to localhost use <code>-p 127.0.0.1:80:80</code>

* <code>-it</code> to be interactive with a pseudo-tty

===stats===

[https://docs.docker.com/engine/reference/commandline/stats/ docker stats]

Returns information about container cpu usage, memory usage, network usage, and disk usage.

==Networking==

Line 71:

Line 94:

In bridge mode, the docker service acts as a NAT and gives each container a separate local IP along with the docker host.

On linux, you can type <code>ip a</code> to see the ip address of the <code>docker0</code> network interface.

On my server, it is <code>172.17.0.1/16</code>.

To access services running on the host (such as MySQL or Postgres), you will need to make these services ~~liston~~ on this network interface and allow it through your firewall.

~~Then make your~~ docker ~~containers connect~~ using the ~~ip address of your docker host~~.

To access services running on the host (such as MySQL or Postgres), you will need to make these services listen on this network interface and allow it through your firewall. I suggest using [https://github.com/qoomon/docker-host qoomon/docker-host] which can redirect network traffic to the host.

When using docker-compose, services can access each other using their service name as the hostname. However, the port needs to be exposed in the compose file.

===host===

Line 91:

Line 116:

See [https://github.com/docker/compose/issues/6691 issue].

<pre>

deploy:

resources:

reservations:

devices:

- ~~capabilities~~:

- driver: nvidia

- gpu

count: 1

capabilities: [gpu]

</pre>

Line 130:

Line 156:

[https://docs.docker.com/compose/compose-file/compose-file-v3/ Compose file reference]

===Accessing Host===

===Compose file===

See [https://github.com/qoomon/docker-host docker-host] for a container which can access the host.

See [https://docs.docker.com/compose/compose-file/compose-file-v3/ Compose file specification].

~~I haven't tried~~ this ~~but~~ you ~~should~~ be ~~able~~ to ~~use this~~ in docker-~~compose~~ to ~~resolve to localhost~~.

Previously, the Compose file (<code>docker-compose.yml</code>) required a version. Version 2 and version 3 had different options and not all options from version 2 were available in version 3. However, as of docker-compose v1.27+, you should no longer specify a version and options from both versions are supported.

{{hidden | Example docker-compose.yml |

services:

web:

image: registry.gitlab.davidl.me/dli7319/davidl_me:latest

restart: unless-stopped

</syntaxhighlight>

}}

==Accessing the Host==

Sometimes you may have services running on the host which you want to access from a container.

See [https://github.com/qoomon/docker-host docker-host] for a container which can access the host.

Add the following to your docker compose to expose port 8201 to other containers:

docker-host:

image: qoomon/docker-host

cap_add:

- NET_ADMIN

- NET_RAW

</syntaxhighlight>

* You do not need to add <code>expose</code>.

By default, networks are allocated with ip ranges:

* 172.17.0.0/12 with size /16

* 192.168.0.0/16 with size /20

If you want this to be more consistent, you can change it as follows:

Set the following in <code>/etc/docker/daemon.json</code>:

{

"default-address-pools":[

{"base":"172.16.0.0/12","size":24}

]

}

</syntaxhighlight>

Then restart your docker: <code>sudo systemctl restart docker</code> and prune networks <code>docker network prune</code>.

Next, in your firewall, allow connections to your localhost from 172.16.0.0/12.

<pre>

ufw allow from 172.16.0.0/12 to any comment "from_docker"

</pre>

==Registries==

The official Docker registry is [https://hub.docker.com/ Docker Hub].

However, [https://www.docker.com/increase-rate-limits/ Docker Hug has rate limits] of 100 pulls per 6 hours.

Alternative public registries:

* [https://gallery.ecr.aws/ AWS ECR Gallery] has a mirror for all official docker containers.

==Caching==

If you want your builds to be fast on CICD, you have to setup [https://docs.docker.com/build/cache/ caching].

In particular you should:

* Enable [https://docs.docker.com/build/buildkit/ buildkit] by setting the environment variable <code>DOCKER_BUILDKIT=1</code>

* Use <code>--cache-from</code> in your build.

* Setup external caching, e.g. with <code>--build-arg BUILDKIT_INLINE_CACHE=1</code>.

** Use <code>--arg BUILDKIT_INLINE_CACHE=1</code> if using <code>docker buildx build</code>.

* For multistage builds, cache each stage in your container registry.

;Resources

* https://michalwojcik.com.pl/2021/01/21/using-cache-in-multi-stage-builds-in-gitlab-ci-docker/

* https://testdriven.io/blog/faster-ci-builds-with-docker-cache/#multi-stage-builds

==Useful Services==

* [https://containrrr.dev/watchtower/ https://containrrr.dev/watchtower/] is a tool which will automatically update your docker containers when new images are published. It also has an http endpoint to trigger checks manually e.g. from CI/CD.

==My Images==

I have a few custom container images below:

* [https://github.com/dli7319/docker-anki-server ghcr.io/dli7319/docker-anki-server:main]

* [https://github.com/dli7319/docker-nextcloud ghcr.io/dli7319/docker-nextcloud:main]

* [https://github.com/dli7319/docker-mediawiki ghcr.io/dli7319/docker-mediawiki:main]

==Resources==

* [https://www.youtube.com/watch?v=fqMOX6JJhGo freeCodeCamp.org Docker Tutorial for Beginners Video]

* [https://www.udacity.com/course/scalable-microservices-with-kubernetes--ud615 Scalable Microservices with Kubernetes Udacity Course]

** I haven't watched this but it's by Google so it's probably good.

** Lessons 1-2 are on Docker and lessons 3-4 are on Kubernetes.

@@ Line 8: / Line 8: @@
 ===Ubuntu===
 [https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-engine---community-1 Reference]
+{{hidden | Install Script |
 <syntaxhighlight lang="bash">
 # Uninstall old docker
@@ Line 36: / Line 37: @@
 sudo apt-get install docker-ce docker-ce-cli containerd.io
 </syntaxhighlight>
+}}
 ===Windows===
@@ Line 44: / Line 46: @@
 ==Guides==
 [https://docs.docker.com/get-started/ Get Started]
+==Dockerfile==
+How to write a dockerfile
 ==CLI Usage==
 ===Images===
+For the most part, you don't need to worry about images as <code>docker run</code> and <code>docker-compose</code> will download and build images for you as needed.
 <syntaxhighlight lang="bash">
+# List images.
+docker image ls
+# Prune unused images.
+docker image prune -a
+# Copy image.
+docker tag $SOURCE $TARGET
+docker push $TARGET
 </syntaxhighlight>
+;Notes
+* Pruning with <code>docker system prune</code> will also delete images.
+* Omitting <code>-a</code> will only prune dangling (untagged) images.
 ===Containers===
@@ Line 64: / Line 83: @@
 ** To restrict listing to localhost use <code>-p 127.0.0.1:80:80</code>
 * <code>-it</code> to be interactive with a pseudo-tty
+===stats===
+[https://docs.docker.com/engine/reference/commandline/stats/ docker stats]
+Returns information about container cpu usage, memory usage, network usage, and disk usage.
 ==Networking==
@@ Line 71: / Line 94: @@
 In bridge mode, the docker service acts as a NAT and gives each container a separate local IP along with the docker host.
 On linux, you can type <code>ip a</code> to see the ip address of the <code>docker0</code> network interface.
 On my server, it is <code>172.17.0.1/16</code>.
-To access services running on the host (such as MySQL or Postgres), you will need to make these services liston on this network interface and allow it through your firewall.
-Then make your docker containers connect using the ip address of your docker host.
+To access services running on the host (such as MySQL or Postgres), you will need to make these services listen on this network interface and allow it through your firewall. I suggest using [https://github.com/qoomon/docker-host qoomon/docker-host] which can redirect network traffic to the host.
+When using docker-compose, services can access each other using their service name as the hostname. However, the port needs to be exposed in the compose file.
 ===host===
@@ Line 91: / Line 116: @@
 See [https://github.com/docker/compose/issues/6691 issue].
 <pre>
-   deploy:
+    deploy:
        resources:
          reservations:
            devices:
-             - capabilities:
+             - driver: nvidia
-               - gpu
+              count: 1
+               capabilities: [gpu]
 </pre>
@@ Line 130: / Line 156: @@
 [https://docs.docker.com/compose/compose-file/compose-file-v3/ Compose file reference]
-===Accessing Host===
+===Compose file===
-See [https://github.com/qoomon/docker-host docker-host] for a container which can access the host.
+See [https://docs.docker.com/compose/compose-file/compose-file-v3/ Compose file specification].
-I haven't tried this but you should be able to use this in docker-compose to resolve to localhost.
+Previously, the Compose file (<code>docker-compose.yml</code>) required a version. Version 2 and version 3 had different options and not all options from version 2 were available in version 3. However, as of docker-compose v1.27+, you should no longer specify a version and options from both versions are supported.
+{{hidden | Example docker-compose.yml |
+<syntaxhighlight lang="yaml">
+services:
+  web:
+    image: registry.gitlab.davidl.me/dli7319/davidl_me:latest
+    restart: unless-stopped
+</syntaxhighlight>
+}}
+==Accessing the Host==
+Sometimes you may have services running on the host which you want to access from a container.<br>
+See [https://github.com/qoomon/docker-host docker-host] for a container which can access the host.<br>
+Add the following to your docker compose to expose port 8201 to other containers:
+<syntaxhighlight lang="yaml">
+  docker-host:
+    image: qoomon/docker-host
+    cap_add:
+      - NET_ADMIN
+      - NET_RAW
+</syntaxhighlight>
+* You do not need to add <code>expose</code>.
+By default, networks are allocated with ip ranges:
+* 172.17.0.0/12 with size /16
+* 192.168.0.0/16 with size /20
+If you want this to be more consistent, you can change it as follows:
+Set the following in <code>/etc/docker/daemon.json</code>:
+<syntaxhighlight lang="json">
+{
+  "default-address-pools":[
+    {"base":"172.16.0.0/12","size":24}
+  ]
+}
+</syntaxhighlight>
+Then restart your docker: <code>sudo systemctl restart docker</code> and prune networks <code>docker network prune</code>.<br>
+Next, in your firewall, allow connections to your localhost from 172.16.0.0/12.
+<pre>
+ufw allow from 172.16.0.0/12 to any comment "from_docker"
+</pre>
+==Registries==
+The official Docker registry is [https://hub.docker.com/ Docker Hub].<br>
+However, [https://www.docker.com/increase-rate-limits/ Docker Hug has rate limits] of 100 pulls per 6 hours.<br>
+Alternative public registries:
+* [https://gallery.ecr.aws/ AWS ECR Gallery] has a mirror for all official docker containers.
+==Caching==
+If you want your builds to be fast on CICD, you have to setup [https://docs.docker.com/build/cache/ caching].
+In particular you should:
+* Enable [https://docs.docker.com/build/buildkit/ buildkit] by setting the environment variable <code>DOCKER_BUILDKIT=1</code>
+* Use <code>--cache-from</code> in your build.
+* Setup external caching, e.g. with <code>--build-arg BUILDKIT_INLINE_CACHE=1</code>.
+** Use <code>--arg BUILDKIT_INLINE_CACHE=1</code> if using <code>docker buildx build</code>.
+* For multistage builds, cache each stage in your container registry.
+;Resources
+* https://michalwojcik.com.pl/2021/01/21/using-cache-in-multi-stage-builds-in-gitlab-ci-docker/
+* https://testdriven.io/blog/faster-ci-builds-with-docker-cache/#multi-stage-builds
+==Useful Services==
+* [https://containrrr.dev/watchtower/ https://containrrr.dev/watchtower/] is a tool which will automatically update your docker containers when new images are published. It also has an http endpoint to trigger checks manually e.g. from CI/CD.
+==My Images==
+I have a few custom container images below:
+* [https://github.com/dli7319/docker-anki-server ghcr.io/dli7319/docker-anki-server:main]
+* [https://github.com/dli7319/docker-nextcloud ghcr.io/dli7319/docker-nextcloud:main]
+* [https://github.com/dli7319/docker-mediawiki ghcr.io/dli7319/docker-mediawiki:main]
 ==Resources==
 * [https://www.youtube.com/watch?v=fqMOX6JJhGo freeCodeCamp.org Docker Tutorial for Beginners Video]
-* [https://www.udacity.com/course/scalable-microservices-with-kubernetes--ud615 Scalable Microservices with Kubernetes Udacity Course]
-** I haven't watched this but it's by Google so it's probably good.
-** Lessons 1-2 are on Docker and lessons 3-4 are on Kubernetes.