Kubernetes
Kubernetes, also known as K8s, is a container orchestration service by Google.
It supposedly has a harder learning curve than docker-swarm but is heavily inspired by Google's internal borg system.
Getting Started
Background
Kubernetes runs applications across nodes which are physical or virtual machines.
Each node contains a kubelet process, a container runtime, and possibly one or more pods.
Pods contain resources needed to host your application including volumes and containers.
Typically you will want one container per pod since deployments scale by creating multiple pods.
Installation
For local development, you can install minikube.
Otherwise, install kubeadm.
kubeadm
kubeadm install
- Notes
Pods per node
How to increase pods per node
By default, Kubernetes allows 110 pods per node.
You may increase this up to a limit of 255 with the default networking subnet.
For reference, GCP GKE uses 110 pods per node and AWS EKS uses 250 pods per node.
kubectl
In general you will want to create a .yaml manifest and use apply, create, or delete to manage your resources.
nodes
kubectl get nodes
pods
kubectl get pods kubectl describe pods # Access a port on a pod kubectl port-forward <pod> <localport:podport>
deployment
kubectl get deployments kubectl logs $POD_NAME kubectl exec -it $POD_NAME -- bash # For one-off deployments of an image. kubectl create deployment <name> --image=<image> [--replicas=1]
proxy
kubectl proxy
service
Services handle routing to your pods.
kubectl get services kubectl expose deployment/<name> --type=<type> --port <port> kubectl describe services/<name>
run
https://gc-taylor.com/blog/2016/10/31/fire-up-an-interactive-bash-pod-within-a-kubernetes-cluster
# Throw up a ubuntu container kubectl run my-shell --rm -i --tty --image ubuntu -- bash kubectl run busybox-shell --rm -i --tty --image odise/busybox-curl -- sh
Services
Services handle networking.
For self-hosted/bare metal deployments, there are two types of services.
- ClusterIP - This creates an IP address on the internal cluster which nodes and pods on the cluster can access. (Default)
- NodePort - This exposes the port on every node. It implicitly creates a ClusterIP and every node will route to that. This allows access from outside the cluster.
- ExternalName - uses a CNAME record. Primarily for accessing other services from within the cluster.
On managed deployments (e.g. AWS EKS, GKE) you also have
- LoadBalancer - fires up the provider's load balancer
By default, ClusterIP is provided by kube-proxy and performs round-robin load-balancing to pods.
Ingress
Ingress | Kubernetes
Ingress is equivalent to having a load-balancer / reverse-proxy pod with a NodePort service.
Installing an Ingress Controller
See ingress-nginx to deploy an ingress controller.
Autoscaling
Horizontal Autoscale Walkthrough
Horizontal Pod Autoscaler
You will need to install metrics-server.
For testing, you may need to allow insecure tls.
Accessing External Services
access mysql on localhost
To access services running outside of your kubernetes cluster, including services running directly on a node, you need to add an endpoint and a service.
Devices
Generic devices
See https://gitlab.com/arm-research/smarter/smarter-device-manager
and https://github.com/kubernetes/kubernetes/issues/7890#issuecomment-766088805
Intel GPU
See https://github.com/intel/intel-device-plugins-for-kubernetes/tree/main/cmd/gpu_plugin
After adding the gpu plugin, add the following to your deployment.
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
resources:
limits:
gpu.intel.com/i915: 1
Variants
minikube
minikube is a tool to quickly set up a local Kubernetes cluster on your PC.
kind
k3s
k3s is a lighter-weight Kubernetes by Rancher Labs.