Kubernetes: Difference between revisions
No edit summary |
|||
(37 intermediate revisions by the same user not shown) | |||
Line 5: | Line 5: | ||
==Getting Started== | ==Getting Started== | ||
===Background=== | ===Background=== | ||
Kubernetes runs applications across nodes which are physical or virtual machines.<br> | Kubernetes runs applications across '''nodes''' which are (physical or virtual) Linux machines.<br> | ||
Each node contains a kubelet process, a container runtime (typically containerd), and any running pods.<br> | Each node contains a kubelet process, a container runtime (typically containerd), and any running pods.<br> | ||
Pods contain resources needed to host your application including volumes and containers.<br> | '''Pods''' contain resources needed to host your application including volumes and containers.<br> | ||
Typically you will want one container per pod since deployments scale by creating multiple pods. | Typically you will want one container per pod since deployments scale by creating multiple pods.<br> | ||
A '''deployment''' is a rule which spawns and manages pods.<br> | |||
A '''service''' is a networking rule which allows connecting to pods. | |||
In addition to standard Kubernetes objects, '''operators''' watch for and allow you to instantiate custom resources (CR). | |||
==Kubeadm Administration== | |||
Notes on administering kubernetes clusters. | |||
Kuberenetes has many parts and administration is very tedious which is why K3S exists. I'd recommend against using kubeadm for a homelab. | |||
===Installation=== | ===Installation=== | ||
Line 148: | Line 157: | ||
See https://metallb.universe.tf/installation/.<br> | See https://metallb.universe.tf/installation/.<br> | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
cat <<EOF > | helm repo add metallb https://metallb.github.io/metallb | ||
helm upgrade --install --create-namespace -n metallb metallb metallb/metallb | |||
cat <<EOF >ipaddresspool.yaml | |||
apiVersion: metallb.io/v1beta1 | |||
kind: IPAddressPool | |||
metadata: | |||
name: default | |||
namespace: metallb | |||
spec: | |||
addresses: | |||
- 192.168.1.2-192.168.1.11 | |||
--- | |||
apiVersion: metallb.io/v1beta1 | |||
kind: L2Advertisement | |||
metadata: | |||
name: example | |||
namespace: metallb | |||
EOF | EOF | ||
kubectl apply -f ipaddresspool.yaml | |||
</syntaxhighlight> | </syntaxhighlight> | ||
}} | }} | ||
Line 164: | Line 183: | ||
The ingress controller is used to forward HTTP requests to the appropriate ingress.<br> | The ingress controller is used to forward HTTP requests to the appropriate ingress.<br> | ||
See https://kubernetes.github.io/ingress-nginx/. | See https://kubernetes.github.io/ingress-nginx/. | ||
}} | |||
{{hidden | cert-manager | | |||
See https://cert-manager.io/docs/installation/helm/ | |||
You may also want to setup DNS challenges to support wildcard certificates.<br> | |||
See https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/ if you are using Cloudflare. | |||
}} | }} | ||
{{hidden | Add worker nodes | | {{hidden | Add worker nodes | | ||
Line 174: | Line 199: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
}} | }} | ||
===Certificates=== | |||
[https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ Certificate Management with kubeadm] | |||
Kubernetes requires several TLS certificates which are automatically generated by Kubeadm. | |||
These expire in one year but are automatically renewed whenever you upgrade your cluster with <code>kubeadm upgrade apply</code> | |||
To renew the certificates manually, run <code>kubeadm certs renew all</code> and restart your control plane services. | |||
Note that if you lets the certificates expire, you will need to setup kubectl again. | |||
Issues connecting with etcd | |||
I ran into this when trying to kubeadm upgrade | |||
;context deadline exceeded remote error: tls: bad certificate | |||
Kubeadm stores etcd certificates in <code>/etc/kubernetes/pki/etcd/</code>. | |||
Follow this to generate new certificates: https://github.com/etcd-io/etcd/issues/9785#issuecomment-432438748 | |||
You will need to create a temporary files for ca-config.json and server.json to generate new keys. | |||
Make sure in the server.json to set the key algo to "rsa" and size to 2048. In the same file, set your CN to 127.0.0.1 and the hosts to [127.0.0.1, your local IP]. | |||
;cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs | |||
This means your hosts in server.json is not correct when you generated the new keys. | |||
===Pods per node=== | ===Pods per node=== | ||
Line 180: | Line 226: | ||
You may increase this up to a limit of 255 with the default networking subnet.<br> | You may increase this up to a limit of 255 with the default networking subnet.<br> | ||
For reference, GCP GKE uses 110 pods per node and AWS EKS uses 250 pods per node. | For reference, GCP GKE uses 110 pods per node and AWS EKS uses 250 pods per node. | ||
===Changing Master Address=== | |||
See https://ystatit.medium.com/how-to-change-kubernetes-kube-apiserver-ip-address-402d6ddb8aa2 | |||
==kubectl== | ==kubectl== | ||
Line 238: | Line 287: | ||
kubectl run busybox-shell --rm -i --tty --image odise/busybox-curl -- sh | kubectl run busybox-shell --rm -i --tty --image odise/busybox-curl -- sh | ||
</syntaxhighlight> | </syntaxhighlight> | ||
==Deployments== | |||
In most cases, you will use deployments to provision pods.<br> | |||
Deployments internally use replicasets to create multiple identical pods.<br> | |||
This is great for things such as webservers or standalone services which are not stateful. | |||
In most cases, you can stick a service in front which will round-robin requests to different pods in your deployment. | |||
{{hidden | Example Deployment | | |||
<syntaxhighlight lang="yaml"> | |||
apiVersion: apps/v1 | |||
kind: Deployment | |||
metadata: | |||
name: nextcloud-app | |||
labels: | |||
app: nextcloud | |||
spec: | |||
replicas: 1 | |||
selector: | |||
matchLabels: | |||
pod-label: nextcloud-app-pod | |||
template: | |||
metadata: | |||
labels: | |||
pod-label: nextcloud-app-pod | |||
spec: | |||
containers: | |||
- name: nextcloud | |||
image: public.ecr.aws/docker/library/nextcloud:stable | |||
ports: | |||
- containerPort: 80 | |||
env: | |||
- name: MYSQL_HOST | |||
value: nextcloud-db-service | |||
- name: MYSQL_DATABASE | |||
value: nextcloud | |||
- name: MYSQL_USER | |||
valueFrom: | |||
secretKeyRef: | |||
name: nextcloud-db-credentials | |||
key: username | |||
- name: MYSQL_PASSWORD | |||
valueFrom: | |||
secretKeyRef: | |||
name: nextcloud-db-credentials | |||
key: password | |||
volumeMounts: | |||
- name: nextcloud-app-storage | |||
mountPath: /var/www/html | |||
volumes: | |||
- name: nextcloud-app-storage | |||
persistentVolumeClaim: | |||
claimName: nextcloud-app-pvc | |||
</syntaxhighlight> | |||
}} | |||
==StatefulSets== | |||
[https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/ StatefulSets basics]<br> | |||
Stateful sets are useful when you need a fixed number of pods with stable identities such as databases.<br> | |||
Pods created by stateful sets have a unique number suffix which allows you to query a specific pod.<br> | |||
Typically, you will want to use a headless service (i.e. without ClusterIP) to give local dns records to each service. | |||
In most cases, you will want to look for a helm chart instead of creating your own stateful sets. | |||
==Services== | ==Services== | ||
Line 243: | Line 354: | ||
Services handle networking. | Services handle networking. | ||
For self-hosted/bare metal | For self-hosted/bare metal clusters, there are two types of services. | ||
* ClusterIP - This creates an IP address on the internal cluster which nodes and pods on the cluster can access. (Default) | * ClusterIP - This creates an IP address on the internal cluster which nodes and pods on the cluster can access. (Default) | ||
* NodePort - This exposes the port on every node. It implicitly creates a ClusterIP and every node will route to that. This allows access from outside the cluster. | * NodePort - This exposes the port on every node. It implicitly creates a ClusterIP and every node will route to that. This allows access from outside the cluster. | ||
* ExternalName - uses a CNAME record. Primarily for accessing other services from within the cluster. | * ExternalName - uses a CNAME record. Primarily for accessing other services from within the cluster. | ||
* LoadBalancer - Creates a clusterip+nodeport and tells the loadbalancer to create an IP and route it to the nodeport. | * LoadBalancer - Creates a clusterip+nodeport and tells the loadbalancer to create an IP and route it to the nodeport. | ||
** On bare-metal | ** On bare-metal clusters you will need to install a loadbalancer such as metallb. | ||
By default, ClusterIP is provided by <code>kube-proxy</code> and performs round-robin load-balancing to pods.<br> | By default, ClusterIP is provided by <code>kube-proxy</code> and performs round-robin load-balancing to pods.<br> | ||
Line 273: | Line 384: | ||
==Ingress== | ==Ingress== | ||
[https://kubernetes.io/docs/concepts/services-networking/ingress/ Ingress | Kubernetes]<br> | [https://kubernetes.io/docs/concepts/services-networking/ingress/ Ingress | Kubernetes]<br> | ||
An ingress is an http endpoint. This configures an ingress controller which is a load-balancer or reverse-proxy pod that integrates with Kubernetes. | |||
===Installing | A common ingress controller is [https://github.com/kubernetes/ingress-nginx ingress-nginx] which is maintained by the Kubernetes team. Alternatives include [https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/ nginx-ingress] [https://doc.traefik.io/traefik/providers/kubernetes-ingress/ traefik], [https://haproxy-ingress.github.io/ haproxy-ingress], and [https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/ others]. | ||
See [https://kubernetes.github.io/ingress-nginx/deploy/ ingress-nginx] to deploy an ingress controller. | |||
===Installing ingress-nginx=== | |||
See [https://kubernetes.github.io/ingress-nginx/deploy/ ingress-nginx] to deploy an ingress controller.<br> | |||
Note that <code>ingress-nginx</code> is managed by the Kubernetes team and <code>nginx-ingress</code> is an different ingress controller by the Nginx team. | |||
Personally, I have: | Personally, I have: | ||
Line 312: | Line 426: | ||
type: LoadBalancer | type: LoadBalancer | ||
loadBalancerIP: 192.168.1.3 | loadBalancerIP: 192.168.1.3 | ||
externalTrafficPolicy: Local | |||
config: | config: | ||
proxy-body-size: 1g | proxy-body-size: 1g | ||
Line 359: | Line 474: | ||
If your backend uses HTTPS, you will need to add the annotation: <code>nginx.ingress.kubernetes.io/backend-protocol: HTTPS</code> | If your backend uses HTTPS, you will need to add the annotation: <code>nginx.ingress.kubernetes.io/backend-protocol: HTTPS</code> | ||
For self-signed SSL certificates, you will also need the annotation: | |||
<syntaxhighlight lang="yaml"> | |||
nginx.ingress.kubernetes.io/configuration-snippet: | | |||
proxy_ssl_name $host; | |||
proxy_ssl_server_name on; | |||
</syntaxhighlight> | |||
===Authentication=== | |||
[https://kubernetes.github.io/ingress-nginx/examples/auth/oauth-external-auth/ ingress-nginx external oauth]<br> | |||
If you like to authenticate using an oauth2 provider (e.g. Google, GitHub), I suggest using [https://github.com/oauth2-proxy/oauth2-proxy oauth2-proxy]. | |||
# First setup a deployment of the oauth2, possibly without an upstream. | |||
# Then you can simply add the following annotations to your ingresses to protect them: | |||
#:<syntaxhighlight lang="yaml"> | |||
nginx.ingress.kubernetes.io/auth-url: "http://oauth2proxy.default.svc.cluster.local/oauth2/auth?allowed_emails=myemail@gmail.com" | |||
nginx.ingress.kubernetes.io/auth-signin: "https://oauth2proxy.davidl.me/oauth2/start?rd=$scheme://$host$request_uri" | |||
</syntaxhighlight> | |||
;Additional things to look into | |||
* Pomerium | |||
* Keycloak | |||
** https://www.talkingquickly.co.uk/webapp-authentication-keycloak-OAuth2-proxy-nginx-ingress-kubernetes | |||
* Authelia - only supports username/password as the first factor | |||
* Authentik - tried this but had too complicated and buggy for me. | |||
If you use Cloudflare, you can also use Cloudflare access, though make sure you prevent other sources from accessing the service directly. | |||
==Autoscaling== | ==Autoscaling== | ||
Line 415: | Line 555: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
}} | }} | ||
==Security Context== | |||
[https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ security context] | |||
If you want to restrict pods to run as a particular UID/GUI while still binding to any port, you can add the following: | |||
<syntaxhighlight lang=yaml> | |||
spec: | |||
securityContext: | |||
runAsUser: 1000 | |||
runAsGroup: 1000 | |||
sysctls: | |||
- name: net.ipv4.ip_unprivileged_port_start | |||
value: "0" | |||
</syntaxhighlight> | |||
==Devices== | ==Devices== | ||
Line 440: | Line 593: | ||
===Scale to 0=== | ===Scale to 0=== | ||
[https://stackoverflow.com/questions/64133011/scale-down-kubernetes-deployments-to-0-and-scale-back-to-original-number-of-repl reference]<br> | [https://stackoverflow.com/questions/64133011/scale-down-kubernetes-deployments-to-0-and-scale-back-to-original-number-of-repl reference]<br> | ||
If you wish to restart your cluster, you can scale your deployments and stateful sets down to 0 and then scale them back up after. | If you wish to restart all nodes of your cluster, you can scale your deployments and stateful sets down to 0 and then scale them back up after. | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
# Annotate existing deployments and statefulsets with replica count. | # Annotate existing deployments and statefulsets with replica count. | ||
Line 458: | Line 611: | ||
==Helm== | ==Helm== | ||
Helm is a method for deploying | Helm is a method for deploying applications using premade kubernetes manifest templates known as helm charts.<br> | ||
Helm charts abstract away manifests, allowing you to focus on only the important configuration values.<br> | |||
Manifests can also be composed into other manifests for applications which require multiple microservices. | Manifests can also be composed into other manifests for applications which require multiple microservices. | ||
[https://artifacthub.io/ https://artifacthub.io/] allows you to search for helm charts others have made.<br> | |||
[https://github.com/bitnami/charts bitnami/charts] contains helm charts for many popular applications. | |||
===Usage=== | ===Usage=== | ||
Line 468: | Line 624: | ||
# Install the application using helm. | # Install the application using helm. | ||
#:<pre>helm upgrade --install $NAME $CHARTNAME -f values.yaml [--version $VERSION]</pre> | #:<pre>helm upgrade --install $NAME $CHARTNAME -f values.yaml [--version $VERSION]</pre> | ||
===Troubleshooting=== | |||
Sometimes, Kubernetes will deprecate APIs, preventing it from managing existing helm releases.<br> | |||
The [https://github.com/helm/helm-mapkubeapis mapkubeapis] helm plugin can help resolve some of these issues. | |||
==Variants== | ==Variants== | ||
===minikube=== | ===minikube=== | ||
[https://minikube.sigs.k8s.io/docs/ minikube] is a tool to quickly set up a local Kubernetes | [https://minikube.sigs.k8s.io/docs/ minikube] is a tool to quickly set up a local Kubernetes dev environment on your PC. | ||
===kind=== | ===kind=== | ||
Line 477: | Line 637: | ||
===k3s=== | ===k3s=== | ||
[https://k3s.io/ k3s] is a lighter-weight Kubernetes by Rancher Labs. | [https://k3s.io/ k3s] is a lighter-weight Kubernetes by Rancher Labs. | ||
It includes Flannel CNI and Traefik Ingress Controller. | |||
==KubeVirt== | ==KubeVirt== | ||
{{main | KubeVirt}} | {{main | KubeVirt}} | ||
KubeVirt allows you to run virtual machines | KubeVirt allows you to run virtual machines on your Kubernetes cluster. | ||
==Resources== | ==Resources== | ||
* [https://kubernetes.io/docs/tutorials/kubernetes-basics/ Kubernetes Basics] | * [https://kubernetes.io/docs/tutorials/kubernetes-basics/ Kubernetes Basics] | ||
* [https://www.udemy.com/course/certified-kubernetes-administrator-with-practice-tests/ Certified Kubernetes Administrator (CKA) with Practice Tests (~$15)] | * [https://www.udemy.com/course/certified-kubernetes-administrator-with-practice-tests/ Certified Kubernetes Administrator (CKA) with Practice Tests (~$15)] | ||
* [https://yolops.net/k8s-dualstack-cilium.html https://yolops.net/k8s-dualstack-cilium.html] |