so that it's available through `<IP_ADDRESS>:xxxxx` on each node .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- ## Deploying a self-hosted registry - We will deploy a registry container, and expose it with a NodePort .exercise[ - Create the registry service: ```bash kubectl create deployment registry --image=registry ``` - Expose it on a NodePort: ```bash kubectl expose deploy/registry --port=5000 --type=NodePort ``` ] .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- ## Connecting to our registry - We need to find out which port has been allocated .exercise[ - View the service details: ```bash kubectl describe svc/registry ``` - Get the port number programmatically: ```bash NODEPORT=$(kubectl get svc/registry -o json | jq .spec.ports[0].nodePort) REGISTRY=<IP_ADDRESS>:$NODEPORT ``` ] .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- ## Testing our registry - A convenient Docker registry API route to remember is `/v2/_catalog` .exercise[ - View the repositories currently held in our registry: ```bash curl $REGISTRY/v2/_catalog ``` ] -- We should see: ```json {"repositories":[]} ``` .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- ## Testing our local registry - We can retag a small image, and push it to the registry .exercise[ - Make sure we have the busybox image, and retag it: ```bash docker pull busybox docker tag busybox $REGISTRY/busybox ``` - Push it: ```bash docker push $REGISTRY/busybox ``` ] .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- ## Checking again what's on our local registry - Let's use the same endpoint as before .exercise[ - Ensure that our busybox image is now in the local registry: ```bash curl $REGISTRY/v2/_catalog ``` ] The curl command should now output: ```json {"repositories":["busybox"]} ``` .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- ## Building and pushing our images - We are going to use a convenient feature of Docker Compose .exercise[ - Go to the `stacks` directory: ```bash cd ~/container.training/stacks ``` - Build and push the images: ```bash export REGISTRY export TAG=v0.1 docker-compose -f dockercoins.yml build docker-compose -f dockercoins.yml push ``` ] Let's have a look at the `dockercoins.yml` file while this is building and pushing. .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- ```yaml version: "3" services: rng: build: dockercoins/rng image: ${REGISTRY-<IP_ADDRESS>:5000}/rng:${TAG-latest} deploy: mode: global ... redis: image: redis ... worker: build: dockercoins/worker image: ${REGISTRY-<IP_ADDRESS>:5000}/worker:${TAG-latest} ... deploy: replicas: 10 ``` .warning[Just in case you were wondering ... Docker "services" are not Kubernetes "services".] .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- class: extra-details ## Avoiding the `latest` tag .warning[Make sure that you've set the `TAG` variable properly!] - If you don't, the tag will default to `latest` - The problem with `latest`: nobody knows what it points to! - the latest commit in the repo? - the latest commit in some branch? (Which one?) - the latest tag? - some random version pushed by a random team member? - If you keep pushing the `latest` tag, how do you roll back? - Image tags should be meaningful, i.e. correspond to code branches, tags, or hashes .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- ## Checking the content of the registry - All our images should now be in the registry .exercise[ - Re-run the same `curl` command as earlier: ```bash curl $REGISTRY/v2/_catalog ``` ] *In these slides, all the commands to deploy DockerCoins will use a $REGISTRY environment variable, so that we can quickly switch from the self-hosted registry to pre-built images hosted on the Docker Hub. So make sure that this $REGISTRY variable is set correctly when running the exercises!* .debug[[k8s/buildshiprun-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-selfhosted.md)] --- ## Using images from the Docker Hub - For everyone's convenience, we took care of building DockerCoins images - We pushed these images to the DockerHub, under the [dockercoins](https://hub.docker.com/u/dockercoins) user - These images are *tagged* with a version number, `v0.1` - The full image names are therefore: - `dockercoins/hasher:v0.1` - `dockercoins/rng:v0.1` - `dockercoins/webui:v0.1` - `dockercoins/worker:v0.1` .debug[[k8s/buildshiprun-dockerhub.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/buildshiprun-dockerhub.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/two-containers-on-a-truck.jpg)] --- name: toc-running-our-application-on-kubernetes class: title Running our application on Kubernetes .nav[ [Previous section](#toc-shipping-images-with-a-registry) | [Back to table of contents](#toc-module-3) | [Next section](#toc-deploying-with-yaml) ] .debug[(automatically generated title slide)] --- # Running our application on Kubernetes - We can now deploy our code (as well as a redis instance) .exercise[ - Deploy `redis`: ```bash kubectl create deployment redis --image=redis ``` - Deploy everything else: ```bash kubectl create deployment hasher --image=dockercoins/hasher:v0.1 kubectl create deployment rng --image=dockercoins/rng:v0.1 kubectl create deployment webui --image=dockercoins/webui:v0.1 kubectl create deployment worker --image=dockercoins/worker:v0.1 ``` ] .debug[[k8s/ourapponkube.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/ourapponkube.md)] --- class: extra-details ## Deploying other images - If we wanted to deploy images from another registry ... - ... Or with a different tag ... - ... We could use the following snippet: ```bash REGISTRY=dockercoins TAG=v0.1 for SERVICE in hasher rng webui worker; do kubectl create deployment $SERVICE --image=$REGISTRY/$SERVICE:$TAG done ``` .debug[[k8s/ourapponkube.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/ourapponkube.md)] --- ## Is this working? - After waiting for the deployment to complete, let's look at the logs! (Hint: use `kubectl get deploy -w` to watch deployment events) .exercise[ - Look at some logs: ```bash kubectl logs deploy/rng kubectl logs deploy/worker ``` ] -- 🤔 `rng` is fine ... But not `worker`. -- 💡 Oh right! We forgot to `expose`. .debug[[k8s/ourapponkube.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/ourapponkube.md)] --- ## Connecting containers together - Three deployments need to be reachable by others: `hasher`, `redis`, `rng` - `worker` doesn't need to be exposed - `webui` will be dealt with later .exercise[ - Expose each deployment, specifying the right port: ```bash kubectl expose deployment redis --port 6379 kubectl expose deployment rng --port 80 kubectl expose deployment hasher --port 80 ``` ] .debug[[k8s/ourapponkube.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/ourapponkube.md)] --- ## Is this working yet? - The `worker` has an infinite loop, that retries 10 seconds after an error .exercise[ - Stream the worker's logs: ```bash kubectl logs deploy/worker --follow ``` (Give it about 10 seconds to recover) ] -- We should now see the `worker`, well, working happily. .debug[[k8s/ourapponkube.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/ourapponkube.md)] --- ## Exposing services for external access - Now we would like to access the Web UI - We will expose it with a `NodePort` (just like we did for the registry) .exercise[ - Create a `NodePort` service for the Web UI: ```bash kubectl expose deploy/webui --type=NodePort --port=80 ``` - Check the port that was allocated: ```bash kubectl get svc ``` ] .debug[[k8s/ourapponkube.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/ourapponkube.md)] --- ## Accessing the web UI - We can now connect to *any node*, on the allocated node port, to view the web UI .exercise[ - Open the web UI in your browser (http://node-ip-address:3xxxx/) ] -- Yes, this may take a little while to update. *(Narrator: it was DNS.)* -- *Alright, we're back to where we started, when we were running on a single node!* ??? :EN:- Running our demo app on Kubernetes :FR:- Faire tourner l'application de démo sur Kubernetes .debug[[k8s/ourapponkube.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/ourapponkube.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/wall-of-containers.jpeg)] --- name: toc-deploying-with-yaml class: title Deploying with YAML .nav[ [Previous section](#toc-running-our-application-on-kubernetes) | [Back to table of contents](#toc-module-3) | [Next section](#toc-setting-up-kubernetes) ] .debug[(automatically generated title slide)] --- # Deploying with YAML - So far, we created resources with the following commands: - `kubectl run` - `kubectl create deployment` - `kubectl expose` - We can also create resources directly with YAML manifests .debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/yamldeploy.md)] --- ## `kubectl apply` vs `create` - `kubectl create -f whatever.yaml` - creates resources if they don't exist - if resources already exist, don't alter them
(and display error message) - `kubectl apply -f whatever.yaml` - creates resources if they don't exist - if resources already exist, update them
(to match the definition provided by the YAML file) - stores the manifest as an *annotation* in the resource .debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/yamldeploy.md)] --- ## Creating multiple resources - The manifest can contain multiple resources separated by `---` ```yaml kind: ... apiVersion: ... metadata: ... name: ... ... --- kind: ... apiVersion: ... metadata: ... name: ... ... ``` .debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/yamldeploy.md)] --- ## Creating multiple resources - The manifest can also contain a list of resources ```yaml apiVersion: v1 kind: List items: - kind: ... apiVersion: ... ... - kind: ... apiVersion: ... ... ``` .debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/yamldeploy.md)] --- ## Deploying dockercoins with YAML - We provide a YAML manifest with all the resources for Dockercoins (Deployments and Services) - We can use it if we need to deploy or redeploy Dockercoins .exercise[ - Deploy or redeploy Dockercoins: ```bash kubectl apply -f ~/container.training/k8s/dockercoins.yaml ``` ] (If we deployed Dockercoins earlier, we will see warning messages, because the resources that we created lack the necessary annotation. We can safely ignore them.) .debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/yamldeploy.md)] --- ## Deleting resources - We can also use a YAML file to *delete* resources - `kubectl delete -f ...` will delete all the resources mentioned in a YAML file (useful to clean up everything that was created by `kubectl apply -f ...`) - The definitions of the resources don't matter (just their `kind`, `apiVersion`, and `name`) .debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/yamldeploy.md)] --- ## Pruning¹ resources - We can also tell `kubectl` to remove old resources - This is done with `kubectl apply -f ... --prune` - It will remove resources that don't exist in the YAML file(s) - But only if they were created with `kubectl apply` in the first place (technically, if they have an annotation `kubectl.kubernetes.io/last-applied-configuration`) .footnote[¹If English is not your first language: *to prune* means to remove dead or overgrown branches in a tree, to help it to grow.] .debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/yamldeploy.md)] --- ## YAML as source of truth - Imagine the following workflow: - do not use `kubectl run`, `kubectl create deployment`, `kubectl expose` ... - define everything with YAML - `kubectl apply -f ... --prune --all` that YAML - keep that YAML under version control - enforce all changes to go through that YAML (e.g. with pull requests) - Our version control system now has a full history of what we deploy - Compares to "Infrastructure-as-Code", but for app deployments .debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/yamldeploy.md)] --- class: extra-details ## Specifying the namespace - When creating resources from YAML manifests, the namespace is optional - If we specify a namespace: - resources are created in the specified namespace - this is typical for things deployed only once per cluster - example: system components, cluster add-ons ... - If we don't specify a namespace: - resources are created in the current namespace - this is typical for things that may be deployed multiple times - example: applications (production, staging, feature branches ...) ??? :EN:- Deploying with YAML manifests :FR:- Déployer avec des *manifests* YAML .debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/yamldeploy.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/Container-Ship-Freighter-Navigation-Elbe-Romance-1782991.jpg)] --- name: toc-setting-up-kubernetes class: title Setting up Kubernetes .nav[ [Previous section](#toc-deploying-with-yaml) | [Back to table of contents](#toc-module-4) | [Next section](#toc-running-a-local-development-cluster) ] .debug[(automatically generated title slide)] --- # Setting up Kubernetes - Kubernetes is made of many components that require careful configuration - Secure operation typically requires TLS certificates and a local CA (certificate authority) - Setting up everything manually is possible, but rarely done (except for learning purposes) - Let's do a quick overview of available options! .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- ## Local development - Are you writing code that will eventually run on Kubernetes? - Then it's a good idea to have a development cluster! - Development clusters only need one node - This simplifies their setup a lot: - pod networking doesn't even need CNI plugins, overlay networks, etc. - they can be fully contained (no pun intended) in an easy-to-ship VM image - some of the security aspects may be simplified (different threat model) - Examples: Docker Desktop, k3d, KinD, MicroK8s, Minikube (some of these also support clusters with multiple nodes) .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- ## Managed clusters - Many cloud providers and hosting providers offer "managed Kubernetes" - The deployment and maintenance of the cluster is entirely managed by the provider (ideally, clusters can be spun up automatically through an API, CLI, or web interface) - Given the complexity of Kubernetes, this approach is *strongly recommended* (at least for your first production clusters) - After working for a while with Kubernetes, you will be better equipped to decide: - whether to operate it yourself or use a managed offering - which offering or which distribution works best for you and your needs .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- ## Managed clusters details - Pricing models differ from one provider to another - nodes are generally charged at their usual price - control plane may be free or incur a small nominal fee - Beyond pricing, there are *huge* differences in features between providers - The "major" providers are not always the best ones! .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- ## Managed clusters differences - Most providers let you pick which Kubernetes version you want - some providers offer up-to-date versions - others lag significantly (sometimes by 2 or 3 minor versions) - Some providers offer multiple networking or storage options - Others will only support one, tied to their infrastructure (changing that is in theory possible, but might be complex or unsupported) - Some providers let you configure or customize the control plane (generally through Kubernetes "feature gates") .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- ## Kubernetes distributions and installers - If you want to run Kubernetes yourselves, there are many options (free, commercial, proprietary, open source ...) - Some of them are installers, while some are complete platforms - Some of them leverage other well-known deployment tools (like Puppet, Terraform ...) - A good starting point to explore these options is this [guide](https://v1-16.docs.kubernetes.io/docs/setup/#production-environment) (it defines categories like "managed", "turnkey" ...) .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- ## kubeadm - kubeadm is a tool part of Kubernetes to facilitate cluster setup - Many other installers and distributions use it (but not all of them) - It can also be used by itself - Excellent starting point to install Kubernetes on your own machines (virtual, physical, it doesn't matter) - It even supports highly available control planes, or "multi-master" (this is more complex, though, because it introduces the need for an API load balancer) .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- ## Manual setup - The resources below are mainly for educational purposes! - [Kubernetes The Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way) by Kelsey Hightower - step by step guide to install Kubernetes on Google Cloud - covers certificates, high availability ... - *“Kubernetes The Hard Way is optimized for learning, which means taking the long route to ensure you understand each task required to bootstrap a Kubernetes cluster.”* - [Deep Dive into Kubernetes Internals for Builders and Operators](https://www.youtube.com/watch?v=3KtEAa7_duA) - conference presentation showing step-by-step control plane setup - emphasis on simplicity, not on security and availability .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- ## About our training clusters - How did we set up these Kubernetes clusters that we're using? -- - We used `kubeadm` on freshly installed VM instances running Ubuntu LTS 1. Install Docker 2. Install Kubernetes packages 3. Run `kubeadm init` on the first node (it deploys the control plane on that node) 4. Set up Weave (the overlay network) with a single `kubectl apply` command 5. Run `kubeadm join` on the other nodes (with the token produced by `kubeadm init`) 6. Copy the configuration file generated by `kubeadm init` - Check the [prepare VMs README](https://github.com/jpetazzo/container.training/blob/master/prepare-vms/README.md) for more details .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- ## `kubeadm` "drawbacks" - Doesn't set up Docker or any other container engine (this is by design, to give us choice) - Doesn't set up the overlay network (this is also by design, for the same reasons) - HA control plane requires [some extra steps](https://kubernetes.io/docs/setup/independent/high-availability/) - Note that HA control plane also requires setting up a specific API load balancer (which is beyond the scope of kubeadm) ??? :EN:- Various ways to install Kubernetes :FR:- Survol des techniques d'installation de Kubernetes .debug[[k8s/setup-overview.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-overview.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/ShippingContainerSFBay.jpg)] --- name: toc-running-a-local-development-cluster class: title Running a local development cluster .nav[ [Previous section](#toc-setting-up-kubernetes) | [Back to table of contents](#toc-module-4) | [Next section](#toc-deploying-a-managed-cluster) ] .debug[(automatically generated title slide)] --- # Running a local development cluster - Let's review some options to run Kubernetes locally - There is no "best option", it depends what you value: - ability to run on all platforms (Linux, Mac, Windows, other?) - ability to run clusters with multiple nodes - ability to run multiple clusters side by side - ability to run recent (or even, unreleased) versions of Kubernetes - availability of plugins - etc. .debug[[k8s/setup-devel.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-devel.md)] --- ## Docker Desktop - Available on Mac and Windows - Gives you one cluster with one node - Rather old version of Kubernetes - Very easy to use if you are already using Docker Desktop: go to Docker Desktop preferences and enable Kubernetes - Ideal for Docker users who need good integration between both platforms .debug[[k8s/setup-devel.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-devel.md)] --- ## [k3d](https://k3d.io/) - Based on [K3s](https://k3s.io/) by Rancher Labs - Requires Docker - Runs Kubernetes nodes in Docker containers - Can deploy multiple clusters, with multiple nodes, and multiple master nodes - As of June 2020, two versions co-exist: stable (1.7) and beta (3.0) - They have different syntax and options, this can be confusing (but don't let that stop you!) .debug[[k8s/setup-devel.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-devel.md)] --- ## k3d in action - Get `k3d` beta 3 binary on https://github.com/rancher/k3d/releases - Create a simple cluster: ```bash k3d create cluster petitcluster --update-kubeconfig ``` - Use it: ```bash kubectl config use-context k3d-petitcluster ``` - Create a more complex cluster with a custom version: ```bash k3d create cluster groscluster --update-kubeconfig \ --image rancher/k3s:v1.18.3-k3s1 --masters 3 --workers 5 --api-port 6444 ``` (note: API port seems to be necessary when running multiple clusters) .debug[[k8s/setup-devel.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-devel.md)] --- ## [KinD](https://kind.sigs.k8s.io/) - Kubernetes-in-Docker - Requires Docker (obviously!) - Deploying a single node cluster using the latest version is simple: ```bash kind create cluster ``` - More advanced scenarios require writing a short [config file](https://kind.sigs.k8s.io/docs/user/quick-start#configuring-your-kind-cluster) (to define multiple nodes, multiple master nodes, set Kubernetes versions ...) - Can deploy multiple clusters .debug[[k8s/setup-devel.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-devel.md)] --- ## [Minikube](https://minikube.sigs.k8s.io/docs/) - The "legacy" option! (note: this is not a bad thing, it means that it's very stable, has lots of plugins, etc.) - Supports many [drivers](https://minikube.sigs.k8s.io/docs/drivers/) (HyperKit, Hyper-V, KVM, VirtualBox, but also Docker and many others) - Can deploy a single cluster; recent versions can deploy multiple nodes - Great option if you want a "Kubernetes first" experience (i.e. if you don't already have Docker and/or don't want/need it) .debug[[k8s/setup-devel.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-devel.md)] --- ## [MicroK8s](https://microk8s.io/) - Available on Linux, and since recently, on Mac and Windows as well - The Linux version is installed through Snap (which is pre-installed on all recent versions of Ubuntu) - Also supports clustering (as in, multiple machines running MicroK8s) - DNS is not enabled by default; enable it with `microk8s enable dns` .debug[[k8s/setup-devel.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-devel.md)] --- ## VM with custom install - Choose your own adventure! - Pick any Linux distribution! - Build your cluster from scratch or use a Kubernetes installer! - Discover exotic CNI plugins and container runtimes! - The only limit is yourself, and the time you are willing to sink in! ??? :EN:- Kubernetes options for local development :FR:- Installation de Kubernetes pour travailler en local .debug[[k8s/setup-devel.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-devel.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/aerial-view-of-containers.jpg)] --- name: toc-deploying-a-managed-cluster class: title Deploying a managed cluster .nav[ [Previous section](#toc-running-a-local-development-cluster) | [Back to table of contents](#toc-module-4) | [Next section](#toc-kubernetes-distributions-and-installers) ] .debug[(automatically generated title slide)] --- # Deploying a managed cluster *"The easiest way to install Kubernetes is to get someone else to do it for you."
([Jérôme Petazzoni](https://twitter.com/jpetazzo))* - Let's see a few options to install managed clusters! - This is not an exhaustive list (the goal is to show the actual steps to get started) - The list is sorted alphabetically - All the options mentioned here require an account with a cloud provider - ... And a credit card .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## AKS (initial setup) - Install the Azure CLI - Login: ```bash az login ``` - Select a [region](https://azure.microsoft.com/en-us/global-infrastructure/services/?products=kubernetes-service®ions=all ) - Create a "resource group": ```bash az group create --name my-aks-group --location westeurope ``` .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## AKS (create cluster) - Create the cluster: ```bash az aks create --resource-group my-aks-group --name my-aks-cluster ``` - Wait about 5-10 minutes - Add credentials to `kubeconfig`: ```bash az aks get-credentials --resource-group my-aks-group --name my-aks-cluster ``` .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## AKS (cleanup) - Delete the cluster: ```bash az aks delete --resource-group my-aks-group --name my-aks-cluster ``` - Delete the resource group: ```bash az group delete --resource-group my-aks-group ``` - Note: delete actions can take a while too! (5-10 minutes as well) .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## AKS (notes) - The cluster has useful components pre-installed, such as the metrics server - There is also a product called [AKS Engine](https://github.com/Azure/aks-engine): - leverages ARM (Azure Resource Manager) templates to deploy Kubernetes - it's "the library used by AKS" - fully customizable - think of it as "half-managed" Kubernetes option .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Amazon EKS (the old way) - [Read the doc](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html) - Create service roles, VPCs, and a bunch of other oddities - Try to figure out why it doesn't work - Start over, following an [official AWS blog post](https://aws.amazon.com/blogs/aws/amazon-eks-now-generally-available/) - Try to find the missing Cloud Formation template -- .footnote[(╯°□°)╯︵ ┻━┻] .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Amazon EKS (the new way) - Install `eksctl` - Set the usual environment variables ([AWS_DEFAULT_REGION](https://docs.aws.amazon.com/general/latest/gr/rande.html#eks_region), AWS_ACCESS_KEY, AWS_SECRET_ACCESS_KEY) - Create the cluster: ```bash eksctl create cluster ``` - Cluster can take a long time to be ready (15-20 minutes is typical) - Add cluster add-ons (by default, it doesn't come with metrics-server, logging, etc.) .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Amazon EKS (cleanup) - Delete the cluster: ```bash eksctl delete cluster
``` - If you need to find the name of the cluster: ```bash eksctl get clusters ``` .footnote[Note: the AWS documentation has been updated and now includes [eksctl instructions](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html).] .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Amazon EKS (notes) - Convenient if you *have to* use AWS - Needs extra steps to be truly production-ready - [Versions tend to be outdated](https://twitter.com/jpetazzo/status/1252948707680686081) - The only officially supported pod network is the [Amazon VPC CNI plugin](https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html) - integrates tightly with security groups and VPC networking - not suitable for high density clusters (with many small pods on big nodes) - other plugins [should still work](https://docs.aws.amazon.com/eks/latest/userguide/alternate-cni-plugins.html) but will require extra work .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Digital Ocean (initial setup) - Install `doctl` - Generate API token (in web console) - Set up the CLI authentication: ```bash doctl auth init ``` (It will ask you for the API token) - Check the list of regions and pick one: ```bash doctl compute region list ``` (If you don't specify the region later, it will use `nyc1`) .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Digital Ocean (create cluster) - Create the cluster: ```bash doctl kubernetes cluster create my-do-cluster [--region xxx1] ``` - Wait 5 minutes - Update `kubeconfig`: ```bash kubectl config use-context do-xxx1-my-do-cluster ``` - The cluster comes with some components (like Cilium) but no metrics server .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Digital Ocean (cleanup) - List clusters (if you forgot its name): ```bash doctl kubernetes cluster list ``` - Delete the cluster: ```bash doctl kubernetes cluster delete my-do-cluster ``` .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## GKE (initial setup) - Install `gcloud` - Login: ```bash gcloud auth init ``` - Create a "project": ```bash gcloud projects create my-gke-project gcloud config set project my-gke-project ``` - Pick a [region](https://cloud.google.com/compute/docs/regions-zones/) (example: `europe-west1`, `us-west1`, ...) .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## GKE (create cluster) - Create the cluster: ```bash gcloud container clusters create my-gke-cluster --region us-west1 --num-nodes=2 ``` (without `--num-nodes` you might exhaust your IP address quota!) - The first time you try to create a cluster in a given project, you get an error - you need to enable the Kubernetes Engine API - the error message gives you a link - follow the link and enable the API (and billing)
(it's just a couple of clicks and it's instantaneous) - Clutser should be ready in a couple of minutes .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## GKE (cleanup) - List clusters (if you forgot its name): ```bash gcloud container clusters list ``` - Delete the cluster: ```bash gcloud container clusters delete my-gke-cluster --region us-west1 ``` - Delete the project (optional): ```bash gcloud projects delete my-gke-project ``` .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## GKE (notes) - Well-rounded product overall (it used to be one of the best managed Kubernetes offerings available; now that many other providers entered the game, that title is debatable) - The cluster comes with many add-ons - Versions lag a bit: - latest minor version (e.g. 1.18) tends to be unsupported - previous minor version (e.g. 1.17) supported through alpha channel - previous versions (e.g. 1.14-1.16) supported .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Scaleway (initial setup) - After creating your account, make sure you set a password or get an API key (by default, it uses email "magic links" to sign in) - Install `scw` (you need [CLI v2](https://github.com/scaleway/scaleway-cli/tree/v2#Installation), which in beta as of May 2020) - Generate the CLI configuration with `scw init` (it will prompt for your API key, or email + password) .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Scaleway (create cluster) - Create the cluster: ```bash k8s cluster create name=my-kapsule-cluster version=1.18.3 cni=cilium \ default-pool-config.node-type=DEV1-M default-pool-config.size=3 ``` - After less than 5 minutes, cluster state will be `ready` (check cluster status with e.g. `scw k8s cluster list` on a wide terminal ) - Add connection information to your `.kube/config` file: ```bash scw k8s kubeconfig install `CLUSTERID` ``` (the cluster ID is shown by `scw k8s cluster list`) .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- class: extra-details ## Scaleway (automation) - If you want to obtain the cluster ID programmatically, this will do it: ```bash scw k8s cluster list # or CLUSTERID=$(scw k8s cluster list -o json | \ jq -r '.[] | select(.name="my-kapsule-cluster") | .id') ``` .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Scaleway (cleanup) - Get cluster ID (e.g. with `scw k8s cluster list`) - Delete the cluster: ```bash scw cluster delete cluster-id=$CLUSTERID ``` - Warning: as of May 2020, load balancers have to be deleted separately! .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## Scaleway (notes) - The `create` command is a bit more complex than with other providers (you must specify the Kubernetes version, CNI plugin, and node type) - To see available versions and CNI plugins, run `scw k8s version list` - As of May 2020, Kapsule supports: - multiple CNI plugins, including: cilium, calico, weave, flannel - Kubernetes versions 1.15 to 1.18 - multiple container runtimes, including: Docker, containerd, CRI-O - To see available node types and their price, check their [pricing page]( https://www.scaleway.com/en/pricing/) .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- ## More options - Alibaba Cloud - [IBM Cloud](https://console.bluemix.net/docs/containers/cs_cli_install.html#cs_cli_install) - [Linode Kubernetes Engine (LKE)](https://www.linode.com/products/kubernetes/) - OVHcloud [Managed Kubernetes Service](https://www.ovhcloud.com/en/public-cloud/kubernetes/) - ... ??? :EN:- Installing a managed cluster :FR:- Installer un cluster infogéré .debug[[k8s/setup-managed.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-managed.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/blue-containers.jpg)] --- name: toc-kubernetes-distributions-and-installers class: title Kubernetes distributions and installers .nav[ [Previous section](#toc-deploying-a-managed-cluster) | [Back to table of contents](#toc-module-4) | [Next section](#toc-the-kubernetes-dashboard) ] .debug[(automatically generated title slide)] --- # Kubernetes distributions and installers - Sometimes, we need to run Kubernetes ourselves (as opposed to "use a managed offering") - Beware: it takes *a lot of work* to set up and maintain Kubernetes - It might be necessary if you have specific security or compliance requirements (e.g. national security for states that don't have a suitable domestic cloud) - There are [countless](https://kubernetes.io/docs/setup/pick-right-solution/) distributions available - We can't review them all - We're just going to explore a few options .debug[[k8s/setup-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-selfhosted.md)] --- ## [kops](https://github.com/kubernetes/kops) - Deploys Kubernetes using cloud infrastructure (supports AWS, GCE, Digital Ocean ...) - Leverages special cloud features when possible (e.g. Auto Scaling Groups ...) .debug[[k8s/setup-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-selfhosted.md)] --- ## kubeadm - Provisions Kubernetes nodes on top of existing machines - `kubeadm init` to provision a single-node control plane - `kubeadm join` to join a node to the cluster - Supports HA control plane [with some extra steps](https://kubernetes.io/docs/setup/independent/high-availability/) .debug[[k8s/setup-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-selfhosted.md)] --- ## [kubespray](https://github.com/kubernetes-incubator/kubespray) - Based on Ansible - Works on bare metal and cloud infrastructure (good for hybrid deployments) - The expert says: ultra flexible; slow; complex .debug[[k8s/setup-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-selfhosted.md)] --- ## RKE (Rancher Kubernetes Engine) - Opinionated installer with low requirements - Requires a set of machines with Docker + SSH access - Supports highly available etcd and control plane - The expert says: fast; maintenance can be tricky .debug[[k8s/setup-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-selfhosted.md)] --- ## Terraform + kubeadm - Sometimes it is necessary to build a custom solution - Example use case: - deploying Kubernetes on OpenStack - ... with highly available control plane - ... and Cloud Controller Manager integration - Solution: Terraform + kubeadm (kubeadm driven by remote-exec) - [GitHub repository](https://github.com/enix/terraform-openstack-kubernetes) - [Blog post (in French)](https://enix.io/fr/blog/deployer-kubernetes-1-13-sur-openstack-grace-a-terraform/) .debug[[k8s/setup-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-selfhosted.md)] --- ## And many more ... - [AKS Engine](https://github.com/Azure/aks-engine) - Docker Enterprise Edition - [Lokomotive](https://github.com/kinvolk/lokomotive), leveraging Terraform and [Flatcar Linux](https://www.flatcar-linux.org/) - Pivotal Container Service (PKS) - [Tarmak](https://github.com/jetstack/tarmak), leveraging Puppet and Terraform - Tectonic by CoreOS (now being integrated into Red Hat OpenShift) - [Typhoon](https://typhoon.psdn.io/), leveraging Terraform - VMware Tanzu Kubernetes Grid (TKG) .debug[[k8s/setup-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-selfhosted.md)] --- ## Bottom line - Each distribution / installer has pros and cons - Before picking one, we should sort out our priorities: - cloud, on-premises, hybrid? - integration with existing network/storage architecture or equipment? - are we storing very sensitive data, like finance, health, military? - how many clusters are we deploying (and maintaining): 2, 10, 50? - which team will be responsible for deployment and maintenance?
(do they need training?) - etc. ??? :EN:- Kubernetes distributions and installers :FR:- L'offre Kubernetes "on premises" .debug[[k8s/setup-selfhosted.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/setup-selfhosted.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/chinook-helicopter-container.jpg)] --- name: toc-the-kubernetes-dashboard class: title The Kubernetes dashboard .nav[ [Previous section](#toc-kubernetes-distributions-and-installers) | [Back to table of contents](#toc-module-4) | [Next section](#toc-security-implications-of-kubectl-apply) ] .debug[(automatically generated title slide)] --- # The Kubernetes dashboard - Kubernetes resources can also be viewed with a web dashboard - That dashboard is usually exposed over HTTPS (this requires obtaining a proper TLS certificate) - Dashboard users need to authenticate - We are going to take a *dangerous* shortcut .debug[[k8s/dashboard.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dashboard.md)] --- ## The insecure method - We could (and should) use [Let's Encrypt](https://letsencrypt.org/) ... - ... but we don't want to deal with TLS certificates - We could (and should) learn how authentication and authorization work ... - ... but we will use a guest account with admin access instead .footnote[.warning[Yes, this will open our cluster to all kinds of shenanigans. Don't do this at home.]] .debug[[k8s/dashboard.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dashboard.md)] --- ## Running a very insecure dashboard - We are going to deploy that dashboard with *one single command* - This command will create all the necessary resources (the dashboard itself, the HTTP wrapper, the admin/guest account) - All these resources are defined in a YAML file - All we have to do is load that YAML file with with `kubectl apply -f` .exercise[ - Create all the dashboard resources, with the following command: ```bash kubectl apply -f ~/container.training/k8s/insecure-dashboard.yaml ``` ] .debug[[k8s/dashboard.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dashboard.md)] --- ## Connecting to the dashboard .exercise[ - Check which port the dashboard is on: ```bash kubectl get svc dashboard ``` ] You'll want the `3xxxx` port. .exercise[ - Connect to http://oneofournodes:3xxxx/ ] The dashboard will then ask you which authentication you want to use. .debug[[k8s/dashboard.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dashboard.md)] --- ## Dashboard authentication - We have three authentication options at this point: - token (associated with a role that has appropriate permissions) - kubeconfig (e.g. using the `~/.kube/config` file from `node1`) - "skip" (use the dashboard "service account") - Let's use "skip": we're logged in! -- .warning[By the way, we just added a backdoor to our Kubernetes cluster!] .debug[[k8s/dashboard.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dashboard.md)] --- ## Running the Kubernetes dashboard securely - The steps that we just showed you are *for educational purposes only!* - If you do that on your production cluster, people [can and will abuse it](https://redlock.io/blog/cryptojacking-tesla) - For an in-depth discussion about securing the dashboard,
check [this excellent post on Heptio's blog](https://blog.heptio.com/on-securing-the-kubernetes-dashboard-16b09b1b7aca) .debug[[k8s/dashboard.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dashboard.md)] --- ## Other dashboards - [Kube Web View](https://codeberg.org/hjacobs/kube-web-view) - read-only dashboard - optimized for "troubleshooting and incident response" - see [vision and goals](https://kube-web-view.readthedocs.io/en/latest/vision.html#vision) for details - [Kube Ops View](https://github.com/hjacobs/kube-ops-view) - "provides a common operational picture for multiple Kubernetes clusters" .debug[[k8s/dashboard.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dashboard.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/container-cranes.jpg)] --- name: toc-security-implications-of-kubectl-apply class: title Security implications of `kubectl apply` .nav[ [Previous section](#toc-the-kubernetes-dashboard) | [Back to table of contents](#toc-module-4) | [Next section](#toc-scaling-our-demo-app) ] .debug[(automatically generated title slide)] --- # Security implications of `kubectl apply` - When we do `kubectl apply -f
`, we create arbitrary resources - Resources can be evil; imagine a `deployment` that ... -- - starts bitcoin miners on the whole cluster -- - hides in a non-default namespace -- - bind-mounts our nodes' filesystem -- - inserts SSH keys in the root account (on the node) -- - encrypts our data and ransoms it -- - ☠☠☠ .debug[[k8s/dashboard.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dashboard.md)] --- ## `kubectl apply` is the new `curl | sh` - `curl | sh` is convenient - It's safe if you use HTTPS URLs from trusted sources -- - `kubectl apply -f` is convenient - It's safe if you use HTTPS URLs from trusted sources - Example: the official setup instructions for most pod networks -- - It introduces new failure modes (for instance, if you try to apply YAML from a link that's no longer valid) ??? :EN:- The Kubernetes dashboard :FR:- Le *dashboard* Kubernetes .debug[[k8s/dashboard.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dashboard.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/container-housing.jpg)] --- name: toc-scaling-our-demo-app class: title Scaling our demo app .nav[ [Previous section](#toc-security-implications-of-kubectl-apply) | [Back to table of contents](#toc-module-4) | [Next section](#toc-daemon-sets) ] .debug[(automatically generated title slide)] --- # Scaling our demo app - Our ultimate goal is to get more DockerCoins (i.e. increase the number of loops per second shown on the web UI) - Let's look at the architecture again: ![DockerCoins architecture](images/dockercoins-diagram.svg) - The loop is done in the worker; perhaps we could try adding more workers? .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- ## Adding another worker - All we have to do is scale the `worker` Deployment .exercise[ - Open a new terminal to keep an eye on our pods: ```bash kubectl get pods -w ``` - Now, create more `worker` replicas: ```bash kubectl scale deployment worker --replicas=2 ``` ] After a few seconds, the graph in the web UI should show up. .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- ## Adding more workers - If 2 workers give us 2x speed, what about 3 workers? .exercise[ - Scale the `worker` Deployment further: ```bash kubectl scale deployment worker --replicas=3 ``` ] The graph in the web UI should go up again. (This is looking great! We're gonna be RICH!) .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- ## Adding even more workers - Let's see if 10 workers give us 10x speed! .exercise[ - Scale the `worker` Deployment to a bigger number: ```bash kubectl scale deployment worker --replicas=10 ``` ] -- The graph will peak at 10 hashes/second. (We can add as many workers as we want: we will never go past 10 hashes/second.) .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- class: extra-details ## Didn't we briefly exceed 10 hashes/second? - It may *look like it*, because the web UI shows instant speed - The instant speed can briefly exceed 10 hashes/second - The average speed cannot - The instant speed can be biased because of how it's computed .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- class: extra-details ## Why instant speed is misleading - The instant speed is computed client-side by the web UI - The web UI checks the hash counter once per second
(and does a classic (h2-h1)/(t2-t1) speed computation) - The counter is updated once per second by the workers - These timings are not exact
(e.g. the web UI check interval is client-side JavaScript) - Sometimes, between two web UI counter measurements,
the workers are able to update the counter *twice* - During that cycle, the instant speed will appear to be much bigger
(but it will be compensated by lower instant speed before and after) .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- ## Why are we stuck at 10 hashes per second? - If this was high-quality, production code, we would have instrumentation (Datadog, Honeycomb, New Relic, statsd, Sumologic, ...) - It's not! - Perhaps we could benchmark our web services? (with tools like `ab`, or even simpler, `httping`) .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- ## Benchmarking our web services - We want to check `hasher` and `rng` - We are going to use `httping` - It's just like `ping`, but using HTTP `GET` requests (it measures how long it takes to perform one `GET` request) - It's used like this: ``` httping [-c count] http://host:port/path ``` - Or even simpler: ``` httping ip.ad.dr.ess ``` - We will use `httping` on the ClusterIP addresses of our services .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- ## Obtaining ClusterIP addresses - We can simply check the output of `kubectl get services` - Or do it programmatically, as in the example below .exercise[ - Retrieve the IP addresses: ```bash HASHER=$(kubectl get svc hasher -o go-template={{.spec.clusterIP}}) RNG=$(kubectl get svc rng -o go-template={{.spec.clusterIP}}) ``` ] Now we can access the IP addresses of our services through `$HASHER` and `$RNG`. .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- ## Checking `hasher` and `rng` response times .exercise[ - Check the response times for both services: ```bash httping -c 3 $HASHER httping -c 3 $RNG ``` ] - `hasher` is fine (it should take a few milliseconds to reply) - `rng` is not (it should take about 700 milliseconds if there are 10 workers) - Something is wrong with `rng`, but ... what? ??? :EN:- Scaling up our demo app :FR:- *Scale up* de l'application de démo .debug[[k8s/scalingdockercoins.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/scalingdockercoins.md)] --- ## Let's draw hasty conclusions - The bottleneck seems to be `rng` - *What if* we don't have enough entropy and can't generate enough random numbers? - We need to scale out the `rng` service on multiple machines! Note: this is a fiction! We have enough entropy. But we need a pretext to scale out. (In fact, the code of `rng` uses `/dev/urandom`, which never runs out of entropy...
...and is [just as good as `/dev/random`](http://www.slideshare.net/PacSecJP/filippo-plain-simple-reality-of-entropy).) .debug[[shared/hastyconclusions.md](https://github.com/jpetazzo/container.training/tree/master/slides/shared/hastyconclusions.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/containers-by-the-water.jpg)] --- name: toc-daemon-sets class: title Daemon sets .nav[ [Previous section](#toc-scaling-our-demo-app) | [Back to table of contents](#toc-module-4) | [Next section](#toc-labels-and-selectors) ] .debug[(automatically generated title slide)] --- # Daemon sets - We want to scale `rng` in a way that is different from how we scaled `worker` - We want one (and exactly one) instance of `rng` per node - We *do not want* two instances of `rng` on the same node - We will do that with a *daemon set* .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Why not a deployment? - Can't we just do `kubectl scale deployment rng --replicas=...`? -- - Nothing guarantees that the `rng` containers will be distributed evenly - If we add nodes later, they will not automatically run a copy of `rng` - If we remove (or reboot) a node, one `rng` container will restart elsewhere (and we will end up with two instances `rng` on the same node) - By contrast, a daemon set will start one pod per node and keep it that way (as nodes are added or removed) .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Daemon sets in practice - Daemon sets are great for cluster-wide, per-node processes: - `kube-proxy` - `weave` (our overlay network) - monitoring agents - hardware management tools (e.g. SCSI/FC HBA agents) - etc. - They can also be restricted to run [only on some nodes](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#running-pods-on-only-some-nodes) .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Creating a daemon set - Unfortunately, as of Kubernetes 1.19, the CLI cannot create daemon sets -- - More precisely: it doesn't have a subcommand to create a daemon set -- - But any kind of resource can always be created by providing a YAML description: ```bash kubectl apply -f foo.yaml ``` -- - How do we create the YAML file for our daemon set? -- - option 1: [read the docs](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#create-a-daemonset) -- - option 2: `vi` our way out of it .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Creating the YAML file for our daemon set - Let's start with the YAML file for the current `rng` resource .exercise[ - Dump the `rng` resource in YAML: ```bash kubectl get deploy/rng -o yaml >rng.yml ``` - Edit `rng.yml` ] .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## "Casting" a resource to another - What if we just changed the `kind` field? (It can't be that easy, right?) .exercise[ - Change `kind: Deployment` to `kind: DaemonSet` - Save, quit - Try to create our new resource: ```bash kubectl apply -f rng.yml ``` ] -- We all knew this couldn't be that easy, right! .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Understanding the problem - The core of the error is: ``` error validating data: [ValidationError(DaemonSet.spec): unknown field "replicas" in io.k8s.api.extensions.v1beta1.DaemonSetSpec, ... ``` -- - *Obviously,* it doesn't make sense to specify a number of replicas for a daemon set -- - Workaround: fix the YAML - remove the `replicas` field - remove the `strategy` field (which defines the rollout mechanism for a deployment) - remove the `progressDeadlineSeconds` field (also used by the rollout mechanism) - remove the `status: {}` line at the end -- - Or, we could also ... .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Use the `--force`, Luke - We could also tell Kubernetes to ignore these errors and try anyway - The `--force` flag's actual name is `--validate=false` .exercise[ - Try to load our YAML file and ignore errors: ```bash kubectl apply -f rng.yml --validate=false ``` ] -- 🎩✨🐇 -- Wait ... Now, can it be *that* easy? .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Checking what we've done - Did we transform our `deployment` into a `daemonset`? .exercise[ - Look at the resources that we have now: ```bash kubectl get all ``` ] -- We have two resources called `rng`: - the *deployment* that was existing before - the *daemon set* that we just created We also have one too many pods.
(The pod corresponding to the *deployment* still exists.) .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## `deploy/rng` and `ds/rng` - You can have different resource types with the same name (i.e. a *deployment* and a *daemon set* both named `rng`) - We still have the old `rng` *deployment* ``` NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/rng 1 1 1 1 18m ``` - But now we have the new `rng` *daemon set* as well ``` NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/rng 2 2 2 2 2
9s ``` .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Too many pods - If we check with `kubectl get pods`, we see: - *one pod* for the deployment (named `rng-xxxxxxxxxx-yyyyy`) - *one pod per node* for the daemon set (named `rng-zzzzz`) ``` NAME READY STATUS RESTARTS AGE rng-54f57d4d49-7pt82 1/1 Running 0 11m rng-b85tm 1/1 Running 0 25s rng-hfbrr 1/1 Running 0 25s [...] ``` -- The daemon set created one pod per node, except on the master node. The master node has [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) preventing pods from running there. (To schedule a pod on this node anyway, the pod will require appropriate [tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/).) .footnote[(Off by one? We don't run these pods on the node hosting the control plane.)] .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Is this working? - Look at the web UI -- - The graph should now go above 10 hashes per second! -- - It looks like the newly created pods are serving traffic correctly - How and why did this happen? (We didn't do anything special to add them to the `rng` service load balancer!) .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/distillery-containers.jpg)] --- name: toc-labels-and-selectors class: title Labels and selectors .nav[ [Previous section](#toc-daemon-sets) | [Back to table of contents](#toc-module-4) | [Next section](#toc-authoring-yaml) ] .debug[(automatically generated title slide)] --- # Labels and selectors - The `rng` *service* is load balancing requests to a set of pods - That set of pods is defined by the *selector* of the `rng` service .exercise[ - Check the *selector* in the `rng` service definition: ```bash kubectl describe service rng ``` ] - The selector is `app=rng` - It means "all the pods having the label `app=rng`" (They can have additional labels as well, that's OK!) .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Selector evaluation - We can use selectors with many `kubectl` commands - For instance, with `kubectl get`, `kubectl logs`, `kubectl delete` ... and more .exercise[ - Get the list of pods matching selector `app=rng`: ```bash kubectl get pods -l app=rng kubectl get pods --selector app=rng ``` ] But ... why do these pods (in particular, the *new* ones) have this `app=rng` label? .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Where do labels come from? - When we create a deployment with `kubectl create deployment rng`,
this deployment gets the label `app=rng` - The replica sets created by this deployment also get the label `app=rng` - The pods created by these replica sets also get the label `app=rng` - When we created the daemon set from the deployment, we re-used the same spec - Therefore, the pods created by the daemon set get the same labels .footnote[Note: when we use `kubectl run stuff`, the label is `run=stuff` instead.] .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Updating load balancer configuration - We would like to remove a pod from the load balancer - What would happen if we removed that pod, with `kubectl delete pod ...`? -- It would be re-created immediately (by the replica set or the daemon set) -- - What would happen if we removed the `app=rng` label from that pod? -- It would *also* be re-created immediately -- Why?!? .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Selectors for replica sets and daemon sets - The "mission" of a replica set is: "Make sure that there is the right number of pods matching this spec!" - The "mission" of a daemon set is: "Make sure that there is a pod matching this spec on each node!" -- - *In fact,* replica sets and daemon sets do not check pod specifications - They merely have a *selector*, and they look for pods matching that selector - Yes, we can fool them by manually creating pods with the "right" labels - Bottom line: if we remove our `app=rng` label ... ... The pod "disappears" for its parent, which re-creates another pod to replace it .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- class: extra-details ## Isolation of replica sets and daemon sets - Since both the `rng` daemon set and the `rng` replica set use `app=rng` ... ... Why don't they "find" each other's pods? -- - *Replica sets* have a more specific selector, visible with `kubectl describe` (It looks like `app=rng,pod-template-hash=abcd1234`) - *Daemon sets* also have a more specific selector, but it's invisible (It looks like `app=rng,controller-revision-hash=abcd1234`) - As a result, each controller only "sees" the pods it manages .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Removing a pod from the load balancer - Currently, the `rng` service is defined by the `app=rng` selector - The only way to remove a pod is to remove or change the `app` label - ... But that will cause another pod to be created instead! - What's the solution? -- - We need to change the selector of the `rng` service! - Let's add another label to that selector (e.g. `active=yes`) .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Complex selectors - If a selector specifies multiple labels, they are understood as a logical *AND* (In other words: the pods must match all the labels) - Kubernetes has support for advanced, set-based selectors (But these cannot be used with services, at least not yet!) .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## The plan 1. Add the label `active=yes` to all our `rng` pods 2. Update the selector for the `rng` service to also include `active=yes` 3. Toggle traffic to a pod by manually adding/removing the `active` label 4. Profit! *Note: if we swap steps 1 and 2, it will cause a short service disruption, because there will be a period of time during which the service selector won't match any pod. During that time, requests to the service will time out. By doing things in the order above, we guarantee that there won't be any interruption.* .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Adding labels to pods - We want to add the label `active=yes` to all pods that have `app=rng` - We could edit each pod one by one with `kubectl edit` ... - ... Or we could use `kubectl label` to label them all - `kubectl label` can use selectors itself .exercise[ - Add `active=yes` to all pods that have `app=rng`: ```bash kubectl label pods -l app=rng active=yes ``` ] .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Updating the service selector - We need to edit the service specification - Reminder: in the service definition, we will see `app: rng` in two places - the label of the service itself (we don't need to touch that one) - the selector of the service (that's the one we want to change) .exercise[ - Update the service to add `active: yes` to its selector: ```bash kubectl edit service rng ``` ] -- ... And then we get *the weirdest error ever.* Why? .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## When the YAML parser is being too smart - YAML parsers try to help us: - `xyz` is the string `"xyz"` - `42` is the integer `42` - `yes` is the boolean value `true` - If we want the string `"42"` or the string `"yes"`, we have to quote them - So we have to use `active: "yes"` .footnote[For a good laugh: if we had used "ja", "oui", "si" ... as the value, it would have worked!] .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Updating the service selector, take 2 .exercise[ - Update the YAML manifest of the service - Add `active: "yes"` to its selector ] This time it should work! If we did everything correctly, the web UI shouldn't show any change. .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Updating labels - We want to disable the pod that was created by the deployment - All we have to do, is remove the `active` label from that pod - To identify that pod, we can use its name - ... Or rely on the fact that it's the only one with a `pod-template-hash` label - Good to know: - `kubectl label ... foo=` doesn't remove a label (it sets it to an empty string) - to remove label `foo`, use `kubectl label ... foo-` - to change an existing label, we would need to add `--overwrite` .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Removing a pod from the load balancer .exercise[ - In one window, check the logs of that pod: ```bash POD=$(kubectl get pod -l app=rng,pod-template-hash -o name) kubectl logs --tail 1 --follow $POD ``` (We should see a steady stream of HTTP logs) - In another window, remove the label from the pod: ```bash kubectl label pod -l app=rng,pod-template-hash active- ``` (The stream of HTTP logs should stop immediately) ] There might be a slight change in the web UI (since we removed a bit of capacity from the `rng` service). If we remove more pods, the effect should be more visible. .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- class: extra-details ## Updating the daemon set - If we scale up our cluster by adding new nodes, the daemon set will create more pods - These pods won't have the `active=yes` label - If we want these pods to have that label, we need to edit the daemon set spec - We can do that with e.g. `kubectl edit daemonset rng` .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- class: extra-details ## We've put resources in your resources - Reminder: a daemon set is a resource that creates more resources! - There is a difference between: - the label(s) of a resource (in the `metadata` block in the beginning) - the selector of a resource (in the `spec` block) - the label(s) of the resource(s) created by the first resource (in the `template` block) - We would need to update the selector and the template (metadata labels are not mandatory) - The template must match the selector (i.e. the resource will refuse to create resources that it will not select) .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Labels and debugging - When a pod is misbehaving, we can delete it: another one will be recreated - But we can also change its labels - It will be removed from the load balancer (it won't receive traffic anymore) - Another pod will be recreated immediately - But the problematic pod is still here, and we can inspect and debug it - We can even re-add it to the rotation if necessary (Very useful to troubleshoot intermittent and elusive bugs) .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- ## Labels and advanced rollout control - Conversely, we can add pods matching a service's selector - These pods will then receive requests and serve traffic - Examples: - one-shot pod with all debug flags enabled, to collect logs - pods created automatically, but added to rotation in a second step
(by setting their label accordingly) - This gives us building blocks for canary and blue/green deployments ??? :EN:- Scaling with Daemon Sets :FR:- Utilisation de Daemon Sets .debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/daemonset.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/lots-of-containers.jpg)] --- name: toc-authoring-yaml class: title Authoring YAML .nav[ [Previous section](#toc-labels-and-selectors) | [Back to table of contents](#toc-module-4) | [Next section](#toc-rolling-updates) ] .debug[(automatically generated title slide)] --- # Authoring YAML - There are various ways to generate YAML with Kubernetes, e.g.: - `kubectl run` - `kubectl create deployment` (and a few other `kubectl create` variants) - `kubectl expose` - When and why do we need to write our own YAML? - How do we write YAML from scratch? .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## The limits of generated YAML - Many advanced (and even not-so-advanced) features require to write YAML: - pods with multiple containers - resource limits - healthchecks - DaemonSets, StatefulSets - and more! - How do we access these features? .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## We don't have to start from scratch - Create a resource (e.g. Deployment) - Dump its YAML with `kubectl get -o yaml ...` - Edit the YAML - Use `kubectl apply -f ...` with the YAML file to: - update the resource (if it's the same kind) - create a new resource (if it's a different kind) - Or: Use The Docs, Luke (the documentation almost always has YAML examples) .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## Generating YAML without creating resources - We can use the `--dry-run` option .exercise[ - Generate the YAML for a Deployment without creating it: ```bash kubectl create deployment web --image nginx --dry-run ``` ] - We can clean up that YAML even more if we want (for instance, we can remove the `creationTimestamp` and empty dicts) .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## Using `--dry-run` with `kubectl apply` - The `--dry-run` option can also be used with `kubectl apply` - However, it can be misleading (it doesn't do a "real" dry run) - Let's see what happens in the following scenario: - generate the YAML for a Deployment - tweak the YAML to transform it into a DaemonSet - apply that YAML to see what would actually be created .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## The limits of `kubectl apply --dry-run` .exercise[ - Generate the YAML for a deployment: ```bash kubectl create deployment web --image=nginx -o yaml > web.yaml ``` - Change the `kind` in the YAML to make it a `DaemonSet`: ```bash sed -i s/Deployment/DaemonSet/ web.yaml ``` - Ask `kubectl` what would be applied: ```bash kubectl apply -f web.yaml --dry-run --validate=false -o yaml ``` ] The resulting YAML doesn't represent a valid DaemonSet. .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## Server-side dry run - Since Kubernetes 1.13, we can use [server-side dry run and diffs](https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/) - Server-side dry run will do all the work, but *not* persist to etcd (all validation and mutation hooks will be executed) .exercise[ - Try the same YAML file as earlier, with server-side dry run: ```bash kubectl apply -f web.yaml --server-dry-run --validate=false -o yaml ``` ] The resulting YAML doesn't have the `replicas` field anymore. Instead, it has the fields expected in a DaemonSet. .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## Advantages of server-side dry run - The YAML is verified much more extensively - The only step that is skipped is "write to etcd" - YAML that passes server-side dry run *should* apply successfully (unless the cluster state changes by the time the YAML is actually applied) - Validating or mutating hooks that have side effects can also be an issue .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## `kubectl diff` - Kubernetes 1.13 also introduced `kubectl diff` - `kubectl diff` does a server-side dry run, *and* shows differences .exercise[ - Try `kubectl diff` on the YAML that we tweaked earlier: ```bash kubectl diff -f web.yaml ``` ] Note: we don't need to specify `--validate=false` here. .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## Advantage of YAML - Using YAML (instead of `kubectl create
`) allows to be *declarative* - The YAML describes the desired state of our cluster and applications - YAML can be stored, versioned, archived (e.g. in git repositories) - To change resources, change the YAML files (instead of using `kubectl edit`/`scale`/`label`/etc.) - Changes can be reviewed before being applied (with code reviews, pull requests ...) - This workflow is sometimes called "GitOps" (there are tools like Weave Flux or GitKube to facilitate it) .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- ## YAML in practice - Get started with `kubectl create deployment` and `kubectl expose` - Dump the YAML with `kubectl get -o yaml` - Tweak that YAML and `kubectl apply` it back - Store that YAML for reference (for further deployments) - Feel free to clean up the YAML: - remove fields you don't know - check that it still works! - That YAML will be useful later when using e.g. Kustomize or Helm .debug[[k8s/dryrun.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/dryrun.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/plastic-containers.JPG)] --- name: toc-rolling-updates class: title Rolling updates .nav[ [Previous section](#toc-authoring-yaml) | [Back to table of contents](#toc-module-5) | [Next section](#toc-healthchecks) ] .debug[(automatically generated title slide)] --- # Rolling updates - By default (without rolling updates), when a scaled resource is updated: - new pods are created - old pods are terminated - ... all at the same time - if something goes wrong, ¯\\\_(ツ)\_/¯ .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Rolling updates - With rolling updates, when a Deployment is updated, it happens progressively - The Deployment controls multiple Replica Sets - Each Replica Set is a group of identical Pods (with the same image, arguments, parameters ...) - During the rolling update, we have at least two Replica Sets: - the "new" set (corresponding to the "target" version) - at least one "old" set - We can have multiple "old" sets (if we start another update before the first one is done) .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Update strategy - Two parameters determine the pace of the rollout: `maxUnavailable` and `maxSurge` - They can be specified in absolute number of pods, or percentage of the `replicas` count - At any given time ... - there will always be at least `replicas`-`maxUnavailable` pods available - there will never be more than `replicas`+`maxSurge` pods in total - there will therefore be up to `maxUnavailable`+`maxSurge` pods being updated - We have the possibility of rolling back to the previous version
(if the update fails or is unsatisfactory in any way) .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Checking current rollout parameters - Recall how we build custom reports with `kubectl` and `jq`: .exercise[ - Show the rollout plan for our deployments: ```bash kubectl get deploy -o json | jq ".items[] | {name:.metadata.name} + .spec.strategy.rollingUpdate" ``` ] .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Rolling updates in practice - As of Kubernetes 1.8, we can do rolling updates with: `deployments`, `daemonsets`, `statefulsets` - Editing one of these resources will automatically result in a rolling update - Rolling updates can be monitored with the `kubectl rollout` subcommand .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Rolling out the new `worker` service .exercise[ - Let's monitor what's going on by opening a few terminals, and run: ```bash kubectl get pods -w kubectl get replicasets -w kubectl get deployments -w ``` - Update `worker` either with `kubectl edit`, or by running: ```bash kubectl set image deploy worker worker=dockercoins/worker:v0.2 ``` ] -- That rollout should be pretty quick. What shows in the web UI? .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Give it some time - At first, it looks like nothing is happening (the graph remains at the same level) - According to `kubectl get deploy -w`, the `deployment` was updated really quickly - But `kubectl get pods -w` tells a different story - The old `pods` are still here, and they stay in `Terminating` state for a while - Eventually, they are terminated; and then the graph decreases significantly - This delay is due to the fact that our worker doesn't handle signals - Kubernetes sends a "polite" shutdown request to the worker, which ignores it - After a grace period, Kubernetes gets impatient and kills the container (The grace period is 30 seconds, but [can be changed](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods) if needed) .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Rolling out something invalid - What happens if we make a mistake? .exercise[ - Update `worker` by specifying a non-existent image: ```bash kubectl set image deploy worker worker=dockercoins/worker:v0.3 ``` - Check what's going on: ```bash kubectl rollout status deploy worker ``` ] -- Our rollout is stuck. However, the app is not dead. (After a minute, it will stabilize to be 20-25% slower.) .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## What's going on with our rollout? - Why is our app a bit slower? - Because `MaxUnavailable=25%` ... So the rollout terminated 2 replicas out of 10 available - Okay, but why do we see 5 new replicas being rolled out? - Because `MaxSurge=25%` ... So in addition to replacing 2 replicas, the rollout is also starting 3 more - It rounded down the number of MaxUnavailable pods conservatively,
but the total number of pods being rolled out is allowed to be 25+25=50% .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- class: extra-details ## The nitty-gritty details - We start with 10 pods running for the `worker` deployment - Current settings: MaxUnavailable=25% and MaxSurge=25% - When we start the rollout: - two replicas are taken down (as per MaxUnavailable=25%) - two others are created (with the new version) to replace them - three others are created (with the new version) per MaxSurge=25%) - Now we have 8 replicas up and running, and 5 being deployed - Our rollout is stuck at this point! .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Checking the dashboard during the bad rollout If you didn't deploy the Kubernetes dashboard earlier, just skip this slide. .exercise[ - Connect to the dashboard that we deployed earlier - Check that we have failures in Deployments, Pods, and Replica Sets - Can we see the reason for the failure? ] .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Recovering from a bad rollout - We could push some `v0.3` image (the pod retry logic will eventually catch it and the rollout will proceed) - Or we could invoke a manual rollback .exercise[ - Cancel the deployment and wait for the dust to settle: ```bash kubectl rollout undo deploy worker kubectl rollout status deploy worker ``` ] .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Rolling back to an older version - We reverted to `v0.2` - But this version still has a performance problem - How can we get back to the previous version? .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Multiple "undos" - What happens if we try `kubectl rollout undo` again? .exercise[ - Try it: ```bash kubectl rollout undo deployment worker ``` - Check the web UI, the list of pods ... ] 🤔 That didn't work. .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Multiple "undos" don't work - If we see successive versions as a stack: - `kubectl rollout undo` doesn't "pop" the last element from the stack - it copies the N-1th element to the top - Multiple "undos" just swap back and forth between the last two versions! .exercise[ - Go back to v0.2 again: ```bash kubectl rollout undo deployment worker ``` ] .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## In this specific scenario - Our version numbers are easy to guess - What if we had used git hashes? - What if we had changed other parameters in the Pod spec? .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Listing versions - We can list successive versions of a Deployment with `kubectl rollout history` .exercise[ - Look at our successive versions: ```bash kubectl rollout history deployment worker ``` ] We don't see *all* revisions. We might see something like 1, 4, 5. (Depending on how many "undos" we did before.) .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Explaining deployment revisions - These revisions correspond to our Replica Sets - This information is stored in the Replica Set annotations .exercise[ - Check the annotations for our replica sets: ```bash kubectl describe replicasets -l app=worker | grep -A3 ^Annotations ``` ] .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- class: extra-details ## What about the missing revisions? - The missing revisions are stored in another annotation: `deployment.kubernetes.io/revision-history` - These are not shown in `kubectl rollout history` - We could easily reconstruct the full list with a script (if we wanted to!) .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- ## Rolling back to an older version - `kubectl rollout undo` can work with a revision number .exercise[ - Roll back to the "known good" deployment version: ```bash kubectl rollout undo deployment worker --to-revision=1 ``` - Check the web UI or the list of pods ] .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- class: extra-details ## Changing rollout parameters - We want to: - revert to `v0.1` - be conservative on availability (always have desired number of available workers) - go slow on rollout speed (update only one pod at a time) - give some time to our workers to "warm up" before starting more The corresponding changes can be expressed in the following YAML snippet: .small[ ```yaml spec: template: spec: containers: - name: worker image: dockercoins/worker:v0.1 strategy: rollingUpdate: maxUnavailable: 0 maxSurge: 1 minReadySeconds: 10 ``` ] .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- class: extra-details ## Applying changes through a YAML patch - We could use `kubectl edit deployment worker` - But we could also use `kubectl patch` with the exact YAML shown before .exercise[ .small[ - Apply all our changes and wait for them to take effect: ```bash kubectl patch deployment worker -p " spec: template: spec: containers: - name: worker image: dockercoins/worker:v0.1 strategy: rollingUpdate: maxUnavailable: 0 maxSurge: 1 minReadySeconds: 10 " kubectl rollout status deployment worker kubectl get deploy -o json worker | jq "{name:.metadata.name} + .spec.strategy.rollingUpdate" ``` ] ] ??? :EN:- Rolling updates :EN:- Rolling back a bad deployment :FR:- Mettre à jour un déploiement :FR:- Concept de *rolling update* et *rollback* :FR:- Paramétrer la vitesse de déploiement .debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/rollout.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-1.jpg)] --- name: toc-healthchecks class: title Healthchecks .nav[ [Previous section](#toc-rolling-updates) | [Back to table of contents](#toc-module-5) | [Next section](#toc-recording-deployment-actions) ] .debug[(automatically generated title slide)] --- # Healthchecks - Kubernetes provides two kinds of healthchecks: liveness and readiness - Healthchecks are *probes* that apply to *containers* (not to pods) - Each container can have two (optional) probes: - liveness = is this container dead or alive? - readiness = is this container ready to serve traffic? - Different probes are available (HTTP, TCP, program execution) - Let's see the difference and how to use them! .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Liveness probe - Indicates if the container is dead or alive - A dead container cannot come back to life - If the liveness probe fails, the container is killed (to make really sure that it's really dead; no zombies or undeads!) - What happens next depends on the pod's `restartPolicy`: - `Never`: the container is not restarted - `OnFailure` or `Always`: the container is restarted .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## When to use a liveness probe - To indicate failures that can't be recovered - deadlocks (causing all requests to time out) - internal corruption (causing all requests to error) - Anything where our incident response would be "just restart/reboot it" .warning[**Do not** use liveness probes for problems that can't be fixed by a restart] - Otherwise we just restart our pods for no reason, creating useless load .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Readiness probe - Indicates if the container is ready to serve traffic - If a container becomes "unready" it might be ready again soon - If the readiness probe fails: - the container is *not* killed - if the pod is a member of a service, it is temporarily removed - it is re-added as soon as the readiness probe passes again .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## When to use a readiness probe - To indicate failure due to an external cause - database is down or unreachable - mandatory auth or other backend service unavailable - To indicate temporary failure or unavailability - application can only service *N* parallel connections - runtime is busy doing garbage collection or initial data load - For processes that take a long time to start (more on that later) .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Dependencies - If a web server depends on a database to function, and the database is down: - the web server's liveness probe should succeed - the web server's readiness probe should fail - Same thing for any hard dependency (without which the container can't work) .warning[**Do not** fail liveness probes for problems that are external to the container] .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Timing and thresholds - Probes are executed at intervals of `periodSeconds` (default: 10) - The timeout for a probe is set with `timeoutSeconds` (default: 1) .warning[If a probe takes longer than that, it is considered as a FAIL] - A probe is considered successful after `successThreshold` successes (default: 1) - A probe is considered failing after `failureThreshold` failures (default: 3) - A probe can have an `initialDelaySeconds` parameter (default: 0) - Kubernetes will wait that amount of time before running the probe for the first time (this is important to avoid killing services that take a long time to start) .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- class: extra-details ## Startup probe - Kubernetes 1.16 introduces a third type of probe: `startupProbe` (it is in `alpha` in Kubernetes 1.16) - It can be used to indicate "container not ready *yet*" - process is still starting - loading external data, priming caches - Before Kubernetes 1.16, we had to use the `initialDelaySeconds` parameter (available for both liveness and readiness probes) - `initialDelaySeconds` is a rigid delay (always wait X before running probes) - `startupProbe` works better when a container start time can vary a lot .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Different types of probes - HTTP request - specify URL of the request (and optional headers) - any status code between 200 and 399 indicates success - TCP connection - the probe succeeds if the TCP port is open - arbitrary exec - a command is executed in the container - exit status of zero indicates success .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Benefits of using probes - Rolling updates proceed when containers are *actually ready* (as opposed to merely started) - Containers in a broken state get killed and restarted (instead of serving errors or timeouts) - Unavailable backends get removed from load balancer rotation (thus improving response times across the board) - If a probe is not defined, it's as if there was an "always successful" probe .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Example: HTTP probe Here is a pod template for the `rng` web service of the DockerCoins app: ```yaml apiVersion: v1 kind: Pod metadata: name: rng-with-liveness spec: containers: - name: rng image: dockercoins/rng:v0.1 livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 10 periodSeconds: 1 ``` If the backend serves an error, or takes longer than 1s, 3 times in a row, it gets killed. .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Example: exec probe Here is a pod template for a Redis server: ```yaml apiVersion: v1 kind: Pod metadata: name: redis-with-liveness spec: containers: - name: redis image: redis livenessProbe: exec: command: ["redis-cli", "ping"] ``` If the Redis process becomes unresponsive, it will be killed. .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Questions to ask before adding healthchecks - Do we want liveness, readiness, both? (sometimes, we can use the same check, but with different failure thresholds) - Do we have existing HTTP endpoints that we can use? - Do we need to add new endpoints, or perhaps use something else? - Are our healthchecks likely to use resources and/or slow down the app? - Do they depend on additional services? (this can be particularly tricky, see next slide) .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Healthchecks and dependencies - Liveness checks should not be influenced by the state of external services - All checks should reply quickly (by default, less than 1 second) - Otherwise, they are considered to fail - This might require to check the health of dependencies asynchronously (e.g. if a database or API might be healthy but still take more than 1 second to reply, we should check the status asynchronously and report a cached status) .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Healthchecks for workers (In that context, worker = process that doesn't accept connections) - Readiness isn't useful (because workers aren't backends for a service) - Liveness may help us restart a broken worker, but how can we check it? - Embedding an HTTP server is a (potentially expensive) option - Using a "lease" file can be relatively easy: - touch a file during each iteration of the main loop - check the timestamp of that file from an exec probe - Writing logs (and checking them from the probe) also works ??? :EN:- Using healthchecks to improve availability :FR:- Utiliser des *healthchecks* pour améliorer la disponibilité .debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks.md)] --- ## Adding healthchecks to an app - Let's add healthchecks to DockerCoins! - We will examine the questions of the previous slide - Then we will review each component individually to add healthchecks .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Liveness, readiness, or both? - To answer that question, we need to see the app run for a while - Do we get temporary, recoverable glitches? → then use readiness - Or do we get hard lock-ups requiring a restart? → then use liveness - In the case of DockerCoins, we don't know yet! - Let's pick liveness .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Do we have HTTP endpoints that we can use? - Each of the 3 web services (hasher, rng, webui) has a trivial route on `/` - These routes: - don't seem to perform anything complex or expensive - don't seem to call other services - Perfect! (See next slides for individual details) .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- - [hasher.rb](https://github.com/jpetazzo/container.training/blob/master/dockercoins/hasher/hasher.rb) ```ruby get '/' do "HASHER running on #{Socket.gethostname}\n" end ``` - [rng.py](https://github.com/jpetazzo/container.training/blob/master/dockercoins/rng/rng.py) ```python @app.route("/") def index(): return "RNG running on {}\n".format(hostname) ``` - [webui.js](https://github.com/jpetazzo/container.training/blob/master/dockercoins/webui/webui.js) ```javascript app.get('/', function (req, res) { res.redirect('/index.html'); }); ``` .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Running DockerCoins - We will run DockerCoins in a new, separate namespace - We will use a set of YAML manifests and pre-built images - We will add our new liveness probe to the YAML of the `rng` DaemonSet - Then, we will deploy the application .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Creating a new namespace - This will make sure that we don't collide / conflict with previous exercises .exercise[ - Create the yellow namespace: ```bash kubectl create namespace yellow ``` - Switch to that namespace: ```bash kns yellow ``` ] .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Retrieving DockerCoins manifests - All the manifests that we need are on a convenient repository: https://github.com/jpetazzo/kubercoins .exercise[ - Clone that repository: ```bash cd ~ git clone https://github.com/jpetazzo/kubercoins ``` - Change directory to the repository: ```bash cd kubercoins ``` ] .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## A simple HTTP liveness probe This is what our liveness probe should look like: ```yaml containers: - name: ... image: ... livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 30 periodSeconds: 5 ``` This will give 30 seconds to the service to start. (Way more than necessary!)
It will run the probe every 5 seconds.
It will use the default timeout (1 second).
It will use the default failure threshold (3 failed attempts = dead).
It will use the default success threshold (1 successful attempt = alive). .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Adding the liveness probe - Let's add the liveness probe, then deploy DockerCoins .exercise[ - Edit `rng-deployment.yaml` and add the liveness probe ```bash vim rng-deployment.yaml ``` - Load the YAML for all the resources of DockerCoins: ```bash kubectl apply -f . ``` ] .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Testing the liveness probe - The rng service needs 100ms to process a request (because it is single-threaded and sleeps 0.1s in each request) - The probe timeout is set to 1 second - If we send more than 10 requests per second per backend, it will break - Let's generate traffic and see what happens! .exercise[ - Get the ClusterIP address of the rng service: ```bash kubectl get svc rng ``` ] .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Monitoring the rng service - Each command below will show us what's happening on a different level .exercise[ - In one window, monitor cluster events: ```bash kubectl get events -w ``` - In another window, monitor the response time of rng: ```bash httping `
` ``` - In another window, monitor pods status: ```bash kubectl get pods -w ``` ] .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Generating traffic - Let's use `ab` to send concurrent requests to rng .exercise[ - In yet another window, generate traffic: ```bash ab -c 10 -n 1000 http://`
`/1 ``` - Experiment with higher values of `-c` and see what happens ] - The `-c` parameter indicates the number of concurrent requests - The final `/1` is important to generate actual traffic (otherwise we would use the ping endpoint, which doesn't sleep 0.1s per request) .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Discussion - Above a given threshold, the liveness probe starts failing (about 10 concurrent requests per backend should be plenty enough) - When the liveness probe fails 3 times in a row, the container is restarted - During the restart, there is *less* capacity available - ... Meaning that the other backends are likely to timeout as well - ... Eventually causing all backends to be restarted - ... And each fresh backend gets restarted, too - This goes on until the load goes down, or we add capacity *This wouldn't be a good healthcheck in a real application!* .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Better healthchecks - We need to make sure that the healthcheck doesn't trip when performance degrades due to external pressure - Using a readiness check would have fewer effects (but it would still be an imperfect solution) - A possible combination: - readiness check with a short timeout / low failure threshold - liveness check with a longer timeout / higher failure threshold .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- ## Healthchecks for redis - A liveness probe is enough (it's not useful to remove a backend from rotation when it's the only one) - We could use an exec probe running `redis-cli ping` .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- class: extra-details ## Exec probes and zombies - When using exec probes, we should make sure that we have a *zombie reaper* 🤔🧐🧟 Wait, what? - When a process terminates, its parent must call `wait()`/`waitpid()` (this is how the parent process retrieves the child's exit status) - In the meantime, the process is in *zombie* state (the process state will show as `Z` in `ps`, `top` ...) - When a process is killed, its children are *orphaned* and attached to PID 1 - PID 1 has the responsibility of *reaping* these processes when they terminate - OK, but how does that affect us? .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- class: extra-details ## PID 1 in containers - On ordinary systems, PID 1 (`/sbin/init`) has logic to reap processes - In containers, PID 1 is typically our application process (e.g. Apache, the JVM, NGINX, Redis ...) - These *do not* take care of reaping orphans - If we use exec probes, we need to add a process reaper - We can add [tini](https://github.com/krallin/tini) to our images - Or [share the PID namespace between containers of a pod](https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/) (and have gcr.io/pause take care of the reaping) - Discussion of this in [Video - 10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You](https://www.youtube.com/watch?v=QKI-JRs2RIE) ??? :EN:- Adding healthchecks to an app :FR:- Ajouter des *healthchecks* à une application .debug[[k8s/healthchecks-more.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/healthchecks-more.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-2.jpg)] --- name: toc-recording-deployment-actions class: title Recording deployment actions .nav[ [Previous section](#toc-healthchecks) | [Back to table of contents](#toc-module-5) | [Next section](#toc-namespaces) ] .debug[(automatically generated title slide)] --- # Recording deployment actions - Some commands that modify a Deployment accept an optional `--record` flag (Example: `kubectl set image deployment worker worker=alpine --record`) - That flag will store the command line in the Deployment (Technically, using the annotation `kubernetes.io/change-cause`) - It gets copied to the corresponding ReplicaSet (Allowing to keep track of which command created or promoted this ReplicaSet) - We can view this information with `kubectl rollout history` .debug[[k8s/record.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/record.md)] --- ## Using `--record` - Let's make a couple of changes to a Deployment and record them .exercise[ - Roll back `worker` to image version 0.1: ```bash kubectl set image deployment worker worker=dockercoins/worker:v0.1 --record ``` - Promote it to version 0.2 again: ```bash kubectl set image deployment worker worker=dockercoins/worker:v0.2 --record ``` - View the change history: ```bash kubectl rollout history deployment worker ``` ] .debug[[k8s/record.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/record.md)] --- ## Pitfall #1: forgetting `--record` - What happens if we don't specify `--record`? .exercise[ - Promote `worker` to image version 0.3: ```bash kubectl set image deployment worker worker=dockercoins/worker:v0.3 ``` - View the change history: ```bash kubectl rollout history deployment worker ``` ] -- It recorded version 0.2 instead of 0.3! Why? .debug[[k8s/record.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/record.md)] --- ## How `--record` really works - `kubectl` adds the annotation `kubernetes.io/change-cause` to the Deployment - The Deployment controller copies that annotation to the ReplicaSet - `kubectl rollout history` shows the ReplicaSets' annotations - If we don't specify `--record`, the annotation is not updated - The previous value of that annotation is copied to the new ReplicaSet - In that case, the ReplicaSet annotation does not reflect reality! .debug[[k8s/record.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/record.md)] --- ## Pitfall #2: recording `scale` commands - What happens if we use `kubectl scale --record`? .exercise[ - Check the current history: ```bash kubectl rollout history deployment worker ``` - Scale the deployment: ```bash kubectl scale deployment worker --replicas=3 --record ``` - Check the change history again: ```bash kubectl rollout history deployment worker ``` ] -- The last entry in the history was overwritten by the `scale` command! Why? .debug[[k8s/record.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/record.md)] --- ## Actions that don't create a new ReplicaSet - The `scale` command updates the Deployment definition - But it doesn't create a new ReplicaSet - Using the `--record` flag sets the annotation like before - The annotation gets copied to the existing ReplicaSet - This overwrites the previous annotation that was there - In that case, we lose the previous change cause! .debug[[k8s/record.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/record.md)] --- ## Updating the annotation directly - Let's see what happens if we set the annotation manually .exercise[ - Annotate the Deployment: ```bash kubectl annotate deployment worker kubernetes.io/change-cause="Just for fun" ``` - Check that our annotation shows up in the change history: ```bash kubectl rollout history deployment worker ``` ] -- Our annotation shows up (and overwrote whatever was there before). .debug[[k8s/record.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/record.md)] --- ## Using change cause - It sounds like a good idea to use `--record`, but: *"Incorrect documentation is often worse than no documentation."*
(Bertrand Meyer) - If we use `--record` once, we need to either: - use it every single time after that - or clear the Deployment annotation after using `--record`
(subsequent changes will show up with a `
` change cause) - A safer way is to set it through our tooling .debug[[k8s/record.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/record.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/two-containers-on-a-truck.jpg)] --- name: toc-namespaces class: title Namespaces .nav[ [Previous section](#toc-recording-deployment-actions) | [Back to table of contents](#toc-module-6) | [Next section](#toc-controlling-a-kubernetes-cluster-remotely) ] .debug[(automatically generated title slide)] --- # Namespaces - We would like to deploy another copy of DockerCoins on our cluster - We could rename all our deployments and services: hasher → hasher2, redis → redis2, rng → rng2, etc. - That would require updating the code - There has to be a better way! -- - As hinted by the title of this section, we will use *namespaces* .debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/namespaces.md)] --- ## Identifying a resource - We cannot have two resources with the same name (or can we...?) -- - We cannot have two resources *of the same kind* with the same name (but it's OK to have an `rng` service, an `rng` deployment, and an `rng` daemon set) -- - We cannot have two resources of the same kind with the same name *in the same namespace* (but it's OK to have e.g. two `rng` services in different namespaces) -- - Except for resources that exist at the *cluster scope* (these do not belong to a namespace) .debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/namespaces.md)] --- ## Uniquely identifying a resource - For *namespaced* resources: the tuple *(kind, name, namespace)* needs to be unique - For resources at the *cluster scope*: the tuple *(kind, name)* needs to be unique .exercise[ - List resource types again, and check the NAMESPACED column: ```bash kubectl api-resources ``` ] .debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/namespaces.md)] --- ## Pre-existing namespaces - If we deploy a cluster with `kubeadm`, we have three or four namespaces: - `default` (for our applications) - `kube-system` (for the control plane) - `kube-public` (contains one ConfigMap for cluster discovery) - `kube-node-lease` (in Kubernetes 1.14 and later; contains Lease objects) - If we deploy differently, we may have different namespaces .debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/namespaces.md)] --- ## Creating namespaces - Let's see two identical methods to create a namespace .exercise[ - We can use `kubectl create namespace`: ```bash kubectl create namespace blue ``` - Or we can construct a very minimal YAML snippet: ```bash kubectl apply -f- <
- Download the `kubectl` binary from one of these links: [Linux](https://storage.googleapis.com/kubernetes-release/release/v1.18.8/bin/linux/amd64/kubectl) | [macOS](https://storage.googleapis.com/kubernetes-release/release/v1.18.8/bin/darwin/amd64/kubectl) | [Windows](https://storage.googleapis.com/kubernetes-release/release/v1.18.8/bin/windows/amd64/kubectl.exe) - On Linux and macOS, make the binary executable with `chmod +x kubectl` (And remember to run it with `./kubectl` or move it to your `$PATH`) ] Note: if you are following along with a different platform (e.g. Linux on an architecture different from amd64, or with a phone or tablet), installing `kubectl` might be more complicated (or even impossible) so feel free to skip this section. .debug[[k8s/localkubeconfig.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/localkubeconfig.md)] --- ## Testing `kubectl` - Check that `kubectl` works correctly (before even trying to connect to a remote cluster!) .exercise[ - Ask `kubectl` to show its version number: ```bash kubectl version --client ``` ] The output should look like this: ``` Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"} ``` .debug[[k8s/localkubeconfig.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/localkubeconfig.md)] --- ## Preserving the existing `~/.kube/config` - If you already have a `~/.kube/config` file, rename it (we are going to overwrite it in the following slides!) - If you never used `kubectl` on your machine before: nothing to do! .exercise[ - Make a copy of `~/.kube/config`; if you are using macOS or Linux, you can do: ```bash cp ~/.kube/config ~/.kube/config.before.training ``` - If you are using Windows, you will need to adapt this command ] .debug[[k8s/localkubeconfig.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/localkubeconfig.md)] --- ## Copying the configuration file from `node1` - The `~/.kube/config` file that is on `node1` contains all the credentials we need - Let's copy it over! .exercise[ - Copy the file from `node1`; if you are using macOS or Linux, you can do: ``` scp `USER`@`X.X.X.X`:.kube/config ~/.kube/config # Make sure to replace X.X.X.X with the IP address of node1, # and USER with the user name used to log into node1! ``` - If you are using Windows, adapt these instructions to your SSH client ] .debug[[k8s/localkubeconfig.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/localkubeconfig.md)] --- ## Updating the server address - There is a good chance that we need to update the server address - To know if it is necessary, run `kubectl config view` - Look for the `server:` address: - if it matches the public IP address of `node1`, you're good! - if it is anything else (especially a private IP address), update it! - To update the server address, run: ```bash kubectl config set-cluster kubernetes --server=https://`X.X.X.X`:6443 # Make sure to replace X.X.X.X with the IP address of node1! ``` .debug[[k8s/localkubeconfig.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/localkubeconfig.md)] --- class: extra-details ## What if we get a certificate error? - Generally, the Kubernetes API uses a certificate that is valid for: - `kubernetes` - `kubernetes.default` - `kubernetes.default.svc` - `kubernetes.default.svc.cluster.local` - the ClusterIP address of the `kubernetes` service - the hostname of the node hosting the control plane (e.g. `node1`) - the IP address of the node hosting the control plane - On most clouds, the IP address of the node is an internal IP address - ... And we are going to connect over the external IP address - ... And that external IP address was not used when creating the certificate! .debug[[k8s/localkubeconfig.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/localkubeconfig.md)] --- class: extra-details ## Working around the certificate error - We need to tell `kubectl` to skip TLS verification (only do this with testing clusters, never in production!) - The following command will do the trick: ```bash kubectl config set-cluster kubernetes --insecure-skip-tls-verify ``` .debug[[k8s/localkubeconfig.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/localkubeconfig.md)] --- ## Checking that we can connect to the cluster - We can now run a couple of trivial commands to check that all is well .exercise[ - Check the versions of the local client and remote server: ```bash kubectl version ``` - View the nodes of the cluster: ```bash kubectl get nodes ``` ] We can now utilize the cluster exactly as if we're logged into a node, except that it's remote. ??? :EN:- Working with remote Kubernetes clusters :FR:- Travailler avec des *clusters* distants .debug[[k8s/localkubeconfig.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/localkubeconfig.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/Container-Ship-Freighter-Navigation-Elbe-Romance-1782991.jpg)] --- name: toc-accessing-internal-services class: title Accessing internal services .nav[ [Previous section](#toc-controlling-a-kubernetes-cluster-remotely) | [Back to table of contents](#toc-module-6) | [Next section](#toc-accessing-the-api-with-kubectl-proxy) ] .debug[(automatically generated title slide)] --- # Accessing internal services - When we are logged in on a cluster node, we can access internal services (by virtue of the Kubernetes network model: all nodes can reach all pods and services) - When we are accessing a remote cluster, things are different (generally, our local machine won't have access to the cluster's internal subnet) - How can we temporarily access a service without exposing it to everyone? -- - `kubectl proxy`: gives us access to the API, which includes a proxy for HTTP resources - `kubectl port-forward`: allows forwarding of TCP ports to arbitrary pods, services, ... .debug[[k8s/accessinternal.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/accessinternal.md)] --- ## Suspension of disbelief The exercises in this section assume that we have set up `kubectl` on our local machine in order to access a remote cluster. We will therefore show how to access services and pods of the remote cluster, from our local machine. You can also run these exercises directly on the cluster (if you haven't installed and set up `kubectl` locally). Running commands locally will be less useful (since you could access services and pods directly), but keep in mind that these commands will work anywhere as long as you have installed and set up `kubectl` to communicate with your cluster. .debug[[k8s/accessinternal.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/accessinternal.md)] --- ## `kubectl proxy` in theory - Running `kubectl proxy` gives us access to the entire Kubernetes API - The API includes routes to proxy HTTP traffic - These routes look like the following: `/api/v1/namespaces/
/services/
/proxy` - We just add the URI to the end of the request, for instance: `/api/v1/namespaces/
/services/
/proxy/index.html` - We can access `services` and `pods` this way .debug[[k8s/accessinternal.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/accessinternal.md)] --- ## `kubectl proxy` in practice - Let's access the `webui` service through `kubectl proxy` .exercise[ - Run an API proxy in the background: ```bash kubectl proxy & ``` - Access the `webui` service: ```bash curl localhost:8001/api/v1/namespaces/default/services/webui/proxy/index.html ``` - Terminate the proxy: ```bash kill %1 ``` ] .debug[[k8s/accessinternal.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/accessinternal.md)] --- ## `kubectl port-forward` in theory - What if we want to access a TCP service? - We can use `kubectl port-forward` instead - It will create a TCP relay to forward connections to a specific port (of a pod, service, deployment...) - The syntax is: `kubectl port-forward service/name_of_service local_port:remote_port` - If only one port number is specified, it is used for both local and remote ports .debug[[k8s/accessinternal.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/accessinternal.md)] --- ## `kubectl port-forward` in practice - Let's access our remote Redis server .exercise[ - Forward connections from local port 10000 to remote port 6379: ```bash kubectl port-forward svc/redis 10000:6379 & ``` - Connect to the Redis server: ```bash telnet localhost 10000 ``` - Issue a few commands, e.g. `INFO server` then `QUIT` - Terminate the port forwarder: ```bash kill %1 ``` ] ??? :EN:- Securely accessing internal services :FR:- Accès sécurisé aux services internes .debug[[k8s/accessinternal.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/accessinternal.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/ShippingContainerSFBay.jpg)] --- name: toc-accessing-the-api-with-kubectl-proxy class: title Accessing the API with `kubectl proxy` .nav[ [Previous section](#toc-accessing-internal-services) | [Back to table of contents](#toc-module-6) | [Next section](#toc-exposing-http-services-with-ingress-resources) ] .debug[(automatically generated title slide)] --- # Accessing the API with `kubectl proxy` - The API requires us to authenticate.red[¹] - There are many authentication methods available, including: - TLS client certificates
(that's what we've used so far) - HTTP basic password authentication
(from a static file; not recommended) - various token mechanisms
(detailed in the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#authentication-strategies)) .red[¹]OK, we lied. If you don't authenticate, you are considered to be user `system:anonymous`, which doesn't have any access rights by default. .debug[[k8s/kubectlproxy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/kubectlproxy.md)] --- ## Accessing the API directly - Let's see what happens if we try to access the API directly with `curl` .exercise[ - Retrieve the ClusterIP allocated to the `kubernetes` service: ```bash kubectl get svc kubernetes ``` - Replace the IP below and try to connect with `curl`: ```bash curl -k https://`<IP_ADDRESS>`/ ``` ] The API will tell us that user `system:anonymous` cannot access this path. .debug[[k8s/kubectlproxy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/kubectlproxy.md)] --- ## Authenticating to the API If we wanted to talk to the API, we would need to: - extract our TLS key and certificate information from `~/.kube/config` (the information is in PEM format, encoded in base64) - use that information to present our certificate when connecting (for instance, with `openssl s_client -key ... -cert ... -connect ...`) - figure out exactly which credentials to use (once we start juggling multiple clusters) - change that whole process if we're using another authentication method 🤔 There has to be a better way! .debug[[k8s/kubectlproxy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/kubectlproxy.md)] --- ## Using `kubectl proxy` for authentication - `kubectl proxy` runs a proxy in the foreground - This proxy lets us access the Kubernetes API without authentication (`kubectl proxy` adds our credentials on the fly to the requests) - This proxy lets us access the Kubernetes API over plain HTTP - This is a great tool to learn and experiment with the Kubernetes API - ... And for serious uses as well (suitable for one-shot scripts) - For unattended use, it's better to create a [service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) .debug[[k8s/kubectlproxy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/kubectlproxy.md)] --- ## Trying `kubectl proxy` - Let's start `kubectl proxy` and then do a simple request with `curl`! .exercise[ - Start `kubectl proxy` in the background: ```bash kubectl proxy & ``` - Access the API's default route: ```bash curl localhost:8001 ``` - Terminate the proxy: ```bash kill %1 ``` ] The output is a list of available API routes. .debug[[k8s/kubectlproxy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/kubectlproxy.md)] --- ## OpenAPI (fka Swagger) - The Kubernetes API serves an OpenAPI Specification (OpenAPI was formerly known as Swagger) - OpenAPI has many advantages (generate client library code, generate test code ...) - For us, this means we can explore the API with [Swagger UI](https://swagger.io/tools/swagger-ui/) (for instance with the [Swagger UI add-on for Firefox](https://addons.mozilla.org/en-US/firefox/addon/swagger-ui-ff/)) .debug[[k8s/kubectlproxy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/kubectlproxy.md)] --- ## `kubectl proxy` is intended for local use - By default, the proxy listens on port 8001 (But this can be changed, or we can tell `kubectl proxy` to pick a port) - By default, the proxy binds to `<IP_ADDRESS>` (Making it unreachable from other machines, for security reasons) - By default, the proxy only accepts connections from: `^localhost$,^127\.0\.0\.1$,^\[::1\]$` - This is great when running `kubectl proxy` locally - Not-so-great when you want to connect to the proxy from a remote machine .debug[[k8s/kubectlproxy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/kubectlproxy.md)] --- class: extra-details ## Running `kubectl proxy` on a remote machine - If we wanted to connect to the proxy from another machine, we would need to: - bind to `INADDR_ANY` instead of `<IP_ADDRESS>` - accept connections from any address - This is achieved with: ``` kubectl proxy --port=8888 --address=<IP_ADDRESS> --accept-hosts=.* ``` .warning[Do not do this on a real cluster: it opens full unauthenticated access!] .debug[[k8s/kubectlproxy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/kubectlproxy.md)] --- class: extra-details ## Security considerations - Running `kubectl proxy` openly is a huge security risk - It is slightly better to run the proxy where you need it (and copy credentials, e.g. `~/.kube/config`, to that place) - It is even better to use a limited account with reduced permissions .debug[[k8s/kubectlproxy.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/kubectlproxy.md)] --- ## Good to know ... - `kubectl proxy` also gives access to all internal services - Specifically, services are exposed as such: ``` /api/v1/namespaces/
/services/