|
Secret SocietiesFor centuries, humans have been trying to keep information from other humans. Paradoxically, many have come to the conclusion that the best way to keep a secret is to tell it to a bunch of other people and then swear them all to secrecy.When this effort is unsuccessful, we call the result a "secret society." When the effort is successful, we don't call the result anything, because we plebians never hear about the effort to begin with. In short, the society part is easy. The secret part is hard. Nevertheless, secret societies have become deeply embedded in the zeitgeist. In some cases, their secrets are so poorly kept that a quick run through Google will yield nearly anything you could possibly want to know. In other cases, the society manages to keep some of its secrets secret, but the group itself becomes known to a greater or lesser extent. There are many different ways to structure a secret society, but there are a few specific models which recur fairly often. In order to qualify as a secret society, a group generally has to be based around initiation rituals, degrees of authority and dramatic oaths of silence. Most groups can arguably be included in more than one of the categories which follow, and probably all of them can be included under the final heading:
PoliticalMany people look at the state of the world and come to the understandable conclusion that they must be missing something. After all, no rational person would make the decisions some world leaders make... unless, of course, they have a hidden agenda that we don't know about.So whose agenda is it, anyway? Some favorite contenders include:
The Carbonari in Italy, a derivative of the Masons, helped forment revolution in the 19th century. Edelweiss, a European group, advanced a pro-Nordic racial agenda and produced such illustrious members as Hermann Goering.  A Russian group, Land and Liberty, used terrorism and assassination to lay the groundwork for revolution. Formally known as the Fists of Righteous Harmony, the Boxers began as a small Chinese nationalist society toward the end of the 19th century before swelling to incredible size, embarking on a reign of terror against foreigners and subsequently getting slaughtered by the U.S. Navy. There's plenty of evidence to show that secret societies have formed to accomplish specific political goals, but those groups that don't get killed in the process tend to fade away after the immediate political crisis is resolved. The idea that a secret society might be running the world can be appealing. It offers the possibility that every stupid, pointless thing done by world leaders might actually be smart and pointful, part of some sort of plan. However, it doesn't take much live experience to realize that individual people are generally stupid and pointless, and Occam's Razor tells us the simplest explanation is most often correct. Nevertheless, people will talk. If all the above secret societies aren't enough to satisfy your paranoid tendencies, you can always look into "The Octopus" -- an uber-secret society which purportedly links all of the other secret societies in one vast conspiracy to control the world. The Octopus was first tenatively identified by a freelance investigative reporter named Danny Casolaro, who believed it linked such conspiracies as Iran-Contra, BCCI, INSLAW to such government agencies as the CIA, FBI and the NSA. Casolaro turned up dead due to an extremely suspicious "suicide" in 1991. His story (or rather, a wildly imaginative telling of his story) has made him a martyr to the conspiracy crowd. Some political secret societies eventually metamorphose into criminal organizations, such as the Mafia, the Japanese Yakuza and the Chinese Triads. You can also make a case that race-based hate groups like the Ku Klux Klan were also working from a political mindset, at least in their formative years.
ReligiousMany secret societies have a religious or occult component, but some are very explicitly devoted to advancing one form of religion or another. Religious secret societies are very real, and they have often had a tremendous impact on history, which explains why you've probably heard of a few:
In order to counter heretical groups, the Catholic Church has created its own secret orders from time to time, under auspices of the pope. The Templars originated as such an officially sanctioned group, but they fell out of favor when their wealth and power challenged the political status quo. Envy and fear led to charges of witchcraft and other wrongdoing, and the Church eventually exterminated the order... or did they? These days, you can't swing a dead cat without hitting some group that claims to be a continuation or an offshoot of the Templars. The history of Islam is also riddled with secretive groups which have had an incalcuable effect on history, from the Assassins of the 11th Century to al Qaeda in the present day. One reason these groups are so effective is their embrace of violence, paired with a complete disregard for the personal safety of their members. The Assassins and the Templars were rumored to have shared trade secrets and engaged in other covert alliances. The list of religious secret societies goes on and on. Buddhists got into the act, with organizations like the White Lotus Groups (of which the Boxers were technically a part), and variations sprouted among Hindus and Jews as well. Africa has been rife with secret societies of all sorts, many of which are based in the rituals of shamanistic tribal religions. An outgrowth of African spiritual beliefs, Voudoun was intimately linked with Hatian secret societies, many of which are political in nature. An American variation on the theme, Santeria, is practically a secret society in itself. You will be the hit of the cocktail party when you hold forth on the origins of the word "Mumbo Jumbo," which is a botched transliteration of the name of a Mandingo secret society. The account originated with the 18th century explorer of Africa Francis Moore, who wrote that the all-male society, bound by terrible oaths, existed primarily to adjudicate disputes between men and women... in favor of men. So how did Moore find out about this "secret" group?
About the year 1727, the king of Jagra having a very inquisitive woman to his wife, was so weak as to disclose to her this secret; and she being a gossip, revealed it to some other women of her acquaintance. This at last coming to the ears of some who were no friends to the king, they, dreading lest if the affair took vent, it should put a period to the subjection of their wives, took the coat, put a man into it, and going to the king's town, sent for him out, and taxed him with it: when he not denying it, they sent for his wife, and killed them both on the spot. Thus the poor king died for his complaisance to his wife, and she for her curiosity. The first rule of Mumbo Jumbo is DON'T TALK ABOUT MUMBO JUMBO!
FraternalSome people are attracted to the bizarre rituals and self-important playtime of a secret organization without necessarily wanting the responsibility of ruling the world or protecting the arcane keys of occult power.
All of these groups are basically clubs for silly boys, to a greater or lesser extent. They are based around "lodges," a word nicked from Masonic practice which means, in this context, a place you go to get drunk in the company of men. Most of these groups feature some sort of thinly veiled homoerotic bondage play as a form of initiation. Generally, these groups are all male, all the time, though some have women's auxiliary groups and others have been forced by American law to open their doors to all comers. Most fraternal organizations require you to pay dues, which entitles you to use of the bar. Fraternal clubs often perform charity work in a vain effort to justify their existence. They can also provide business networking opportunities for those who are insufficiently ambitious to hook up with the Masons, or even better, the Trilateral Commission.
FictionalMost secret societies have a fictional history, concocted to make them look important. Some groups are more fictional than others, however.
Many authors have discovered that once you create a secret society with a sufficiently intriguing premise, people will automatically assume it's based on something real. If the author tries to deny it later, well, that just means someone got to them. The Illuminati are technically not fictional, but so many fictional things have been written about them that they might as well be. In addition to many earnest flights of fancy composed by the slightly deranged, the Illuminati got the most ink in the famous Illuminatus! trilogy written by Robert Anton Wilson. Illuminatus! was so successful that many of its yarns are now taken as gospel truth by such illustrious conspiratorial minds as David Icke. H.P. Lovecraft created a fictional cult known as Cthulhu, which he shared with several other horror writers of the day. The cult was based around a series of shapeless, nameless, writhing monstrosities and an entirely mythical grimoire known as the Necronomicon.  There was an 18th century Italian secret society called The Invisibles, but the name was adopted by comic book auteur Grant Morrison for a 1994-2000 comic book series which set out with the goal of making The Invisibles real. If someone hands you a blank badge, you'll know Morrison's quest succeeded. Other notable fictional secret societies include the Talamasca (a vampire-hunting group of scholars in Anne Rice's books), the Clandestine Watchers Council (a vampire-hunting group of scholars in Buffy the Vampire Slayer), the E-Branch (a vampire-hunting group of secret agents in Brian Lumley's Necroscope books), and the Millennium Group (an apocalyptic organization inexplicably unconcerned with vampire hunting, and the brainchild of X-Files creator Chris Carter).
InsaneBetween the paddling and the dreams of world domination, any given secret society is going to attract a certain element of the deranged. Some groups are way out there... even relative to other secret societies.The Ku Klux Klan originated as a politcal white supremacy group in 1886. At the time, it was not that far out of the political mainsteam. Racism was rampant after the Civil War, and many people resented the North's exploitation of the South. Nevertheless, the Klan was rooted in bizarre behavior. The original group dressed in white hoods and pretended to be ghosts in order to frighten freed slaves. Subsequent iterations were not much more sophisticated and today the Klan is populated with fringe personalities with too much time on their hands. The Klan also had its sworn enemies, such as the memorably named Order of Anti-Poke-Noses, which formed to oppose "any organization that attends to everyone's business but their own." The bad guys outlived these intrepid crusaders, however. Strange behavior hardly begins and ends with the Klan. A quick browse through the pages of The International Encyclopedia of Secret Societies and Fraternal Orders by Alan Axelrod yields up a smorgasbord of strangeness. Fils d'Adam is an au courant French secret society devoted to the joys of similating necrophilia and performing actual bestiality on some extremely loosely contrived premise having to do with Original Sin. The Abecadarians, also French, formed to battle the insidious evil of the printing press during the 15th century. The theory was that ignorance is divinely mandated and that everyone should strive to the pinnacle of ignorance of everything, including the letters of the alphabet -- thus their name comes from A-B-C-D... Clever, you might think, until you realize that anyone who knows the group's name now knows the first four letters of the alphabet and therefore can no longer be saved. (The doctrine was later supposedly adopted by the Illuminati.) (The real ones.) The Society of Goats in 18th century Germany wore goat masks in order to frighten the local peasantry and accomplish various acts of crime. (And people think Batman is implausible.) Their initiation involved riding a wooden goat rigged up approximately in the manner of a mechanical bull. The goat itself may also have been a Baphomet-style symbol of Satan. They were eventually exterminated by the local authorities, and bore no relation to the even stranger Order of Pink Goats, which arose in the 20th century. The Hermanos Penitentes were originally a European order which enjoyed self-flagellation. They celebrated Good Friday each year by crucifying one of the sect's members, though the lucky victim was removed from the cross before dying. Despite being the recipient of a papal censure, the group not only survived, but continues its tradition of crucifixion, much to the bemusement of the occasional newspaper feature writer who stumbles across the event. The Concatenated Order of Hoo-Hoo... well, OK, the Hoo-Hoos were largely just kidding around. The members were lumberjacks and people whose jobs related in some vague manner related to lumber. Many of their titles and rituals were based on Lewis Carroll's Jabberwocky. Amazingly, the organization continues to day under a different name, despite what Darwin's theories would suggest.
|


(in fact, from kubelet's point of view, there is no such thing as an "invalid" address) - We must delete our pods and recreate them .exercise[ - Delete all the pods, and let the ReplicaSet recreate them: ```bash kubectl delete pods --all ``` - Wait for the pods to be up again: ```bash kubectl get pods -o wide -w ``` ] .debug[[k8s/multinode.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/multinode.md)] --- ## Adding kube-proxy - Let's start kube-proxy to provide internal load balancing - Then see if we can create a Service and use it to contact our pods .exercise[ - Start kube-proxy: ```bash sudo kube-proxy --kubeconfig ~/.kube/config ``` - Expose our Deployment: ```bash kubectl expose deployment web --port=80 ``` ] .debug[[k8s/multinode.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/multinode.md)] --- ## Test internal load balancing .exercise[ - Retrieve the ClusterIP address: ```bash kubectl get svc web ``` - Send a few requests to the ClusterIP address (with `curl`) ] -- Sometimes it works, sometimes it doesn't. Why? .debug[[k8s/multinode.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/multinode.md)] --- ## Routing traffic - Our pods have new, distinct IP addresses - But they are on host-local, isolated networks - If we try to ping a pod on a different node, it won't work - kube-proxy merely rewrites the destination IP address - But we need that IP address to be reachable in the first place - How do we fix this? (hint: check the title of this slide!) .debug[[k8s/multinode.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/multinode.md)] --- ## Important warning - The technique that we are about to use doesn't work everywhere - It only works if: - all the nodes are directly connected to each other (at layer 2) - the underlying network allows the IP addresses of our pods - If we are on physical machines connected by a switch: OK - If we are on virtual machines in a public cloud: NOT OK - on AWS, we need to disable "source and destination checks" on our instances - on OpenStack, we need to disable "port security" on our network ports .debug[[k8s/multinode.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/multinode.md)] --- ## Routing basics - We need to tell *each* node: "The subnet 10.C.N.0/24 is located on node N" (for all values of N) - This is how we add a route on Linux: ```bash ip route add 10.C.N.0/24 via W.X.Y.Z ``` (where `W.X.Y.Z` is the internal IP address of node N) - We can see the internal IP addresses of our nodes with: ```bash kubectl get nodes -o wide ``` .debug[[k8s/multinode.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/multinode.md)] --- ## Firewalling - By default, Docker prevents containers from using arbitrary IP addresses (by setting up iptables rules) - We need to allow our containers to use our pod CIDR - For simplicity, we will insert a blanket iptables rule allowing all traffic: `iptables -I FORWARD -j ACCEPT` - This has to be done on every node .debug[[k8s/multinode.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/multinode.md)] --- ## Setting up routing .exercise[ - Create all the routes on all the nodes - Insert the iptables rule allowing traffic - Check that you can ping all the pods from one of the nodes - Check that you can `curl` the ClusterIP of the Service successfully ] .debug[[k8s/multinode.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/multinode.md)] --- ## What's next? - We did a lot of manual operations: - allocating subnets to nodes - adding command-line flags to kubelet - updating the routing tables on our nodes - We want to automate all these steps - We want something that works on all networks ??? :EN:- Connecting nodes ands pods :FR:- Interconnecter les nœuds et les pods .debug[[k8s/multinode.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/multinode.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/plastic-containers.JPG)] --- name: toc-the-container-network-interface class: title The Container Network Interface .nav[ [Previous section](#toc-adding-nodes-to-the-cluster) | [Back to table of contents](#toc-module-13) | [Next section](#toc-api-server-availability) ] .debug[(automatically generated title slide)] --- # The Container Network Interface - Allows us to decouple network configuration from Kubernetes - Implemented by *plugins* - Plugins are executables that will be invoked by kubelet - Plugins are responsible for: - allocating IP addresses for containers - configuring the network for containers - Plugins can be combined and chained when it makes sense .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Combining plugins - Interface could be created by e.g. `vlan` or `bridge` plugin - IP address could be allocated by e.g. `dhcp` or `host-local` plugin - Interface parameters (MTU, sysctls) could be tweaked by the `tuning` plugin The reference plugins are available [here]. Look in each plugin's directory for its documentation. [here]: https://github.com/containernetworking/plugins/tree/master/plugins .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## How does kubelet know which plugins to use? - The plugin (or list of plugins) is set in the CNI configuration - The CNI configuration is a *single file* in `/etc/cni/net.d` - If there are multiple files in that directory, the first one is used (in lexicographic order) - That path can be changed with the `--cni-conf-dir` flag of kubelet .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## CNI configuration in practice - When we set up the "pod network" (like Calico, Weave...) it ships a CNI configuration (and sometimes, custom CNI plugins) - Very often, that configuration (and plugins) is installed automatically (by a DaemonSet featuring an initContainer with hostPath volumes) - Examples: - Calico [CNI config](https://github.com/projectcalico/calico/blob/1372b56e3bfebe2b9c9cbf8105d6a14764f44159/v2.6/getting-started/kubernetes/installation/hosted/calico.yaml#L25) and [volume](https://github.com/projectcalico/calico/blob/1372b56e3bfebe2b9c9cbf8105d6a14764f44159/v2.6/getting-started/kubernetes/installation/hosted/calico.yaml#L219) - kube-router [CNI config](https://github.com/cloudnativelabs/kube-router/blob/c2f893f64fd60cf6d2b6d3fee7191266c0fc0fe5/daemonset/generic-kuberouter.yaml#L10) and [volume](https://github.com/cloudnativelabs/kube-router/blob/c2f893f64fd60cf6d2b6d3fee7191266c0fc0fe5/daemonset/generic-kuberouter.yaml#L73) .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## Conf vs conflist - There are two slightly different configuration formats - Basic configuration format: - holds configuration for a single plugin - typically has a `.conf` name suffix - has a `type` string field in the top-most structure - [examples](https://github.com/containernetworking/cni/blob/master/SPEC.md#example-configurations) - Configuration list format: - can hold configuration for multiple (chained) plugins - typically has a `.conflist` name suffix - has a `plugins` list field in the top-most structure - [examples](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration-lists) .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## How plugins are invoked - Parameters are given through environment variables, including: - CNI_COMMAND: desired operation (ADD, DEL, CHECK, or VERSION) - CNI_CONTAINERID: container ID - CNI_NETNS: path to network namespace file - CNI_IFNAME: what the network interface should be named - The network configuration must be provided to the plugin on stdin (this avoids race conditions that could happen by passing a file path) .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## In practice: kube-router - We are going to set up a new cluster - For this new cluster, we will use kube-router - kube-router will provide the "pod network" (connectivity with pods) - kube-router will also provide internal service connectivity (replacing kube-proxy) .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## How kube-router works - Very simple architecture - Does not introduce new CNI plugins (uses the `bridge` plugin, with `host-local` for IPAM) - Pod traffic is routed between nodes (no tunnel, no new protocol) - Internal service connectivity is implemented with IPVS - Can provide pod network and/or internal service connectivity - kube-router daemon runs on every node .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## What kube-router does - Connect to the API server - Obtain the local node's `podCIDR` - Inject it into the CNI configuration file (we'll use `/etc/cni/net.d/10-kuberouter.conflist`) - Obtain the addresses of all nodes - Establish a *full mesh* BGP peering with the other nodes - Exchange routes over BGP .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## What's BGP? - BGP (Border Gateway Protocol) is the protocol used between internet routers - It [scales](https://www.cidr-report.org/as2.0/) pretty [well](https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp-active%2etxt&descr=Active%20BGP%20entries%20%28FIB%29&ylabel=Active%20BGP%20entries%20%28FIB%29&with=step) (it is used to announce the 700k CIDR prefixes of the internet) - It is spoken by many hardware routers from many vendors - It also has many software implementations (Quagga, Bird, FRR...) - Experienced network folks generally know it (and appreciate it) - It also used by Calico (another popular network system for Kubernetes) - Using BGP allows us to interconnect our "pod network" with other systems .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## The plan - We'll work in a new cluster (named `kuberouter`) - We will run a simple control plane (like before) - ... But this time, the controller manager will allocate `podCIDR` subnets (so that we don't have to manually assign subnets to individual nodes) - We will create a DaemonSet for kube-router - We will join nodes to the cluster - The DaemonSet will automatically start a kube-router pod on each node .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Logging into the new cluster .exercise[ - Log into node `kuberouter1` - Clone the workshop repository: ```bash git clone https://github.com/jpetazzo/container.training ``` - Move to this directory: ```bash cd container.training/compose/kube-router-k8s-control-plane ``` ] .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## Checking the CNI configuration - By default, kubelet gets the CNI configuration from `/etc/cni/net.d` .exercise[ - Check the content of `/etc/cni/net.d` ] (On most machines, at this point, `/etc/cni/net.d` doesn't even exist).) .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Our control plane - We will use a Compose file to start the control plane - It is similar to the one we used with the `kubenet` cluster - The API server is started with `--allow-privileged` (because we will start kube-router in privileged pods) - The controller manager is started with extra flags too: `--allocate-node-cidrs` and `--cluster-cidr` - We need to edit the Compose file to set the Cluster CIDR .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Starting the control plane - Our cluster CIDR will be `10.C.0.0/16` (where `C` is our cluster number) .exercise[ - Edit the Compose file to set the Cluster CIDR: ```bash vim docker-compose.yaml ``` - Start the control plane: ```bash docker-compose up ``` ] .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## The kube-router DaemonSet - In the same directory, there is a `kuberouter.yaml` file - It contains the definition for a DaemonSet and a ConfigMap - Before we load it, we also need to edit it - We need to indicate the address of the API server (because kube-router needs to connect to it to retrieve node information) .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Creating the DaemonSet - The address of the API server will be `http://A.B.C.D:8080` (where `A.B.C.D` is the public address of `kuberouter1`, running the control plane) .exercise[ - Edit the YAML file to set the API server address: ```bash vim kuberouter.yaml ``` - Create the DaemonSet: ```bash kubectl create -f kuberouter.yaml ``` ] Note: the DaemonSet won't create any pods (yet) since there are no nodes (yet). .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Generating the kubeconfig for kubelet - This is similar to what we did for the `kubenet` cluster .exercise[ - Generate the kubeconfig file (replacing `X.X.X.X` with the address of `kuberouter1`): ```bash kubectl config set-cluster cni --server http://`X.X.X.X`:8080 kubectl config set-context cni --cluster cni kubectl config use-context cni cp ~/.kube/config ~/kubeconfig ``` ] .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Distributing kubeconfig - We need to copy that kubeconfig file to the other nodes .exercise[ - Copy `kubeconfig` to the other nodes: ```bash for N in 2 3; do scp ~/kubeconfig kuberouter$N: done ``` ] .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Starting kubelet - We don't need the `--pod-cidr` option anymore (the controller manager will allocate these automatically) - We need to pass `--network-plugin=cni` .exercise[ - Join the first node: ```bash sudo kubelet --kubeconfig ~/kubeconfig --network-plugin=cni ``` - Open more terminals and join the other nodes: ```bash ssh kuberouter2 sudo kubelet --kubeconfig ~/kubeconfig --network-plugin=cni ssh kuberouter3 sudo kubelet --kubeconfig ~/kubeconfig --network-plugin=cni ``` ] .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## Checking the CNI configuration - At this point, kuberouter should have installed its CNI configuration (in `/etc/cni/net.d`) .exercise[ - Check the content of `/etc/cni/net.d` ] - There should be a file created by kuberouter - The file should contain the node's podCIDR .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Setting up a test - Let's create a Deployment and expose it with a Service .exercise[ - Create a Deployment running a web server: ```bash kubectl create deployment web --image=jpetazzo/httpenv ``` - Scale it so that it spans multiple nodes: ```bash kubectl scale deployment web --replicas=5 ``` - Expose it with a Service: ```bash kubectl expose deployment web --port=8888 ``` ] .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- ## Checking that everything works .exercise[ - Get the ClusterIP address for the service: ```bash kubectl get svc web ``` - Send a few requests there: ```bash curl `X.X.X.X`:8888 ``` ] Note that if you send multiple requests, they are load-balanced in a round robin manner. This shows that we are using IPVS (vs. iptables, which picked random endpoints). .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## Troubleshooting - What if we need to check that everything is working properly? .exercise[ - Check the IP addresses of our pods: ```bash kubectl get pods -o wide ``` - Check our routing table: ```bash route -n ip route ``` ] We should see the local pod CIDR connected to `kube-bridge`, and the other nodes' pod CIDRs having individual routes, with each node being the gateway. .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## More troubleshooting - We can also look at the output of the kube-router pods (with `kubectl logs`) - kube-router also comes with a special shell that gives lots of useful info (we can access it with `kubectl exec`) - But with the current setup of the cluster, these options may not work! - Why? .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## Trying `kubectl logs` / `kubectl exec` .exercise[ - Try to show the logs of a kube-router pod: ```bash kubectl -n kube-system logs ds/kube-router ``` - Or try to exec into one of the kube-router pods: ```bash kubectl -n kube-system exec kube-router-xxxxx bash ``` ] These commands will give an error message that includes: ``` dial tcp: lookup kuberouterX on <IP_ADDRESS>:53: no such host ``` What does that mean? .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## Internal name resolution - To execute these commands, the API server needs to connect to kubelet - By default, it creates a connection using the kubelet's name (e.g. `http://kuberouter1:...`) - This requires our nodes names to be in DNS - We can change that by setting a flag on the API server: `--kubelet-preferred-address-types=InternalIP` .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## Another way to check the logs - We can also ask the logs directly to the container engine - First, get the container ID, with `docker ps` or like this: ```bash CID=$(docker ps -q \ --filter label=io.kubernetes.pod.namespace=kube-system \ --filter label=io.kubernetes.container.name=kube-router) ``` - Then view the logs: ```bash docker logs $CID ``` .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: extra-details ## Other ways to distribute routing tables - We don't need kube-router and BGP to distribute routes - The list of nodes (and associated `podCIDR` subnets) is available through the API - This shell snippet generates the commands to add all required routes on a node: ```bash NODES=$(kubectl get nodes -o name | cut -d/ -f2) for DESTNODE in $NODES; do if [ "$DESTNODE" != "$HOSTNAME" ]; then echo $(kubectl get node $DESTNODE -o go-template=" route add -net {{.spec.podCIDR}} gw {{(index .status.addresses 0).address}}") fi done ``` - This could be useful for embedded platforms with very limited resources (or lab environments for learning purposes) ??? :EN:- Configuring CNI plugins :FR:- Configurer des plugins CNI .debug[[k8s/cni.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cni.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-1.jpg)] --- name: toc-api-server-availability class: title API server availability .nav[ [Previous section](#toc-the-container-network-interface) | [Back to table of contents](#toc-module-13) | [Next section](#toc-static-pods) ] .debug[(automatically generated title slide)] --- # API server availability - When we set up a node, we need the address of the API server: - for kubelet - for kube-proxy - sometimes for the pod network system (like kube-router) - How do we ensure the availability of that endpoint? (what if the node running the API server goes down?) .debug[[k8s/apilb.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/apilb.md)] --- ## Option 1: external load balancer - Set up an external load balancer - Point kubelet (and other components) to that load balancer - Put the node(s) running the API server behind that load balancer - Update the load balancer if/when an API server node needs to be replaced - On cloud infrastructures, some mechanisms provide automation for this (e.g. on AWS, an Elastic Load Balancer + Auto Scaling Group) - [Example in Kubernetes The Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/08-bootstrapping-kubernetes-controllers.md#the-kubernetes-frontend-load-balancer) .debug[[k8s/apilb.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/apilb.md)] --- ## Option 2: local load balancer - Set up a load balancer (like NGINX, HAProxy...) on *each* node - Configure that load balancer to send traffic to the API server node(s) - Point kubelet (and other components) to `localhost` - Update the load balancer configuration when API server nodes are updated .debug[[k8s/apilb.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/apilb.md)] --- ## Updating the local load balancer config - Distribute the updated configuration (push) - Or regularly check for updates (pull) - The latter requires an external, highly available store (it could be an object store, an HTTP server, or even DNS...) - Updates can be facilitated by a DaemonSet (but remember that it can't be used when installing a new node!) .debug[[k8s/apilb.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/apilb.md)] --- ## Option 3: DNS records - Put all the API server nodes behind a round-robin DNS - Point kubelet (and other components) to that name - Update the records when needed - Note: this option is not officially supported (but since kubelet supports reconnection anyway, it *should* work) .debug[[k8s/apilb.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/apilb.md)] --- ## Option 4: .................... - Many managed clusters expose a high-availability API endpoint (and you don't have to worry about it) - You can also use HA mechanisms that you're familiar with (e.g. virtual IPs) - Tunnels are also fine (e.g. [k3s](https://k3s.io/) uses a tunnel to allow each node to contact the API server) ??? :EN:- Ensuring API server availability :FR:- Assurer la disponibilité du serveur API .debug[[k8s/apilb.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/apilb.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-2.jpg)] --- name: toc-static-pods class: title Static pods .nav[ [Previous section](#toc-api-server-availability) | [Back to table of contents](#toc-module-13) | [Next section](#toc-upgrading-clusters) ] .debug[(automatically generated title slide)] --- # Static pods - Hosting the Kubernetes control plane on Kubernetes has advantages: - we can use Kubernetes' replication and scaling features for the control plane - we can leverage rolling updates to upgrade the control plane - However, there is a catch: - deploying on Kubernetes requires the API to be available - the API won't be available until the control plane is deployed - How can we get out of that chicken-and-egg problem? .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## A possible approach - Since each component of the control plane can be replicated... - We could set up the control plane outside of the cluster - Then, once the cluster is fully operational, create replicas running on the cluster - Finally, remove the replicas that are running outside of the cluster *What could possibly go wrong?* .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## Sawing off the branch you're sitting on - What if anything goes wrong? (During the setup or at a later point) - Worst case scenario, we might need to: - set up a new control plane (outside of the cluster) - restore a backup from the old control plane - move the new control plane to the cluster (again) - This doesn't sound like a great experience .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## Static pods to the rescue - Pods are started by kubelet (an agent running on every node) - To know which pods it should run, the kubelet queries the API server - The kubelet can also get a list of *static pods* from: - a directory containing one (or multiple) *manifests*, and/or - a URL (serving a *manifest*) - These "manifests" are basically YAML definitions (As produced by `kubectl get pod my-little-pod -o yaml`) .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## Static pods are dynamic - Kubelet will periodically reload the manifests - It will start/stop pods accordingly (i.e. it is not necessary to restart the kubelet after updating the manifests) - When connected to the Kubernetes API, the kubelet will create *mirror pods* - Mirror pods are copies of the static pods (so they can be seen with e.g. `kubectl get pods`) .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## Bootstrapping a cluster with static pods - We can run control plane components with these static pods - They can start without requiring access to the API server - Once they are up and running, the API becomes available - These pods are then visible through the API (We cannot upgrade them from the API, though) *This is how kubeadm has initialized our clusters.* .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## Static pods vs normal pods - The API only gives us read-only access to static pods - We can `kubectl delete` a static pod... ...But the kubelet will re-mirror it immediately - Static pods can be selected just like other pods (So they can receive service traffic) - A service can select a mixture of static and other pods .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## From static pods to normal pods - Once the control plane is up and running, it can be used to create normal pods - We can then set up a copy of the control plane in normal pods - Then the static pods can be removed - The scheduler and the controller manager use leader election (Only one is active at a time; removing an instance is seamless) - Each instance of the API server adds itself to the `kubernetes` service - Etcd will typically require more work! .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## From normal pods back to static pods - Alright, but what if the control plane is down and we need to fix it? - We restart it using static pods! - This can be done automatically with the [Pod Checkpointer] - The Pod Checkpointer automatically generates manifests of running pods - The manifests are used to restart these pods if API contact is lost (More details in the [Pod Checkpointer] documentation page) - This technique is used by [bootkube] [Pod Checkpointer]: https://github.com/kubernetes-incubator/bootkube/blob/master/cmd/checkpoint/README.md [bootkube]: https://github.com/kubernetes-incubator/bootkube .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## Where should the control plane run? *Is it better to run the control plane in static pods, or normal pods?* - If I'm a *user* of the cluster: I don't care, it makes no difference to me - What if I'm an *admin*, i.e. the person who installs, upgrades, repairs... the cluster? - If I'm using a managed Kubernetes cluster (AKS, EKS, GKE...) it's not my problem (I'm not the one setting up and managing the control plane) - If I already picked a tool (kubeadm, kops...) to set up my cluster, the tool decides for me - What if I haven't picked a tool yet, or if I'm installing from scratch? - static pods = easier to set up, easier to troubleshoot, less risk of outage - normal pods = easier to upgrade, easier to move (if nodes need to be shut down) .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- ## Static pods in action - On our clusters, the `staticPodPath` is `/etc/kubernetes/manifests` .exercise[ - Have a look at this directory: ```bash ls -l /etc/kubernetes/manifests ``` ] We should see YAML files corresponding to the pods of the control plane. .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- class: static-pods-exercise ## Running a static pod - We are going to add a pod manifest to the directory, and kubelet will run it .exercise[ - Copy a manifest to the directory: ```bash sudo cp ~/container.training/k8s/just-a-pod.yaml /etc/kubernetes/manifests ``` - Check that it's running: ```bash kubectl get pods ``` ] The output should include a pod named `hello-node1`. .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- class: static-pods-exercise ## Remarks In the manifest, the pod was named `hello`. ```yaml apiVersion: v1 kind: Pod metadata: name: hello namespace: default spec: containers: - name: hello image: nginx ``` The `-node1` suffix was added automatically by kubelet. If we delete the pod (with `kubectl delete`), it will be recreated immediately. To delete the pod, we need to delete (or move) the manifest file. ??? :EN:- Static pods :FR:- Les *static pods* .debug[[k8s/staticpods.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/staticpods.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/two-containers-on-a-truck.jpg)] --- name: toc-upgrading-clusters class: title Upgrading clusters .nav[ [Previous section](#toc-static-pods) | [Back to table of contents](#toc-module-14) | [Next section](#toc-backing-up-clusters) ] .debug[(automatically generated title slide)] --- # Upgrading clusters - It's *recommended* to run consistent versions across a cluster (mostly to have feature parity and latest security updates) - It's not *mandatory* (otherwise, cluster upgrades would be a nightmare!) - Components can be upgraded one at a time without problems .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Checking what we're running - It's easy to check the version for the API server .exercise[ - Log into node `test1` - Check the version of kubectl and of the API server: ```bash kubectl version ``` ] - In a HA setup with multiple API servers, they can have different versions - Running the command above multiple times can return different values .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Node versions - It's also easy to check the version of kubelet .exercise[ - Check node versions (includes kubelet, kernel, container engine): ```bash kubectl get nodes -o wide ``` ] - Different nodes can run different kubelet versions - Different nodes can run different kernel versions - Different nodes can run different container engines .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Control plane versions - If the control plane is self-hosted (running in pods), we can check it .exercise[ - Show image versions for all pods in `kube-system` namespace: ```bash kubectl --namespace=kube-system get pods -o json \ | jq -r ' .items[] | [.spec.nodeName, .metadata.name] + (.spec.containers[].image | split(":")) | @tsv ' \ | column -t ``` ] .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## What version are we running anyway? - When I say, "I'm running Kubernetes 1.15", is that the version of: - kubectl - API server - kubelet - controller manager - something else? .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Other versions that are important - etcd - kube-dns or CoreDNS - CNI plugin(s) - Network controller, network policy controller - Container engine - Linux kernel .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## General guidelines - To update a component, use whatever was used to install it - If it's a distro package, update that distro package - If it's a container or pod, update that container or pod - If you used configuration management, update with that .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Know where your binaries come from - Sometimes, we need to upgrade *quickly* (when a vulnerability is announced and patched) - If we are using an installer, we should: - make sure it's using upstream packages - or make sure that whatever packages it uses are current - make sure we can tell it to pin specific component versions .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Important questions - Should we upgrade the control plane before or after the kubelets? - Within the control plane, should we upgrade the API server first or last? - How often should we upgrade? - How long are versions maintained? - All the answers are in [the documentation about version skew policy](https://kubernetes.io/docs/setup/release/version-skew-policy/)! - Let's review the key elements together ... .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Kubernetes uses semantic versioning - Kubernetes versions look like MAJOR.MINOR.PATCH; e.g. in 1.17.2: - MAJOR = 1 - MINOR = 17 - PATCH = 2 - It's always possible to mix and match different PATCH releases (e.g. 1.16.1 and 1.16.6 are compatible) - It is recommended to run the latest PATCH release (but it's mandatory only when there is a security advisory) .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Version skew - API server must be more recent than its clients (kubelet and control plane) - ... Which means it must always be upgraded first - All components support a difference of one¹ MINOR version - This allows live upgrades (since we can mix e.g. 1.15 and 1.16) - It also means that going from 1.14 to 1.16 requires going through 1.15 .footnote[¹Except kubelet, which can be up to two MINOR behind API server, and kubectl, which can be one MINOR ahead or behind API server.] .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Release cycle - There is a new PATCH relese whenever necessary (every few weeks, or "ASAP" when there is a security vulnerability) - There is a new MINOR release every 3 months (approximately) - At any given time, three MINOR releases are maintained - ... Which means that MINOR releases are maintained approximately 9 months - We should expect to upgrade at least every 3 months (on average) .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## In practice - We are going to update a few cluster components - We will change the kubelet version on one node - We will change the version of the API server - We will work with cluster `test` (nodes `test1`, `test2`, `test3`) .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Updating the API server - This cluster has been deployed with kubeadm - The control plane runs in *static pods* - These pods are started automatically by kubelet (even when kubelet can't contact the API server) - They are defined in YAML files in `/etc/kubernetes/manifests` (this path is set by a kubelet command-line flag) - kubelet automatically updates the pods when the files are changed .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Changing the API server version - We will edit the YAML file to use a different image version .exercise[ - Log into node `test1` - Check API server version: ```bash kubectl version ``` - Edit the API server pod manifest: ```bash sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml ``` - Look for the `image:` line, and update it to e.g. `v1.16.0` ] .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Checking what we've done - The API server will be briefly unavailable while kubelet restarts it .exercise[ - Check the API server version: ```bash kubectl version ``` ] .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Was that a good idea? -- **No!** -- - Remember the guideline we gave earlier: *To update a component, use whatever was used to install it.* - This control plane was deployed with kubeadm - We should use kubeadm to upgrade it! .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Updating the whole control plane - Let's make it right, and use kubeadm to upgrade the entire control plane (note: this is possible only because the cluster was installed with kubeadm) .exercise[ - Check what will be upgraded: ```bash sudo kubeadm upgrade plan ``` ] Note 1: kubeadm thinks that our cluster is running 1.16.0.
It is confused by our manual upgrade of the API server! Note 2: kubeadm itself is still version 1.15.9.
It doesn't know how to upgrade do 1.16.X. .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Upgrading kubeadm - First things first: we need to upgrade kubeadm .exercise[ - Upgrade kubeadm: ``` sudo apt install kubeadm ``` - Check what kubeadm tells us: ``` sudo kubeadm upgrade plan ``` ] Problem: kubeadm doesn't know know how to handle upgrades from version 1.15. This is because we installed version 1.17 (or even later). We need to install kubeadm version 1.16.X. .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Downgrading kubeadm - We need to go back to version 1.16.X (e.g. 1.16.6) .exercise[ - View available versions for package `kubeadm`: ```bash apt show kubeadm -a | grep ^Version | grep 1.16 ``` - Downgrade kubeadm: ``` sudo apt install kubeadm=1.16.6-00 ``` - Check what kubeadm tells us: ``` sudo kubeadm upgrade plan ``` ] kubeadm should now agree to upgrade to 1.16.6. .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Upgrading the cluster with kubeadm - Ideally, we should revert our `image:` change (so that kubeadm executes the right migration steps) - Or we can try the upgrade anyway .exercise[ - Perform the upgrade: ```bash sudo kubeadm upgrade apply v1.16.6 ``` ] .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Updating kubelet - These nodes have been installed using the official Kubernetes packages - We can therefore use `apt` or `apt-get` .exercise[ - Log into node `test3` - View available versions for package `kubelet`: ```bash apt show kubelet -a | grep ^Version ``` - Upgrade kubelet: ```bash sudo apt install kubelet=1.16.6-00 ``` ] .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Checking what we've done .exercise[ - Log into node `test1` - Check node versions: ```bash kubectl get nodes -o wide ``` - Create a deployment and scale it to make sure that the node still works ] .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Was that a good idea? -- **Almost!** -- - Yes, kubelet was installed with distribution packages - However, kubeadm took care of configuring kubelet (when doing `kubeadm join ...`) - We were supposed to run a special command *before* upgrading kubelet! - That command should be executed on each node - It will download the kubelet configuration generated by kubeadm .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Upgrading kubelet the right way - We need to upgrade kubeadm, upgrade kubelet config, then upgrade kubelet (after upgrading the control plane) .exercise[ - Download the configuration on each node, and upgrade kubelet: ```bash for N in 1 2 3; do ssh test$N " sudo apt install kubeadm=1.16.6-00 && sudo kubeadm upgrade node && sudo apt install kubelet=1.16.6-00" done ``` ] .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- ## Checking what we've done - All our nodes should now be updated to version 1.16.6 .exercise[ - Check nodes versions: ```bash kubectl get nodes -o wide ``` ] .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- class: extra-details ## Skipping versions - This example worked because we went from 1.15 to 1.16 - If you are upgrading from e.g. 1.14, you will have to go through 1.15 first - This means upgrading kubeadm to 1.15.X, then using it to upgrade the cluster - Then upgrading kubeadm to 1.16.X, etc. - **Make sure to read the release notes before upgrading!** ??? :EN:- Best practices for cluster upgrades :EN:- Example: upgrading a kubeadm cluster :FR:- Bonnes pratiques pour la mise à jour des clusters :FR:- Exemple : mettre à jour un cluster kubeadm .debug[[k8s/cluster-upgrade.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-upgrade.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/wall-of-containers.jpeg)] --- name: toc-backing-up-clusters class: title Backing up clusters .nav[ [Previous section](#toc-upgrading-clusters) | [Back to table of contents](#toc-module-14) | [Next section](#toc-the-cloud-controller-manager) ] .debug[(automatically generated title slide)] --- # Backing up clusters - Backups can have multiple purposes: - disaster recovery (servers or storage are destroyed or unreachable) - error recovery (human or process has altered or corrupted data) - cloning environments (for testing, validation...) - Let's see the strategies and tools available with Kubernetes! .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Important - Kubernetes helps us with disaster recovery (it gives us replication primitives) - Kubernetes helps us clone / replicate environments (all resources can be described with manifests) - Kubernetes *does not* help us with error recovery - We still need to back up/snapshot our data: - with database backups (mysqldump, pgdump, etc.) - and/or snapshots at the storage layer - and/or traditional full disk backups .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## In a perfect world ... - The deployment of our Kubernetes clusters is automated (recreating a cluster takes less than a minute of human time) - All the resources (Deployments, Services...) on our clusters are under version control (never use `kubectl run`; always apply YAML files coming from a repository) - Stateful components are either: - stored on systems with regular snapshots - backed up regularly to an external, durable storage - outside of Kubernetes .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Kubernetes cluster deployment - If our deployment system isn't fully automated, it should at least be documented - Litmus test: how long does it take to deploy a cluster... - for a senior engineer? - for a new hire? - Does it require external intervention? (e.g. provisioning servers, signing TLS certs...) .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Plan B - Full machine backups of the control plane can help - If the control plane is in pods (or containers), pay attention to storage drivers (if the backup mechanism is not container-aware, the backups can take way more resources than they should, or even be unusable!) - If the previous sentence worries you: **automate the deployment of your clusters!** .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Managing our Kubernetes resources - Ideal scenario: - never create a resource directly on a cluster - push to a code repository - a special branch (`production` or even `master`) gets automatically deployed - Some folks call this "GitOps" (it's the logical evolution of configuration management and infrastructure as code) .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## GitOps in theory - What do we keep in version control? - For very simple scenarios: source code, Dockerfiles, scripts - For real applications: add resources (as YAML files) - For applications deployed multiple times: Helm, Kustomize... (staging and production count as "multiple times") .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## GitOps tooling - Various tools exist (Weave Flux, GitKube...) - These tools are still very young - You still need to write YAML for all your resources - There is no tool to: - list *all* resources in a namespace - get resource YAML in a canonical form - diff YAML descriptions with current state .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## GitOps in practice - Start describing your resources with YAML - Leverage a tool like Kustomize or Helm - Make sure that you can easily deploy to a new namespace (or even better: to a new cluster) - When tooling matures, you will be ready .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Plan B - What if we can't describe everything with YAML? - What if we manually create resources and forget to commit them to source control? - What about global resources, that don't live in a namespace? - How can we be sure that we saved *everything*? .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Backing up etcd - All objects are saved in etcd - etcd data should be relatively small (and therefore, quick and easy to back up) - Two options to back up etcd: - snapshot the data directory - use `etcdctl snapshot` .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Making an etcd snapshot - The basic command is simple: ```bash etcdctl snapshot save
``` - But we also need to specify: - an environment variable to specify that we want etcdctl v3 - the address of the server to back up - the path to the key, certificate, and CA certificate
(if our etcd uses TLS certificates) .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Snapshotting etcd on kubeadm - The following command will work on clusters deployed with kubeadm (and maybe others) - It should be executed on a master node ```bash docker run --rm --net host -v $PWD:/vol \ -v /etc/kubernetes/pki/etcd:/etc/kubernetes/pki/etcd:ro \ -e ETCDCTL_API=3 k8s.gcr.io/etcd:3.3.10 \ etcdctl --endpoints=https://[<IP_ADDRESS>]:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \ --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \ snapshot save /vol/snapshot ``` - It will create a file named `snapshot` in the current directory .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## How can we remember all these flags? - Older versions of kubeadm did add a healthcheck probe with all these flags - That healthcheck probe was calling `etcdctl` with all the right flags - With recent versions of kubeadm, we're on our own! - Exercise: write the YAML for a batch job to perform the backup (how will you access the key and certificate required to connect?) .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Restoring an etcd snapshot - ~~Execute exactly the same command, but replacing `save` with `restore`~~ (Believe it or not, doing that will *not* do anything useful!) - The `restore` command does *not* load a snapshot into a running etcd server - The `restore` command creates a new data directory from the snapshot (it's an offline operation; it doesn't interact with an etcd server) - It will create a new data directory in a temporary container (leaving the running etcd node untouched) .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## When using kubeadm 1. Create a new data directory from the snapshot: ```bash sudo rm -rf /var/lib/etcd docker run --rm -v /var/lib:/var/lib -v $PWD:/vol \ -e ETCDCTL_API=3 k8s.gcr.io/etcd:3.3.10 \ etcdctl snapshot restore /vol/snapshot --data-dir=/var/lib/etcd ``` 2. Provision the control plane, using that data directory: ```bash sudo kubeadm init \ --ignore-preflight-errors=DirAvailable--var-lib-etcd ``` 3. Rejoin the other nodes .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## The fine print - This only saves etcd state - It **does not** save persistent volumes and local node data - Some critical components (like the pod network) might need to be reset - As a result, our pods might have to be recreated, too - If we have proper liveness checks, this should happen automatically .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## More information about etcd backups - [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#built-in-snapshot) about etcd backups - [etcd documentation](https://coreos.com/etcd/docs/latest/op-guide/recovery.html#snapshotting-the-keyspace) about snapshots and restore - [A good blog post by elastisys](https://elastisys.com/2018/12/10/backup-kubernetes-how-and-why/) explaining how to restore a snapshot - [Another good blog post by consol labs](https://labs.consol.de/kubernetes/2018/05/25/kubeadm-backup.html) on the same topic .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Don't forget ... - Also back up the TLS information (at the very least: CA key and cert; API server key and cert) - With clusters provisioned by kubeadm, this is in `/etc/kubernetes/pki` - If you don't: - you will still be able to restore etcd state and bring everything back up - you will need to redistribute user certificates .warning[**TLS information is highly sensitive!
Anyone who has it has full access to your cluster!**] .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Stateful services - It's totally fine to keep your production databases outside of Kubernetes *Especially if you have only one database server!* - Feel free to put development and staging databases on Kubernetes (as long as they don't hold important data) - Using Kubernetes for stateful services makes sense if you have *many* (because then you can leverage Kubernetes automation) .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## Snapshotting persistent volumes - Option 1: snapshot volumes out of band (with the API/CLI/GUI of our SAN/cloud/...) - Option 2: storage system integration (e.g. [Portworx](https://docs.portworx.com/portworx-install-with-kubernetes/storage-operations/create-snapshots/) can [create snapshots through annotations](https://docs.portworx.com/portworx-install-with-kubernetes/storage-operations/create-snapshots/snaps-annotations/#taking-periodic-snapshots-on-a-running-pod)) - Option 3: [snapshots through Kubernetes API](https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/) (now in alpha for a few storage providers: GCE, OpenSDS, Ceph, Portworx) .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- ## More backup tools - [Stash](https://appscode.com/products/stash/) back up Kubernetes persistent volumes - [ReShifter](https://github.com/mhausenblas/reshifter) cluster state management - ~~Heptio Ark~~ [Velero](https://github.com/heptio/velero) full cluster backup - [kube-backup](https://github.com/pieterlange/kube-backup) simple scripts to save resource YAML to a git repository - [bivac](https://github.com/camptocamp/bivac) Backup Interface for Volumes Attached to Containers ??? :EN:- Backing up clusters :FR:- Politiques de sauvegarde .debug[[k8s/cluster-backup.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cluster-backup.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/Container-Ship-Freighter-Navigation-Elbe-Romance-1782991.jpg)] --- name: toc-the-cloud-controller-manager class: title The Cloud Controller Manager .nav[ [Previous section](#toc-backing-up-clusters) | [Back to table of contents](#toc-module-14) | [Next section](#toc-git-based-workflows) ] .debug[(automatically generated title slide)] --- # The Cloud Controller Manager - Kubernetes has many features that are cloud-specific (e.g. providing cloud load balancers when a Service of type LoadBalancer is created) - These features were initially implemented in API server and controller manager - Since Kubernetes 1.6, these features are available through a separate process: the *Cloud Controller Manager* - The CCM is optional, but if we run in a cloud, we probably want it! .debug[[k8s/cloud-controller-manager.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cloud-controller-manager.md)] --- ## Cloud Controller Manager duties - Creating and updating cloud load balancers - Configuring routing tables in the cloud network (specific to GCE) - Updating node labels to indicate region, zone, instance type... - Obtain node name, internal and external addresses from cloud metadata service - Deleting nodes from Kubernetes when they're deleted in the cloud - Managing *some* volumes (e.g. ELBs, AzureDisks...) (Eventually, volumes will be managed by the Container Storage Interface) .debug[[k8s/cloud-controller-manager.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cloud-controller-manager.md)] --- ## In-tree vs. out-of-tree - A number of cloud providers are supported "in-tree" (in the main kubernetes/kubernetes repository on GitHub) - More cloud providers are supported "out-of-tree" (with code in different repositories) - There is an [ongoing effort](https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider) to move everything to out-of-tree providers .debug[[k8s/cloud-controller-manager.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cloud-controller-manager.md)] --- ## In-tree providers The following providers are actively maintained: - Amazon Web Services - Azure - Google Compute Engine - IBM Cloud - OpenStack - VMware vSphere These ones are less actively maintained: - Apache CloudStack - oVirt - VMware Photon .debug[[k8s/cloud-controller-manager.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cloud-controller-manager.md)] --- ## Out-of-tree providers The list includes the following providers: - DigitalOcean - keepalived (not exactly a cloud; provides VIPs for load balancers) - Linode - Oracle Cloud Infrastructure (And possibly others; there is no central registry for these.) .debug[[k8s/cloud-controller-manager.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cloud-controller-manager.md)] --- ## Audience questions - What kind of clouds are you using/planning to use? - What kind of details would you like to see in this section? - Would you appreciate details on clouds that you don't / won't use? .debug[[k8s/cloud-controller-manager.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cloud-controller-manager.md)] --- ## Cloud Controller Manager in practice - Write a configuration file (typically `/etc/kubernetes/cloud.conf`) - Run the CCM process (on self-hosted clusters, this can be a DaemonSet selecting the control plane nodes) - Start kubelet with `--cloud-provider=external` - When using managed clusters, this is done automatically - There is very little documentation on writing the configuration file (except for OpenStack) .debug[[k8s/cloud-controller-manager.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cloud-controller-manager.md)] --- ## Bootstrapping challenges - When a node joins the cluster, it needs to obtain a signed TLS certificate - That certificate must contain the node's addresses - These addresses are provided by the Cloud Controller Manager (at least the external address) - To get these addresses, the node needs to communicate with the control plane - ...Which means joining the cluster (The problem didn't occur when cloud-specific code was running in kubelet: kubelet could obtain the required information directly from the cloud provider's metadata service.) .debug[[k8s/cloud-controller-manager.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cloud-controller-manager.md)] --- ## More information about CCM - CCM configuration and operation is highly specific to each cloud provider (which is why this section remains very generic) - The Kubernetes documentation has *some* information: - [architecture and diagrams](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) - [configuration](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/) (mainly for OpenStack) - [deployment](https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/) ??? :EN:- The Cloud Controller Manager :FR:- Le *Cloud Controller Manager* .debug[[k8s/cloud-controller-manager.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/cloud-controller-manager.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/ShippingContainerSFBay.jpg)] --- name: toc-git-based-workflows class: title Git-based workflows .nav[ [Previous section](#toc-the-cloud-controller-manager) | [Back to table of contents](#toc-module-14) | [Next section](#toc-last-words) ] .debug[(automatically generated title slide)] --- # Git-based workflows - Deploying with `kubectl` has downsides: - we don't know *who* deployed *what* and *when* - there is no audit trail (except the API server logs) - there is no easy way to undo most operations - there is no review/approval process (like for code reviews) - We have all these things for *code*, though - Can we manage cluster state like we manage our source code? .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Reminder: Kubernetes is *declarative* - All we do is create/change resources - These resources have a perfect YAML representation - All we do is manipulating these YAML representations (`kubectl run` generates a YAML file that gets applied) - We can store these YAML representations in a code repository - We can version that code repository and maintain it with best practices - define which branch(es) can go to qa/staging/production - control who can push to which branches - have formal review processes, pull requests ... .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Enabling git-based workflows - There are a few tools out there to help us do that - We'll see demos of two of them: [Flux] and [Gitkube] - There are *many* other tools, some of them with even more features - There are also *many* integrations with popular CI/CD systems (e.g.: GitLab, Jenkins, ...) [Flux]: https://www.weave.works/oss/flux/ [Gitkube]: https://gitkube.sh/ .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Flux overview - We put our Kubernetes resources as YAML files in a git repository - Flux polls that repository regularly (every 5 minutes by default) - The resources described by the YAML files are created/updated automatically - Changes are made by updating the code in the repository .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Preparing a repository for Flux - We need a repository with Kubernetes YAML files - I have one: https://github.com/jpetazzo/kubercoins - Fork it to your GitHub account - Create a new branch in your fork; e.g. `prod` (e.g. by adding a line in the README through the GitHub web UI) - This is the branch that we are going to use for deployment .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Setting up Flux - Clone the Flux repository: ``` git clone https://github.com/fluxcd/flux ``` - Edit `deploy/flux-deployment.yaml` - Change the `--git-url` and `--git-branch` parameters: ```yaml - --git-url=git@github.com:your-git-username/kubercoins - --git-branch=prod ``` - Apply all the YAML: ``` kubectl apply -f deploy/ ``` .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Allowing Flux to access the repository - When it starts, Flux generates an SSH key - Display that key: ``` kubectl logs deployment/flux | grep identity ``` - Then add that key to the repository, giving it **write** access (some Flux features require write access) - After a minute or so, DockerCoins will be deployed to the current namespace .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Making changes - Make changes (on the `prod` branch), e.g. change `replicas` in `worker` - After a few minutes, the changes will be picked up by Flux and applied .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Other features - Flux can keep a list of all the tags of all the images we're running - The `fluxctl` tool can show us if we're running the latest images - We can also "automate" a resource (i.e. automatically deploy new images) - And much more! .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Gitkube overview - We put our Kubernetes resources as YAML files in a git repository - Gitkube is a git server (or "git remote") - After making changes to the repository, we push to Gitkube - Gitkube applies the resources to the cluster .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Setting up Gitkube - Install the CLI: ``` sudo curl -L -o /usr/local/bin/gitkube \ https://github.com/hasura/gitkube/releases/download/v0.2.1/gitkube_linux_amd64 sudo chmod +x /usr/local/bin/gitkube ``` - Install Gitkube on the cluster: ``` gitkube install --expose ClusterIP ``` .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Creating a Remote - Gitkube provides a new type of API resource: *Remote* (this is using a mechanism called Custom Resource Definitions or CRD) - Create and apply a YAML file containing the following manifest: ```yaml apiVersion: gitkube.sh/v1alpha1 kind: Remote metadata: name: example spec: authorizedKeys: - `ssh-rsa AAA...` manifests: path: "." ``` (replace the `ssh-rsa AAA...` section with the content of `~/.ssh/id_rsa.pub`) .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Pushing to our remote - Get the `gitkubed` IP address: ``` kubectl -n kube-system get svc gitkubed IP=$(kubectl -n kube-system get svc gitkubed -o json | jq -r .spec.clusterIP) ``` - Get ourselves a sample repository with resource YAML files: ``` git clone git://github.com/jpetazzo/kubercoins cd kubercoins ``` - Add the remote and push to it: ``` git remote add k8s ssh://default-example@$IP/~/git/default-example git push k8s master ``` .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Making changes - Edit a local file - Commit - Push! - Make sure that you push to the `k8s` remote .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- ## Other features - Gitkube can also build container images for us (see the [documentation](https://github.com/hasura/gitkube/blob/master/docs/remote.md) for more details) - Gitkube can also deploy Helm charts (instead of raw YAML files) ??? :EN:- GitOps :FR:- GitOps .debug[[k8s/gitworkflows.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/gitworkflows.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/aerial-view-of-containers.jpg)] --- name: toc-last-words class: title Last words .nav[ [Previous section](#toc-git-based-workflows) | [Back to table of contents](#toc-module-15) | [Next section](#toc-links-and-resources) ] .debug[(automatically generated title slide)] --- # Last words - Congratulations! - We learned a lot about Kubernetes, its internals, its advanced concepts -- - That was just the easy part - The hard challenges will revolve around *culture* and *people* -- - ... What does that mean? .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## Running an app involves many steps - Write the app - Tests, QA ... - Ship *something* (more on that later) - Provision resources (e.g. VMs, clusters) - Deploy the *something* on the resources - Manage, maintain, monitor the resources - Manage, maintain, monitor the app - And much more .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## Who does what? - The old "devs vs ops" division has changed - In some organizations, "ops" are now called "SRE" or "platform" teams (and they have very different sets of skills) - Do you know which team is responsible for each item on the list on the previous page? - Acknowledge that a lot of tasks are outsourced (e.g. if we add "buy/rack/provision machines" in that list) .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## What do we ship? - Some organizations embrace "you build it, you run it" - When "build" and "run" are owned by different teams, where's the line? - What does the "build" team ship to the "run" team? - Let's see a few options, and what they imply .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## Shipping code - Team "build" ships code (hopefully in a repository, identified by a commit hash) - Team "run" containerizes that code ✔ no extra work for developers ❌ very little advantage of using containers .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## Shipping container images - Team "build" ships container images (hopefully built automatically from a source repository) - Team "run" uses theses images to create e.g. Kubernetes resources ✔ universal artefact (support all languages uniformly) ✔ easy to start a single component (good for monoliths) ❌ complex applications will require a lot of extra work ❌ adding/removing components in the stack also requires extra work ❌ complex applications will run very differently between dev and prod .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## Shipping Compose files (Or another kind of dev-centric manifest) - Team "build" ships a manifest that works on a single node (as well as images, or ways to build them) - Team "run" adapts that manifest to work on a cluster ✔ all teams can start the stack in a reliable, deterministic manner ❌ adding/removing components still requires *some* work (but less than before) ❌ there will be *some* differences between dev and prod .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## Shipping Kubernetes manifests - Team "build" ships ready-to-run manifests (YAML, Helm charts, Kustomize ...) - Team "run" adjusts some parameters and monitors the application ✔ parity between dev and prod environments ✔ "run" team can focus on SLAs, SLOs, and overall quality ❌ requires *a lot* of extra work (and new skills) from the "build" team ❌ Kubernetes is not a very convenient development platform (at least, not yet) .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## What's the right answer? - It depends on our teams - existing skills (do they know how to do it?) - availability (do they have the time to do it?) - potential skills (can they learn to do it?) - It depends on our culture - owning "run" often implies being on call - do we reward on-call duty without encouraging hero syndrome? - do we give people resources (time, money) to learn? .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- class: extra-details ## Tools to develop on Kubernetes *If we decide to make Kubernetes the primary development platform, here are a few tools that can help us.* - Docker Desktop - Draft - Minikube - Skaffold - Tilt - ... .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## Where do we run? - Managed vs. self-hosted - Cloud vs. on-premises - If cloud: public vs. private - Which vendor/distribution to pick? - Which versions/features to enable? .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## Developer experience *These questions constitute a quick "smoke test" for our strategy:* - How do we on-board a new developer? - What do they need to install to get a dev stack? - How does a code change make it from dev to prod? - How does someone add a component to a stack? .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- ## Some guidelines - Start small - Outsource what we don't know - Start simple, and stay simple as long as possible (try to stay away from complex features that we don't need) - Automate (regularly check that we can successfully redeploy by following scripts) - Transfer knowledge (make sure everyone is on the same page/level) - Iterate! .debug[[k8s/lastwords.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/lastwords.md)] --- class: pic .interstitial[![Image separating from the next module](https://gallant-turing-d0d520.netlify.com/containers/blue-containers.jpg)] --- name: toc-links-and-resources class: title Links and resources .nav[ [Previous section](#toc-last-words) | [Back to table of contents](#toc-module-15) | [Next section](#toc-) ] .debug[(automatically generated title slide)] --- # Links and resources All things Kubernetes: - [Kubernetes Community](https://kubernetes.io/community/) - Slack, Google Groups, meetups - [Kubernetes on StackOverflow](https://stackoverflow.com/questions/tagged/kubernetes) - [Play With Kubernetes Hands-On Labs](https://medium.com/@marcosnils/introducing-pwk-play-with-k8s-159fcfeb787b) All things Docker: - [Docker documentation](http://docs.docker.com/) - [Docker Hub](https://hub.docker.com) - [Docker on StackOverflow](https://stackoverflow.com/questions/tagged/docker) - [Play With Docker Hands-On Labs](http://training.play-with-docker.com/) Everything else: - [Local meetups](https://www.meetup.com/) .footnote[These slides (and future updates) are on → http://container.training/] .debug[[k8s/links.md](https://github.com/jpetazzo/container.training/tree/master/slides/k8s/links.md)] --- class: title, self-paced Thank you! .debug[[shared/thankyou.md](https://github.com/jpetazzo/container.training/tree/master/slides/shared/thankyou.md)] --- class: title, in-person That's all, folks!
Questions? ![end](images/end.jpg) .debug[[shared/thankyou.md](https://github.com/jpetazzo/container.training/tree/master/slides/shared/thankyou.md)]


Participation Activity 4
Last name:
First name:
Std#:
Memory Allocation Example:
Where does eveything go?
Indicate with an arrow in which memory segment the content of each box will be found one we start executing the program.
Code (box is indicated as a comment):
#include ... // cyan box
char hugeArray[1 << 31]; /* 2^31 = 2GB */ // purple box
int global = 0; // purple box
int main()
{
void *ptr1, *ptr2; // red box
int local = 0; // red box
ptr1 = malloc(1 << 28); /* 2^28 = 256 MB */ // blue box
ptr2 = malloc(1 << 8); /* 2^8 = 256 B*/ // blue box
/* some print statements */
}
| M[] |
|---|
| Kernel |
| Stack (arrow pointing down) |
| ... |
| Shared Libraries |
| ... |
| Heap (arrow pointing up) |
| Data |
| Text |
| ... |


MLlib - Clustering
Clustering
Clustering is an unsupervised learning problem whereby we aim to group subsets of entities with one another based on some notion of similarity. Clustering is often used for exploratory analysis and/or as a component of a hierarchical supervised learning pipeline (in which distinct classifiers or regression models are trained for each cluster).
MLlib supports k-means clustering, one of the most commonly used clustering algorithms that clusters the data points into predefined number of clusters. The MLlib implementation includes a parallelized variant of the k-means++ method called kmeans||. The implementation in MLlib has the following parameters:
- k is the number of desired clusters.
- maxIterations is the maximum number of iterations to run.
- initializationMode specifies either random initialization or initialization via k-means||.
- runs is the number of times to run the k-means algorithm (k-means is not guaranteed to find a globally optimal solution, and when run multiple times on a given dataset, the algorithm returns the best clustering result).
- initializationSteps determines the number of steps in the k-means|| algorithm.
- epsilon determines the distance threshold within which we consider k-means to have converged.
Examples
The following code snippets can be executed in spark-shell
.
In the following example after loading and parsing data, we use the
KMeans
object to cluster the data
into two clusters. The number of desired clusters is passed to the algorithm. We then compute Within
Set Sum of Squared Error (WSSSE). You can reduce this error measure by increasing k. In fact the
optimal k is usually one where there is an “elbow” in the WSSSE graph.
import org.apache.spark.mllib.clustering.KMeans
import org.apache.spark.mllib.linalg.Vectors
// Load and parse the data
val data = sc.textFile("data/mllib/kmeans_data.txt")
val parsedData = data.map(s => Vectors.dense(s.split(' ').map(_.toDouble))).cache()
// Cluster the data into two classes using KMeans
val numClusters = 2
val numIterations = 20
val clusters = KMeans.train(parsedData, numClusters, numIterations)
// Evaluate clustering by computing Within Set Sum of Squared Errors
val WSSSE = clusters.computeCost(parsedData)
println("Within Set Sum of Squared Errors = " + WSSSE)
All of MLlib’s methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate JavaRDD
class. You can convert a Java RDD to a Scala one by
calling .rdd()
on your JavaRDD
object. A self-contained application example
that is equivalent to the provided example in Scala is given below:
import org.apache.spark.api.java.*;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.mllib.clustering.KMeans;
import org.apache.spark.mllib.clustering.KMeansModel;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
import org.apache.spark.SparkConf;
public class KMeansExample {
public static void main(String[] args) {
SparkConf conf = new SparkConf().setAppName("K-means Example");
JavaSparkContext sc = new JavaSparkContext(conf);
// Load and parse data
String path = "data/mllib/kmeans_data.txt";
JavaRDD<String> data = sc.textFile(path);
JavaRDD<Vector> parsedData = data.map(
new Function<String, Vector>() {
public Vector call(String s) {
String[] sarray = s.split(" ");
double[] values = new double[sarray.length];
for (int i = 0; i < sarray.length; i++)
values[i] = Double.parseDouble(sarray[i]);
return Vectors.dense(values);
}
}
);
parsedData.cache();
// Cluster the data into two classes using KMeans
int numClusters = 2;
int numIterations = 20;
KMeansModel clusters = KMeans.train(parsedData.rdd(), numClusters, numIterations);
// Evaluate clustering by computing Within Set Sum of Squared Errors
double WSSSE = clusters.computeCost(parsedData.rdd());
System.out.println("Within Set Sum of Squared Errors = " + WSSSE);
}
}
The following examples can be tested in the PySpark shell.
In the following example after loading and parsing data, we use the KMeans object to cluster the data into two clusters. The number of desired clusters is passed to the algorithm. We then compute Within Set Sum of Squared Error (WSSSE). You can reduce this error measure by increasing k. In fact the optimal k is usually one where there is an “elbow” in the WSSSE graph.
from pyspark.mllib.clustering import KMeans
from numpy import array
from math import sqrt
# Load and parse the data
data = sc.textFile("data/mllib/kmeans_data.txt")
parsedData = data.map(lambda line: array([float(x) for x in line.split(' ')]))
# Build the model (cluster the data)
clusters = KMeans.train(parsedData, 2, maxIterations=10,
runs=10, initializationMode="random")
# Evaluate clustering by computing Within Set Sum of Squared Errors
def error(point):
center = clusters.centers[clusters.predict(point)]
return sqrt(sum([x**2 for x in (point - center)]))
WSSSE = parsedData.map(lambda point: error(point)).reduce(lambda x, y: x + y)
print("Within Set Sum of Squared Error = " + str(WSSSE))
In order to run the above application, follow the instructions provided in the Self-Contained Applications section of the Spark Quick Start guide. Be sure to also include spark-mllib to your build file as a dependency.
Streaming clustering
When data arrive in a stream, we may want to estimate clusters dynamically, updating them as new data arrive. MLlib provides support for streaming k-means clustering, with parameters to control the decay (or “forgetfulness”) of the estimates. The algorithm uses a generalization of the mini-batch k-means update rule. For each batch of data, we assign all points to their nearest cluster, compute new cluster centers, then update each cluster using:
\begin{equation}
c_{t+1} = \frac{c_tn_t\alpha + x_tm_t}{n_t\alpha+m_t}
\end{equation}
\begin{equation}
n_{t+1} = n_t + m_t
\end{equation}
Where $c_t$
is the previous center for the cluster, $n_t$
is the number of points assigned
to the cluster thus far, $x_t$
is the new cluster center from the current batch, and $m_t$
is the number of points added to the cluster in the current batch. The decay factor $\alpha$
can be used to ignore the past: with $\alpha$=1
all data will be used from the beginning;
with $\alpha$=0
only the most recent data will be used. This is analogous to an
exponentially-weighted moving average.
The decay can be specified using a halfLife
parameter, which determines the
correct decay factor a
such that, for data acquired
at time t
, its contribution by time t + halfLife
will have dropped to 0.5.
The unit of time can be specified either as batches
or points
and the update rule
will be adjusted accordingly.
Examples
This example shows how to estimate clusters on streaming data.
First we import the neccessary classes.
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.clustering.StreamingKMeans
Then we make an input stream of vectors for training, as well as a stream of labeled data
points for testing. We assume a StreamingContext ssc
has been created, see
Spark Streaming Programming Guide for more info.
val trainingData = ssc.textFileStream("/training/data/dir").map(Vectors.parse)
val testData = ssc.textFileStream("/testing/data/dir").map(LabeledPoint.parse)
We create a model with random clusters and specify the number of clusters to find
val numDimensions = 3
val numClusters = 2
val model = new StreamingKMeans()
.setK(numClusters)
.setDecayFactor(1.0)
.setRandomCenters(numDimensions, 0.0)
Now register the streams for training and testing and start the job, printing the predicted cluster assignments on new data points as they arrive.
model.trainOn(trainingData)
model.predictOnValues(testData.map(lp => (lp.label, lp.features))).print()
ssc.start()
ssc.awaitTermination()
As you add new text files with data the cluster centers will update. Each training
point should be formatted as [x1, x2, x3]
, and each test data point
should be formatted as (y, [x1, x2, x3])
, where y
is some useful label or identifier
(e.g. a true category assignment). Anytime a text file is placed in /training/data/dir
the model will update. Anytime a text file is placed in /testing/data/dir
you will see predictions. With new data, the cluster centers will change!


| PostgreSQL 9.0.10 Documentation | ||||
|---|---|---|---|---|
| Prev | Up | Chapter 45. System Catalogs | Next |
The catalog pg_database stores information about the available databases. Databases are created with the CREATE DATABASE command. Consult Chapter 21 for details about the meaning of some of the parameters.
Unlike most system catalogs, pg_database is shared across all databases of a cluster: there is only one copy of pg_database per cluster, not one per database.
Table 45-14. pg_database Columns
| Name | Type | References | Description |
|---|---|---|---|
| datname | name | Database name | |
| datdba | oid | pg_authid.oid | Owner of the database, usually the user who created it |
| encoding | int4 | Character encoding for this database
(pg_encoding_to_char() can translate
this number to the encoding name) | |
| datcollate | name | LC_COLLATE for this database | |
| datctype | name | LC_CTYPE for this database | |
| datistemplate | bool | If true then this database can be used in the TEMPLATE clause of CREATE DATABASE to create a new database as a clone of this one | |
| datallowconn | bool | If false then no one can connect to this database. This is used to protect the template0 database from being altered. | |
| datconnlimit | int4 | Sets maximum number of concurrent connections that can be made to this database. -1 means no limit. | |
| datlastsysoid | oid | Last system OID in the database; useful particularly to pg_dump | |
| datfrozenxid | xid | All transaction IDs before this one have been replaced with a permanent ("frozen") transaction ID in this database. This is used to track whether the database needs to be vacuumed in order to prevent transaction ID wraparound or to allow pg_clog to be shrunk. It is the minimum of the per-table pg_class.relfrozenxid values. | |
| dattablespace | oid | pg_tablespace.oid | The default tablespace for the database. Within this database, all tables for which pg_class.reltablespace is zero will be stored in this tablespace; in particular, all the non-shared system catalogs will be there. |
| datacl | aclitem[] | Access privileges; see GRANT and REVOKE for details |