CKA Prep - Cluster Architecture, Installation and Configuration

This post is part of a series which contains my study notes for the Certified Kubernetes Administrator (CKA) exam.

Note: Unless specifically indicated, text and examples in this post all come directly from the official Kubernetes documentation. I attempted to locate and extract the relevant portions of the kubernetes.io documentation that applied to the exam objective. However, I encourage you to do your own reading. I cannot guarantee that I got all of the important sections.

Cluster Architecture, Installation & Configuration

The Exam Curriculum breaks down the first exam topic into the following objectives:

• Manage role based access control (RBAC)
• Use Kubeadm to install a basic cluster
• Manage a highly-available Kubernetes cluster
• Provision underlying infrastructure to deploy a Kubernetes cluster
• Perform a version upgrade on a Kubernetes cluster using Kubeadm
• Implement etcd backup and restore

Manage Role Based Access Control (RBAC)

Relevant search terms for Kubernetes Documentation: RBAC, service account

Kubernetes Documentation Links

• Using RBAC Authorization
• Configure service accounts for Pods
• Authorization Overview
• Certificate Signing Requests - Normal User

API Objects

• Role - contains rules that represent a set of permissions within a particular namespace
• ClusterRole - contains rules that represent a set of permissions that can be applied within a given namespace or cluster wide depending on whether it is connected to a subject via a RoleBinding or ClusterRoleBinding
• RoleBinding - grants the permissions defined in a role to users, groups, or service accounts in the specified namespace
• ClusterRoleBinding - grants the permissions defined in a clusterrole to users, groups, or service accounts across the entire cluster
• ServiceAccount - (shortname sa) A service account provides an identity for processes that run in a Pod.
• CertificateSigningRequest - used to request that a certificate be signed by a denoted signer, after which the request may be approved or denied before finally being signed.

Concepts

• “A Role Binding may reference any Role in the same namespace. Alternatively, a Role Binding can reference a Cluster Role and bind that Cluster Role to the namespace of the Role Binding. If you want to bind a Cluster Role to all the namespaces in your cluster, you use a Cluster Role Binding.”
• There are three supported combinations of Roles, Role Bindings, Cluster Roles, and Cluster Role Bindings.
  • Create a Role in a namespace and connect a subject with it using a Role Binding. In this case, the Role only exists in a specific namespace and the permissions that are assigned to the role may only be exercised by the subject in the specified namespace.
  • Create a Cluster Role and connect a subject with it using a Cluster Role Binding. In this case, the Cluster Role is a cluster-level entity and the permissions that are assigned to the Cluster Role via the Cluster Role Binding may be exercised on cluster level resources and on namespaced resources in any namespace.
  • Create a Cluster Role and connect a subject with it using a Role Binding. In this case, the Cluster Role is a cluster-level entity; however, the permissions that are assigned to the Cluster Role via the Role Binding may only be exercised in the namespace where the Role Binding has been created.
• As indicated above, bindings can be applied to users, groups, or service accounts. These are called subjects. service accounts are always created within a namespace. Therefore, when you bind a service account to a Role or Cluster Role you must specify the namespace and the service account (e.g., my-namespace:my-sa).
• “After you create a binding, you cannot change the Role or Cluster Role that it refers to. If you try to change a binding’s roleRef, you get a validation error. If you do want to change the roleRef for a binding, you need to remove the binding object and create a replacement.”
• When you create a binding with kubectl, the system does not verify the validity of the role or cluster role you specify. If you find that you specified an incorrect Role or Cluster Role then you will need to delete and then recreate the binding.
• “You can aggregate several Cluster Roles into one combined Cluster Role. A controller, running as part of the cluster control plane, watches for Cluster Role objects with an aggregationRule set. The aggregationRule defines a label selector that the controller uses to match other Cluster Role objects that should be combined into the rules field of this one.”
• A default service account is created in each namespace; this service account is automatically assigned to each pod created in the namespace unless another service account is specified when creating the pod.
• The token for the service account is automatically stored as a Kubernetes secret in the same namespace. This token is mounted inside the pod when the pod is created and can be used by processes running in the pod to access the Kubernetes API.
• “A few steps are required in order to get a normal user to be able to authenticate and invoke an API. First, this user must have a certificate issued by the Kubernetes cluster, and then present that certificate to the Kubernetes API.”
  • Create a private key
  • Create a Kubernetes Certificate Signing Request
  • Approve the Kubernetes Certificate Signing Request
  • Get the certificate from Kubernetes
  • Create a Role or Cluster Role and Role Binding or Cluster Role Binding
  • Validate the permissions using kubectl auth can-i <API ACTION> --as <USER>
  • Optionally, add the new credentials to the Kube.config file

Relevant Commands

Working with Roles

List and describe roles in the namespace “default”

# list the roles
kubectl get roles -n default
# describe role "web-app"
kubectl describe role web-app -n default

Delete Role pod-reader

kubectl delete role pod-reader

Create a Role called “pod-reader” with access to get, list, and watch pods in the default namespace

kubectl create role pod-reader \
--verb=get --verb=list --verb=watch --resource=pods

Create a Role with resource specified**

kubectl create role pod-reader \
--verb=get --resource=pods \
--resource-name=readablepod --resource-name=anotherpod

Create Role with api groups specified

kubectl create role foo \
--verb=get,list,watch --resource=replicasets.apps

Create Role with subresource permissions

kubectl create role foo \
--verb=get,list,watch --resource=pods,pods/status

Working with Role Bindings

List and describe Role Bindings in the namespace “default”

# list role bindings
kubectl get rolebindings -n default
# describe the role dinding pod-reader-binding
kubectl describe rolebinding pod-reader-binding -n default

Delete Role Binding “pod-reader-binding”

kubectl delete rolebinding pod-reader-binding

Create Role Binding in namespace “acme” to bind user “bob” to a Cluster Role called “admin”

kubectl create rolebinding bob-admin-binding \
--clusterrole=admin --user=bob --namespace=acme

Create Role Binding in namespace “acme” to bind service account “myapp” to Cluster Role “view”

kubectl create rolebinding myapp-view-binding --clusterrole=view \
--serviceaccount=acme:myapp --namespace=acme

Create Role Binding in namespace “acme” to bind service account “web” to Role “foo”

kubectl create rolebinding web-foo-binding --role=foo \
--serviceaccount=web  --namespace=acme

Working with Cluster Roles

List and describe Cluster Roles

# list the cluster roles
kubectl get clusterroles
# describe the cluster role "pod-reader"
kubectl describe clusterrole pod-reader

Delete Cluster Role “pod-reader”

kubectl delete clusterrole pod-reader

Create Cluster Role

kubectl create clusterrole pod-reader \
--verb=get,list,watch --resource=pods

Create Cluster Role with an aggregation rule specified

kubectl create clusterrole monitoring \
--aggregation-rule="rbac.example.com/aggregate-to-monitoring=true"  

Working with Cluster Role Bindings

List and describe Cluster Role Bindings

# List cluster role bindings
kubectl get clusterrolebindings
# describe the cluster role binding root-cluster-admin-binding
kubectl describe clusterrolebinding root-cluster-admin-binding

Delete Cluster Role Binding “root-cluster-admin-binding”

kubectl delete clusterrole root-cluster-admin-binding

Create Cluster Role Binding to grant user “root” permission to the Cluster Role “cluster-admin” across the entire cluster

kubectl create clusterrolebinding root-cluster-admin-binding \
--clusterrole=cluster-admin --user=root

Create Cluster Role Binding to grant service account “myapp” in namespace “acme” permission to the Cluster Role “view” across the entire cluster

kubectl create clusterrolebinding myapp-view-binding \
--clusterrole=view --serviceaccount=acme:myapp

Working with service accounts

List and describe service accounts

# list the service accounts in the current namespace
kubectl get serviceaccounts
# or use the shortname
kubectl get sa
# describe the service account web-app
kubectl describe sa web-app

Delete service account “web-app”

kubectl delete serviceaccount web-app
# or use the shortname
kubectl delete sa web-app

Create service account web-app

kubectl create serviceaccount web-app
# or use the shortname
kubectl create sa web-app

Start a Pod using a service account called “web-app”

kubectl run nginx --image=nginx --port=80 \
--serviceaccount=web-app

Validating RBAC Permissions

Verify whether the service account “web-app” in the default namespace has permissions to list pods

# returns yes or no
kubectl auth can-i get pods \
--as system:serviceaccount:default:web-app

Verify whether user “dave” has permissions to list Secrets in the namespace “dev”

# returns yes or no
kubectl auth can-i list secrets --namespace dev --as dave

Setup Kubernetes Authentication for a Normal User

As described above, the following steps must be completed to configure a normal user with kubernetes API access.

Create a Private Key

“The following scripts show how to generate PKI private key and CSR. It is important to set CN and O attribute of the CSR. CN is the name of the user and O is the group that this user will belong to. You can refer to RBAC for standard groups.”

openssl genrsa -out myuser.key 2048
openssl req -new -key myuser.key -out myuser.csr

Create a Kubernetes Certificate Signing Request

Before a CertificateSigningRequest can be submitted to Kubernetes, a base64 encoded version of the CSR must be generated using the following command:

cat myuser.csr | base64 | tr -d "\n"

The base64 encoded CSR must then be incorporated into the CertificateSigningRequest as the request parameter in the spec.

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: myuser
spec:
  request: # base64 encoded CSR HERE!
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400  # one day
  usages:
  - client auth

• usages has to be ‘client auth’
• expirationSeconds could be made longer (i.e. 864000 for ten days) or shorter (i.e. 3600 for one hour)
• request is the base64 encoded value of the CSR file content.

To submit the request, first save the file with a name like certsigningrequest.yaml. After the file has been saved, create the request using the kubectl apply -f certsigningrequest.yaml command.

Approve the Kubernetes Certificate Signing Request

Get the list of CSRs:

kubectl get csr

Approve the CSR:

kubectl certificate approve myuser

Get the certificate from Kubernetes

The certificate value is in Base64-encoded format under status.certificate in the approved CSR.

Export the issued certificate from the CertificateSigningRequest.

kubectl get csr myuser -o \
jsonpath='{.status.certificate}'| base64 -d > myuser.crt

Create a Role or Cluster Role and Role Binding or Cluster Role Binding

With the certificate created it is time to define the Role and RoleBinding for this user to access Kubernetes cluster resources.

This is a sample command to create a Role for this new user:

kubectl create role developer --verb=create,get,list,update,delete \
--resource=pods

This is a sample command to create a RoleBinding for this new user:

kubectl create rolebinding developer-binding-myuser \
--role=developer --user=myuser

Validate Permissions Using the kubectl auth can-i Command

After the Role and RoleBinding have been created, verify that the newly created user has the required permissions. The following is an example:

kubectl auth can-i list pods --as john.smith

Optionally, Add the New Credential to the Kube.config File

First, you need to add new credentials:

kubectl config set-credentials myuser --client-key=myuser.key \
--client-certificate=myuser.crt \
--embed-certs=true

Then, you need to add the context:

kubectl config set-context myuser \
--cluster=kubernetes --user=myuser

To test it, change the context to myuser:

kubectl config use-context myuser

top

Provision and Upgrade Kubernetes Clusters

This section will address the following exam objectives as they are all interrelated:

• Use Kubeadm to install a basic cluster
• Manage a highly-available Kubernetes cluster
• Provision underlying infrastructure to deploy a Kubernetes cluster
• Perform a version upgrade on a Kubernetes cluster using Kubeadm

Relevant search terms for Kubernetes Documentation: kubeadm, create cluster, upgrade cluster, kubeadm init

Kubernetes Documentation Links

• Creating a cluster with kubeadm
• Creating Highly Available Clusters with kubeadm
• kubeadm init
• Cluster Networking
• Upgrading kubeadm clusters

API Objects

• Node - a virtual or physical machine that runs pods. When you provision or upgrade clusters you can run the kubetcl get nodes command to check the status of the nodes.

Concepts

• The following are the high level steps for creating a cluster using kubeadm:

  • Initialize the cluster by running kubeadm init <options> on the control plane node
  • Install a pod network add-on (CNI)
  • join another node to the cluster by remoting to the node using ssh and then running the kubeadm join <options> command which was displayed when you originally initialized the cluster using kubeadm init

• The following are the high level steps for upgrading a cluster using kubeadm

  • Upgrade the control plane node

    • Check current kubernetes versions available for upgrade
    • Update the version of kubeadm on the control plane node to match the selected upgrade version of Kubernetes
    • Plan and then apply the upgrade
    • Drain and cordon the control plane node so that you can safely update the kubelet
    • Update the version of kubeadm and kubelet to match the selected upgrade version of Kubernetes
    • Restart the kubelet process
    • Uncordon the control plane node so that pods can be scheduled on it again
    • Verify the control plane node updated successfully

  • Upgrade worker nodes

    • Update the version of kubeadm on the worker node to match the selected upgrade version of Kubernetes
    • Upgrade the worker node
    • Drain and cordon the worker node so that you can safely update the kubelet
    • Update the version of kubeadm and kubelet to match the selected upgrade version of Kubernetes
    • Restart the kubelet process
    • Uncordon the worker node so that pods can be scheduled on it again
    • Verify the worker node updated successfully

Relevant Commands

Use Kubeadm to Create a Cluster with a Control Plane Node and a Worker Node

Step 1: Initialize the Control Plane Node

# Run Kubeadm on the control plane node
kubeadm init --pod-network-cidr=10.200.0.0/16 \
 --kubernetes-version=1.24.3

After this command completes successfully, you will see output that looks like this:

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as
a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options
listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the
following on each as root:

kubeadm join 10.200.0.3:6443 --token fzymy3.3yh9kjtdryupifgl \
        --discovery-token-ca-cert-hash \
        sha256:e6af09985bd91bbd43eac39b4c0d4d250f . . . 

Follow the directions in the output to get the /.kube/config file setup on the host.

Step 2: Install a Pod Network Add-On (CNI)

NOTE: The kuberernetes.io documentation has links to several different Pod Network Add-ons. However, installation steps are not documented on kubernetes.io.

Step 3: Join Another Node to the Cluster

Copy the Node command provided in the output from kubeadm init and then ssh into the new node and run the command:

kubeadm join 10.0.0.11:6443 --token fzymy3.3yh9kjtdryupifgl \
        --discovery-token-ca-cert-hash

Perform a Version Upgrade on a Kubernetes Cluster Using Kubeadm

Step 1: Check Current Kubernetes Version Prior to Upgrade

Verify the current kubernetes version on all nodes in the cluster:

kubectl get nodes

Step 2: Determine Available Versions for the Upgrade

Update the apt cache and then check the available versions of kubeadm:

# update the apt cache
sudo apt update
# query for latest kubeadm releases
sudo apt-cache madison kubeadm

Step 3: Upgrade the Control Plane Node

Upgrade kubeadm:

 # replace x in 1.24.x-00 with the latest patch version
 apt-mark unhold kubeadm && \
 apt-get update && \
 apt-get install -y kubeadm=1.24.x-00 && \
 apt-mark hold kubeadm

Verify that kubeadm was updated successfully:

kubeadm version

Verify the upgrade plan:

kubeadm upgrade plan

After verification, run the upgrade:

# replace x with the patch version you picked for this upgrade
sudo kubeadm upgrade apply v1.24.x

Drain the control plane node so that you can safely update the kubelet:

# replace <node-to-drain> with the name of your node you are draining
kubectl drain <node-to-drain> --ignore-daemonsets

After the command to drain the node is complete, update kubelet and kubectl:

# replace x in 1.24.x-00 with the latest patch version
apt-mark unhold kubelet kubectl && \
apt-get update && \
apt-get install -y kubelet=1.24.x-00 kubectl=1.24.x-00 && \
apt-mark hold kubelet kubectl

Then restart the kubelet:

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Uncordon the control plane node so that pods can be scheduled on it again:

# replace <node-to-drain> with the name of your node
kubectl uncordon <node-to-drain>

Run the kubectl get nodes command and check the version of the control plane node to verify that the control plane node update was successful.

NOTE: After the upgrade of the first control plane node is finished, you will need to upgrade the remaining control plane nodes if there are any using the sudo kubeadm upgrade node command.

Step 4: Upgrade Each Worker Node

Connect to the worker node using ssh and then upgrade kubeadm:

# replace x in 1.24.x-00 with the latest patch version
apt-mark unhold kubeadm && \
apt-get update && \
apt-get install -y kubeadm=1.24.x-00 && \
apt-mark hold kubeadm

Verify that kubeadm was updated successfully:

kubeadm version

Upgrade the worker node:

sudo kubeadm upgrade node

Drain the worker node so that you can safely update the kubelet:

# replace <node-to-drain> with the name of your node you are draining
kubectl drain <node-to-drain> --ignore-daemonsets

After the command to drain the node is complete, update the kubelet and kubectl:

# replace x in 1.24.x-00 with the latest patch version
apt-mark unhold kubelet kubectl && \
apt-get update && \
apt-get install -y kubelet=1.24.x-00 kubectl=1.24.x-00 && \
apt-mark hold kubelet kubectl

Restart the kubelet:

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Uncordon the worker node so that pods can be scheduled on it again:

# replace <node-to-drain> with the name of your node
kubectl uncordon <node-to-drain>

Run the kubectl get nodes command and check the version of the worker node to verify that the control plane node update was successful.

top

Implement etcd Backup and Restore

Relevant search terms for Kubernetes Documentation: etcd

Kubernetes Documentation Links

• Operating etcd clusters for Kubernetes
• Options for Highly Available Topology
• Create static Pods

Note: The documentation regarding etcd backup and restore in the kubernetes documentation is limited. The process for restoring the etcd snapshot to a new file, updating the etcd.yaml manifrest file to point to the restored snapshot, and then restarting the kubelet service came from the Certified Kubernetes Administrator (CKA) Study Guide by Benjamin Muschko.

Concepts

• According to the Kubernetes documentation:

  You can set up an HA cluster:

  • With stacked control plane nodes, where etcd nodes are colocated with control plane nodes
  • With external etcd nodes, where etcd runs on separate nodes from the control plane

• Typically, when the etcd nodes are collocated with the control plane nodes then each etcd node is running as a static pod on each control plane node.

• “Static Pods are managed directly by the kubelet daemon on a specific node, without the API server observing them. Unlike Pods that are managed by the control plane (for example, a Deployment); instead, the kubelet watches each static Pod (and restarts it if it fails).”

• The default path for static pods is /etc/kubernetes/manifests/. The default name for the etcd pod manifest is etcd.yaml. The parameters needed to run the etcdctl snapshot save command can be found in the file.

• To restore an etcd snapshot, first restore the snapshot to a new folder, update the etcd.yaml file to point to the newly restored snapshot, then restart the kubelet.

Relevant Commands

• Review the spec section of the /etc/kubernetes/manifests/etcd.yaml file:

  $ cat /etc/kubernetes/manifests/etcd.yaml
  
  . . . 
  spec:
    containers:
    - command:
      - etcd
      - --advertise-client-urls=https://10.69.191.9:2379
      - --cert-file=/etc/kubernetes/pki/etcd/server.crt
      - --client-cert-auth=true
      - --data-dir=/var/lib/etcd
      - --initial-advertise-peer-urls=https://10.69.191.9:2380
      - --initial-cluster=controlplane=https://10.69.191.9:2380
      - --key-file=/etc/kubernetes/pki/etcd/server.key
      - --listen-client-urls=https://127.0.0.1:2379,https://10.69.191.9:2379
      - --listen-metrics-urls=http://127.0.0.1:2381
      - --listen-peer-urls=https://10.69.191.9:2380
      - --name=controlplane
      - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
      - --peer-client-cert-auth=true
      - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
      - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
      - --snapshot-count=10000
      - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
  . . . 
  

• The parameters from the etcd command in the etcd.yaml must be combined with the following template command for taking an etcd snapshot from the kubernetes.io documentation:

  ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
    --cacert=<trusted-ca-file> \
    --cert=<cert-file> \
    --key=<key-file> \
    snapshot save <backup-file-location>
  

  The following table details the mapping:

• Combining the values from the etcd.yaml file with the template command yields the following result:

  ETCDCTL_API=3 etcdctl \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    snapshot save <backup-file-location>
  

• After taking an etcd snapshot, use the following command to validate it:

  ETCDCTL_API=3 etcdctl --write-out=table \
  snapshot status <backup-file-location>
  

• In order to restore a snapshot, first use the etcdctl snapshot restore command:

  ETCDCTL_API=3 etcdctl --data-dir <new-data-dir-location> \
  snapshot restore <backup-file-location>
  

• Next, update the path for the etcd-data hostPath volume in the /etc/kubernetes/manifests/etcd.yaml file:

  . . . 
    volumes:
    - hostPath:
        path: /etc/kubernetes/pki/etcd
        type: DirectoryOrCreate
      name: etcd-certs
    - hostPath:
        path: <PATH TO SNAPSHOT>
        type: DirectoryOrCreate
      name: etcd-data
  . . .
  

• Save the etcd.yaml file after making the change and then restart the kubelet service:

  sudo systemctl restart kubelet.service
  

• After restarting the kubelet service, check the status of the etcd-controlplane pod:

  kubectl get pods -n kube-system
  

• If the pod doesn’t transition to a Running state then delete the pod by running:

  kubectl delete pod etcd-controlplane -n kube-system
  

• The recreated pod should transition to a Running state.

top

And that’s a wrap for this topic.

Return to the series overview page