1.28.2 升级和更新的操作步骤
标签
计算机/操作系统/Linux
操作系统/Debian
操作系统/Debian/Debian-11
操作系统/Unix
运维/云原生/Kubernetes/K8s
运维/云原生/Kubernetes
命令行/kubectl
软件/云原生/kubectl
命令行/kubeadm
软件/云原生/kubeadm
软件/云原生/kubelet
命令行/apt
计算机/网络/Cilium
运维/Cilium
命令行/cilium
软件/云原生/Cilium
字数
2104 字
阅读时间
12 分钟
先决条件
检查 kubelet，kubeadm 和 kubectl 的依赖状态
我们先检查一下当前依赖对应的 Repository 配置是否正确：
shell
sudo apt info kubeletshell
sudo apt info kubeadmshell
sudo apt info kubectl一般而言，1.28 的集群对应的 apt Repository 应该都是
https://pkgs.k8s.io/core:/stable:/v1.28/deb如果确定 apt 的 Repository 是预期内的结果，比如下面这样：
shell
$ sudo apt info kubectl
Package: kubectl
Version: 1.28.4-1.1
Priority: optional
Section: admin
Maintainer: Kubernetes Authors <[email protected]o>
Installed-Size: 49.9 MB
Homepage: https://kubernetes.io
Download-Size: 10.3 MB
APT-Sources: https://pkgs.k8s.io/core:/stable:/v1.28/deb Packages
Description: Command-line utility for interacting with a Kubernetes cluster
Command-line utility for interacting with a Kubernetes cluster.
N: 有 4 条附加记录。请加上 ‘-a’ 参数来查看它们那么我们再继续。
升级第一个控制平面节点的 kubeadm
请先参考通过 apt 升级和更新 1.28.4 版本的 kubeadm 完成这部分的步骤。
准备 kubeadm 升级计划
shell
sudo kubeadm upgrade plan它会有这样的输出
shell
$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.28.4
[upgrade/versions] Target version: v1.28.4
W1201 12:27:33.656466 4866 version.go:104] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.28.txt": Get "https://cdn.dl.k8s.io/release/stable-1.28.txt": EOF
W1201 12:27:33.656531 4866 version.go:105] falling back to the local client version: v1.28.4
[upgrade/versions] Latest version in the v1.28 series: v1.28.4
W1201 12:27:34.039994 4866 configset.go:78] Warning: No kubeproxy.config.k8s.io/v1alpha1 config is loaded. Continuing without it: configmaps "kube-proxy" not found
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 2 x v1.28.2 v1.28.4
1 x v1.28.4 v1.28.4
Upgrade to the latest version in the v1.28 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.28.0 v1.28.4
kube-controller-manager v1.28.0 v1.28.4
kube-scheduler v1.28.0 v1.28.4
kube-proxy v1.28.0 v1.28.4
CoreDNS v1.10.1 v1.10.1
etcd 3.5.9-0 3.5.9-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.28.4
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io - v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________准备无误之后就可以升级了。
应用 kubeadm 升级
shell
sudo kubeadm upgrade apply v1.28.4输出就像这样：
shell
$ sudo kubeadm upgrade apply v1.28.4
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1201 12:29:14.365811 5109 configset.go:78] Warning: No kubeproxy.config.k8s.io/v1alpha1 config is loaded. Continuing without it: configmaps "kube-proxy" not found
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.28.4"
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.28.4
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.28.4" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests829415407"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-01-12-30-22/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-01-12-30-22/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-01-12-30-22/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config2580934401/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
W1201 12:32:07.596874 5109 postupgrade.go:181] the ConfigMap "kube-proxy" in the namespace "kube-system" was not found. Assuming that kube-proxy was not deployed for this cluster. Note that once 'kubeadm upgrade apply' supports phases you will have to skip the kube-proxy upgrade manually
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.28.4". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.升级安装的 CNI：Cilium
我安装的是 1.14.2，所以应该可以直接升级到 1.14.4，先执行 cilium-preflight 的 Helm Chart 安装来对集群进行预检：
shell
sudo helm install cilium-preflight cilium/cilium \
--version 1.14.4 \
--namespace=kube-system \
--set preflight.enabled=true \
--set agent=false \
--set operator.enabled=false然后等待 Pod 状态正常
shell
sudo watch kubectl get pods -n kube-system就像这样
shell
$ sudo kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
cilium-762f6 1/1 Running 1 (93m ago) 50d
cilium-8fv96 1/1 Running 1 (97m ago) 50d
cilium-operator-657845c48c-lplcg 1/1 Running 5 (13m ago) 50d
cilium-pg92v 1/1 Running 2 (92m ago) 50d
cilium-pre-flight-check-6cb5b47c6d-cd8dw 1/1 Running 0 14m
cilium-pre-flight-check-fpp7f 1/1 Running 0 14m
cilium-pre-flight-check-r5vts 1/1 Running 0 14m
cilium-pre-flight-check-vn9mf 1/1 Running 0 14m
coredns-5dd5756b68-k9qcl 1/1 Running 1 (92m ago) 50d
coredns-5dd5756b68-rbjnm 1/1 Running 1 (92m ago) 50d
etcd-node1 1/1 Running 24 (97m ago) 50d
hubble-relay-68b65978cb-82dsf 1/1 Running 1 (93m ago) 50d
hubble-ui-6cc5887659-ngf8z 2/2 Running 2 (92m ago) 50d
kube-apiserver-node1 1/1 Running 0 33m
kube-controller-manager-node1 1/1 Running 1 (13m ago) 32m
kube-scheduler-node1 1/1 Running 1 (13m ago) 32m等待 cilium-pre-flight-check Deployment 状态正常
shell
sudo watch kubectl get deployment -n kube-system cilium-pre-flight-check就像这样
shell
$ sudo kubectl get deployment -n kube-system cilium-pre-flight-check
NAME READY UP-TO-DATE AVAILABLE AGE
cilium-pre-flight-check 1/1 1 1 14m预检完成之后就可以删掉了：
shell
sudo helm delete cilium-preflight --namespace=kube-system编辑我们之前在通过 Helm 安装 Cilium 的时候用的 values，并且添加一行：
yaml
cluster:
name: kubernetes
upgradeCompatibility: "1.14"
enableIPv4Masquerade: true
enableIPv6Masquerade: true
hubble:
relay:
enabled: true
ui:
enabled: true
ipam:
mode: kubernetes
operator:
clusterPoolIPv4PodCIDRList: 10.244.0.0/16
ipv4NativeRoutingCIDR: 10.244.0.0/16
k8sServiceHost: 10.24.0.2
k8sServicePort: 6443
kubeProxyReplacement: strict
operator:
replicas: 1
serviceAccounts:
cilium:
name: cilium
operator:
name: cilium-operator
tunnel: vxlan然后执行更新
shell
sudo helm upgrade cilium cilium/cilium \
--version 1.14.4 \
--namespace=kube-system \
-f cilium-values-initial.yaml然后等待资源准备完成：
shell
sudo watch kubectl get pods -n kube-system完成之后再次执行 cilium status 查看状态
shell
$ sudo cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode)
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
Containers: cilium Running: 3
cilium-operator Running: 1
hubble-ui Running: 1
hubble-relay Running: 1
Cluster Pods: 8/8 managed by Cilium
Helm chart version: 1.14.4
Image versions hubble-relay quay.io/cilium/hubble-relay:v1.14.4@sha256:ca81622fd9f04c1316bf4144bde5dbce613758810f6022f6c706b14c9c0815db: 1
cilium quay.io/cilium/cilium:v1.14.4@sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e: 3
cilium-operator quay.io/cilium/operator-generic:v1.14.4@sha256:f0f05e4ba3bb1fe0e4b91144fa4fea637701aba02e6c00b23bd03b4a7e1dfd55: 1
hubble-ui quay.io/cilium/hubble-ui:v0.12.1@sha256:9e5f81ee747866480ea1ac4630eb6975ff9227f9782b7c93919c081c33f38267: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.12.1@sha256:1f86f3400827a0451e6332262467f894eeb7caf0eb8779bd951e2caa9d027cbe: 1升级其他控制平面节点的 kubeadm
和之前升级第一个控制平面节点不同的是，我们需要用下面的命令来升级其他剩下的控制平面节点：
shell
sudo kubeadm upgrade node逐步升级所有控制平面的节点的 kubelet 和 kubectl
驱逐 Pod
shell
sudo kubectl drain <node-to-drain> --ignore-daemonsets对于我而言的话，按步骤执行
shell
sudo kubectl drain node1 --ignore-daemonsets
sudo kubectl drain node2 --ignore-daemonsets
sudo kubectl drain node3 --ignore-daemonsets就好了。
shell
$ sudo kubectl drain node1 --ignore-daemonsets
node/node2 cordoned
error: unable to drain node "node2" due to error:cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/hubble-ui-6f48889749-zhgx9, continuing command...
There are pending nodes to be drained:
node2
cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/hubble-ui-6f48889749-zhgx9如果执行的时候遇到上面这样的问题，可以通过
shell
sudo kubectl drain node2 --ignore-daemonsets --delete-emptydir-data来尝试解决。
请先参考通过 apt 升级和更新 1.28.4 版本的 kubelet 和 kubectl 完成这部分的步骤。
取消驱逐
shell
sudo kubectl uncordon node1检查节点状态，看看版本是否对齐：
shell
$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready control-plane 50d v1.28.4
node2 Ready <none> 50d v1.28.2
node3 Ready <none> 50d v1.28.2状态为 Ready 就表示完成了。
升级负载节点
升级 kubeadm
请先参考通过 apt 升级和更新 1.28.4 版本的 kubeadm 完成这部分的步骤。
shell
sudo kubeadm upgrade node驱逐 Pod
shell
sudo kubectl drain node2 --ignore-daemonsets升级 kubelet 和 kubectl
请先参考通过 apt 升级和更新 1.28.4 版本的 kubelet 和 kubectl 完成这部分的步骤。
取消驱逐
sudo kubectl uncordon node2检查节点状态，看看版本是否对齐：
shell
$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready control-plane 50d v1.28.4
node2 Ready <none> 50d v1.28.4
node3 Ready <none> 50d v1.28.4状态为 Ready 就表示完成了。
絢香猫