Skip to content

1.28.2 升级和更新的操作步骤

标签
计算机/操作系统/Linux
操作系统/Debian
操作系统/Debian/Debian-11
操作系统/Unix
运维/云原生/Kubernetes/K8s
运维/云原生/Kubernetes
命令行/kubectl
软件/云原生/kubectl
命令行/kubeadm
软件/云原生/kubeadm
软件/云原生/kubelet
命令行/apt
计算机/网络/Cilium
运维/Cilium
命令行/cilium
软件/云原生/Cilium
字数
2113 字
阅读时间
12 分钟

先决条件

检查 kubeletkubeadmkubectl 的依赖状态

我们先检查一下当前依赖对应的 Repository 配置是否正确:

shell
sudo apt info kubelet
shell
sudo apt info kubeadm
shell
sudo apt info kubectl

一般而言,1.28 的集群对应的 apt Repository 应该都是

https://pkgs.k8s.io/core:/stable:/v1.28/deb

如果确定 apt 的 Repository 是预期内的结果,比如下面这样:

shell
$ sudo apt info kubectl
Package: kubectl
Version: 1.28.4-1.1
Priority: optional
Section: admin
Maintainer: Kubernetes Authors <[email protected]o>
Installed-Size: 49.9 MB
Homepage: https://kubernetes.io
Download-Size: 10.3 MB
APT-Sources: https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
Description: Command-line utility for interacting with a Kubernetes cluster
 Command-line utility for interacting with a Kubernetes cluster.

N: 4 条附加记录。请加上 ‘-a’ 参数来查看它们

那么我们再继续。

升级第一个控制平面节点的 kubeadm

请先参考通过 apt 升级和更新 1.28.4 版本的 kubeadm 完成这部分的步骤。

准备 kubeadm 升级计划

shell
sudo kubeadm upgrade plan

它会有这样的输出

shell
$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.28.4
[upgrade/versions] Target version: v1.28.4
W1201 12:27:33.656466    4866 version.go:104] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.28.txt": Get "https://cdn.dl.k8s.io/release/stable-1.28.txt": EOF
W1201 12:27:33.656531    4866 version.go:105] falling back to the local client version: v1.28.4
[upgrade/versions] Latest version in the v1.28 series: v1.28.4

W1201 12:27:34.039994    4866 configset.go:78] Warning: No kubeproxy.config.k8s.io/v1alpha1 config is loaded. Continuing without it: configmaps "kube-proxy" not found
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     2 x v1.28.2   v1.28.4
            1 x v1.28.4   v1.28.4

Upgrade to the latest version in the v1.28 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.28.0   v1.28.4
kube-controller-manager   v1.28.0   v1.28.4
kube-scheduler            v1.28.0   v1.28.4
kube-proxy                v1.28.0   v1.28.4
CoreDNS                   v1.10.1   v1.10.1
etcd                      3.5.9-0   3.5.9-0

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply v1.28.4

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   -                 v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

准备无误之后就可以升级了。

应用 kubeadm 升级

shell
sudo kubeadm upgrade apply v1.28.4

输出就像这样:

shell
$ sudo kubeadm upgrade apply v1.28.4
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1201 12:29:14.365811    5109 configset.go:78] Warning: No kubeproxy.config.k8s.io/v1alpha1 config is loaded. Continuing without it: configmaps "kube-proxy" not found
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.28.4"
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.28.4
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.28.4" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests829415407"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-01-12-30-22/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-01-12-30-22/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-01-12-30-22/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config2580934401/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
W1201 12:32:07.596874    5109 postupgrade.go:181] the ConfigMap "kube-proxy" in the namespace "kube-system" was not found. Assuming that kube-proxy was not deployed for this cluster. Note that once 'kubeadm upgrade apply' supports phases you will have to skip the kube-proxy upgrade manually

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.28.4". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

升级安装的 CNI:Cilium

我安装的是 1.14.2,所以应该可以直接升级到 1.14.4,先执行 cilium-preflight 的 Helm Chart 安装来对集群进行预检:

shell
sudo helm install cilium-preflight cilium/cilium \
	--version 1.14.4 \
	--namespace=kube-system \
	--set preflight.enabled=true \
	--set agent=false \
	--set operator.enabled=false

然后等待 Pod 状态正常

shell
sudo watch kubectl get pods -n kube-system

就像这样

shell
$ sudo kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS       AGE
cilium-762f6                               1/1     Running   1 (93m ago)    50d
cilium-8fv96                               1/1     Running   1 (97m ago)    50d
cilium-operator-657845c48c-lplcg           1/1     Running   5 (13m ago)    50d
cilium-pg92v                               1/1     Running   2 (92m ago)    50d
cilium-pre-flight-check-6cb5b47c6d-cd8dw   1/1     Running   0              14m
cilium-pre-flight-check-fpp7f              1/1     Running   0              14m
cilium-pre-flight-check-r5vts              1/1     Running   0              14m
cilium-pre-flight-check-vn9mf              1/1     Running   0              14m
coredns-5dd5756b68-k9qcl                   1/1     Running   1 (92m ago)    50d
coredns-5dd5756b68-rbjnm                   1/1     Running   1 (92m ago)    50d
etcd-node1                                 1/1     Running   24 (97m ago)   50d
hubble-relay-68b65978cb-82dsf              1/1     Running   1 (93m ago)    50d
hubble-ui-6cc5887659-ngf8z                 2/2     Running   2 (92m ago)    50d
kube-apiserver-node1                       1/1     Running   0              33m
kube-controller-manager-node1              1/1     Running   1 (13m ago)    32m
kube-scheduler-node1                       1/1     Running   1 (13m ago)    32m

等待 cilium-pre-flight-check Deployment 状态正常

shell
sudo watch kubectl get deployment -n kube-system cilium-pre-flight-check

就像这样

shell
$ sudo kubectl get deployment -n kube-system cilium-pre-flight-check
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cilium-pre-flight-check   1/1     1            1           14m

预检完成之后就可以删掉了:

shell
sudo helm delete cilium-preflight --namespace=kube-system

编辑我们之前在通过 Helm 安装 Cilium 的时候用的 values,并且添加一行:

yaml
cluster:
  name: kubernetes
upgradeCompatibility: "1.14"
enableIPv4Masquerade: true
enableIPv6Masquerade: true
hubble:
  relay:
    enabled: true
  ui:
    enabled: true
ipam:
  mode: kubernetes
  operator:
    clusterPoolIPv4PodCIDRList: 10.244.0.0/16
ipv4NativeRoutingCIDR: 10.244.0.0/16
k8sServiceHost: 10.24.0.2
k8sServicePort: 6443
kubeProxyReplacement: strict
operator:
  replicas: 1
serviceAccounts:
  cilium:
    name: cilium
  operator:
    name: cilium-operator
tunnel: vxlan

然后执行更新

shell
sudo helm upgrade cilium cilium/cilium \
	--version 1.14.4 \
	--namespace=kube-system \
	-f cilium-values-initial.yaml

然后等待资源准备完成:

shell
sudo watch kubectl get pods -n kube-system

完成之后再次执行 cilium status 查看状态

shell
$ sudo cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
Containers:            cilium             Running: 3
                       cilium-operator    Running: 1
                       hubble-ui          Running: 1
                       hubble-relay       Running: 1
Cluster Pods:          8/8 managed by Cilium
Helm chart version:    1.14.4
Image versions         hubble-relay       quay.io/cilium/hubble-relay:v1.14.4@sha256:ca81622fd9f04c1316bf4144bde5dbce613758810f6022f6c706b14c9c0815db: 1
                       cilium             quay.io/cilium/cilium:v1.14.4@sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.4@sha256:f0f05e4ba3bb1fe0e4b91144fa4fea637701aba02e6c00b23bd03b4a7e1dfd55: 1
                       hubble-ui          quay.io/cilium/hubble-ui:v0.12.1@sha256:9e5f81ee747866480ea1ac4630eb6975ff9227f9782b7c93919c081c33f38267: 1
                       hubble-ui          quay.io/cilium/hubble-ui-backend:v0.12.1@sha256:1f86f3400827a0451e6332262467f894eeb7caf0eb8779bd951e2caa9d027cbe: 1

升级其他控制平面节点的 kubeadm

和之前升级第一个控制平面节点不同的是,我们需要用下面的命令来升级其他剩下的控制平面节点:

shell
sudo kubeadm upgrade node

逐步升级所有控制平面的节点的 kubeletkubectl

驱逐 Pod

shell
sudo kubectl drain <node-to-drain> --ignore-daemonsets

对于我而言的话,按步骤执行

shell
sudo kubectl drain node1 --ignore-daemonsets
sudo kubectl drain node2 --ignore-daemonsets
sudo kubectl drain node3 --ignore-daemonsets

就好了。

shell
$ sudo kubectl drain node1 --ignore-daemonsets
node/node2 cordoned
error: unable to drain node "node2" due to error:cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/hubble-ui-6f48889749-zhgx9, continuing command...
There are pending nodes to be drained:
 node2
cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/hubble-ui-6f48889749-zhgx9

如果执行的时候遇到上面这样的问题,可以通过

shell
sudo kubectl drain node2 --ignore-daemonsets --delete-emptydir-data

来尝试解决。

请先参考通过 apt 升级和更新 1.28.4 版本的 kubelet 和 kubectl 完成这部分的步骤。

取消驱逐

shell
sudo kubectl uncordon node1

检查节点状态,看看版本是否对齐:

shell
$ sudo kubectl get nodes
NAME    STATUS   ROLES           AGE   VERSION
node1   Ready    control-plane   50d   v1.28.4
node2   Ready    <none>          50d   v1.28.2
node3   Ready    <none>          50d   v1.28.2

状态为 Ready 就表示完成了。

升级负载节点

升级 kubeadm

请先参考通过 apt 升级和更新 1.28.4 版本的 kubeadm 完成这部分的步骤。

shell
sudo kubeadm upgrade node

驱逐 Pod

shell
sudo kubectl drain node2 --ignore-daemonsets

升级 kubeletkubectl

请先参考通过 apt 升级和更新 1.28.4 版本的 kubelet 和 kubectl 完成这部分的步骤。

取消驱逐

sudo kubectl uncordon node2

检查节点状态,看看版本是否对齐:

shell
$ sudo kubectl get nodes
NAME    STATUS   ROLES           AGE   VERSION
node1   Ready    control-plane   50d   v1.28.4
node2   Ready    <none>          50d   v1.28.4
node3   Ready    <none>          50d   v1.28.4

状态为 Ready 就表示完成了。

贡献者

文件历史

撰写