Upgrade Kubernetes cluster from v1.23 to v1.24 or later

Prepare

1. Modify configmap kubeadm-config

kubectl -n kube-system edit cm kubeadm-config

## Delete the removed featuregate: TTLAfterFinished=true
## There are three places for feature-gates in this configmap.

2. Modify configmap kubelet-config-1.23

kubectl -n kube-system edit cm kubelet-config-1.23 

## Delete the removed featuregate: TTLAfterFinished=true

3. Change container runtime (all nodes)

   Notice: Kubernetes no longer supports docker by default from v1.24. So if your cluster is using docker as a container runtime, you need to change it, which can be contained, crio, iSula，cri-docker and other cri runtime.

   Change the containerd runtime can refer to https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim

   This page will use cri-dockerd as an example.

## 1. install cri-dockerd

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.4/cri-dockerd-0.3.4.amd64.tgz
tar -zxvf cri-dockerd-0.3.4.amd64.tgz && mv cri-dockerd/cri-dockerd  /usr/bin/


## 2. create systemd configuration for cri-dockerd 

cat << EOF > /etc/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com

[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --pod-infra-container-image docker.io/kubesphere/pause:3.8
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target

EOF

## 3. install crictl
VERSION="v1.26.0" # check latest version in /releases page
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz

## 4. create crictl configuration
cat << EOF > /etc/crictl.yaml
runtime-endpoint: unix:///var/run/cri-dockerd.sock
timeout: 5
debug: false
pull-image-on-create: false
EOF

## 5. start cri-docker
systemctl enable cri-docker --now

4. Migrate Docker Engine nodes from dockershim to cri-dockerd (all nodes)

## 1. Cordon the node to stop new Pods scheduling on it:
kubectl cordon <NODE_NAME>

## 2. Drain the node to safely evict running Pods:
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data

## 3. Configure the kubelet to use cri-dockerd
vim /var/lib/kubelet/kubeadm-flags.env
# Add flag: --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock
# Delete the removed flag: --network-plugin=cni  

## 4. Restart kubelet
systemctl restart kubelet

## 5. Change kubeadm.alpha.kubernetes.io/cri-socket from /var/run/dockershim.sock to unix:///var/run/cri-dockerd.sock
kubectl patch node <NODE_NAME> -p '{"metadata": {"annotations": {"kubeadm.alpha.kubernetes.io/cri-socket": "unix:///var/run/cri-dockerd.sock"}}}'

## 6. Verify that the node is healthy
#     To check whether the node uses the cri-dockerd endpoint, follow the instructions in [Find out which runtime you use](https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/). The --container-runtime-endpoint flag for the kubelet should be unix:///var/run/cri-dockerd.sock.

## 7. Uncordon the node to let Pods schedule on it
kubectl uncordon <NODE_NAME>

Upgrade

1. Update kubekey configuration

vim config-sample.yaml

## add config `spec.kubernetes.containerRuntimeEndpoint: unix:///var/run/cri-dockerd.sock`

2. Download v3.0.10 or later kubekey

curl -sSL https://get-kk.kubesphere.io | sh -

3. Start upgrade

./kk upgrade -f config-sample.yaml --with-kubernetes v1.26.5