目录

Kubernetes集群(v1.28)搭建部署实践

本文记录了在作者本地使用三台CentOS虚拟机,搭建部署单Master双Slave Kubernetes集群。

题图源:Understanding Your Kubernetes Deployment Lifecycle

1 系统准备

搭建部署包含1个Master节点和2个Slave节点的Kubernetes集群,需要准备3台Linux虚拟机作为集群节点,可以通过VMware Worksation快速创建三台CentOS虚拟机。

1.1 工具安装

1.2 网络配置

使用VMware工具本身的界面操作起来并不方便,对CentOS虚拟机固定IP后,可使虚拟机每次启动时IP不会发生变化,可以使用XShellTabby通过SSH访问虚拟机,进行后续的配置操作,会节省很多时间。

此配置保证后续Kubernetes集群节点IP不会随意改变,这里不做过多介绍,具体可参考这篇文章:Vmware虚拟机Linux配置固定IP地址(详细版)

通过以上配置,在各个节点执行ip addr获取到三个节点的IP信息:

192.168.198.129
192.168.198.130
192.168.198.131

1.3 CentOS系统配置

1.3.1 主机名配置

设置192.168.198.129主机名 apple

1
hostnamectl set-hostname apple

设置192.168.198.130主机名 banana

1
hostnamectl set-hostname banana

设置192.168.198.131主机名 cherry

1
hostnamectl set-hostname cherry

1.3.2 配置主机IP地址

设置Apple IP地址为 192.168.198.129

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
vim /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="529588ee-2c22-4dd4-99c1-c823fbf96642"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.198.129
GATEWAY=192.168.198.2
NETMASK=255.255.255.0

设置Banana IP地址为 192.168.198.130

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
vim /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="529588ee-2c22-4dd4-99c1-c823fbf96642"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.198.130
GATEWAY=192.168.198.2
NETMASK=255.255.255.0

设置Cherry IP地址为 192.168.198.131

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
vim /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="529588ee-2c22-4dd4-99c1-c823fbf96642"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.198.131
GATEWAY=192.168.198.2
NETMASK=255.255.255.0

1.3.3 配置DNS网络信息

在每个节点执行

1
2
3
4
5
6
# 缺少此配置可能无法访问外网
cat >> /etc/resolv.conf << EOF
# Generated by NetworkManager
nameserver 8.8.8.8
nameserver 114.114.114.114
EOF

重启生效

1
2
3
4
5
# 重启网络服务
systemctl restart network

# 验证网络
ping www.baidu.com

1.3.4 配置主机名与IP地址解析

在每个节点执行

1
2
3
4
5
cat >> /etc/hosts << EOF
192.168.198.129 apple
192.168.198.130 banana
192.168.198.131 cherry
EOF

1.3.5 防火墙配置

在每个节点执行

1
2
3
4
5
# 关闭防火墙
systemctl stop firewalld

# 禁止防火墙开机自启
systemctl disable firewalld

1.3.6 SELinux配置

在每个节点执行,修改后需要重启

1
2
3
4
5
# 永久关闭
sed -i 's/enforcing/disabled/' /etc/selinux/config

#临时关闭
setenforce 0

重启生效

1
reboot

1.3.7 swap分区配置

在每个节点执行,修改后需重启

1
2
3
4
5
# 永久关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab

# 临时关闭
swapoff -a

重启生效

1
reboot

1.3.8 配置内核转发及网桥过滤

在每个节点执行

1
2
3
4
5
6
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF
1
2
3
4
5
6
7
8
# 加载br_netfilter模块
modprobe br_netfilter

# 查看是否加载
lsmod | grep br_netfilter

# 生效
sysctl --system  

1.3.9 时间同步配置

在每个节点执行

1
2
3
4
5
# 安装 ntpdate
yum install ntpdate -y

# 同步时间
ntpdate time.windows.com

1.3.10 IPVS配置

在每个节点执行

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# 安装ipset和ipvsadm
yum -y install ipset ipvsadm

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

# 添加执行权限
chmod 755 /etc/sysconfig/modules/ipvs.modules

# 运行
bash /etc/sysconfig/modules/ipvs.modules

# 检查是否加载
lsmod | grep -e ipvs -e nf_conntrack_ipv4

1.3.11 升级操作系统内核

在每个节点执行,升级后需重启

安装YUM源

1
2
3
4
5
# 导入elrepo gpg key
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

# 安装elrepo YUM源仓库
yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm

升级内核版本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# 稳定版kernel-ml 长期维护版本kernel-lt  
yum -y --enablerepo=elrepo-kernel  install  kernel-ml

# 查看已安装的内核
rpm -qa | grep kernel

# 查看默认内核
grubby --default-kernel

# 若不是最新版本时执行
grubby --set-default $(ls /boot/vmlinuz-* | grep elrepo)

重启生效

1
reboot

1.4 创建快照

技巧
 在VMware对3台虚拟机创建一份快照,便于后续配置出现问题时回滚!

2 容器运行时准备

在每个节点执行

2.1 安装Containred

采用二进制文件安装

2.1.1 下载安装cni-plugins和cri-containerd-cni

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# 下载 cni-plugins-linux-amd64-v1.3.0.tgz
curl -O https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz

#创建cni插件所需目录
mkdir -p /etc/cni/net.d /opt/cni/bin 

#解压cni二进制包
tar xf cni-plugins-linux-amd64-v*.tgz -C /opt/cni/bin/

# 下载 cri-containerd-cni-1.7.8-linux-amd64.tar.gz
curl -O https://github.com/containerd/containerd/releases/download/v1.7.8/cri-containerd-cni-1.7.8-linux-amd64.tar.gz

#解压
tar -xzf cri-containerd-cni-*-linux-amd64.tar.gz -C /

#创建服务启动文件
cat > /etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

2.1.2 配置Containerd所需的模块

1
2
3
4
5
6
7
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

# 加载模块
systemctl restart systemd-modules-load.service

2.1.3 配置Containerd所需的内核

1
2
3
4
5
6
7
8
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# 加载内核
sysctl --system

2.1.4 创建Containerd的配置文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# 创建默认配置文件
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml

# 修改Containerd的配置文件
sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
cat /etc/containerd/config.toml | grep SystemdCgroup
sed -i "s#registry.k8s.io#m.daocloud.io/registry.k8s.io#g" /etc/containerd/config.toml
cat /etc/containerd/config.toml | grep sandbox_image
sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml
cat /etc/containerd/config.toml | grep certs.d

# 配置加速器
mkdir /etc/containerd/certs.d/docker.io -pv
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://docker.mirrors.ustc.edu.cn"]
  capabilities = ["pull", "resolve"]
EOF

2.1.5 启动并设置为开机自启

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# 用于重新加载systemd管理的单位文件
systemctl daemon-reload

# 启用并立即启动docker.service单元
systemctl enable --now containerd.service

# 停止运行中的docker.service单元
systemctl stop containerd.service

# 启动docker.service单元
systemctl start containerd.service

# 重启docker.service单元
systemctl restart containerd.service

# 显示docker.service单元的当前状态
systemctl status containerd.service

2.1.6 安装配置crictl客户端连接的运行时位置

采用二进制文件安装

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# 选择下载版本 1.28.0
curl -O https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.28.0/crictl-v1.28.0-linux-amd64.tar.gz

#解压
tar xf crictl-v*-linux-amd64.tar.gz -C /usr/bin/

#生成配置文件
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

#测试
systemctl restart  containerd
crictl info

2.2 安装 Docker

采用二进制文件安装

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# 查看已安装的docker相关包
yum list installed | grep docker

# 卸载docker相关包
yum remove docker-ce*
yum remove containerd.io.x86_64

rm -rf /etc/docker
rm -rf /run/docker
rm -rf /var/lib/dockershim
rm -rf /var/lib/docker

2.2.1 下载、解压 Docker

1
2
3
4
5
6
7
8
# 选择下载最新版本 24.0.7
curl -O https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/static/stable/x86_64/docker-24.0.7.tgz

# 解压
tar xf docker-24.0.7.tgz

# 拷贝二进制文件
cp docker/* /usr/bin

2.1.2 创建containerd.service文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cat >/etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

# 设置为开机自启服务
systemctl enable --now containerd.service

2.2.3 创建docker.service文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
cat > /etc/systemd/system/docker.service <<EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
OOMScoreAdjust=-500

[Install]
WantedBy=multi-user.target
EOF

2.2.4 创建docker.socket文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
cat > /etc/systemd/system/docker.socket <<EOF
[Unit]
Description=Docker Socket for the API

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target
EOF

2.2.5 配置阿里云镜像加速

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# 配置加速器
mkdir /etc/docker/ -pv

cat >/etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker"
}
EOF

2.2.6 启动 docker

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#创建docker组
groupadd docker

# 用于重新加载systemd管理的单位文件
systemctl daemon-reload

# 启用并立即启动docker.socket单元
systemctl enable --now docker.socket

# 启用并立即启动docker.service单元
systemctl enable --now docker.service

# 停止运行中的docker.service单元
systemctl stop docker.service

# 启动docker.service单元
systemctl start docker.service

# 重启docker.service单元
systemctl restart docker.service

# 显示docker.service单元的当前状态
systemctl status docker.service

# 打印docker配置
docker info

2.2.7 安装cri-docker

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# 选择安装版本 0.3.7
curl -O https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.7/cri-dockerd-0.3.7.amd64.tgz

# 解压cri-docker
tar xvf cri-dockerd-*.amd64.tgz 
cp -r cri-dockerd/  /usr/bin/
chmod +x /usr/bin/cri-dockerd/cri-dockerd

# 写入启动cri-docker配置文件
cat >  /usr/lib/systemd/system/cri-docker.service <<EOF
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

# 写入socket配置文件
cat > /usr/lib/systemd/system/cri-docker.socket <<EOF
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target
EOF

# 启动cri-docker

# 用于重新加载systemd管理的单位文件
systemctl daemon-reload

# 启用并立即启动cri-docker.service单元
systemctl enable --now cri-docker.service

# 重启cri-docker.service单元
systemctl restart cri-docker.service

# 显示docker.service单元的当前状态
systemctl status docker.service

3 部署Kubernetes集群

3.1 安装Kubernetes集群软件

在每个节点执行

3.1.1 配置阿里云镜像源代理

1
2
3
4
5
6
7
8
9
cat > /etc/yum.repos.d/k8s.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

3.1.2 安装kubeadm、kubelet、kubectl

1
2
3
4
5
6
7
8

# 查看指定版本
yum list kubeadm.x86_64 --showduplicates | sort -r
yum list kubelet.x86_64 --showduplicates | sort -r
yum list kubectl.x86_64 --showduplicates | sort -r

# 这里安装最新版本 1.28.2-0
yum -y install  kubeadm-1.28.2-0  kubelet-1.28.2-0 kubectl-1.28.2-0

3.1.3 拉取Kubernetes组件镜像

1
kubeadm config images pull --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --cri-socket unix:///var/run/cri-dockerd.sock --kubernetes-version=v1.28.2

3.1.4 初始化Master节点

1
kubeadm init --kubernetes-version=v1.28.2 --pod-network-cidr=10.244.0.0/16 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.198.129  --cri-socket unix:///var/run/cri-dockerd.sock

3.1.5 将Slave节点加入集群

1
kubeadm join 192.168.198.129:6443 --token <token>