k8s高可用（多master）集群搭建
环境准备： 192.168.2.117 node117 192.168.2.123 node123 192.168.2.124 master124 192.168.2.125 master125 192.168.2.126 master126 keeplive vip 192.168.2.130 cluster-endpoint k8s 1.28.2 containerd 1.7.8 runc.amd64 1.1.9 anolis 8.8 ubuntu 22.04 在所有节点修改环境 修改主机名 hostnamectl set-hostname 修改host vim /etc/hosts 192.168.2.117 node117 192.168.2.123 node123 192.168.2.124 master124 192.168.2.125 master125 192.168.2.126 master126 192.168.2.130 cluster-endpoint #anolis服务器 systemctl disable firewalld systemctl stop firewalld #ubuntu服务器 sudo ufw status 启用 / 禁用 sudo ufw enable / disable
setenforce 0 sed -i ’s/SELINUX=permissive/SELINUX=disabled/’ /etc/sysconfig/selinux sed -i “s/SELINUX=enforcing/SELINUX=disabled/g” /etc/selinux/config swapoff -a sed -i ’s/.swap./#&/’ /etc/fstab cat «EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sysctl –system modprobe br_netfilter echo 1 > /proc/sys/net/ipv4/ip_forward sysctl -p /etc/sysctl.d/k8s.conf chronyc -a makestep #还要配置免密登录（暂时还不知道什么用处所以所有服务器我都配置了） ssh-keygen -t rsa ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.2.123 安装配置containerd tar xf containerd-1.6.6-linux-amd64.tar.gz cp -r bin/* /usr/local/bin vim /etc/systemd//system/containerd.service
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the “License”);
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
[Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target local-fs.target [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5
Having non-zero Limit*s causes performance problems due to accounting overhead
in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity LimitCORE=infinity LimitNOFILE=infinity
Comment TasksMax if your systemd version does not supports it.
Only systemd 226 and above support this version.
TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target systemctl daemon-reload mkdir /etc/containerd containerd config default > /etc/containerd/config.toml
vim config.toml vim下搜索/mirrors，添加镜像加速，使用docker镜像源即可，上下级配置，缩进两个空格。 [plugins.“io.containerd.grpc.v1.cri”.registry.mirrors] [plugins.“io.containerd.grpc.v1.cri”.registry.mirrors.“docker.io”] endpoint = [“https://docker.mirrors.ustc.edu.cn”] 搜索 sandbox_image 将 registry.k8s.io/pause:3.8 替换为 registry.aliyuncs.com/google_containers/pause:3.8 sandbox_image = “registry.aliyuncs.com/google_containers/pause:3.8” 把配置文件中的 SystemdCgroup 修改为 true SystemdCgroup = true systemctl enable containerd –now systemctl restart containerd 安装runc chmod +x runc.amd64 cp runc.amd64 /usr/local/bin/runc 安装配置haproxy yum install -y haproxy vim /etc/haproxy/haproxy.cfg [root@master126 ~]# haproxy -v HAProxy version 2.4.22-f8e3218 2023/02/14 - https://haproxy.org/ Status: long-term supported branch - will stop receiving fixes around Q2 2026. Known bugs: http://www.haproxy.org/bugs/bugs-2.4.22.html Running on: Linux 5.10.134-13.an8.x86_64 #1 SMP Mon Jan 9 10:39:46 CST 2023 x86_64 haproxy用于反向代理兼顾负载均衡，代理后端的api-server
配置文件修改只需要配置一个前端代理端口一个后端代理， 修改完的配置文件如下 global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats ssl-default-bind-ciphers PROFILE=SYSTEM ssl-default-server-ciphers PROFILE=SYSTEM defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000
#前端代理配置将6443 统一代理成16443 frontend kube-api bind *:16443
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
use_backend static if url_static
default_backend k8smaster #backend static
balance roundrobin
server static 127.0.0.1:4331 check
#后端服务配置 backend k8smaster balance roundrobin server master124 192.168.2.124:6443 check server master125 192.168.2.125:6443 check server master126 192.168.2.126:6443 check systemctl enable –now keepalived systemctl status keepalived 安装配置keepalived yum install -y keepalived vim /etc/keepalived/keepalived.conf keepalived -v Keepalived v2.2.4 (08/21,2021) Keepalived Keepalived一个基于VRRP 协议来实现的 LVS 服务高可用方案，可以利用其来解决单点故障。 一个LVS服务会有2台服务器运行Keepalived，一台为主服务器（MASTER），一台为备份服务器（BACKUP），
但是对外表现为一个虚拟IP，主服务器会发送特定的消息给备份服务器，当备份服务器收不到这个消息的时 候，即主服务器宕机的时候， 备份服务器就会接管虚拟IP，继续提供服务，从而保证了高可用性。 配置keepalived ! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id master126 #修改成本服务器的ip或者hostname（推荐hostname） vrrp_skip_check_adv_addr
vrrp_strict #初始化启动的时候不能注释不然不会配置Vip但是此时vip不生效，等vip配置后再
注释掉重启，vip才能生效 vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_script chklive { # 引入自定义的检活脚本 script /etc/keepalived/chkhaproxy.sh interval 1 weight -3 timeout 5 } vrrp_instance VI_1 { state MASTER interface eth0 #网卡名称 virtual_router_id 51 #集群id 同一个集群必须相同 priority 100 #权重 advert_int 1 authentication {
auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.2.130/24 #vip } track_script { chklive #引入自定义的检活脚本 } } 自定义检活脚本 vim /etc/keepalived/chkhaproxy.sh #! /bin/bash if ! netstat -nlpt | grep -w “16443” &> /dev/null then systemctl stop keepalived fi exit 0 然后记得赋予脚本执行权限 chmod +x /etc/keepalived/chkhaproxy.sh systemctl enable –now haproxy systemctl status haproxy 安装k8s ======== anolis ========== cat «EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes]
name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF ======== ubuntu ========== apt install -y apt-transport-https gnupg gnupg2 gnupg1 curl lsb-release curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - cat «EOF >/etc/apt/sources.list.d/kubernetes.list deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main EOF yum install -y kubelet-1.28.2 kubeadm-1.28.2 kubectl-1.28.2 systemctl enable kubelet systemctl start kubelet kubelet –version crictl config runtime-endpoint unix:///run/containerd/containerd.sock crictl config image-endpoint unix:///run/containerd/containerd.sock
查看所需的镜像列表
kubeadm config images list –kubernetes-version=v1.28.2 # –image-repository registry.aliyuncs.com/google_containers
1. 使用默认容器镜像仓库
sudo kubeadm config images pull –kubernetes-version=v1.28.2
2. 使用指定容器镜像仓库
sudo kubeadm config images pull –kubernetes-version=v1.28.2 –image-repository
registry.aliyuncs.com/google_containers
初始化k8s
kubeadm init
–image-repository registry.aliyuncs.com/google_containers
–kubernetes-version=v1.28.2
–pod-network-cidr=10.244.0.0/16
–service-cidr=10.96.0.0/12 \
–apiserver-advertise-address=192.168.2.124
–node-name=master124 \ #执行初始化脚本的服务器 hostname
–control-plane-endpoint=cluster-endpoint:16443 \ #注意此处是harpoxy配置的端口
vip的地址
–cri-socket unix:///var/run/containerd/containerd.sock
直到出现
kubeadm join cluster-endpoint:16443 –token ylfz8v.00ez7yjocfwmrc3q
–discovery-token-ca-cert-hash
sha256:3edb939a5c71834ebf161e3f0fb9112963b6fcdce87d57b4afd7362aa3fd668f
–control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join cluster-endpoint:16443 –token ylfz8v.00ez7yjocfwmrc3q
–discovery-token-ca-cert-hash
sha256:3edb939a5c71834ebf161e3f0fb9112963b6fcdce87d57b4afd7362aa3fd668f
【温馨提示】kubeadm 不支持将没有 –control-plane-endpoint 参数的单个控制平面集群转换为高可用性
集群。
重置再初始化
控制节点加入集群
首先需要传送令牌
scp -rp /etc/kubernetes/pki/ca.* master126:/etc/kubernetes/pki
scp -rp /etc/kubernetes/pki/sa.* master126:/etc/kubernetes/pki
scp -rp /etc/kubernetes/pki/front-proxy-ca.* master126:/etc/kubernetes/pki
#这一句会找不到文件夹手动创建
scp -rp /etc/kubernetes/pki/etcd/ca.* master126:/etc/kubernetes/pki/etcd
scp -rp /etc/kubernetes/admin.conf master126:/etc/kubernetes
以控制面节点加入集群
kubeadm join cluster-endpoint:16443 –token ylfz8v.00ez7yjocfwmrc3q
–discovery-token-ca-cert-hash
sha256:3edb939a5c71834ebf161e3f0fb9112963b6fcdce87d57b4afd7362aa3fd668f
–control-plane
以工作节点加入
kubeadm join api.k8s.local:6443 –token abcdef.0123456789abcdef
–discovery-token-ca-cert-hash
sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866
安装calico网络插件
#目前测试3.23版本的可用 3.24-3.27都报错
curl https://docs.projectcalico.org/v3.23/manifests/calico.yaml -O
kubectl apply -f calico.yaml
calico 开头的以及 coredns 都跑起来就算完成。
如果网络插件起不来请查看网卡配置
因为有节点是多网卡，所以需要在资源清单文件中指定内网网卡
vim calico.yaml
……
spec:
containers:
- env:
Cluster type to identify the deployment type
- name: CLUSTER_TYPE value: “k8s,bgp”
value 就是指定你的网卡名字，我这里网卡是 eth0，然后直接配置的通配符 eth.*
- name: IP_AUTODETECTION_METHOD value: “interface=eth.*”
Auto-detect the BGP IP address.
- name: IP value: “autodetect”
Enable IPIP
- name: CALICO_IPV4POOL_IPIP value: “Always”
Enable or Disable VXLAN on the default IP pool.
- name: CALICO_IPV4POOL_VXLAN value: “Never”
Enable or Disable VXLAN on the default IPv6 IP pool.
- name: CALICO_IPV6POOL_VXLAN value: “Never” …… 已修改好的calico.yaml calico.yaml