Kubernetes 部署高可用 Ingress-Nginx

环境说明

这里的 Ingress-Nginx 采用的是 Helm 包方式部署的，服务部署类型为 DaemonSet，采用宿主机网络，所以会在节点上监听 80，443 端口。高可用方案采用 keepalived 方式，在每个节点上测试本地的 80 端口是否能正常访问，能正常访问则说明 ingress-nginx 服务正常，否则停止 keepalived 服务，将 VIP 漂移到其他的 ingress-nginx 节点。

主机名	IP 地址	备注
k8snode01	10.66.211.14	部署 ingress 节点，Master 节点
k8snode02	10.66.211.15	部署 ingress 节点，Backup 节点

VIP 地址：10.66.211.30

部署 Keepalived 服务

1. 所有 ingress-nginx 节点安装 keepalived

   apt install -y  keepalived

2. 创建 keepalived 配置文件

   export INTERFACE=$(ip route show |grep default |cut -d ' ' -f5)
   export IPADDR=$(ifconfig |grep -A1 $INTERFACE |grep inet |awk '{print $2}')
   
   cat > /etc/keepalived/keepalived.conf <<EOF
   ! Configuration File for keepalived
   global_defs {
       router_id LVS_DEVEL
   }
   
   vrrp_script chk_ingress {
       script "/etc/keepalived/check_ingress.sh"
       interval    2
       weight      -5
       fall    3
       rise    2
   }
   
   vrrp_instance VI_30 {
       state MASTER
       interface ${INTERFACE}
       mcast_src_ip ${IPADDR}
       virtual_router_id 30
       priority 100
       advert_int 2
       authentication {
           auth_type PASS
           auth_pass K8SHA_KA_AUTH
       }
       virtual_ipaddress {
           10.66.211.30
       }
       track_script {
           chk_ingress
       }
   }
   EOF

   注意修改其它节点的 priority 以及 state 的值，

3. 创建服务检查脚本 /etc/keepalived/check_ingress.sh

   cat > /etc/keepalived/check_ingress.sh <<EOF
   #!/bin/bash
   
   err=0
   for k in \$(seq 1 3); do
       nc -z localhost 80
       if [ \$? -ne 0 ]; then
           err=\$((err + 1))
           sleep 1
       else
           err=0
           break
       fi
   done
   
   if [ \$err -ne 0 ]; then
       echo "Stopping keepalived service"
       sudo /usr/bin/systemctl stop keepalived
       exit 1
   else
       echo "Ingress-Nginx is running. Keepalived service remains active."
       exit 0
   fi
   EOF

4. 所有节点启动 keepalived 服务

   systemctl enable --now keepalived

5. 检查 VIP 是否正常在 MASTER 节点

   ip a |grep '10.66.211.30'

验证服务高可用

1. 将 Master 节点的 ingress-Nginx 服务停掉，查看 VIP 是否会正常漂移到其他节点

   kubectl label nodes k8snode01.north.yubang168.cn ingress-

2. 将 Master 节点的 ingress-nginx 服务恢复运行，查看 VIP 是否会回到 Master 节点

   kubectl label nodes k8snode01.north.yubang168.cn ingress=true