文章目录

实验环境部署master02节点部署nginx负载均衡集群

实验环境

Master01：14.0.0.50 Master02：14.0.0.80 Node01： 14.0.0.60 Node02： 14.0.0.70 Nginx01+keepalived：14.0.0.90 Nginx02+keepalived：14.0.0.100

部署master02节点

关闭防火墙，关闭核心防护，关闭网络管理功能（生成环境中一定要关闭它） [root@localhost ~]# hostnamectl set-hostname master02 #修改主机名 [root@localhost ~]# su [root@master02 ~]# systemctl stop firewalld #关闭防火墙 [root@master02 ~]# setenforce 0 && sed -i "s/SELINUX=enforcing/SELNIUX=disabled/g" /etc/selinux/config #关闭核心防护 [root@master02 ~]# systemctl stop NetworkManager && systemctl disable NetworkManager #关闭网络管理功能 将master01节点的kubernetes配置文件和启动脚本复制到master02节点 [root@master ~]# scp -r /opt/kubernetes/ root@14.0.0.80:/opt/ [root@master ~]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@14.0.0.80:/usr/lib/systemd/system/ 将master01节点的etcd证书复制到master02节点（master02上一定要有etcd证书，用来与etcd通信） [root@master ~]# scp -r /opt/etcd/ root@14.0.0.80:/opt master02上修改apiserver配置文件中的IP地址 [root@master02 ~]# cd /opt/kubernetes/cfg/ [root@master02 cfg]# ls kube-apiserver kube-controller-manager kube-scheduler token.csv [root@master02 cfg]# vim kube-apiserver KUBE_APISERVER_OPTS="--logtostderr=true \ --v=4 \ --etcd-servers=https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379 \ --bind-address=14.0.0.80 \ #修改此处的绑定IP地址 --secure-port=6443 \ --advertise-address=14.0.0.80 \ #修改此处的IP地址 ...省略 启动master02中的三个组件服务 [root@localhost cfg]# systemctl start kube-apiserver.service [root@localhost cfg]# systemctl start kube-controller-manager.service [root@localhost cfg]# systemctl start kube-scheduler.service 添加环境变量并查看状态 [root@master02 ~]# echo export PATH=$PATH:/opt/kubernetes/bin >> /etc/profile [root@master02 ~]# source /etc/profile [root@master02 ~]# kubectl get node NAME STATUS ROLES AGE VERSION 14.0.0.60 Ready <none> 23h v1.12.3 14.0.0.70 Ready <none> 23h v1.12.3 #看到两个node节点Ready说明master02部署成功

部署nginx负载均衡集群

两个nginx主机关闭防火墙和核心防护，编辑nginx的yum源 [root@localhost ~]# hostnamectl set-hostname nginx01 #修改主机名 [root@localhost ~]# su [root@nginx01 ~]# systemctl stop firewalld && systemctl disable firewalld #关闭防火墙与核心防护 [root@nginx01 ~]# setenforce 0 && sed -i "s/SELINUX=enforcing/SELNIUX=disabled/g" /etc/selinux/config [root@nginx01 ~]# vi /etc/yum.repos.d/nginx.repo #编辑nginx的yum源 [nginx] name=nginx.repo baseurl=http://nginx.org/packages/centos/7/$basearch/ enabled=1 gpgcheck=0 [root@nginx01 ~]# yum list 两台nginx主机安装nginx并开启四层转发（仅展示nginx01的操作） [root@nginx01 ~]# yum -y install nginx '//安装nginx' [root@nginx01 ~]# vi /etc/nginx/nginx.conf ...省略内容 events { worker_connections 1024; } stream { log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent'; #定义日志格式； access_log /var/log/nginx/access.log main; upstream k8s-apiserver { #定义代理的IP地址及端口 server 14.0.0.50:6443; server 14.0.0.80:6443; } server { listen 6443; proxy_pass k8s-apiserver; } } http { include /etc/nginx/mime.types; default_type application/octet-stream; ...省略内容 启动nginx服务 [root@nginx01 ~]# nginx -t #检查nginx语法 nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@nginx01 ~]# systemctl start nginx #开启服务 [root@nginx01 ~]# systemctl status nginx [root@nginx01 ~]# netstat -ntap |grep nginx #会检测出来6443端口 tcp 0 0 0.0.0.0:6443 0.0.0.0:* LISTEN 1849/nginx: master tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1849/nginx: master 两台nginx主机部署keepalived服务（仅展示节点nginx01的配置） [root@nginx01 ~]# yum -y install keepalived [root@nginx01 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived vrrp_script check_nginx { #定义一个函数check_nginx script "/usr/local/nginx/sbin/check_nginx.sh" #函数内容为一个检测nginx服务是否存活的脚本 } global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL01 #定义该主机在群集中的id，nginx02需要命名为不一样的 } vrrp_instance VI_1 { state MASTER #nginx02节点命名为BACKUP interface ens33 #修改网卡名，centos7开始为ens33，centos6为eth0 virtual_router_id 51 priority 100 #nginx02节点优先级设为90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { #设置VIP 14.0.0.88 } track_script { #该vrrp实例VI_1调用上面定义的函数check_nginx check_nginx } } 创建监控nginx进程的脚本，启动keepalived服务，查看VIP地址 [root@nginx01 ~]# mkdir -p /usr/local/nginx/sbin/ #创建监控脚本目录 [root@nginx01 ~]# vim /usr/local/nginx/sbin/check_nginx.sh #编写监控脚本配置文件 count=$(ps -ef |grep nginx |egrep -cv "grep|$$") #查看nginx进程 if [ "$count" -eq 0 ];then #如果nginx进程关闭了，则关闭keepalived服务 systemctl stop keepalived fi [root@nginx01 ~]# chmod +x /usr/local/nginx/sbin/check_nginx.sh [root@nginx01 ~]# systemctl start keepalived #开启服务 [root@nginx01 ~]# systemctl status keepalived [root@nginx01 ~]# ip a #两个nginx服务器查看IP地址 可以发现VIP在节点nginx01上 [root@nginx02 ~]# ip a 验证漂移地址 [root@nginx01 ~]# pkill nginx #关闭nginx01节点的nginx服务 [root@nginx01 ~]# systemctl status keepalived #发现keepalived服务关闭了 [root@nginx02 ~]# ip a #现在发现VIP地址漂移到nginx02上了 恢复漂移地址的操作 [root@nginx01 ~]# systemctl start nginx [root@nginx01 ~]# systemctl start keepalived #先开启nginx，在启动keepalived服务 [root@nginx01 ~]# ip a #再次查看，发现VIP又回到了nginx01节点上 修改两个node节点配置文件（bootstrap.kubeconfig ），使用VIP地址，仅展示node01节点的操作 [root@node01 ~]# vi /opt/kubernetes/cfg/bootstrap.kubeconfig server: https://14.0.0.88:6443 #此地址修改为VIP地址 [root@node01 ~]# vi /opt/kubernetes/cfg/kubelet.kubeconfig server: https://14.0.0.88:6443 #此地址修改为VIP地址 [root@node01 ~]# vi /opt/kubernetes/cfg/kube-proxy.kubeconfig server: https://14.0.0.88:6443 #此地址修改为VIP地址 重启两个node节点的服务 [root@node01 ~]# systemctl restart kubelet [root@node01 ~]# systemctl restart kube-proxy [root@node01 ~]# cd /opt/kubernetes/cfg/ [root@node01 cfg]# grep 88 * #过滤当前目录下所有内容中是否包含88，如下代表VIP修改成功 bootstrap.kubeconfig: server: https://14.0.0.88:6443 kubelet.kubeconfig: server: https://14.0.0.88:6443 kube-proxy.kubeconfig: server: https://14.0.0.88:6443 在节点nginx01上查看nginx的日志，查看负载均衡是否生效 [root@nginx01 ~]# vim /var/log/nginx/access.log #下面的日志是重启服务后产生的 14.0.0.60 14.0.0.50:6443 - [30/Sep/2020:11:01:22 +0800] 200 15319 14.0.0.60 14.0.0.50:6443 - [30/Sep/2020:11:01:23 +0800] 200 1115 14.0.0.60 14.0.0.80:6443 - [30/Sep/2020:11:01:23 +0800] 200 1115 14.0.0.60 14.0.0.80:6443 - [30/Sep/2020:11:01:31 +0800] 200 3010 14.0.0.70 14.0.0.50:6443 - [30/Sep/2020:11:01:38 +0800] 200 1115 14.0.0.70 14.0.0.50:6443 - [30/Sep/2020:11:01:38 +0800] 200 1114 #nginx负载均衡生效后，会进行状态检查 master节点测试创建pod [root@master01 ~]# kubectl run nginx --image=nginx #创建一个运行nginx服务的pod kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead. deployment.apps/nginx created [root@master01 ~]# kubectl get pods #查看状态，是正在创建 NAME READY STATUS RESTARTS AGE nginx-dbddb74b8-5s6h7 0/1 ContainerCreating 0 13s [root@master01 ~]# kubectl get pods #过会儿再次查看，发现pod已经创建完成，在master02节点也可以查看到 NAME READY STATUS RESTARTS AGE nginx-dbddb74b8-5s6h7 1/1 Running 0 23s 查看刚才创建的运行nginx服务的pod的日志 [root@master01 ~]# kubectl logs nginx-dbddb74b8-5s6h #查看pod日志 Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-5s6h7) #发现是因为使用了system:anonymous（匿名）用户进行操作，没有权限 [root@master01 ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous #将集群中的匿名用户绑定到管理员用户，使其拥有权限 [root@master ~]# kubectl logs nginx-dbddb74b8-5s6h #此时可以查看，这时没有日志产生 访问node节点的pod中的web业务，从而产生日志，并在两个master节点查看 [root@master ~]# kubectl get pods -o wide #查看pod的完整信息（IP信息） NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE nginx-dbddb74b8-5s6h7 1/1 Running 0 6m29s 172.17.26.2 14.0.0.60 <none> [root@node01 ~]# curl 172.17.26.2 #在对应的node节点访问pod [root@master ~]# kubectl logs nginx-dbddb74b8-5s6h7 #再次在master节点查看日志情况，master02节点同样可以查看到 172.17.26.1 - - [30/Apr/2020:17:38:48 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"