下载安装cfssl证书工具到/usr/local/bin/cfssl，并赋予可执行权限，方便直接使用
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo [root@localhost k8s]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssl-certinfo /usr/local/bin/cfssljson命令解读
cfssl：生成证书的工具命令 cfssljson：通过传入json文件生成证书 cfssl-certinfo：查看证书信息为etcd的证书创建一个目录
[root@localhost k8s]# mkdir etcd-cert创建根证书（自建CA，etcd的证书由它签发）
[root@localhost k8s]# cat > ca-config.json <<EOF > { > "signing": { > "default": { > "expiry": "87600h" //指定了证书的过期时间，87600h为10年 > }, > "profiles": { //可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数 > "www": { > "expiry": "87600h", > "usages": [ > "signing", //表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE > "key encipherment", > "server auth", //表示client可以用该CA对server提供的证书进行验证 > "client auth" //表示server可以用该CA对client提供的证书进行验证，这里最后一个字段后面没有逗号 > ] > } > } > } > } > EOF创建CA证书请求
[root@localhost k8s]# cat > ca-csr.json <<EOF > { > "CN": "etcd CA", //Common Name，自定义，这里用于表示用于etcd认证的根证书 > "key": { > "algo": "rsa", //加密算法 > "size": 2048 //加密长度，rsa最小2048 > }, > "names": [ > { > "C": "CN", //国家 > "L": "Beijing", //地区，城市 > "ST": "Beijing" //省 > } > ] > } > EOF签发CA证书和私钥
cfssl gencert -initca ca-csr.json | cfssl-json -bare ca这样表示也可以
[root@localhost k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - 2020/09/28 16:11:54 [INFO] generating a new CA key and certificate from CSR 2020/09/28 16:11:54 [INFO] generate received request 2020/09/28 16:11:54 [INFO] received CSR 2020/09/28 16:11:54 [INFO] generating key: rsa-2048 2020/09/28 16:11:55 [INFO] encoded CSR 2020/09/28 16:11:55 [INFO] signed certificate with serial number 590206773442412657075217441716999534465487365861生成的文件后续要用到，保存好：
ca-key.pem 根证书私钥 ca.pem 根证书 etcd所需要的证书： 1、Etcd对外提供服务，要有一套etcd server证书 2、Etcd各节点之间进行通信，要有一套etcd peer证书 3、Kube-APIserver访问Etcd，要有一套etcd client证书 这里全部都使用同一种证书认证创建etcd服务器端证书请求文件
[root@localhost k8s]# cat > server-csr.json <<EOF > { > "CN": "etcd", > "hosts": [ //需要指定所有etcd集群的节点ip或主机名 > "192.168.245.211", > "192.168.245.209", > "192.168.245.210" > ], > "key": { > "algo": "rsa", > "size": 2048 > }, > "names": [ > { > "C": "CN", > "L": "BeiJing", > "ST": "BeiJing" > } > ] > } > EOF签发etcd服务端证书和私钥
[root@localhost k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server 2020/09/28 16:16:17 [INFO] generate received request 2020/09/28 16:16:17 [INFO] received CSR 2020/09/28 16:16:17 [INFO] generating key: rsa-2048 2020/09/28 16:16:18 [INFO] encoded CSR 2020/09/28 16:16:18 [INFO] signed certificate with serial number 621191565914440988284456398609893537198453121189 2020/09/28 16:16:18 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements").生成的文件后面需要用到，保存好：
server-key.pem 服务器的私钥 server.pem 服务器的数字签名证书将所有证书和密钥都移动到etcd-cert目录中便于管理
[root@localhost k8s]# mv ca.csr ca-config.json server-csr.json server-key.pem ca-csr.json ca-key.pem ca.pem server.csr server.pem etcd-cert/解压etcd
[root@localhost k8s]# tar zxfv etcd-v3.3.10-linux-amd64.tar.gz解压完成之后将etcd和etcdctl可执行文件移动到/opt/etcd/bin/下
[root@localhost k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/ 注：etcd就是etcd server的启动命令，后面可跟各种启动参数 etcdctl主要为etcd server提供了命令行操作拷贝etcd需要认证的证书到/opt/etcd/ssl/下
[root@localhost k8s]# mkdir /opt/etcd/{cfg,bin,ssl} -p [root@localhost k8s]# ls /opt/etcd bin cfg ssl [root@localhost k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/配置etcd（配置文件/opt/etcd/cfg/etcd）并启动etcd 为了方便，我们将etcd启动和system启动都做成了etcd.sh脚本，执行此脚本启动etcd服务
[root@localhost k8s]# bash etcd.sh etcd01 192.168.245.211 etcd02=https://192.168.245.209:2380,etcd03=https://192.168.245.210:2380注：这里需要三台etcd服务同时启动，这里因为先启动了其中一台etcd节点，所以服务会卡在那里，这是正常的
把etcd相关文件全部分发到另外两个etcd集群节点供使用
[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.245.209:/opt/ [root@localhost k8s]# scp -r /opt/etcd/ root@192.168.245.210:/opt/ 注：etcd认证只需要ca.pem、server-key.pem、server.pem 三个证书即可 一般情况下，K8S中证书只需要创建一次，以后在向集群中添加新节点时只要将/opt/etcd/ssl/目录下的证书拷贝到新节点上即可。两台etcd节点修改etcd配置文件，都改成自己的名称和ip地址
[root@node1 ~]# cat /opt/etcd/cfg/etcd #[Member] ETCD_NAME="etcd02" ETCD_LISTEN_PEER_URLS="https://192.168.245.209:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.245.209:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.245.209:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.245.209:2379" --------------------------------------------------------------------- [root@node2 ~]# cat /opt/etcd/cfg/etcd #[Member] ETCD_NAME="etcd03" ETCD_LISTEN_PEER_URLS="https://192.168.245.210:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.245.210:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.245.210:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.245.210:2379"把etcd的system启动脚本分发到另外两台etcd群集节点上
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.245.209:/usr/lib/systemd/system/ root@192.168.245.209's password: etcd.service 100% 923 1.2MB/s 00:00 [root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.245.210:/usr/lib/systemd/system/ root@192.168.245.210's password: etcd.service 100% 923 749.1KB/s 00:00 [root@localhost k8s]#两台节点用systemctl启动etcd
[root@node1 ~]# systemctl start etcd [root@node1 ~]# systemctl status etcdetcd 提供了 etcdctl 命令行工具 和 HTTP API 两种交互方法。
etcdctl命令行工具用 go 语言编写，也是对 HTTP API 的封装，日常使用起来也更容易，所以这里主要使用etcdctl命令
master上验证集群健康状态，需要带证书的参数否则会报错
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.245.211:2379,https://192.168.245.209:2379,https://192.168.245.210:2379" cluster-health member 15239980699a4b97 is healthy: got healthy result from https://192.168.245.210:2379 member 77360e8f1ec8f43b is healthy: got healthy result from https://192.168.245.211:2379 member da0a263e34e24794 is healthy: got healthy result from https://192.168.245.209:2379 cluster is healthy --endpoints：集群中以逗号分隔的机器地址列表 cluster-health：检查etcd集群的运行状况 其他选项参数可以etcdctl --help查看etcd集群就部署好了
Flannel是Core0S团队针对Kubernetes设计的一个网络规划服务，简单来说，它的功能是让集群中的不同节点主机创建的Docker容器都具有全集群唯一的虚拟IP地址，也就是为了容器之间可以互相通信，而且它提供的ip地址是通过etcd主从存储的
添加flannel网络配置信息到Etcd集群
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.245.211:2379,https://192.168.245.209:2379,https://192.168.245.210:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}' { "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}查看写入的信息
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.245.211:2379,https://192.168.245.209:2379,https://192.168.245.210:2379" get /coreos.com/network/config { "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}} set /coreos.com/network/config 添加一条网络配置记录，这个配置将用于flannel分配给每个Docker的虚拟IP地址段 get /coreos.com/network/config 获取网络配置记录，后面不用再跟参数了 Network：用于指定Flannel地址池 Backend：用于指定数据包以什么方式转发，默认为udp模式，backend为vxlan比起预设的udp性能相对好一些解压
[root@node1 ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz flanneld //flanneld为主要的执行文件 mk-docker-opts.sh //sh脚本用于生成Docker启动参数 README.md [root@node2 ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz flanneld mk-docker-opts.sh README.md将mk-docker-opts.sh脚本移动到/opt/kubernetes/bin/便于使用
[root@node1 ~]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p [root@node1 ~]# mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/ [root@node2 ~]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p [root@node2 ~]# mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/配置启动flannel，为了使用方便，将启动命令和system启动写成了flannel.sh脚本，执行此脚本启动flannel
[root@node1 ~]# bash flannel.sh https://192.168.245.211:2379,https://192.168.245.209:2379,https://192.168.245.210:2379 Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service更改Docker的启动参数，使其能够使用flannel进行IP分配，以及网络通讯。
配置docker连接flannel(ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env 因为启动脚本里有这一步，所以要在docker的启动文件里引用DOCKER_NETWORK_OPTIONS)
[root@node1 ~]# vim /usr/lib/systemd/system/docker.service EnvironmentFile=/run/flannel/subnet.env ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sockflannel运行后会生成一个环境变量文件/run/flannel/subnet.env，包含了当前主机要使用flannel通讯的相关参数
[root@node1 ~]# cat /run/flannel/subnet.env DOCKER_OPT_BIP="--bip=172.17.5.1/24" DOCKER_OPT_IPMASQ="--ip-masq=false" DOCKER_OPT_MTU="--mtu=1450" DOCKER_NETWORK_OPTIONS=" --bip=172.17.5.1/24 --ip-masq=false --mtu=1450" [root@node2 ~]# [root@node2 ~]# cat /run/flannel/subnet.env DOCKER_OPT_BIP="--bip=172.17.10.1/24" DOCKER_OPT_IPMASQ="--ip-masq=false" DOCKER_OPT_MTU="--mtu=1450" DOCKER_NETWORK_OPTIONS=" --bip=172.17.10.1/24 --ip-masq=false --mtu=1450"Flannel启动过程解析：
1、从etcd中获取network的配置信息 2、划分subnet，并在etcd中进行注册 3、将子网信息记录到/run/flannel/subnet.env中
node1节点分配到的ip为172.17.5.1
[root@node1 ~]# ifconfig docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.5.1 netmask 255.255.255.0 broadcast 172.17.5.255 ether 02:42:0d:2f:33:a7 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.245.209 netmask 255.255.255.0 broadcast 192.168.245.255 inet6 fe80::20c:29ff:fea4:e64b prefixlen 64 scopeid 0x20<link> ether 00:0c:29:a4:e6:4b txqueuelen 1000 (Ethernet) RX packets 805062 bytes 1075564084 (1.0 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 209586 bytes 19408651 (18.5 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 172.17.5.0 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::842b:ecff:fedd:8d5f prefixlen 64 scopeid 0x20<link> ether 86:2b:ec:dd:8d:5f txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 35 overruns 0 carrier 0 collisions 0node2节点分配到的ip为172.17.10.1
[root@node2 ~]# ifconfig docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.10.1 netmask 255.255.255.0 broadcast 172.17.10.255 ether 02:42:d9:e8:0c:66 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.245.210 netmask 255.255.255.0 broadcast 192.168.245.255 inet6 fe80::20c:29ff:fe9f:6979 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:9f:69:79 txqueuelen 1000 (Ethernet) RX packets 804058 bytes 1074727380 (1.0 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 226147 bytes 19694949 (18.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 172.17.10.0 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::2c39:49ff:fe70:9f2 prefixlen 64 scopeid 0x20<link> ether 2e:39:49:70:09:f2 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 36 overruns 0 carrier 0 collisions 0测试两个节点互相可以ping通docker0
[root@node2 ~]# ping 172.17.5.1 PING 172.17.5.1 (172.17.5.1) 56(84) bytes of data. 64 bytes from 172.17.5.1: icmp_seq=1 ttl=64 time=1.53 ms 64 bytes from 172.17.5.1: icmp_seq=2 ttl=64 time=0.692 ms 64 bytes from 172.17.5.1: icmp_seq=3 ttl=64 time=3.33 ms ^C --- 172.17.5.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.692/1.854/3.332/1.100 ms创建一个存放证书的路径/opt/kubernetes/k8s-cert
[root@localhost k8s]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p [root@localhost k8s]# mkdir k8s-cert [root@localhost k8s]# cd k8s-cert/生成k8s组件所需要的证书，这里全部写在脚本里了
[root@localhost k8s-cert]# vim k8s-cert.sh [root@localhost k8s-cert]# bash k8s-cert.sh [root@localhost k8s-cert]# ll *.pem -rw------- 1 root root 1679 9月 29 15:19 admin-key.pem -rw-r--r-- 1 root root 1399 9月 29 15:19 admin.pem //kubectl的证书，admin权限，具有访问kubernetes所有api的权限。 -rw------- 1 root root 1675 9月 29 15:19 ca-key.pem -rw-r--r-- 1 root root 1359 9月 29 15:19 ca.pem //专门用来签发其他组件证书的根证书 -rw------- 1 root root 1675 9月 29 15:19 kube-proxy-key.pem -rw-r--r-- 1 root root 1403 9月 29 15:19 kube-proxy.pem //kube-proxy的证书 -rw------- 1 root root 1675 9月 29 15:19 server-key.pem -rw-r--r-- 1 root root 1643 9月 29 15:19 server.pem //apiserver的证书scheduler和controller-manager这里是用非安全端口来和apiserver通信的，所以不需要证书 kubelet的证书是由controller-manager签发的，会自动生成，所以这里也不需要
[root@localhost k8s-cert]# cp ca*pem server*pem /opt/kubernetes/ssl/安装k8s，解压之后就包含了master上所需要的组件
[root@localhost k8s]# tar zxvf kubernetes-server-linux-amd64.tar.gz [root@localhost k8s]# cd /root/k8s/kubernetes/server/bin [root@localhost bin]# cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/生成api-server所需要的TLS Token（apiserver要开启token认证）
[root@localhost ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 49973b42a12aa4d513019a15e3661e2d将token保存到csv文件，apiserver启动时会调用
[root@localhost bin]# cd /root/k8s/ [root@localhost k8s]# vim /opt/kubernetes/cfg/token.csv 49973b42a12aa4d513019a15e3661e2d,kubelet-bootstrap,10001,"system:kubelet-bootstrap" Token，用户名，用户的UID，用户组启动apiserver
[root@localhost k8s]# bash apiserver.sh 192.168.245.211 https://192.168.245.211:2379,https://192.168.245.209:2379,https://192.168.245.210:2379 Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service. [root@localhost k8s]#查看进程
[root@localhost k8s]# ps aux | grep kube root 73461 30.8 16.7 397888 311924 ? Ssl 15:29 0:08 /opt/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://192.168.245.211:2379,https://192.168.245.209:2379,https://192.168.245.210:2379 --bind-address=192.168.245.211 --secure-port=6443 --advertise-address=192.168.245.211 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/opt/kubernetes/ssl/server.pem --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/etcd/ssl/ca.pem --etcd-certfile=/opt/etcd/ssl/server.pem --etcd-keyfile=/opt/etcd/ssl/server-key.pem root 73481 0.0 0.0 112724 988 pts/1 S+ 15:30 0:00 grep --color=auto kubek8s通过kube-apiserver这个进程提供服务，该进程运行在单个k8s-master节点上。默认有两个端口 本地端口8080用于接收HTTP请求，非认证或授权的HTTP请求通过该端口访问API Server 安全端口6443用于接收HTTPS请求，用于基于Tocken文件或客户端证书及HTTP Base的认证，默认IP地址为非本地（Non-Localhost）网络端口，通过启动参数“–bind-address”设置该值；用于基于策略的授权，默认不启动HTTPS安全访问控制
[root@localhost k8s]# netstat -ntap | grep 6443 tcp 0 0 192.168.245.211:6443 0.0.0.0:* LISTEN 73461/kube-apiserve tcp 0 0 192.168.245.211:6443 192.168.245.211:52656 ESTABLISHED 73461/kube-apiserve tcp 0 0 192.168.245.211:52656 192.168.245.211:6443 ESTABLISHED 73461/kube-apiserve [root@localhost k8s]# netstat -ntap | grep 8080 tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 73461/kube-apiserve查看版本信息（必须保证apiserver启动正常，不然无法查询到server的版本信息）
[root@master01 ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}Client Version：kubectl版本信息 Server Version：k8s版本
查看集群健康状态
[root@localhost k8s]# /opt/kubernetes/bin/kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-2 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"}将命令分发到node节点：
[root@localhost bin]# pwd /root/k8s/kubernetes/server/bin [root@localhost bin]# scp kubelet kube-proxy root@192.168.245.209:/opt/kubernetes/bin/ root@192.168.245.209's password: kubelet 100% 168MB 121.2MB/s 00:01 kube-proxy 100% 48MB 107.3MB/s 00:00 [root@localhost bin]# scp kubelet kube-proxy root@192.168.245.210:/opt/kubernetes/bin/ root@192.168.245.210's password: kubelet 100% 168MB 130.4MB/s 00:01 kube-proxy 100% 48MB 138.5MB/s 00:00新建一个kubeconfig目录存放kubelet的配置文件
[root@localhost k8s]# mkdir kubeconfig [root@localhost k8s]# cd kubeconfig/将写好的kubeconfig.sh拷贝进去并改名
[root@localhost kubeconfig]# ls kubeconfig.sh [root@localhost kubeconfig]# mv kubeconfig.sh kubeconfig修改参数，指定token
[root@localhost kubeconfig]# vim kubeconfig # 设置客户端认证参数 kubectl config set-credentials kubelet-bootstrap \ --token=49973b42a12aa4d513019a15e3661e2d \ [root@localhost kubeconfig]# export PATH=$PATH:/opt/kubernetes/bin/配置kubelet，会生成bootstrap.kubeconfig和kube-proxy.kubeconfig配置文件
kubelet通过bootstrap.kubeconfig知道apiserver的地址，拿着token去找apiserver认证身份，然后apiserver给它授权，绑定角色，这时候它就可以向apiserver申请证书认证了，apiserver会自动给它生成证书，实际上是controller-manager签发的，然后kubelet以后就拿着这个证书和apiserver通信了（具体步骤在后面）
[root@localhost kubeconfig]# bash kubeconfig 192.168.245.211 /root/k8s/k8s-cert/ Cluster "kubernetes" set. User "kubelet-bootstrap" set. Context "default" created. Switched to context "default". Cluster "kubernetes" set. User "kube-proxy" set. Context "default" created. Switched to context "default". [root@localhost kubeconfig]# ls bootstrap.kubeconfig kubeconfig kube-proxy.kubeconfig把这2个文件分发到集群2个node节点上
[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig [root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig[root@localhost kubeconfig]#
kubelet采用TLS Bootstrapping 机制，自动完成到kube-apiserver的注册，在node节点量较大或者后期自动扩容时非常有用。 Master apiserver启用TLS认证后，Node节点kubelet组件想要加入集群，必须使用CA签发的有效证书才能与apiserver通信， 当Node节点很多时，签署证书是一件很繁琐的事情，因此有了TLS Bootstrapping机制，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。
kubelet启动时向kube-apiserver发送TLS bootstrapping请求，需要先将bootstrap token文件中的kubelet-bootstrap用户赋予ClusterRole system:node-bootstrapper 角色（通过kubectl get clusterroles可查询） 然后kubelet才有权限创建认证请求，通过创建ClusterRoleBinding可实现；
# --user=kubelet-bootstrap 指定用户名，这里即文件/opt/kubernetes/cfg/token.csv中指定的用户名同时也需要写入kubeconfig文件/opt/kubernetes/cfg/bootstrap.kubeconfig
TLS bootstrap用户授权：
[root@localhost kubeconfig]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap creatednode01节点启动kubelet服务
[root@node1 bin]# bash kubelet.sh 192.168.245.209 Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@node1 bin]# ps aux | grep kube root 9112 17.3 2.5 473792 46952 ? Ssl 17:39 0:01 /opt/kubernetes/bin/kubelet --logtostderr=true --v=4 --hostname-override=192.168.245.209 --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --config=/opt/kubernetes/cfg/kubelet.config --cert-dir=/opt/kubernetes/ssl --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0 root 9127 0.0 0.0 112724 988 pts/3 S+ 17:39 0:00 grep --color=auto kubekubelet 首次启动向 kube-apiserver 发送证书签名请求，必须由 kubernetes 系统允许通过后，才会将该 node 加入到集群。
查看未授权的集群认证请求，处于”Pending”状态
[root@localhost ~]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-5zTiKXjy4yGAbkEHJOlepjY4o7_PWA14HX2x0cu4_R4 77s kubelet-bootstrap Pending授权node1加入群集
[root@localhost ~]# kubectl certificate approve node-csr-5zTiKXjy4yGAbkEHJOlepjY4o7_PWA14HX2x0cu4_R4 certificatesigningrequest.certificates.k8s.io/node-csr-5zTiKXjy4yGAbkEHJOlepjY4o7_PWA14HX2x0cu4_R4 approved在node01节点操作，启动proxy服务
[root@node1 ~]# bash proxy.sh 192.168.245.209 Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.把/opt/kubernetes/下的所有文件拷贝到node2
[root@node1 ~]# scp -r /opt/kubernetes/ root@192.168.245.210:/opt/把服务启动文件也拷贝到node2
[root@node1 ~]# scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.245.210:/usr/lib/systemd/system/ root@192.168.245.210's password: kubelet.service 100% 264 129.3KB/s 00:00 kube-proxy.service 100% 231 260.7KB/s 00:00部署node2上的kubelet和kube-proxy服务
[root@node2 ~]# cd /opt/kubernetes/ssl/ [root@node2 ssl]# rm -rf * [root@node2 ssl]# cd ../cfg/ [root@node2 cfg]# vim kubelet KUBELET_OPTS="--logtostderr=true \ --v=4 \ --hostname-override=192.168.245.210 \ //修改为自己的节点ip [root@node2 cfg]# vim kubelet.config kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 192.168.245.210 //修改为自己的节点ip [root@node2 cfg]# vim kube-proxy KUBE_PROXY_OPTS="--logtostderr=true \ --v=4 \ --hostname-override=192.168.245.210 \ //修改为自己的节点ip启动kubelet和kube-proxy并设置开机自启
[root@node2 cfg]# systemctl start kubelet.service [root@node2 cfg]# systemctl enable kubelet.service Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@node2 cfg]# systemctl start kube-proxy.service [root@node2 cfg]# systemctl enable kube-proxy.service Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.到master节点上发现未授权的node2请求，授权node2加入集群
[root@localhost ~]# kubectl certificate approve node-csr-P3996HQxx_2PLeo9bxBu7TVPcWgbAWqla5yj8Wa_5ks certificatesigningrequest.certificates.k8s.io/node-csr-P3996HQxx_2PLeo9bxBu7TVPcWgbAWqla5yj8Wa_5ks approved通过csr请求，状态变更为”Approved, Issued”
[root@localhost ~]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-5zTiKXjy4yGAbkEHJOlepjY4o7_PWA14HX2x0cu4_R4 27m kubelet-bootstrap Approved,Issued node-csr-P3996HQxx_2PLeo9bxBu7TVPcWgbAWqla5yj8Wa_5ks 111s kubelet-bootstrap Approved,Issued这时可以查看集群节点已ready
[root@localhost ~]# kubectl get nodes //kubectl get node也行 NAME STATUS ROLES AGE VERSION 192.168.245.209 Ready <none> 14m v1.12.3 192.168.245.210 Ready <none> 34s v1.12.3从master01上拷贝证书文件和各服务配置文件到master02
[root@master01 ~]# scp -r /opt/kubernetes/ root@192.168.245.206:/opt从master01拷贝服务启动文件到master02
[root@master01 ~]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.245.206:/usr/lib/systemd/system/ root@192.168.245.206's password: kube-apiserver.service 100% 282 266.8KB/s 00:00 kube-controller-manager.service 100% 317 404.2KB/s 00:00 kube-scheduler.service 100% 281 564.1KB/s 00:00apiserver要改成自己的ip地址
[root@master02 cfg]# vim kube-apiserver --bind-address=192.168.245.206 \ --advertise-address=192.168.245.206 \拷贝etcd证书和etcd通讯
[root@master01 ~]# scp -r /opt/etcd/ root@192.168.245.206:/opt/ root@192.168.245.206's password: etcd 100% 523 146.4KB/s 00:00 etcd 100% 18MB 114.7MB/s 00:00 etcdctl 100% 15MB 110.3MB/s 00:00 ca-key.pem 100% 1679 1.1MB/s 00:00 ca.pem 100% 1265 344.2KB/s 00:00 server-key.pem 100% 1679 754.6KB/s 00:00 server.pem 100% 1338 678.1KB/s 00:00master02上直接启动各项服务并都设置开机自启
[root@master02 cfg]# systemctl start kube-apiserver.service [root@master02 cfg]# systemctl enable kube-apiserver.service Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service. [root@master02 cfg]# systemctl start kube-controller-manager.service [root@master02 cfg]# systemctl enable kube-controller-manager.service Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service. [root@master02 cfg]# systemctl start kube-scheduler.service [root@master02 cfg]# systemctl enable kube-scheduler.service Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.设置环境变量可直接使用命令
[root@master02 cfg]# vim /etc/profile export PATH=$PATH:/opt/kubernetes/bin/ [root@master02 cfg]# source /etc/profile查看节点情况
[root@master02 cfg]# kubectl get node NAME STATUS ROLES AGE VERSION 192.168.245.209 Ready <none> 151m v1.12.3 192.168.245.210 Ready <none> 137m v1.12.3