目录
1、目录
1.1、什么是 Kubernetes?
Kubernetes
,简称 k8s(k,8 个字符,s)或者 kube
,是一个开源的 Linux 容器自动化运维平台,它消除了容器化应用程序在部署、伸缩时涉及到的许多手动操作。
Kubernetes
最开始是由 Google
的工程师设计开发的。Google
作为 Linux 容器技术的早期贡献者之一,曾公开演讲介绍 Google 如何将一切都运行于容器之中(这是 Google 的云服务背后的技术)。Google 一周内的容器部署超过 20 亿次,全部的工作都由内部平台 Borg 支撑。Borg 是 Kubernetes 的前身,几年来开发 Borg 的经验教训也成了影响 Kubernetes 中许多技术的主要因素。
1.2、Kubernetes 有哪些优势?
使用 Kubernetes
,你可以快速、高效地满足用户以下的需求:
- 快速精准地部署应用程序
- 即时伸缩你的应用程序
- 无缝展现新特征
- 限制硬件用量仅为所需资源
Kubernetes
的优势
- 可移动: 公有云、私有云、混合云、多态云
- 可扩展: 模块化、插件化、可挂载、可组合
- 自修复: 自动部署、自动重启、自动复制、自动伸缩
Google
公司于 2014 年启动了 Kubernetes
项目。Kubernetes
是在 Google 的长达 15 年的成规模的产品级任务的经验下构建的,结合了来自社区的最佳创意和实践经验
2、环境准备
本文中的案例会有两台机器,他们的Host和IP地址如下
IP地址 | 主机名 |
---|---|
10.0.0.100 | c0(master) |
10.0.0.101 | c1(master) |
10.0.0.102 | c2 |
10.0.0.103 | c3 |
四台机器的 host
以 c0
为例:
[root@c0 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.100 c0
10.0.0.101 c1
10.0.0.102 c2
10.0.0.103 c3
2.1、网络配置
以下以c0
为例
[root@c0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eth0
UUID=6d8d9ad6-37b5-431a-ab16-47d0aa00d01f
DEVICE=eth0
ONBOOT=yes
IPADDR0=10.0.0.100
PREFIXO0=24
GATEWAY0=10.0.0.1
DNS1=10.0.0.1
DNS2=8.8.8.8
重启网络:
[root@c0 ~]# service network restart
更改源为阿里云
[root@c0 ~]# yum install -y wget
[root@c0 ~]# cd /etc/yum.repos.d/
[root@c0 yum.repos.d]# mv CentOS-Base.repo CentOS-Base.repo.bak
[root@c0 yum.repos.d]# wget http://mirrors.aliyun.com/repo/Centos-7.repo
[root@c0 yum.repos.d]# wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
[root@c0 yum.repos.d]# yum clean all
[root@c0 yum.repos.d]# yum makecache
安装网络工具包和基础工具包
[root@c0 ~]# yum install net-tools checkpolicy gcc dkms foomatic openssh-server bash-completion -y
2.2、更改 HOSTNAME
在四台机器上依次设置 hostname
,以下以c0
为例
[root@c0 ~]# hostnamectl --static set-hostname c0
[root@c0 ~]# hostnamectl status
Static hostname: c0
Icon name: computer-vm
Chassis: vm
Machine ID: 04c3f6d56e788345859875d9f49bd4bd
Boot ID: ba02919abe4245aba673aaf5f778ad10
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-957.el7.x86_64
Architecture: x86-64
2.3、配置ssh免密码登录登录
每一台机器都单独生成
[root@c0 ~]# ssh-keygen
#一路按回车到最后
将 ssh-keygen
生成的密钥,分别复制到其他三台机器,以下以 c0
为例
[root@c0 ~]# ssh-copy-id c0
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c0 (10.0.0.100)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c0's password:
[root@c0 ~]# rm -rf ~/.ssh/known_hosts
[root@c0 ~]# clear
[root@c0 ~]# ssh-copy-id c0
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c0 (10.0.0.100)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c0's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'c0'"
and check to make sure that only the key(s) you wanted were added.
[root@c0 ~]# ssh-copy-id c1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c1 (10.0.0.101)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'c1'"
and check to make sure that only the key(s) you wanted were added.
[root@c0 ~]# ssh-copy-id c2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c2 (10.0.0.102)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c2's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'c2'"
and check to make sure that only the key(s) you wanted were added.
[root@c0 ~]# ssh-copy-id c3
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c3 (10.0.0.103)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c3's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'c3'"
and check to make sure that only the key(s) you wanted were added.
测试密钥是否配置成功
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N hostname; done;
c0
c1
c2
c3
2.4、关闭防火墙
在每一台机器上运行以下命令,以 c0
为例:
[root@c0 ~]# systemctl stop firewalld && systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
2.5、关闭交换分区
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N swapoff -a; done;
关闭前和关闭后,可以使用
free -h
命令查看swap的状态,关闭后的total应该是0
在每一台机器上编辑配置文件: /etc/fstab
, 注释最后一条/dev/mapper/centos-swap swap
,以c0
为例
[root@c0 ~]# sed -i "s//dev/mapper/centos-swap/# /dev/mapper/centos-swap/" /etc/fstab
[root@c1 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Jan 28 11:49:11 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=93572ab6-90da-4cfe-83a4-93be7ad8597c /boot xfs defaults 0 0
# /dev/mapper/centos-swap swap swap defaults 0 0
2.6、关闭 SeLinux
在每一台机器上,关闭 SeLinux
,以 c0
为例
[root@c0 ~]# sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
[root@c0 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
SELinux就是安全加强的Linux
2.7、安装 NTP
安装 NTP
时间同步工具,并启动 NTP
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N yum install ntp -y; done;
在每一台机器上,设置 NTP
开机启动
[root@c0 ~]# systemctl enable ntpd && systemctl start ntpd
依次查看每台机器上的时间:
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N date; done;
Sat Feb 9 18:11:48 CST 2019
Sat Feb 9 18:11:48 CST 2019
Sat Feb 9 18:11:49 CST 2019
Sat Feb 9 18:11:49 CST 2019
2.8、安装及配置 CFSSL
使用 CFSSL
能够构建本地CA,生成后面需要使用的证书。
[root@c0 ~]# mkdir -p /home/work/_src
[root@c0 ~]# cd /home/work/_src
[root@c0 _src]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
[root@c0 _src]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
[root@c0 _src]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
[root@c0 _src]# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
[root@c0 _src]# mv cfssl_linux-amd64 /usr/local/bin/cfssl
[root@c0 _src]# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
[root@c0 _src]# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
2.9、创建安装目录
创建后面要用到的 ETCD
和 Kubernetes
使用目录
[root@c0 _src]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_app/k8s/etcd/{bin,cfg,ssl} -p; done;
[root@c0 _src]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_app/k8s/kubernetes/{bin,cfg,ssl,ssl_cert} -p; done;
[root@c0 _src]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_data/etcd -p; done;
2.10、升级内核
因为3.10版本内核且缺少 ip_vs_fo.ko
模块,将导致 kube-proxy
无法开启ipvs模式。ip_vs_fo.ko
模块的最早版本为3.19版本,这个内核版本在 RedHat
系列发行版的常见RPM源中是不存在的。
[root@c0 ~]# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
[root@c0 ~]# yum --enablerepo=elrepo-kernel install kernel-ml-devel kernel-ml -y
重启系统 reboot
后,手动选择新内核,然后输入以下命令,可以查看新内核的状态:
[root@c0 ~]# hostnamectl
Static hostname: c0
Icon name: computer-vm
Chassis: vm
Machine ID: 04c3f6d56e788345859875d9f49bd4bd
Boot ID: 40a19388698f4907bd233a8cff76f36e
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 4.20.7-1.el7.elrepo.x86_64
Architecture: x86-64
3、安装 Docker 18.06.1-ce
3.1、删除旧版本的 Docker
官方提供的删除方法
$ sudo yum remove docker
docker-client
docker-client-latest
docker-common
docker-latest
docker-latest-logrotate
docker-logrotate
docker-engine
另外一种删除旧版的 Docker
方法,先查询安装过的 Docker
[root@c0 ~]# yum list installed | grep docker
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
containerd.io.x86_64 1.2.2-3.el7 @docker-ce-stable
docker-ce.x86_64 3:18.09.1-3.el7 @docker-ce-stable
docker-ce-cli.x86_64 1:18.09.1-3.el7 @docker-ce-stable
删除已安装的 Docker
[root@c0 ~]# yum -y remove docker-ce.x86_64 docker-ce-cli.x86_64 containerd.io.x86_64
删除 Docker
镜像/容器
[root@c0 ~]# rm -rf /var/lib/docker
3.2、设置存储库
安装所需要的包,yum-utils
提供了 yum-config-manager
实用程序, device-mapper-persistent-data
和 lvm2
是 devicemapper
需要的存储驱动程序。
在每一台机器上操作,以 c0
为例
[root@c0 ~]# sudo yum install -y yum-utils device-mapper-persistent-data lvm2
[root@c0 ~]# sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
3.3、安装 Docker
[root@c0 ~]# sudo yum install docker-ce-18.06.1.ce-3.el7 -y
3.4、启动 Docker
[root@c0 ~]# systemctl enable docker && systemctl start docker
4、安装 ETCD 3.3.10
4.1、创建 ETCD 证书
4.1.1、生成 ETCD SERVER 证书用到的JSON请求文件
[root@c0 ~]# mkdir -p /home/work/_src/ssl_etcd
[root@c0 ~]# cd /home/work/_src/ssl_etcd
[root@c0 ssl_etcd]# cat << EOF | tee ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"etcd": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
默认策略,指定了证书的有效期是10年(87600h)
etcd策略,指定了证书的用途
signing, 表示该证书可用于签名其它证书;生成的 ca.pem 证书中 CA=TRUE
server auth:表示 client 可以用该 CA 对 server 提供的证书进行验证
client auth:表示 server 可以用该 CA 对 client 提供的证书进行验证
4.1.2、创建 ETCD CA 证书配置文件
[root@c0 ssl_etcd]# cat << EOF | tee ca-csr.json
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
4.1.3、创建 ETCD SERVER 证书配置文件
[root@c0 ssl_etcd]# cat << EOF | tee server-csr.json
{
"CN": "etcd",
"hosts": [
"10.0.0.100",
"10.0.0.101",
"10.0.0.102",
"10.0.0.103"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
4.1.4、生成 ETCD CA 证书和私钥
[root@c0 ssl_etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2019/02/14 18:44:37 [INFO] generating a new CA key and certificate from CSR
2019/02/14 18:44:37 [INFO] generate received request
2019/02/14 18:44:37 [INFO] received CSR
2019/02/14 18:44:37 [INFO] generating key: rsa-2048
2019/02/14 18:44:38 [INFO] encoded CSR
2019/02/14 18:44:38 [INFO] signed certificate with serial number 384346866475232855604658229421854651219342845660
[root@c0 ssl_etcd]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem server-csr.json
4.1.5、生成 ETCD SERVER 证书和私钥
[root@c0 ssl_etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server
2019/02/09 20:52:57 [INFO] generate received request
2019/02/09 20:52:57 [INFO] received CSR
2019/02/09 20:52:57 [INFO] generating key: rsa-2048
2019/02/09 20:52:57 [INFO] encoded CSR
2019/02/09 20:52:57 [INFO] signed certificate with serial number 373071566605311458179949133441319838683720611466
2019/02/09 20:52:57 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@c0 ssl_etcd]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem server.csr server-csr.json server-key.pem server.pem
[root@c0 _src]# cp server.pem server-key.pem /home/work/_app/k8s/etcd/ssl/
将生成的证书,复制到 etchd
使用目录
[root@c0 ssl_etcd]# cp *.pem /home/work/_app/k8s/etcd/ssl/
4.2、安装 ETCD
4.2.1、下载 ETCD
[root@c0 ssl_etcd]# cd /home/work/_src/
[root@c0 _src]# wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
[root@c0 _src]# tar -xvf etcd-v3.3.10-linux-amd64.tar.gz
[root@c0 _src]# cd etcd-v3.3.10-linux-amd64
[root@c0 etcd-v3.3.10-linux-amd64]# cp etcd etcdctl /home/work/_app/k8s/etcd/bin/
4.2.2、创建 ETCD 系统启动文件
创建 /usr/lib/systemd/system/etcd.service
文件并保存,内容如下:
[root@c0 etcd-v3.3.10-linux-amd64]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/home/work/_app/k8s/etcd/cfg/etcd.conf
ExecStart=/home/work/_app/k8s/etcd/bin/etcd
--name=${ETCD_NAME}
--data-dir=${ETCD_DATA_DIR}
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS}
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS}
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS}
--initial-cluster=${ETCD_INITIAL_CLUSTER}
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN}
--initial-cluster-state=new
--cert-file=/home/work/_app/k8s/etcd/ssl/server.pem
--key-file=/home/work/_app/k8s/etcd/ssl/server-key.pem
--peer-cert-file=/home/work/_app/k8s/etcd/ssl/server.pem
--peer-key-file=/home/work/_app/k8s/etcd/ssl/server-key.pem
--trusted-ca-file=/home/work/_app/k8s/etcd/ssl/ca.pem
--peer-trusted-ca-file=/home/work/_app/k8s/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
4.2.3、将 ETCD 启动文件、证书文件、系统启动文件复制到其他节点
[root@c0 ~]# for N in $(seq 0 3); do scp -r /home/work/_app/k8s/etcd c$N:/home/work/_app/k8s/; done;
[root@c0 ~]# for N in $(seq 0 3); do scp -r /usr/lib/systemd/system/etcd.service c$N:/usr/lib/systemd/system/etcd.service; done;
4.2.4、ETCD 主配置文件
在 c0
创建 /home/work/_app/k8s/etcd/cfg/etcd.conf
文件,内容如下:
[root@c0 _src]# cat << EOF | tee /home/work/_app/k8s/etcd/cfg/etcd.conf
#[Member]
# ETCD的节点名
ETCD_NAME="etcd00"
# ETCD的数据存储目录
ETCD_DATA_DIR="/home/work/_data/etcd"
# 该节点与其他节点通信时所监听的地址列表,多个地址使用逗号隔开,其格式可以划分为scheme://IP:PORT,这里的scheme可以是http、https
ETCD_LISTEN_PEER_URLS="https://10.0.0.100:2380"
# 该节点与客户端通信时监听的地址列表
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.100:2379"
#[Clustering]
# 该成员节点在整个集群中的通信地址列表,这个地址用来传输集群数据的地址。因此这个地址必须是可以连接集群中所有的成员的。
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.100:2380"
# 配置集群内部所有成员地址,其格式为:ETCD_NAME=ETCD_INITIAL_ADVERTISE_PEER_URLS,如果有多个使用逗号隔开
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.100:2379"
ETCD_INITIAL_CLUSTER="etcd00=https://10.0.0.100:2380,etcd01=https://10.0.0.101:2380,etcd02=https://10.0.0.102:2380,etcd03=https://10.0.0.103:2380"
# 初始化集群token
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
# 初始化集群状态,new表示新建
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/home/work/_app/k8s/etcd/ssl/server.pem"
ETCD_KEY_FILE="/home/work/_app/k8s/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/home/work/_app/k8s/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/home/work/_app/k8s/etcd/ssl/server.pem"
ETCD_PEER_KEY_FILE="/home/work/_app/k8s/etcd/ssl/server-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/home/work/_app/k8s/etcd/ssl/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF
在 c1
创建 /home/work/_app/k8s/etcd/cfg/etcd.conf
文件,内容如下:
[root@c1 _src]# cat << EOF | tee /home/work/_app/k8s/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/home/work/_data/etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.101:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.101:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.101:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.101:2379"
ETCD_INITIAL_CLUSTER="etcd00=https://10.0.0.100:2380,etcd01=https://10.0.0.101:2380,etcd02=https://10.0.0.102:2380,etcd03=https://10.0.0.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/home/work/_app/k8s/etcd/ssl/server.pem"
ETCD_KEY_FILE="/home/work/_app/k8s/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/home/work/_app/k8s/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/home/work/_app/k8s/etcd/ssl/server.pem"
ETCD_PEER_KEY_FILE="/home/work/_app/k8s/etcd/ssl/server-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/home/work/_app/k8s/etcd/ssl/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF
在 c2
创建 /home/work/_app/k8s/etcd/cfg/etcd.conf
文件,内容如下:
[root@c2 _src]# cat << EOF | tee /home/work/_app/k8s/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/home/work/_data/etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.102:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.102:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.102:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.102:2379"
ETCD_INITIAL_CLUSTER="etcd00=https://10.0.0.100:2380,etcd01=https://10.0.0.101:2380,etcd02=https://10.0.0.102:2380,etcd03=https://10.0.0.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/home/work/_app/k8s/etcd/ssl/server.pem"
ETCD_KEY_FILE="/home/work/_app/k8s/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/home/work/_app/k8s/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/home/work/_app/k8s/etcd/ssl/server.pem"
ETCD_PEER_KEY_FILE="/home/work/_app/k8s/etcd/ssl/server-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/home/work/_app/k8s/etcd/ssl/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF
在 c3
创建 /home/work/_app/k8s/etcd/cfg/etcd.conf
文件,内容如下:
[root@c3 _src]# cat << EOF | tee /home/work/_app/k8s/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/home/work/_data/etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.103:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.103:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.103:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.103:2379"
ETCD_INITIAL_CLUSTER="etcd00=https://10.0.0.100:2380,etcd01=https://10.0.0.101:2380,etcd02=https://10.0.0.102:2380,etcd03=https://10.0.0.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/home/work/_app/k8s/etcd/ssl/server.pem"
ETCD_KEY_FILE="/home/work/_app/k8s/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/home/work/_app/k8s/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/home/work/_app/k8s/etcd/ssl/server.pem"
ETCD_PEER_KEY_FILE="/home/work/_app/k8s/etcd/ssl/server-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/home/work/_app/k8s/etcd/ssl/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF
4.2.5、启动 ETCD 服务
在每一台节点机器上单独运行
[root@c0 _src]# systemctl daemon-reload && systemctl enable etcd && systemctl start etcd
4.2.6、检查 ETCD 服务运行状态
[root@c0 _src]# /home/work/_app/k8s/etcd/bin/etcdctl --ca-file=/home/work/_app/k8s/etcd/ssl/ca.pem --cert-file=/home/work/_app/k8s/etcd/ssl/server.pem --key-file=/home/work/_app/k8s/etcd/ssl/server-key.pem cluster-health
member 2cba54b8e3ba988a is healthy: got healthy result from https://10.0.0.103:2379
member 7c12135a398849e3 is healthy: got healthy result from https://10.0.0.102:2379
member 99c2fd4fe11e28d9 is healthy: got healthy result from https://10.0.0.100:2379
member f2fd0c12369e0d75 is healthy: got healthy result from https://10.0.0.101:2379
cluster is healthy
4.2.7、查看 ETCD 集群成员信息
[root@c0 _src]# /home/work/_app/k8s/etcd/bin/etcdctl --ca-file=/home/work/_app/k8s/etcd/ssl/ca.pem --cert-file=/home/work/_app/k8s/etcd/ssl/server.pem --key-file=/home/work/_app/k8s/etcd/ssl/server-key.pem member list
2cba54b8e3ba988a: name=etcd03 peerURLs=https://10.0.0.103:2380 clientURLs=https://10.0.0.103:2379 isLeader=false
7c12135a398849e3: name=etcd02 peerURLs=https://10.0.0.102:2380 clientURLs=https://10.0.0.102:2379 isLeader=false
99c2fd4fe11e28d9: name=etcd00 peerURLs=https://10.0.0.100:2380 clientURLs=https://10.0.0.100:2379 isLeader=true
f2fd0c12369e0d75: name=etcd01 peerURLs=https://10.0.0.101:2380 clientURLs=https://10.0.0.101:2379 isLeader=false
5、安装 Flannel v0.11.0
5.1、Flanneld 网络安装
Flannel
实质上是一种“覆盖网络(overlay network)”,也就是将TCP数据包装在另一种网络包里面进行路由转发和通信,目前已经支持UDP、VxLAN、AWS VPC和GCE路由等数据转发方式。Flannel
在 Kubernetes
中用于配置第三层(网络层)网络结构。
Flannel
负责在集群中的多个节点之间提供第 3 层 IPv4 网络。Flannel
不控制容器如何与主机联网,只负责主机之间如何传输流量。但是,Flannel
确实为 Kubernetes
提供了 CNI
插件,并提供了与 Docker
集成的指导。
没有
Flanneld
网络,Node
节点间的 pod 不能通信,只能Node
内通信。
有Flanneld
服务启动时主要做了以下几步的工作: 从ETCD
中获取NetWork
的配置信息划分Subnet
,并在ETCD
中进行注册,将子网信息记录到/run/flannel/subnet.env
中
5.2、向 ETCD 集群写入网段信息
[root@c0 _src]# /home/work/_app/k8s/etcd/bin/etcdctl --ca-file=/home/work/_app/k8s/etcd/ssl/ca.pem --cert-file=/home/work/_app/k8s/etcd/ssl/server.pem --key-file=/home/work/_app/k8s/etcd/ssl/server-key.pem --endpoints="https://10.0.0.100:2379,https://10.0.0.101:2379,https://10.0.0.102:2379,https://10.0.0.103:2379" set /coreos.com/network/config '{ "Network": "10.244.0.0/16", "Backend": {"Type": "vxlan"}}'
{ "Network": "10.244.0.0/16", "Backend": {"Type": "vxlan"}}
Flanneld
当前版本 (v0.11.0) 不支持 ETCD v3,所以使用 ETCD v2 API 写入配置 key 和网段数据;
写入的Pod
网段 ${CLUSTER_CIDR} 必须是 /16 段地址,必须与 kube-controller-manager 的 –cluster-cidr 参数值一致;
5.3、安装 Flannel
[root@c0 _src]# pwd
/home/work/_src
[root@c0 _src]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@c0 _src]# tar -xvf flannel-v0.11.0-linux-amd64.tar.gz
[root@c0 _src]# mv flanneld mk-docker-opts.sh /home/work/_app/k8s/kubernetes/bin/
5.4、配置 Flannel
创建 /home/work/_app/k8s/kubernetes/cfg/flanneld
文件并保存,写入以下内容:
[root@c0 _src]# cat /home/work/_app/k8s/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://10.0.0.100:2379,https://10.0.0.101:2379,https://10.0.0.102:2379,https://10.0.0.103:2379 -etcd-cafile=/home/work/_app/k8s/etcd/ssl/ca.pem -etcd-certfile=/home/work/_app/k8s/etcd/ssl/server.pem -etcd-keyfile=/home/work/_app/k8s/etcd/ssl/server-key.pem"
5.5、创建 Flannel 系统启动文件
创建 /usr/lib/systemd/system/flanneld.service
文件并保存,内容如下:
[root@c0 _src]# cat /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/home/work/_app/k8s/kubernetes/cfg/flanneld
ExecStart=/home/work/_app/k8s/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/home/work/_app/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
mk-docker-opts.sh
脚本将分配给Flanneld
的 Pod 子网网段信息写入/run/flannel/docker
文件,后续Docker
启动时 使用这个文件中的环境变量配置docker0
网桥.Flanneld
使用系统缺省路由所在的接口与其它节点通信,对于有多个网络接口(如内网和公网)的节点,可以用 -iface 参数指定通信接口;
5.6、配置 Docker 启动指定子网段
编辑 /usr/lib/systemd/system/docker.service
文件,内容如下:
[root@c0 _src]# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
# 加入环境变量的配件文件,并在 ExecStart 附加参数
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
5.7、将 Flannel 相关文件复制到其他机器
主要复制 Flannel
执行文件、Flannel
配置文件、Flannel
系统启动文件、Docker
系统启动文件
[root@c0 _src]# for N in $(seq 0 3); do scp -r /home/work/_app/k8s/kubernetes/* c$N:/home/work/_app/k8s/kubernetes/; done;
[root@c0 _src]# for N in $(seq 0 3); do scp -r /usr/lib/systemd/system/docker.service c$N:/usr/lib/systemd/system/docker.service; done;
[root@c0 _src]# for N in $(seq 0 3); do scp -r /usr/lib/systemd/system/flanneld.service c$N:/usr/lib/systemd/system/flanneld.service; done;
5.8、启动服务
在每一台机器上单独运行,以 c0
为例:
[root@c0 _src]# systemctl daemon-reload && systemctl stop docker && systemctl enable flanneld && systemctl start flanneld && systemctl start docker
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
启动 Flannel 前要关闭 Docker 及相关的 kubelet 这样 Flannel 才会覆盖 docker0 网桥
5.9、查看 Flannel 服务设置 docker0 网桥状态
[root@c0 _src]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:1c:42:50:8c:6a brd ff:ff:ff:ff:ff:ff
inet 10.0.0.100/8 brd 10.255.255.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::49d:e3e6:c623:9582/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 3e:80:5d:97:53:c4 brd ff:ff:ff:ff:ff:ff
inet 10.172.46.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::3c80:5dff:fe97:53c4/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:9e:df:b9:87 brd ff:ff:ff:ff:ff:ff
inet 10.172.46.1/24 brd 10.172.46.255 scope global docker0
valid_lft forever preferred_lft forever
5.10、验证 Flannel 服务
[root@c0 _src]# for N in $(seq 0 3); do ssh c$N cat /run/flannel/subnet.env ; done;
DOCKER_OPT_BIP="--bip=10.172.46.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.172.46.1/24 --ip-masq=false --mtu=1450"
DOCKER_OPT_BIP="--bip=10.172.90.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.172.90.1/24 --ip-masq=false --mtu=1450"
DOCKER_OPT_BIP="--bip=10.172.5.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.172.5.1/24 --ip-masq=false --mtu=1450"
DOCKER_OPT_BIP="--bip=10.172.72.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.172.72.1/24 --ip-masq=false --mtu=1450"
6、安装Kubernetes
6.1、创建 Kubernetes 需要的证书
6.1.1、生成 Kubernetes 证书请求的JSON请求文件
[root@c0 ~]# cd /home/work/_app/k8s/kubernetes/ssl/
[root@c0 ssl]# cat << EOF | tee ca-config.json
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"server": {
"usages": [
"signing",
"key encipherment",
"server auth"
],
"expiry": "8760h"
},
"client": {
"usages": [
"signing",
"key encipherment",
"client auth"
],
"expiry": "8760h"
}
}
}
}
EOF
6.1.2、生成 Kubernetes CA 配置文件和证书
[root@c0 ssl]# cat << EOF | tee ca-csr.json
{
"CN": "kubernetes CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
初始化一个 Kubernetes CA
证书
[root@c0 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2019/02/15 00:33:49 [INFO] generating a new CA key and certificate from CSR
2019/02/15 00:33:49 [INFO] generate received request
2019/02/15 00:33:49 [INFO] received CSR
2019/02/15 00:33:49 [INFO] generating key: rsa-2048
2019/02/15 00:33:49 [INFO] encoded CSR
2019/02/15 00:33:49 [INFO] signed certificate with serial number 19178419085322799829088564182237651657158569707
[root@c0 ssl]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
6.1.3、生成 Kube API Server 配置文件和证书
创建证书配置文件
[root@c0 ssl]# cat << EOF | tee kube-apiserver-server-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"10.0.0.1",
"10.0.0.100",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "API Server"
}
]
}
EOF
生成 kube-apiserver
证书
[root@c0 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kube-apiserver-server-csr.json | cfssljson -bare kube-apiserver-server
2019/02/15 00:40:17 [INFO] generate received request
2019/02/15 00:40:17 [INFO] received CSR
2019/02/15 00:40:17 [INFO] generating key: rsa-2048
2019/02/15 00:40:17 [INFO] encoded CSR
2019/02/15 00:40:17 [INFO] signed certificate with serial number 73791614256163825800646464302566039201359288928
2019/02/15 00:40:17 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@c0 ssl]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem kube-apiserver-server.csr kube-apiserver-server-csr.json kube-apiserver-server-key.pem kube-apiserver-server.pem
6.1.4、生成 kubelet client 配置文件和证书
创建证书配置文件
[root@c0 ssl]# cat << EOF | tee kubelet-client-csr.json
{
"CN": "kubelet",
"hosts": [""],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"O": "k8s",
"OU": "Kubelet",
"ST": "Beijing"
}
]
}
EOF
生成 kubelet client
证书
[root@c0 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kubelet-client-csr.json | cfssljson -bare kubelet-client
2019/02/15 00:44:43 [INFO] generate received request
2019/02/15 00:44:43 [INFO] received CSR
2019/02/15 00:44:43 [INFO] generating key: rsa-2048
2019/02/15 00:44:43 [INFO] encoded CSR
2019/02/15 00:44:43 [INFO] signed certificate with serial number 285651868701760571162897366975202301612567414209
2019/02/15 00:44:43 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@c0 ssl]# ls
ca-config.json ca-csr.json ca.pem kube-apiserver-server-csr.json kube-apiserver-server.pem kubelet-client-csr.json kubelet-client.pem
ca.csr ca-key.pem kube-apiserver-server.csr kube-apiserver-server-key.pem kubelet-client.csr kubelet-client-key.pem
6.1.5、生成 Kube-Proxy 配置文件和证书
创建证书配置文件
[root@c0 ssl]# cat << EOF | tee kube-proxy-client-csr.json
{
"CN": "system:kube-proxy",
"hosts": [""],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"O": "k8s",
"OU": "System",
"ST": "Beijing"
}
]
}
EOF
生成 Kube-Proxy
证书
[root@c0 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-client-csr.json | cfssljson -bare kube-proxy-client
2019/02/15 01:14:39 [INFO] generate received request
2019/02/15 01:14:39 [INFO] received CSR
2019/02/15 01:14:39 [INFO] generating key: rsa-2048
2019/02/15 01:14:39 [INFO] encoded CSR
2019/02/15 01:14:39 [INFO] signed certificate with serial number 535503934939407075396917222976858989138817338004
2019/02/15 01:14:39 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@c0 ssl]# ls
ca-config.json ca-csr.json ca.pem kube-apiserver-server-csr.json kube-apiserver-server.pem kubelet-client-csr.json kubelet-client.pem kube-proxy-client-csr.json kube-proxy-client.pem
ca.csr ca-key.pem kube-apiserver-server.csr kube-apiserver-server-key.pem kubelet-client.csr kubelet-client-key.pem kube-proxy-client.csr kube-proxy-client-key.pem
6.1.6、生成 kubectl 管理员配置文件和证书
创建 kubectl
管理员证书配置文件
[root@c0 ssl]# cat << EOF | tee kubernetes-admin-user.csr.json
{
"CN": "admin",
"hosts": [""],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"O": "k8s",
"OU": "Cluster Admins",
"ST": "Beijing"
}
]
}
EOF
生成 kubectl
管理员证书
[root@c0 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kubernetes-admin-user.csr.json | cfssljson -bare kubernetes-admin-user
2019/02/15 01:23:22 [INFO] generate received request
2019/02/15 01:23:22 [INFO] received CSR
2019/02/15 01:23:22 [INFO] generating key: rsa-2048
2019/02/15 01:23:22 [INFO] encoded CSR
2019/02/15 01:23:22 [INFO] signed certificate with serial number 724413523889121871668676123719532667068182658276
2019/02/15 01:23:22 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@c0 ssl]# ls
ca-config.json ca-key.pem kube-apiserver-server-csr.json kubelet-client.csr kubelet-client.pem kube-proxy-client-key.pem kubernetes-admin-user.csr.json
ca.csr ca.pem kube-apiserver-server-key.pem kubelet-client-csr.json kube-proxy-client.csr kube-proxy-client.pem kubernetes-admin-user-key.pem
ca-csr.json kube-apiserver-server.csr kube-apiserver-server.pem kubelet-client-key.pem kube-proxy-client-csr.json kubernetes-admin-user.csr kubernetes-admin-user.pem
6.1.7、将相关证书复制到 Kubernetes Node 节点
[root@c0 ~]# for N in $(seq 0 3); do scp -r /home/work/_app/k8s/kubernetes/ssl/*.pem c$N:/home/work/_app/k8s/kubernetes/ssl/; done;
6.2、部署 Kubernetes Master 节点并加入集群
Kubernetes Master 节点运行如下组件:
- APIServer
APIServer负责对外提供RESTful的kubernetes API的服务,它是系统管理指令的统一接口,任何对资源的增删该查都要交给APIServer处理后再交给etcd,如图,kubectl(kubernetes提供的客户端工具,该工具内部是对kubernetes API的调用)是直接和APIServer交互的。 - Schedule
schedule负责调度Pod到合适的Node上,如果把scheduler看成一个黑匣子,那么它的输入是pod和由多个Node组成的列表,输出是Pod和一个Node的绑定。 kubernetes目前提供了调度算法,同样也保留了接口。用户根据自己的需求定义自己的调度算法。 - Controller manager
如果APIServer做的是前台的工作的话,那么controller manager就是负责后台的。每一个资源都对应一个控制器。而control manager就是负责管理这些控制器的,比如我们通过APIServer创建了一个Pod,当这个Pod创建成功后,APIServer的任务就算完成了。 - ETCD
etcd是一个高可用的键值存储系统,kubernetes使用它来存储各个资源的状态,从而实现了Restful的API。 - Flannel
默认没有flanneld网络,Node节点间的pod不能通信,只能Node内通信,Flannel从etcd中获取network的配置信息 划分subnet,并在etcd中进行注册 将子网信息记录
kube-scheduler 和 kube-controller-manager 可以以集群模式运行,通过 leader 选举产生一个工作进程,其它进程处于阻塞模式。
6.2.1、下载文件并安装 Kubernetes Server
[root@c0 ~]# cd /home/work/_src/
[root@c0 _src]# wget https://dl.k8s.io/v1.13.0/kubernetes-server-linux-amd64.tar.gz
[root@c0 _src]# tar -xzvf kubernetes-server-linux-amd64.tar.gz
[root@c0 _src]# cd kubernetes/server/bin/
[root@c0 bin]# cp kube-scheduler kube-apiserver kube-controller-manager kubectl kubelet kube-proxy /home/work/_app/k8s/kubernetes/bin/
从 c0
复制 kubelet、kubectl、kube-proxy,同时复制到其他节点
[root@c0 ~]# cd /home/work/_src/kubernetes/server/bin/
[root@c0 bin]# for N in $(seq 1 3); do scp -r kubelet kubectl kube-proxy c$N:/home/work/_app/k8s/kubernetes/bin/; done;
kubelet 100% 108MB 120.3MB/s 00:00
kubectl 100% 37MB 120.0MB/s 00:00
kube-proxy 100% 33MB 113.7MB/s 00:00
kubelet 100% 108MB 108.0MB/s 00:00
kubectl 100% 37MB 108.6MB/s 00:00
kube-proxy 100% 33MB 106.1MB/s 00:00
kubelet 100% 108MB 117.8MB/s 00:00
kubectl 100% 37MB 116.6MB/s 00:00
kube-proxy 100% 33MB 119.0MB/s 00:00
6.2.2、部署 Apiserver
创建 TLS Bootstrapping Token
[root@c0 ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
4470210dbf9d9c57f8543bce4683c3ce
这里我们生成的随机Token是
4470210dbf9d9c57f8543bce4683c3ce
,记下来后面要用到
创建 /home/work/_app/k8s/kubernetes/cfg/token-auth-file
文件并保存,内容如下:
[root@c0 ~]# cat /home/work/_app/k8s/kubernetes/cfg/token-auth-file
4470210dbf9d9c57f8543bce4683c3ce,kubelet-bootstrap,10001,"system:kubelet-bootstrap"