使用kubeadm 部署多master 高可用集群
环境清单
- OS： Debian-9.5
- Docker 版本： 18.06
- Master 节点IP：
- 192.168.11.81
- 192.168.11.82
- 192.168.11.83
- LB 节点IP：192.168.11.80 (HAProxy)
- Pod网络： 10.30.0.0/16
- kubernetes版本： 1.13.3
在LB节点安装HAProxy
- 远程到LB节点安装haproxy
sudo apt-get install haproxy
- 配置
$ sudo vim /etc/haproxy/haproxy.cfg
global
...
default
...
frontend kubernetes
bind 192.168.11.80:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes
backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server k8s-master-01 192.168.11.81:6443 check fall 3 rise 2
server k8s-master-02 192.168.11.82:6443 check fall 3 rise 2
server k8s-master-03 192.168.11.83:6443 check fall 3 rise 2
- 重新启动服务
sudo systemctl restart haproxy
创建集群证书
- 下载创建证书工具
curl -s -L -o /usr/local/bin/cfssl https://down.vqiu.cn/package/container/cfssl/cfssl_linux-amd64
curl -s -L -o /usr/local/bin/cfssljson https://down.vqiu.cn/package/container/cfssl/cfssljson
chmod +x /usr/local/bin/{cfssl,cfssljson}
- 创建CA 证书配置文件
mkdir cas && cd cas
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": ["signing", "key encipherment", "server auth", "client auth"],
"expiry": "87600h"
}
}
}
}
EOF
cat > ca-csr.json <<EOF
{
"CN": "Kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Guangdong",
"L": "Guangzhou",
"O": "kube",
"OU": "vqiu co."
}
]
}
EOF
- 生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
- 为集群生成证书
cat > kubernetes-csr.json <<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Guangdong",
"L": "Guangzhou",
"O": "Kubernetes",
"OU": "Kubernetes"
}
]
}
EOF
- 生成证书与私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-hostname=192.168.11.81,192.168.11.82,192.168.11.83,192.168.11.80,127.0.0.1,localhost,kubernetes.default \
-profile=kubernetes kubernetes-csr.json | \
cfssljson -bare kubernetes
- 目前已经创建证书有：
# ls -la
total 44
drwxr-xr-x 2 root root 4096 Feb 11 19:50 .
drwx------ 7 root root 4096 Feb 11 19:38 ..
-rw-r--r-- 1 root root 234 Feb 11 19:38 ca-config.json
-rw-r--r-- 1 root root 1009 Feb 11 19:40 ca.csr
-rw-r--r-- 1 root root 212 Feb 11 19:38 ca-csr.json
-rw------- 1 root root 1679 Feb 11 19:40 ca-key.pem
-rw-r--r-- 1 root root 1379 Feb 11 19:40 ca.pem
-rw-r--r-- 1 root root 1021 Feb 11 19:50 kubernetes.csr
-rw-r--r-- 1 root root 218 Feb 11 19:49 kubernetes-csr.json
-rw------- 1 root root 1679 Feb 11 19:50 kubernetes-key.pem
-rw-r--r-- 1 root root 1521 Feb 11 19:50 kubernetes.pem
- 将证书复制到各master节点：
scp ca.pem kubernetes.pem kubernetes-key.pem qiush@192.168.11.81:~
scp ca.pem kubernetes.pem kubernetes-key.pem qiush@192.168.11.82:~
scp ca.pem kubernetes.pem kubernetes-key.pem qiush@192.168.11.83:~
etcd 集群
- 创建etcd 目录，分别为配置目录(证书)，与数据库目录
$ sudo mkdir /etc/etcd /var/lib/etcd
- 将证书移动到
/etc/etcd目录
$ sudo mv ~/ca.pem ~/kubernetes.pem ~/kubernetes-key.pem /etc/etcd
- 下载etcd二进制文件
$ wget https://down.vqiu.cn/package/container/etcd-v3.3.11-linux-amd64.tar.gz
- 解压文件，并复制到
/usr/local/bin目录
$ tar xvzf etcd-v3.3.11-linux-amd64.tar.gz
$ sudo mv etcd-v3.3.11-linux-amd64/etcd* /usr/local/bin/
5.为各节点配置 systemd
- kube-master-01
cat >/etc/systemd/system/etcd.service<<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos
[Service]
ExecStart=/usr/local/bin/etcd \
--name 192.168.11.81 \
--cert-file=/etc/etcd/kubernetes.pem \
--key-file=/etc/etcd/kubernetes-key.pem \
--peer-cert-file=/etc/etcd/kubernetes.pem \
--peer-key-file=/etc/etcd/kubernetes-key.pem \
--trusted-ca-file=/etc/etcd/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ca.pem \
--peer-client-cert-auth \
--client-cert-auth \
--initial-advertise-peer-urls https://192.168.11.81:2380 \
--listen-peer-urls https://192.168.11.81:2380 \
--listen-client-urls https://192.168.11.81:2379,http://127.0.0.1:2379 \
--advertise-client-urls https://192.168.11.81:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster 192.168.11.81=https://192.168.11.81:2380,192.168.11.82=https://192.168.11.82:2380,192.168.11.83=https://192.168.11.83:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
- kube-master-02
cat >/etc/systemd/system/etcd.service<<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos
[Service]
ExecStart=/usr/local/bin/etcd \
--name 192.168.11.82 \
--cert-file=/etc/etcd/kubernetes.pem \
--key-file=/etc/etcd/kubernetes-key.pem \
--peer-cert-file=/etc/etcd/kubernetes.pem \
--peer-key-file=/etc/etcd/kubernetes-key.pem \
--trusted-ca-file=/etc/etcd/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ca.pem \
--peer-client-cert-auth \
--client-cert-auth \
--initial-advertise-peer-urls https://192.168.11.82:2380 \
--listen-peer-urls https://192.168.11.82:2380 \
--listen-client-urls https://192.168.11.82:2379,http://127.0.0.1:2379 \
--advertise-client-urls https://192.168.11.82:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster 192.168.11.81=https://192.168.11.81:2380,192.168.11.82=https://192.168.11.82:2380,192.168.11.83=https://192.168.11.83:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
- kube-master-03
cat >/etc/systemd/system/etcd.service<<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos
[Service]
ExecStart=/usr/local/bin/etcd \
--name 192.168.11.83 \
--cert-file=/etc/etcd/kubernetes.pem \
--key-file=/etc/etcd/kubernetes-key.pem \
--peer-cert-file=/etc/etcd/kubernetes.pem \
--peer-key-file=/etc/etcd/kubernetes-key.pem \
--trusted-ca-file=/etc/etcd/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ca.pem \
--peer-client-cert-auth \
--client-cert-auth \
--initial-advertise-peer-urls https://192.168.11.83:2380 \
--listen-peer-urls https://192.168.11.83:2380 \
--listen-client-urls https://192.168.11.83:2379,http://127.0.0.1:2379 \
--advertise-client-urls https://192.168.11.83:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster 192.168.11.81=https://192.168.11.81:2380,192.168.11.82=https://192.168.11.82:2380,192.168.11.83=https://192.168.11.83:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
- 启动服务
sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd
为所有master及worker节点安装Docker环境
- 配置sysctl内核参数
cat > /etc/sysctl.d/docker.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
- 添加镜像仓库
sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
- 安装docker镜像包
sudo apt update && sudo apt-get -y install docker-ce=18.06.1~ce~3-0~debian
- 配置镜像仓库
export repos=https://registry.docker-cn.com
cat >/etc/docker/daemon.json<<EOF
{
"registry-mirrors": ["${repos}"],
"log-driver": "json-file",
"log-opts": {
"max-size": "20m",
"max-file": "3",
"labels": "production_status",
"env": "os,customer"
}
}
EOF
安装 kubernetes环境
- 关闭 swap
sudo swapoff -a
内核不要开启 reuse
- 为系统添加kubunetes APT仓库
apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat >/etc/apt/sources.list.d/kubernetes.list<<EOF
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
初始化kubernetes master 集群
kube-master-01
- 远程到kube-master-01
ssh kube@kube-master-01
- 为kubeadm创建配置文件
cat > kube-master-config.yaml<<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 192.168.11.80
controlPlaneEndpoint: "192.168.11.80:6443"
etcd:
external:
endpoints:
- https://192.168.11.81:2379
- https://192.168.11.82:2379
- https://192.168.11.83:2379
caFile: /etc/etcd/ca.pem
certFile: /etc/etcd/kubernetes.pem
keyFile: /etc/etcd/kubernetes-key.pem
networking:
podSubnet: 10.30.0.0/16
apiServerExtraArgs:
apiserver-count: "3"
EOF
- 使用
kube-master-config.yaml文件初始化 master
sudo kubeadm init --config=kube-master-config.yaml
- 复制证书到其它2个master 节点
sudo scp -r /etc/kubernetes/pki kube@192.168.11.82:~
sudo scp -r /etc/kubernetes/pki kube@192.168.11.83:~
kube-master-02
-
远程到
kube-master-02 -
删除 apiserver.crt 和 apiserver.key.
$ rm ~/pki/apiserver.*
- 将证书文件移到
/etc/kubernetes/目录
$ sudo mv ~/pki /etc/kubernetes/
- 为kubeadm创建配置文件
cat > kube-master-config.yaml<<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 192.168.11.80
controlPlaneEndpoint: "192.168.11.80:6443"
etcd:
external:
endpoints:
- https://192.168.11.81:2379
- https://192.168.11.82:2379
- https://192.168.11.83:2379
caFile: /etc/etcd/ca.pem
certFile: /etc/etcd/kubernetes.pem
keyFile: /etc/etcd/kubernetes-key.pem
networking:
podSubnet: 10.30.0.0/16
apiServerExtraArgs:
apiserver-count: "3"
EOF
- 使用
kube-master-config.yaml文件初始化 master
$ sudo kubeadm init --config=kube-master-config.yaml
kube-master-03
- 远程到
kube-master-03 - 删除 apiserver.crt 和 apiserver.key.
$ rm ~/pki/apiserver.*
- 将证书文件移到
/etc/kubernetes/目录
$ sudo mv ~/pki /etc/kubernetes/
- 为kubeadm创建配置文件
cat > kube-master-config.yaml<<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 192.168.11.80
controlPlaneEndpoint: "192.168.11.80:6443"
etcd:
external:
endpoints:
- https://192.168.11.81:2379
- https://192.168.11.82:2379
- https://192.168.11.83:2379
caFile: /etc/etcd/ca.pem
certFile: /etc/etcd/kubernetes.pem
keyFile: /etc/etcd/kubernetes-key.pem
networking:
podSubnet: 10.30.0.0/16
apiServerExtraArgs:
apiserver-count: "3"
EOF
- 在机器初始化master
$ sudo kubeadm init --config=kube-master-config.yaml
- 复制
kubeadm join命令行，到worker 节点执行
kubeadm join 192.168.11.80:6443 --token 9snqmx.y797mx53ucma8jxo --discovery-token-ca-cert-hash sha256:d1c40642612935be7a9dcdc3446e2a8746235a972e4a9b7a5252f162b5b06340
检查集群节点
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master-01 NotReady master 6m15s v1.13.3
kube-master-02 NotReady master 4m23s v1.13.3
kube-master-03 NotReady master 2m44s v1.13.3
客户端机器配置 kubectl
- 远程到任意master节点
$ ssh kube@192.168.11.81
- 为 admin.conf 添加权限，仅读.
$ sudo chmod +r /etc/kubernetes/admin.conf
- 将 admin.conf 文件复制到客户端
$ scp kube@192.168.11.81:/etc/kubernetes/admin.conf .
- 将文件放置
.kube目录下
$ mkdir ~/.kube
$ mv admin.conf ~/.kube/config
$ chmod 600 ~/.kube/config
- 下载
kubectl工具到本机
$ curl -s -L -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/v1.13.3/bin/linux/amd64/kubectl
$ chmod +x /usr/local/bin/kubectl
- 在机器上通过API查看集群上的pod
$ kubectl get nodes -n kube-system
部署网络
Flannel
也可以考虑其它的网络组件，比如 wave、calico、contiv
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
修改pod 网络
$ vim kube-flannel.yml
76 行
net-conf.json: |
{
"Network": "10.30.0.0/16",
"Backend": {
"Type": "vxlan"
}
国内网络问题，可以使用以下镜像来代替:
$ sudo docker pull registry.cn-shenzhen.aliyuncs.com/shuhui/flannel:latest
更改tag 名称，也可以直接更改 kue-flannel.yaml里面对应的镜像
$ sudo docker tag registry.cn-shenzhen.aliyuncs.com/shuhui/flannel:latest quay.io/coreos/flannel:v0.11.0-amd64
$ sudo docker rmi registry.cn-shenzhen.aliyuncs.com/shuhui/flannel:latest
部署
$ kubectl apply -f kube-flannel.yml
再查看集群状态
$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-86c58d9df4-7qdlp 1/1 Running 0 54m
coredns-86c58d9df4-9sslk 1/1 Running 0 54m
kube-apiserver-kube-master-01 1/1 Running 0 53m
kube-apiserver-kube-master-02 1/1 Running 0 49m
kube-apiserver-kube-master-03 1/1 Running 0 46m
kube-controller-manager-kube-master-01 1/1 Running 0 54m
kube-controller-manager-kube-master-02 1/1 Running 0 49m
kube-controller-manager-kube-master-03 1/1 Running 0 46m
kube-flannel-ds-amd64-5jxfq 1/1 Running 0 17m
kube-flannel-ds-amd64-5nxsb 1/1 Running 0 17m
kube-flannel-ds-amd64-gphk2 1/1 Running 0 17m
kube-proxy-2n646 1/1 Running 0 54m
kube-proxy-9zql6 1/1 Running 0 46m
kube-proxy-nl2x6 1/1 Running 0 49m
kube-scheduler-kube-master-01 1/1 Running 0 53m
kube-scheduler-kube-master-02 1/1 Running 0 49m
kube-scheduler-kube-master-03 1/1 Running 0 46m
kube-master-01:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master-01 Ready master 55m v1.13.3
kube-master-02 Ready master 49m v1.13.3
kube-master-03 Ready master 47m v1.13.3
部署 metrics-server
参考： https://vqiu.cn/k8s-metric-server-deployment/
QA
1. node 节点的ROLES 显示为<none> ，该如何指定一个标记?
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master-01 Ready master 84m v1.13.3
kube-master-02 Ready master 78m v1.13.3
kube-master-03 Ready master 75m v1.13.3
kube-worker-01 Ready <none> 8m43s v1.13.3
实际上对应的是一个标签而已，我们为其它添加对应的标签即可
$ kubectl label node kube-worker-01 node-role.kubernetes.io/node=
node/kube-worker-01 labeled
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master-01 Ready master 86m v1.13.3
kube-master-02 Ready master 80m v1.13.3
kube-master-03 Ready master 77m v1.13.3
kube-worker-01 Ready worker 10m v1.13.3
如何需要进行删除可以
$ kubectl label node kube-worker-01 node-role.kubernetes.io/worker-
2. 如何使用快速补全功能
alias k=kubectl
source <(kubectl completion bash | sed s/kubectl/k/g)
3. 如何快速切换到其它的namespace？
缺省情况下，我们处于名为“default”的namespace，我们可以配置一个系统别名，比如 kcd
$ alias kcd='kubectl config set-context $(kubectl config current-context) --namespace'
# 然后可以使用以下命令来进行快速切换到其它namespace
$ kcd some-namespace