使用kubeadm 部署多master 高可用集群

环境清单

  • OS: Debian-9.5
  • Docker 版本: 18.06
  • Master 节点IP:
    • 192.168.11.81
    • 192.168.11.82
    • 192.168.11.83
  • LB 节点IP:192.168.11.80 (HAProxy)
  • Pod网络: 10.30.0.0/16
  • kubernetes版本: 1.13.3

在LB节点安装HAProxy

  1. 远程到LB节点安装haproxy
sudo apt-get install haproxy
  1. 配置
$ sudo vim /etc/haproxy/haproxy.cfg
global
...
default
...
frontend kubernetes
    bind 192.168.11.80:6443
    option tcplog
    mode tcp
    default_backend kubernetes-master-nodes

backend kubernetes-master-nodes
    mode tcp
    balance roundrobin
    option tcp-check
    server k8s-master-01 192.168.11.81:6443 check fall 3 rise 2
    server k8s-master-02 192.168.11.82:6443 check fall 3 rise 2
    server k8s-master-03 192.168.11.83:6443 check fall 3 rise 2

  1. 重新启动服务
sudo systemctl restart haproxy

创建集群证书

  1. 下载创建证书工具
curl -s -L -o /usr/local/bin/cfssl https://down.vqiu.cn/package/container/cfssl/cfssl_linux-amd64
curl -s -L -o /usr/local/bin/cfssljson https://down.vqiu.cn/package/container/cfssl/cfssljson
chmod +x /usr/local/bin/{cfssl,cfssljson}
  1. 创建CA 证书配置文件
mkdir cas &&  cd cas
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "87600h"
      }
    }
  }
}
EOF
cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "kube",
      "OU": "vqiu co."
  }
 ]
}
EOF
  1. 生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
  1. 为集群生成证书
cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "Kubernetes",
      "OU": "Kubernetes"
  }
 ]
}
EOF
  1. 生成证书与私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-hostname=192.168.11.81,192.168.11.82,192.168.11.83,192.168.11.80,127.0.0.1,localhost,kubernetes.default \
-profile=kubernetes kubernetes-csr.json | \
cfssljson -bare kubernetes
  1. 目前已经创建证书有:
# ls -la
total 44
drwxr-xr-x 2 root root 4096 Feb 11 19:50 .
drwx------ 7 root root 4096 Feb 11 19:38 ..
-rw-r--r-- 1 root root  234 Feb 11 19:38 ca-config.json
-rw-r--r-- 1 root root 1009 Feb 11 19:40 ca.csr
-rw-r--r-- 1 root root  212 Feb 11 19:38 ca-csr.json
-rw------- 1 root root 1679 Feb 11 19:40 ca-key.pem
-rw-r--r-- 1 root root 1379 Feb 11 19:40 ca.pem
-rw-r--r-- 1 root root 1021 Feb 11 19:50 kubernetes.csr
-rw-r--r-- 1 root root  218 Feb 11 19:49 kubernetes-csr.json
-rw------- 1 root root 1679 Feb 11 19:50 kubernetes-key.pem
-rw-r--r-- 1 root root 1521 Feb 11 19:50 kubernetes.pem
  1. 将证书复制到各master节点:
scp ca.pem kubernetes.pem kubernetes-key.pem qiush@192.168.11.81:~
scp ca.pem kubernetes.pem kubernetes-key.pem qiush@192.168.11.82:~
scp ca.pem kubernetes.pem kubernetes-key.pem qiush@192.168.11.83:~

etcd 集群

  1. 创建etcd 目录,分别为配置目录(证书),与数据库目录
$ sudo mkdir /etc/etcd /var/lib/etcd
  1. 将证书移动到/etc/etcd目录
$ sudo mv ~/ca.pem ~/kubernetes.pem ~/kubernetes-key.pem /etc/etcd
  1. 下载etcd二进制文件
$ wget https://down.vqiu.cn/package/container/etcd-v3.3.11-linux-amd64.tar.gz
  1. 解压文件,并复制到/usr/local/bin目录
$ tar xvzf etcd-v3.3.11-linux-amd64.tar.gz
$ sudo mv etcd-v3.3.11-linux-amd64/etcd* /usr/local/bin/

5.为各节点配置 systemd

  • kube-master-01
cat >/etc/systemd/system/etcd.service<<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos

[Service]
ExecStart=/usr/local/bin/etcd \
  --name 192.168.11.81 \
  --cert-file=/etc/etcd/kubernetes.pem \
  --key-file=/etc/etcd/kubernetes-key.pem \
  --peer-cert-file=/etc/etcd/kubernetes.pem \
  --peer-key-file=/etc/etcd/kubernetes-key.pem \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://192.168.11.81:2380 \
  --listen-peer-urls https://192.168.11.81:2380 \
  --listen-client-urls https://192.168.11.81:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.11.81:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster 192.168.11.81=https://192.168.11.81:2380,192.168.11.82=https://192.168.11.82:2380,192.168.11.83=https://192.168.11.83:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
  • kube-master-02
cat >/etc/systemd/system/etcd.service<<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos

[Service]
ExecStart=/usr/local/bin/etcd \
  --name 192.168.11.82 \
  --cert-file=/etc/etcd/kubernetes.pem \
  --key-file=/etc/etcd/kubernetes-key.pem \
  --peer-cert-file=/etc/etcd/kubernetes.pem \
  --peer-key-file=/etc/etcd/kubernetes-key.pem \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://192.168.11.82:2380 \
  --listen-peer-urls https://192.168.11.82:2380 \
  --listen-client-urls https://192.168.11.82:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.11.82:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster 192.168.11.81=https://192.168.11.81:2380,192.168.11.82=https://192.168.11.82:2380,192.168.11.83=https://192.168.11.83:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
  • kube-master-03
cat >/etc/systemd/system/etcd.service<<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos

[Service]
ExecStart=/usr/local/bin/etcd \
  --name 192.168.11.83 \
  --cert-file=/etc/etcd/kubernetes.pem \
  --key-file=/etc/etcd/kubernetes-key.pem \
  --peer-cert-file=/etc/etcd/kubernetes.pem \
  --peer-key-file=/etc/etcd/kubernetes-key.pem \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://192.168.11.83:2380 \
  --listen-peer-urls https://192.168.11.83:2380 \
  --listen-client-urls https://192.168.11.83:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.11.83:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster 192.168.11.81=https://192.168.11.81:2380,192.168.11.82=https://192.168.11.82:2380,192.168.11.83=https://192.168.11.83:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
  1. 启动服务
sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd

为所有master及worker节点安装Docker环境

  1. 配置sysctl内核参数
cat > /etc/sysctl.d/docker.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
  1. 添加镜像仓库
sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
  1. 安装docker镜像包
sudo apt update && sudo apt-get -y install docker-ce=18.06.1~ce~3-0~debian
  1. 配置镜像仓库
export repos=https://registry.docker-cn.com
cat >/etc/docker/daemon.json<<EOF
 {
   "registry-mirrors": ["${repos}"],
   "log-driver": "json-file",
   "log-opts": {
     "max-size": "20m",
     "max-file": "3",
     "labels": "production_status",
     "env": "os,customer"
   }
 }
EOF

安装 kubernetes环境

  • 关闭 swap
sudo swapoff -a

内核不要开启 reuse

  • 为系统添加kubunetes APT仓库
apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat >/etc/apt/sources.list.d/kubernetes.list<<EOF
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

初始化kubernetes master 集群

kube-master-01

  1. 远程到kube-master-01
ssh kube@kube-master-01
  1. 为kubeadm创建配置文件
cat > kube-master-config.yaml<<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 192.168.11.80
controlPlaneEndpoint: "192.168.11.80:6443"
etcd:
  external:
    endpoints:
    - https://192.168.11.81:2379
    - https://192.168.11.82:2379
    - https://192.168.11.83:2379
      caFile: /etc/etcd/ca.pem
      certFile: /etc/etcd/kubernetes.pem
      keyFile: /etc/etcd/kubernetes-key.pem
networking:
  podSubnet: 10.30.0.0/16
apiServerExtraArgs:
  apiserver-count: "3"
EOF
  1. 使用kube-master-config.yaml文件初始化 master
sudo kubeadm init --config=kube-master-config.yaml
  1. 复制证书到其它2个master 节点
sudo scp -r /etc/kubernetes/pki kube@192.168.11.82:~
sudo scp -r /etc/kubernetes/pki kube@192.168.11.83:~

kube-master-02

  1. 远程到kube-master-02

  2. 删除 apiserver.crt 和 apiserver.key.

$ rm ~/pki/apiserver.*
  1. 将证书文件移到/etc/kubernetes/目录
$ sudo mv ~/pki /etc/kubernetes/
  1. 为kubeadm创建配置文件
cat > kube-master-config.yaml<<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 192.168.11.80
controlPlaneEndpoint: "192.168.11.80:6443"
etcd:
  external:
    endpoints:
    - https://192.168.11.81:2379
    - https://192.168.11.82:2379
    - https://192.168.11.83:2379
      caFile: /etc/etcd/ca.pem
      certFile: /etc/etcd/kubernetes.pem
      keyFile: /etc/etcd/kubernetes-key.pem
networking:
  podSubnet: 10.30.0.0/16
apiServerExtraArgs:
  apiserver-count: "3"
EOF
  1. 使用kube-master-config.yaml文件初始化 master
$ sudo kubeadm init --config=kube-master-config.yaml

kube-master-03

  1. 远程到kube-master-03
  2. 删除 apiserver.crt 和 apiserver.key.
$ rm ~/pki/apiserver.*
  1. 将证书文件移到/etc/kubernetes/目录
$ sudo mv ~/pki /etc/kubernetes/
  1. 为kubeadm创建配置文件
cat > kube-master-config.yaml<<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 192.168.11.80
controlPlaneEndpoint: "192.168.11.80:6443"
etcd:
  external:
    endpoints:
    - https://192.168.11.81:2379
    - https://192.168.11.82:2379
    - https://192.168.11.83:2379
      caFile: /etc/etcd/ca.pem
      certFile: /etc/etcd/kubernetes.pem
      keyFile: /etc/etcd/kubernetes-key.pem
networking:
  podSubnet: 10.30.0.0/16
apiServerExtraArgs:
  apiserver-count: "3"
EOF
  1. 在机器初始化master
$ sudo kubeadm init --config=kube-master-config.yaml
  1. 复制 kubeadm join 命令行,到worker 节点执行
kubeadm join 192.168.11.80:6443 --token 9snqmx.y797mx53ucma8jxo --discovery-token-ca-cert-hash sha256:d1c40642612935be7a9dcdc3446e2a8746235a972e4a9b7a5252f162b5b06340

检查集群节点

$ kubectl get nodes
NAME             STATUS     ROLES    AGE     VERSION
kube-master-01   NotReady   master   6m15s   v1.13.3
kube-master-02   NotReady   master   4m23s   v1.13.3
kube-master-03   NotReady   master   2m44s   v1.13.3

客户端机器配置 kubectl

  1. 远程到任意master节点
$ ssh kube@192.168.11.81
  1. 为 admin.conf 添加权限,仅读.
$ sudo chmod +r /etc/kubernetes/admin.conf
  1. 将 admin.conf 文件复制到客户端
$ scp kube@192.168.11.81:/etc/kubernetes/admin.conf .
  1. 将文件放置.kube目录下
$ mkdir ~/.kube
$ mv admin.conf ~/.kube/config
$ chmod 600 ~/.kube/config
  1. 下载kubectl工具到本机
$ curl -s -L -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/v1.13.3/bin/linux/amd64/kubectl
$ chmod +x /usr/local/bin/kubectl
  1. 在机器上通过API查看集群上的pod
$ kubectl get nodes -n kube-system

部署网络

Flannel

也可以考虑其它的网络组件,比如 wave、calico、contiv

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

修改pod 网络

$  vim kube-flannel.yml
76 行
  net-conf.json: |
    {
      "Network": "10.30.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }

国内网络问题,可以使用以下镜像来代替:

$ sudo docker pull registry.cn-shenzhen.aliyuncs.com/shuhui/flannel:latest

更改tag 名称,也可以直接更改 kue-flannel.yaml里面对应的镜像

$ sudo docker tag registry.cn-shenzhen.aliyuncs.com/shuhui/flannel:latest quay.io/coreos/flannel:v0.11.0-amd64
$ sudo docker rmi registry.cn-shenzhen.aliyuncs.com/shuhui/flannel:latest 

部署

$ kubectl apply -f kube-flannel.yml

再查看集群状态

$ kubectl get pod -n kube-system
NAME                                     READY   STATUS    RESTARTS   AGE
coredns-86c58d9df4-7qdlp                 1/1     Running   0          54m
coredns-86c58d9df4-9sslk                 1/1     Running   0          54m
kube-apiserver-kube-master-01            1/1     Running   0          53m
kube-apiserver-kube-master-02            1/1     Running   0          49m
kube-apiserver-kube-master-03            1/1     Running   0          46m
kube-controller-manager-kube-master-01   1/1     Running   0          54m
kube-controller-manager-kube-master-02   1/1     Running   0          49m
kube-controller-manager-kube-master-03   1/1     Running   0          46m
kube-flannel-ds-amd64-5jxfq              1/1     Running   0          17m
kube-flannel-ds-amd64-5nxsb              1/1     Running   0          17m
kube-flannel-ds-amd64-gphk2              1/1     Running   0          17m
kube-proxy-2n646                         1/1     Running   0          54m
kube-proxy-9zql6                         1/1     Running   0          46m
kube-proxy-nl2x6                         1/1     Running   0          49m
kube-scheduler-kube-master-01            1/1     Running   0          53m
kube-scheduler-kube-master-02            1/1     Running   0          49m
kube-scheduler-kube-master-03            1/1     Running   0          46m

kube-master-01:~$ kubectl get nodes
NAME             STATUS   ROLES    AGE   VERSION
kube-master-01   Ready    master   55m   v1.13.3
kube-master-02   Ready    master   49m   v1.13.3
kube-master-03   Ready    master   47m   v1.13.3

部署 metrics-server

参考: https://vqiu.cn/k8s-metric-server-deployment/

QA

1. node 节点的ROLES 显示为<none> ,该如何指定一个标记?

$ kubectl get nodes
NAME             STATUS   ROLES    AGE     VERSION
kube-master-01   Ready    master   84m     v1.13.3
kube-master-02   Ready    master   78m     v1.13.3
kube-master-03   Ready    master   75m     v1.13.3
kube-worker-01   Ready    <none>   8m43s   v1.13.3

实际上对应的是一个标签而已,我们为其它添加对应的标签即可

$ kubectl label node kube-worker-01 node-role.kubernetes.io/node=
node/kube-worker-01 labeled

$ kubectl get nodes
NAME             STATUS   ROLES    AGE   VERSION
kube-master-01   Ready    master   86m   v1.13.3
kube-master-02   Ready    master   80m   v1.13.3
kube-master-03   Ready    master   77m   v1.13.3
kube-worker-01   Ready    worker   10m   v1.13.3

如何需要进行删除可以

$ kubectl label node kube-worker-01 node-role.kubernetes.io/worker-

2. 如何使用快速补全功能

alias k=kubectl
source <(kubectl completion bash | sed s/kubectl/k/g)

3. 如何快速切换到其它的namespace?

缺省情况下,我们处于名为“default”的namespace,我们可以配置一个系统别名,比如 kcd

$ alias kcd='kubectl config set-context $(kubectl config current-context) --namespace'
# 然后可以使用以下命令来进行快速切换到其它namespace
$ kcd some-namespace

参考引用