Kubernetes 部署私有云镜像harbor

树辉 -

前言

Harbor是一个可以在Kubernetes上运行的容器存储库。我认为对于那些希望在私有云中拥有自己的容器存储库的人来说，这将是一种选择。功能包括多租户，映像复制和容器映像漏洞检查。

本部署文档的PV未使用共享存储，仅使用HostPath，作为生产应该使用共享存储。

环境

  • Kubernetes v1.14.6-aliyun.1
  • Helm v2.12.3

为保证应用调度到指定节点，为node新增一个标签

kubectl label nodes xx_node app.kubernetes.io/name=harbor
kubectl get node -l "app.kubernetes.io/name=harbor"

部署步骤

部署 Harbor 的前置条件步骤如下

  • Ingress Controller
  • 创建namespace
  • 创建 Harbor PV(PersistentVolume)

Ingress Controller

如果阿里云 SaaS，可以忽略。

这里我们使用 nginx作为集群的负载，实际你也可以使用其它，比如 HAProxy、Traefik等

$ helm install stable/nginx-ingress --name my-nginx --set rbac.create=true --name nginx-ingress

查看部署的版本

POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version

https://kubernetes.github.io/ingress-nginx/deploy/

创建 Namespace

为harbor 创建一个名为 harbor-system命名空间

kubectl create ns harbor-system

创建 PersistentVolume（PV）以存储应用数据

在xx_node主机中创建目录

[root@kube-node-03 harbor]# ls
for dir in chartmuseum  database  jobservice  redis  redistry
do
    mkdir -pv /data/harbor/$dir;
done

创建pv文件

  • chartmuseum.pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: chartmuseum-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/harbor/chartmuseum
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values:
          - harbor
  • chartmuseum.pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: chartmuseum-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  resources:
    requests:
      storage: 5Gi
  • database.pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: database-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/harbor/database
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values:
          - harbor
  • database.pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: database-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  resources:
    requests:
      storage: 1Gi
  • jobservice.pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jobservice-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/harbor/jobservice
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values:
          - harbor
  • jobservice.pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jobservice-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  resources:
    requests:
      storage: 1Gi
  • redis.pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/harbor/redis
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values:
          - harbor
  • redis.pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: redis-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  resources:
    requests:
      storage: 1Gi
  • registry.pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: registry-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/harbor/redistry
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values:
          - harbor
  • registry.pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: registry-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  resources:
    requests:
      storage: 10Gi

资源创建

kubectl -n harbor-system allply -f .

查看创建的PV资源

kubectl get pv,pvc --namespace=harbor-system 
NAME                              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS    REASON   AGE
persistentvolume/chartmuseum-pv   5Gi        RWO            Delete           Bound    harbor-system/chartmuseum-pvc   local-storage            12h
persistentvolume/database-pv      1Gi        RWO            Delete           Bound    harbor-system/database-pvc      local-storage            12h
persistentvolume/jobservice-pv    1Gi        RWO            Delete           Bound    harbor-system/jobservice-pvc    local-storage            12h
persistentvolume/redis-pv         1Gi        RWO            Delete           Bound    harbor-system/redis-pvc         local-storage            12h
persistentvolume/registry-pv      10Gi       RWO            Delete           Bound    harbor-system/registry-pvc      local-storage            12h

NAME                                    STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS    AGE
persistentvolumeclaim/chartmuseum-pvc   Bound    chartmuseum-pv   5Gi        RWO            local-storage   12h
persistentvolumeclaim/database-pvc      Bound    database-pv      1Gi        RWO            local-storage   12h
persistentvolumeclaim/jobservice-pvc    Bound    jobservice-pv    1Gi        RWO            local-storage   12h
persistentvolumeclaim/redis-pvc         Bound    redis-pv         1Gi        RWO            local-storage   12h
persistentvolumeclaim/registry-pvc      Bound    registry-pv      10Gi       RWO            local-storage   12h

部署harbor

$ git clone https://github.com/goharbor/harbor-helm.git
$ cd harbor-helm

为harbor配置pv，实现单机持久性存储

# diff values.yaml values-origin.yaml 
29,30c29,30
<       core: hub.haid.com.cn
<       notary: hub.haid.com.cn
---
>       core: core.harbor.domain
>       notary: notary.harbor.domain
101c101
< externalURL: https://harbor.haid.com.cn
---
> externalURL: https://core.harbor.domain
119c119
<       existingClaim: "registry-pvc"
---
>       existingClaim: ""
126c126
<       size: 10Gi
---
>       size: 5Gi
128c128
<       existingClaim: "chartmuseum-pvc"
---
>       existingClaim: ""
132c132
<       size: 1Gi
---
>       size: 5Gi
134c134
<       existingClaim: "jobservice-pvc"
---
>       existingClaim: ""
142c142
<       existingClaim: "database-pvc"
---
>       existingClaim: ""
150c150
<       existingClaim: "redis-pvc"
---
>       existingClaim: ""

使用helm 部署至集群

helm install --wait --name harbor --namespace harbor-system . \
  --set expose.ingress.hosts.core=harbor.haid.com.cn \
  --set expose.ingress.hosts.notary=notary.haid.com.cn \
  --set expose.tls.secretName=ingress-cert-with-haid.com.cn \
  --set persistence.enabled=enable \
  --set externalURL=https://harbor.haid.com.cn \
  --set harborAdminPassword=PWD#Admin \
  --set nodeSelector.app.kubernetes.io/name=harbor

完成