趣玩 OSS + docker registry + consul-template 组合

树辉 -

把用了6年多的ECS操作系统（Debian8）升级了一番，百无聊赖，找点乐子来打发下。于是想到工作上时常拉取容器镜像那个龟速，就折腾它啦，顺便掺和些奇技淫巧，不亦乐乎。

docker-registry

docker-registry是docker官方出品的轻量级别服务，就咱们这种穷人，ECS资源毕竟有限，要是资源有足，咱也不折腾了，直接部署个Habor、Nexus就完事了。

同时考虑到本地存储也是捉襟见肘，直接将容器镜像存储在阿里的OSS上面，docker-registry + 阿里云 OSS是妥妥支持的。

OSS 配置

  1. 新建个RAM子账号，生成AccessKeyID、AccessKeySecret记录之
  2. 与ECS同个区创建个Bucket
  3. 将Bukket授权给新建的子账号

根据https://docs.docker.com/registry/storage-drivers/oss/页面，生成以下：

storage:
  cache:
    blobdescriptor: inmemory
  oss:
    accesskeyid: <ID>
    accesskeysecret: <Secret>
    region: oss-cn-hongkong
    internal: yes
    bucket: vqiu-pkg
    rootdirectory: /container/hub.vqiu.top

也可以使用环境变量：

    environment:
      - REGISTRY_STORAGE=oss
      - REGISTRY_STORAGE_OSS_ACCESSKEYID=<ID>
      - REGISTRY_STORAGE_OSS_ACCESSKEYSECRET=<Secret>
      - REGISTRY_STORAGE_OSS_REGION=oss-cn-hongkong
      - REGISTRY_STORAGE_OSS_INTERNAL=false
      - REGISTRY_STORAGE_OSS_BUCKET=vqiu-pkg
      - REGISTRY_STORAGE_OSS_ROOTDIRECTORY=/container/hub.vqiu.top

Tips：REGISTRY_STORAGE_OSS_INTERNAL标记是否允许公网访问

docker-registry 配置

目录结构

# tree ./docker-distributing/
./docker-distributing/
└── docker-compose.yml

1 directory, 1 files
  • docker-compose.yaml
version: '3'
services:
  docker_registry:
    image: registry:2.8.1
    container_name: registry
    networks:
      - proxy
    restart: always
    ports:
      - 127.0.0.1:5000:5000/tcp
    environment:
      - REGISTRY_STORAGE=oss
      - REGISTRY_STORAGE_OSS_ACCESSKEYID=<ID>
      - REGISTRY_STORAGE_OSS_ACCESSKEYSECRET=<Secret>
      - REGISTRY_STORAGE_OSS_REGION=oss-cn-hongkong
      - REGISTRY_STORAGE_OSS_INTERNAL=false
      - REGISTRY_STORAGE_OSS_BUCKET=vqiu-pkg
      - REGISTRY_STORAGE_OSS_ROOTDIRECTORY=/container/hub.vqiu.top

networks:
  proxy:
    name: proxy-network
  • 这里预留了个本地回环地址的监听，方便后续搞事情
  • Basic认证在负载均衡上面配置

服务启动

docker-compose up -d

nginx 配置

<略>

触发镜像拉取策略

定义一组镜像列表，由consul-template监听自动触发镜像拉取动作，完成之后我们再去上面去取。

consul-template

  • config.hcl
log_level = "info"

consul {
  address = "127.0.0.1:8500"
  auth {
    enabled  = false
    username = "test"
    password = "test"
  }
  retry {
    enabled = true
    attempts = 3
  }
}

template {
  contents = "{{ key \"images/hub.vqiu.top/pullList\" }}"
  destination = "/etc/consul-template.d/image_pull_list.txt"
  perms = "0640"
  backup = false
  command = "/etc/consul-template.d/image_mirrors.sh"
}
  • image_mirrors.sh
#!/usr/bin/env bash

#!/usr/bin/env bash

export registry_addr="127.0.0.1:5000"
export consul_addr="127.0.0.1:8500"
export pull_list="/etc/consul-template.d/image_pull_list.txt"

cat $pull_list | tr -s '\n' | while read -r line
do
    image_long_name=$(echo $line | awk -F: '{print $1}')
    image_name=${image_long_name##*/}
    image_tag=$(echo $line | awk -F: '{print $2}')
    image_full_name="${registry_addr}/${image_name}:${image_tag}"
    curl -s -XGET ${registry_addr}/v2/${image_name}/tags/list |grep ${image_tag} >/dev/null 2>/dev/null
    if [[ $? -ne "0" ]]
    then
        docker pull $line
        docker tag $line $image_full_name
        docker push $image_full_name
        if [[ $? -eq "0" ]]
        then
            curl -XPUT ${consul_addr}/v1/kv/images/hub.vqiu.top/pullStatus/${image_full_name} -d 'ok'
            docker rmi -f $line $image_full_name
        else
            curl -XPUT ${CONSUL}/v1/kv/images/hub.vqiu.top/pullStatus/$line -d 'failed'
            docker rmi -f $line $image_full_name
        fi
    else
        echo "[INFO] ${image_name}:${image_tag} does exist!"
    fi
done

Tips:镜像拉取之前去查询该镜像是否在registry，避免重复执行，毕竟这网络也是要钱的，穷人法则--能省就省吧。实际环境，应使用skopeo。嗯，这是后话。

  • consul-template.service
# /etc/systemd/system/consul-template.service
[Unit]
Description=consul-template
Requires=network-online.target
After=network-online.target

[Service]
EnvironmentFile=-/etc/default/consul-template
Restart=on-failure
ExecStart=/usr/local/bin/consul-template $OPTIONS -config=/etc/consul-template.d/config.hcl
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

测试

  1. 往consul 中填入需要拉取的镜像

  1. 查看consul-template日志
Sep 11 17:40:52 hk-www consul-template[83514]: 2022-09-11T17:40:52.086+0800 [INFO] (runner) rendered "(dynamic)" => "/etc/consul-template.d/image_pull_list.txt"
Sep 11 17:40:52 hk-www consul-template[83514]: 2022-09-11T17:40:52.086+0800 [INFO] (runner) executing command "[\"/etc/consul-template.d/image_mirrors.sh\"]" from "(dynamic)" => "/etc/consul-template.d/ima>
Sep 11 17:40:52 hk-www consul-template[83514]: 2022-09-11T17:40:52.086+0800 [INFO] (child) spawning: /etc/consul-template.d/image_mirrors.sh
Sep 11 17:40:52 hk-www consul-template[83538]: 3.2: Pulling from google_containers/pause-amd64
Sep 11 17:40:52 hk-www consul-template[83538]: Digest: sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
Sep 11 17:40:52 hk-www consul-template[83538]: Status: Downloaded newer image for gcr.io/google_containers/pause-amd64:3.2
Sep 11 17:40:52 hk-www consul-template[83538]: gcr.io/google_containers/pause-amd64:3.2
Sep 11 17:40:52 hk-www consul-template[83548]: The push refers to repository [127.0.0.1:5000/pause-amd64]
Sep 11 17:40:52 hk-www consul-template[83548]: ba0dae6243cc: Preparing
Sep 11 17:40:53 hk-www consul-template[83548]: ba0dae6243cc: Layer already exists
Sep 11 17:40:53 hk-www consul-template[83548]: 3.2: digest: sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108 size: 526
Sep 11 17:40:53 hk-www consul-template[83553]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Sep 11 17:40:53 hk-www consul-template[83553]:                                  Dload  Upload   Total   Spent    Left  Speed
Sep 11 17:40:53 hk-www consul-template[83553]: [158B blob data]
Sep 11 17:40:53 hk-www consul-template[83553]: true
Sep 11 17:40:53 hk-www consul-template[83554]: Untagged: gcr.io/google_containers/pause-amd64:3.2
Sep 11 17:40:53 hk-www consul-template[83554]: Untagged: gcr.io/google_containers/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
Sep 11 17:40:53 hk-www consul-template[83554]: Untagged: 127.0.0.1:5000/pause-amd64:3.2
Sep 11 17:40:53 hk-www consul-template[83554]: Untagged: 127.0.0.1:5000/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108