MinIO 生产部署要录
环境
- OS: CentOS-7.9
- 平台: x86
- 部署模式：distributed
- 配置规格: 8C/16/50G/内网带宽3G
- 数据盘：50G *2
- MinIO版本：2021-04-22T15-44-28Z.hotfix.2cc0aaed8
- 压测工具：warp
部署记录
系统配置
配置主机名称
# hostnamectl set-hostname minio-01
# hostnamectl set-hostname minio-02
# hostnamectl set-hostname minio-03
# hostnamectl set-hostname minio-04
配置FQDN
# vim /etc/hosts
...
192.168.48.53 minio-01
192.168.48.54 minio-02
192.168.48.55 minio-03
192.168.48.56 minio-04
创建minio专属用户组
groupadd --system minio
useradd -s /sbin/nologin --system -g minio minio
系统内核参数
# maximum number of open files/file descriptors
fs.file-max = 4194303
# use as little swap space as possible
vm.swappiness = 1
# prioritize application RAM against disk/swap cache
vm.vfs_cache_pressure = 10
# minimum free memory
vm.min_free_kbytes = 1000000
# maximum receive socket buffer (bytes)
net.core.rmem_max = 268435456
# maximum send buffer socket buffer (bytes)
net.core.wmem_max = 268435456
# default receive buffer socket size (bytes)
net.core.rmem_default = 67108864
# default send buffer socket size (bytes)
net.core.wmem_default = 67108864
# maximum number of packets in one poll cycle
net.core.netdev_budget = 1200
# maximum ancillary buffer size per socket
net.core.optmem_max = 134217728
# maximum number of incoming connections
net.core.somaxconn = 65535
# maximum number of packets queued
net.core.netdev_max_backlog = 250000
# maximum read buffer space
net.ipv4.tcp_rmem = 67108864 134217728 268435456
# maximum write buffer space
net.ipv4.tcp_wmem = 67108864 134217728 268435456
# enable low latency mode
net.ipv4.tcp_low_latency = 1
# socket buffer portion used for TCP window
net.ipv4.tcp_adv_win_scale = 1
# queue length of completely established sockets waiting for accept
net.ipv4.tcp_max_syn_backlog = 30000
# maximum number of sockets in TIME_WAIT state
net.ipv4.tcp_max_tw_buckets = 2000000
# reuse sockets in TIME_WAIT state when safe
net.ipv4.tcp_tw_reuse = 1
# time to wait (seconds) for FIN packet
net.ipv4.tcp_fin_timeout = 5
# disable icmp send redirects
net.ipv4.conf.all.send_redirects = 0
# disable icmp accept redirect
net.ipv4.conf.all.accept_redirects = 0
# drop packets with LSR or SSR
net.ipv4.conf.all.accept_source_route = 0
# MTU discovery, only enable when ICMP blackhole detected
net.ipv4.tcp_mtu_probing = 1
创建目录
mkdir -pv /etc/minio /mnt/minio-data
格式化分区
mkfs.xfs /dev/vdb
mkfs.xfs /dev/vdc
mount -t /dev/vdb /mnt/minio-data/1
mount -t /dev/vdc /mnt/minio-data/2
权限分配
chown -R minio:minio /mnt/minio-data /etc/minio
MinIO服务配置
下载MinIO二进制文件
curl -o /usr/local/bin/minio https://dl.min.io/server/minio/hotfixes/linux-amd64/archive/minio.RELEASE.2021-04-22T15-44-28Z.hotfix.2cc0aaed8
chmod +x /usr/bin/minio
创建minio配置文件
cat > /etc/sysconfig/minio <<EOF
MINIO_ACCESS_KEY="AKaHEgQ4II097B4T6DjAUDA4BX"
MINIO_SECRET_KEY="SKbRoq5iDoQgF7gyPYRFhzNMY3vY6ZFMpH"
MINIO_VOLUMES=http://minio-1{1...4}/mnt/minio-data/{1...2}
MINIO_OPTS="-C /etc/minio --address 0.0.0.0:9000"
EOF
创建systemd文件
cat > /etc/systemd/system/minio.service<<EOF
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio
[Service]
WorkingDirectory=/usr/local
User=minio
Group=minio
EnvironmentFile=/etc/sysconfig/minio
ExecStartPre=/bin/bash -c "if [ -z \"\${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/sysconfig/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server \$MINIO_OPTS \$MINIO_VOLUMES
# Let systemd restart this service always
Restart=always
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=1048576
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
EOF
服务启动
# systemctl daemon-reload
# systemctl enable minio
# systemctl start minio.service
查看服务状态
# systemctl status minio
● minio.service - MinIO
Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2023-02-21 17:41:49 CST; 13s ago
Docs: https://docs.min.io
Process: 11785 ExecStartPre=/bin/bash -c if [ -z "${MINIO_VOLUMES}" ]; then echo "Variable MINIO_VOLUMES not set in /etc/sysconfig/minio"; exit 1; fi (code=exited, status=0/SUCCESS)
Main PID: 11788 (minio)
CGroup: /system.slice/minio.service
└─11788 /usr/local/bin/minio server -C /etc/minio --address 0.0.0.0:9000 http://minio-0{1...4}/mnt/minio-data/{1...2}
Feb 21 17:41:53 minio-01 minio[11788]: Browser Access:
Feb 21 17:41:53 minio-01 minio[11788]: http://0.0.0.0:9000
Feb 21 17:41:53 minio-01 minio[11788]: Object API (Amazon S3 compatible):
Feb 21 17:41:53 minio-01 minio[11788]: Go: https://docs.min.io/docs/golang-client-quickstart-guide
Feb 21 17:41:53 minio-01 minio[11788]: Java: https://docs.min.io/docs/java-client-quickstart-guide
Feb 21 17:41:53 minio-01 minio[11788]: Python: https://docs.min.io/docs/python-client-quickstart-guide
Feb 21 17:41:53 minio-01 minio[11788]: Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
Feb 21 17:41:53 minio-01 minio[11788]: JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
Feb 21 17:41:53 minio-01 minio[11788]: .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
Feb 21 17:41:53 minio-01 minio[11788]: IAM initialization complete
# mc admin info myminio
● minio-01:9000
Uptime: 2 minutes
Version: 2021-04-22T15:44:28Z
Network: 4/4 OK
Drives: 2/2 OK
Pool: 1
● minio-02:9000
Uptime: 2 minutes
Version: 2021-04-22T15:44:28Z
Network: 4/4 OK
Drives: 2/2 OK
Pool: 1
● minio-03:9000
Uptime: 2 minutes
Version: 2021-04-22T15:44:28Z
Network: 4/4 OK
Drives: 2/2 OK
Pool: 1
● minio-04:9000
Uptime: 2 minutes
Version: 2021-04-22T15:44:28Z
Network: 4/4 OK
Drives: 2/2 OK
Pool: 1
Pools:
1st, Erasure sets: 1, Drives per erasure set: 8
3.8 MiB Used, 2 Buckets, 3 Objects
8 drives online, 0 drives offline
压力测试
下载warp
# rpm -ivh https://github.com/minio/warp/releases/download/v0.8.0/warp_Linux_x86_64.rpm
在压力测试节点启动client模式
# warp client
warp: Listening on:7761
发起性能测试
# warp mixed --warp-client=192.168.80.8:7761,192.168.80.9:7761 \
--access-key=AKaHEgQ4II097B4T6DjAUDA4BX \
--secret-key=SKbRoq5iDoQgF7gyPYRFhzNMY3vY6ZFMpH \
--host=minio-0{1...4}:9000 \
--duration=120s \
--autoterm
warp: Benchmark data written to "warp-remote-2023-02-21[222124]-afNE.csv.zst"
Mixed operations.
Operation: DELETE, 10%, Concurrency: 40, Ran 1m53s.
* Throughput:13.92 obj/s
Operation: GET, 45%, Concurrency: 40, Ran 1m53s.
* Throughput:625.42 MiB/s, 62.54 obj/s
Operation: PUT, 15%, Concurrency: 40, Ran 1m53s.
* Throughput:209.50 MiB/s, 20.95 obj/s
Operation: STAT, 30%, Concurrency: 40, Ran 1m53s.
* Throughput:41.75 obj/s
Cluster Total: 834.16 MiB/s, 139.04 obj/s over 1m53s.
warp: Cleanup done.
受限于网络因素，压测时CPU百分比在10%之间，但网络带宽已跑满。
Prometheus监控
想偷懒直接加入以下环境变量，可以做到不需要凭证即可访问metrics接口
Environment=MINIO_PROMETHEUS_AUTH_TYPE=public
若考虑安全的操作如下
# mc admin prometheus generate myminio
scrape_configs:
- job_name: minio-job
bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoibWluaW8iLCJleHAiOjQ4NjIxMjcyNzd9.AmA98ij3hi4J-X5VMzLaTJvc5dp6Ke9K4iE_dOtwHxqHaHwhWwdfXA-p1xd3f19oFKVyROymC8I8u2RUOw1oMg
metrics_path: /minio/v2/metrics/cluster
scheme: http
static_configs:
- targets: ['localhost:9000']
此时可以带凭证访问prometheus metrics接口
# curl -v -sSL -H 'Authorization: Bearer eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoibWluaW8iLCJleHAiOjQ4NjIxMjcyNzd9.AmA98ij3hi4J-X5VMzLaTJvc5dp6Ke9K4iE_dOtwHxqHaHwhWwdfXA-p1xd3f19oFKVyROymC8I8u2RUOw1oMg' \
http://minio-01:9000/minio/v2/metrics/cluster
<省略若干行>
prometheus若部署在kubernetes集群中，则创建对应的ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: minio
labels:
app: minio
version: 2021.04.22
release: minio
spec:
endpoints:
- port: http
path: /minio/v2/metrics/cluster
bearerTokenSecret:
name: gitee-minio-prometheus
key: token
namespaceSelector:
matchNames:
- "monitoring"
selector:
matchLabels:
app: minio
release: minio