Kafka kraft 模式非容器化部署

树辉 -

版本

  • OpenJDK：jdk-17.0.17+10
  • Kafka：3.9.1

节点基础配置

系统内核参数

# cat > /etc/sysctl.d/99-kafka.conf<<EOF
# /etc/sysctl.d/99-kafka.conf

# ==================== 内存策略 ====================
vm.overcommit_memory = 1
vm.swappiness = 1
vm.dirty_ratio = 5
vm.dirty_background_ratio = 3

# ==================== 大节点资源扩展 ====================
fs.inotify.max_user_watches = 1048576
fs.inotify.max_user_instances = 16384
fs.nr_open = 2097152
fs.file-max = 2097152
vm.max_map_count = 262144
fs.aio-max-nr = 1048576

# ==================== 网络优化 ====================
net.core.somaxconn = 32768
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.ip_local_port_range = 10240 65535
net.core.rmem_max = 4194304
net.core.wmem_max = 4194304
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 65536 4194304

# ==================== Conntrack ====================
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
EOF

关闭透明大页（THP）

# echo never > /sys/kernel/mm/transparent_hugepage/enabled
# echo never > /sys/kernel/mm/transparent_hugepage/defrag
提示：该特性也可以追加到systemd里面，前提需要root权限

配置JDK环境

直接将包解压并创建软链接

tar acvf OpenJDK17U-jdk_x64_linux_hotspot_17.0.17_10.tar.gz -C /usr/local
cd /usr/local && ln -svf /usr/local/jdk-17.0.17+10 jdk

设置系统环境变量

cat >/etc/profile.d/java.sh<<EOF
export PATH=\$PATH:/usr/local/jdk/bin
EOF

完成验证如下：

$ java -version
openjdk version "17.0.17" 2025-10-21
OpenJDK Runtime Environment Temurin-17.0.17+10 (build 17.0.17+10)
OpenJDK 64-Bit Server VM Temurin-17.0.17+10 (build 17.0.17+10, mixed mode, sharing)

安装Kafka

下载并解压

tar axvf kafka_2.13-3.9.1.tgz -C /usr/local
cd /usr/local && ln -svf kafka_2.13-3.9.1 kafka

配置Kafka环境

单点配置

适用于开发、测试环境。

编辑 /usr/local/kafka/config/kraft/server.properties配置文件，内容如下：

process.roles=broker,controller

node.id=41

controller.quorum.voters=41@110.10.2.41

# 监听节点所有网卡
listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093

# 客户端连接时使用的地址。必须设置为客户端能从其网络环境访问到的地址（如公网IP、域名、主机名），不填写将继承listeners
advertised.listeners=PLAINTEXT://110.10.2.41:9092,CONTROLLER://110.10.2.41:9092

inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER

listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

num.partitions=1
num.recovery.threads.per.data.dir=1
num.network.threads=3
num.io.threads=8

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.dirs=/data/kafka
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

集群配置

适用于生产环境。

编辑 /usr/local/kafka/config/kraft/server.properties配置文件，内容如下：


# ==================== Broker 标识 ====================
process.roles=broker,controller
node.id=93

# ==================== Controller 仲裁 ====================
controller.quorum.voters=93@10.138.98.93:9093,94@10.138.98.94:9093,95@10.138.98.95:9093
controller.listener.names=CONTROLLER

# ==================== 监听器 ====================
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://10.138.98.93:9092,CONTROLLER://10.138.98.93:9093
inter.broker.listener.name=PLAINTEXT

# ==================== 安全（防火墙兜底） ====================
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# ==================== Socket 限流 ====================
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

# ==================== 线程模型 ====================
num.network.threads=8
num.io.threads=8
num.recovery.threads.per.data.dir=1

# ==================== 分区与副本 ====================
num.partitions=3
default.replication.factor=3
min.insync.replicas=2

# ==================== 选举策略 ====================
unclean.leader.election.enable=false
auto.leader.rebalance.enable=false

# ==================== 内部 Topic ====================
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2

# ==================== 数据存储 ====================
log.dirs=/data/kafka
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

# ==================== 消息包大小 ====================
message.max.bytes=10485760
replica.fetch.max.bytes=10485800

# ==================== 压缩 ====================
compression.type=lz4

# ==================== 消费者协调 ====================
group.initial.rebalance.delay.ms=6000
offsets.retention.minutes=10080

# ==================== 生产者/消费者配额 ====================
quota.producer.default=104857600
quota.consumer.default=104857600

初始化集群数据

生成UUID

$ bin/kafka-storage.sh random-uuid
s9xNEMd-QkeBCwmMkyXwCw

创建数据目录

# install -d -o vqiu -g vqiu /data/kafka

初始化数据目录

$ bin/kafka-storage.sh format -t s9xNEMd-QkeBCwmMkyXwCw  -c /usr/local/kafka/config/kraft/server.properties
如果为集群，每节点都需要执行。

添加环境变量

cat >/etc/profile.d/kafka.sh<<EOF
export KAFKA_HOME=/usr/local/kafka
export PATH=\$PATH:${KAFKA_HOME}/bin
EOF

使用systemd来控制kafka

cat  >/usr/lib/systemd/system/kafka.service<<EOF
[Unit]
Description=Apache Kafka server with Kraft mode
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=vqiu
Group=vqiu
LimitAS=infinity
LimitRSS=infinity
LimitCORE=infinity
LimitNOFILE=1048576
Environment=JAVA_HOME=/usr/local/jdk
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh \
          /usr/local/kafka/config/kraft/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target
EOF

检查 Kafka 是否正在运行：

jcmd | grep kafka

查看集群全景

  1. 列出集群中所有的 Broker 节点（这是首要检查项）
kafka-broker-api-versions.sh --bootstrap-server <当前节点IP>:9092

2.查看所有 Topic 的详细状态（分区、副本、Leader分布）

bin/kafka-topics.sh --bootstrap-server <当前节点IP>:9092 --describe

关于堆内存分配

默认通常只分配1G大小，如果需要额外分配JVM 堆内存，可以编辑 bin/kafka-server-start.sh 文件。