英文:
Cannot produce kafka message on kubernetes
问题
我在Kafka上遇到错误:[2020-05-04 12:46:59,477] ERROR [KafkaApi-1001] 存活的代理数量 '0' 未满足偏移量主题的所需复制因子 '1'(通过 'offsets.topic.replication.factor' 配置)。如果集群正在启动并且并非所有代理都已启动,则可以忽略此错误。 (kafka.server.KafkaApis)
在尝试生成消息时出现错误:2020-05-04 12:47:45.221 WARN 1 --- org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] 获取关联 ID 10 的元数据时出错: {activate-user=LEADER_NOT_AVAILABLE}
在使用docker-compose时一切正常,但我也尝试将其移植到Kubernetes。我使用 kompose convert
工具开始了这个过程并修改了输出。
以下是docker-compose的片段:
zookeeper:
container_name: zookeeper
image: wurstmeister/zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- "2181:2181"
mail-sender-kafka:
container_name: mail-sender-kafka
image: wurstmeister/kafka:2.12-2.2.1
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_HOST_NAME: mail-sender-kafka
KAFKA_CREATE_TOPICS: "activate-user:1:1"
ports:
- 9092:9092
depends_on:
- zookeeper
account-service:
image: szastarek/food-delivery-account-service:${TAG}
container_name: account-service
environment:
- KAFKA_URI=mail-sender-kafka:9092
depends_on:
- config-server
- account-service-db
mail-sender:
image: szastarek/food-delivery-mail-sender:${TAG}
container_name: mail-sender
environment:
- KAFKA_URI=mail-sender-kafka:9092
depends_on:
- config-server
转换为Kubernetes后,我得到了 zookeeper-deployment
、zookeeper-service
、mail-sender-deployment
、mail-sender-kafka-deployment
、mail-sender-kafka-service
。
我还尝试添加了一些环境变量,目前看起来是这样的:
spec:
containers:
- env:
- name: KAFKA_ADVERTISED_HOST_NAME
value: mail-sender-kafka
- name: KAFKA_ADVERTISED_PORT
value: '9092'
- name: ADVERTISED_LISTENERS
value: PLAINTEXT://mail-sender-kafka:9092
- name: KAFKA_CREATE_TOPICS
value: activate-user:1:1
- name: KAFKA_ZOOKEEPER_CONNECT
value: zookeeper:2181
我找到了一个可能与问题相关的事项。当我在Docker上运行 ping mail-sender-kafka
时,我可以连接到自己。但是,如果我连接到Kubernetes的 mail-sender-kafka
Pod,则无法ping自己。
在更新了主机文件后,问题得到解决。之前是:
172.18.0.24 mail-sender-kafka-xxxxxxx
我将其更改为:
172.18.0.24 mail-sender-kafka
有关如何修复此问题的任何提示?
英文:
I'm getting error on kafka: [2020-05-04 12:46:59,477] ERROR [KafkaApi-1001] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
And error when I'm trying to produce message: 2020-05-04 12:47:45.221 WARN 1 --- org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Error while fetching metadata with correlation id 10 : {activate-user=LEADER_NOT_AVAILABLE}
Using docker-compose everything works fine but I'm trying to move it also to k8s. I started that process with kompose convert
tool and modify the output.
Here is a fragment of the docker-compse:
zookeeper:
container_name: zookeeper
image: wurstmeister/zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- "2181:2181"
mail-sender-kafka:
container_name: mail-sender-kafka
image: wurstmeister/kafka:2.12-2.2.1
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_HOST_NAME: mail-sender-kafka
KAFKA_CREATE_TOPICS: "activate-user:1:1"
ports:
- 9092:9092
depends_on:
- zookeeper
account-service:
image: szastarek/food-delivery-account-service:${TAG}
container_name: account-service
environment:
- KAFKA_URI=mail-sender-kafka:9092
depends_on:
- config-server
- account-service-db
mail-sender:
image: szastarek/food-delivery-mail-sender:${TAG}
container_name: mail-sender
environment:
- KAFKA_URI=mail-sender-kafka:9092
depends_on:
- config-server
After converting it to k8s I've got zookeeper-deployment, zookeeper-service, mail-sender-deployment, mail-sender-kafka-deployment, mail-sender-kafka-service.
I've also tried to add some env variables and for now, it looks like that:
spec:
containers:
- env:
- name: KAFKA_ADVERTISED_HOST_NAME
value: mail-sender-kafka
- name: KAFKA_ADVERTISED_PORT
value: '9092'
- name: ADVERTISED_LISTENERS
value: PLAINTEXT://mail-sender-kafka:9092
- name: KAFKA_CREATE_TOPICS
value: activate-user:1:1
- name: KAFKA_ZOOKEEPER_CONNECT
value: zookeeper:2181
I've found one thing that probably is connected with the problem.
When I run ping mail-sender-kafka
on docker I can reach myself. But if I connect to kubernetes mail-sender-kafka pod then I cannot ping myself.
After update the hosts file it works. There was something like:
172.18.0.24 mail-sender-kafka-xxxxxxx
And I changed it to
172.18.0.24 mail-sender-kafka
Any tips about how should I fix it?
专注分享java语言的经验与见解,让所有开发者获益!
评论